I think it’s a combination of the replies. The bagman doesn’t look properly seated when the saddle is attached to the seat pin. This could be because:
a) the saddle is too far forward on the pin which means there isn’t enough rail left for the bagman to clamp on to properly, or;
b) the clamp on the pin is taking up too much of the rails.
If it’s a), you need to slide the saddle back (which I think you’ve already done, hence the new stem).
If it’s b) you need either a different saddle, a different seat pin or a different method of carrying your saddle bag.