[Jack]

"We are shooting it from a Caldwell rest and put the front swivel stud against the front of the rest every time."
I think that may be part of your problem, if I read that right.
If you're putting the swivel stud against the front of the front rest, as the rifle recoils, the stud will catch on the front bag.
The rifle is attempting to move rearward during recoil, and the stud catching on the front bag is causing the rifle to snag and move around- differently every time.
Try this: unscrew the front swivel and repeat your tests. You may find that your groups improve quite a bit.
When bench shooting from a rest, I always remove the front sling swivel, so that the rifle slides smoothly on the bag, the same each time.
Bench rest shooters go to significant trouble to make this happen, like putting talcum powder on their bags to make the rifle slide smoothly, and coating their stock with teflon tape for the same reason.