Check your mounts and ring screws to insure they are tight. Don`t forget to make sure the action screws are tight also. Look for small cracks in the stock inside and out. I doubt you have any but I once had a rifle go south and found a hair line crack about 3/4" long in the grip area where it met the trigger guard. The rifle went from a .5-.75" shooter to shotgun patterns in the same range session.
If your scope has an ajustable objective lens be sure it is set properly for the distance you are shooting.
Make sure there are no dings or chips in the muzzle crown.
The rifle should be placed on the bags in the same maner for each shot. Keep the rear bag in the same position and watch the sling swivels don`t contact the front rest during recoil. If you hold the forearm on one shot do it for all, keep the same stock to shoulder tension.
If you can`t find any other possible causes, and the demons are still there, maybe you are makeing the wrong offerings to rifle gods. Virgins and beer don`t work, I`ve tried. (well beer anyway) You`re on your own figuring what to do with that one.