Sounds like the bolt isn't traveling back all the way all the time. It's most likely dirt,lead accumulation. Clean it!
It could also be a stuck or sticky extractor...the ejector could be worn or there could be a burr or wear in the action that's slowing down the bolt speed. Have you tried some high velocity ammo like CCI Stingers?
RG