I used to have the same issue, it was caused by the guard twisting on the chainstay bridge bolt, I added a second one using the clip supplied with new guards and it's been fine since. If you have a rack, you can also stiffen up the guards with an additional attachment point and shorter stays to that rather than the dropout.