Over a 4-day Audax that would happen to my HR too. The max would descend as well - 158 at start, 132 at end. I always used to be in great form on day 3, between 600 & 900 km in, then dead-ass mutton on the last day.
I mentioned it once to my cardiologist (happy the man who cannot say "my cardiologist"): he shrugged and said "adaptation, then fatigue".