Some misconceptions above - aerodynamics are very important, even - or especially - for slow riders.
The main thing you are working against, over the course of a long ride, is wind resistance.
Aerodynamics becomes significant when you are moving at 20+km/h vs the air. As QG says, air speed isn't always the same as speed vs the land, as winds exist. So if you ever ride into a headwind, then aerodynamics matter.
Also, if you ride at an average speed of 20km, or 15km, or whatever, you don't go at a steady speed but your speed varies, mostly according to hills. So, on a long ride, as well as a chunk of time at very low speeds, you will spend a fair amount of time at high speeds.
Finally, while fast riders receive a larger % benefit from aerodynamic improvements, slower riders receive a greater absolute time benefit over a given distance than faster riders, because they are in the wind for longer.