Actually, if you are talking about dry cell batteries don't bother with a multimeter. Get a cheap battery meter from Amazon.
This is a valid point. A multimeter's voltage range measures voltage, while drawing as little current from whatever you're probing as is practical. This is usually what you want for diagnosing circuits, taking measurements etc.
But the main electrical change that batteries undergo as they discharge isn't so much that their voltage drops, as their internal resistance increases. A partially discharged battery that's been sitting around for a while may give a nice high voltage when measured with a voltmeter, but this drops as soon as it's put under load and some of that voltage is lost to overcoming the internal resistance.
Cheap battery meters incorporate a resistor that puts a load on the battery while measuring its voltage. This gives a much better indication of the battery's state of charge.
Some multimeters might incorporate a battery testing mode that does this, but it's not a normal feature.