We used to train even before power meters became available.
Sure, but training with power was a huge leap in terms of structured training, it's also now not beyond the cost of mere mortals. My power meters were ~£250 each (one geared, one for fixed) which is relatively small cost compared to, say, a £2k bike.
VAM can be measured with a stopwatch and it is done automatically with things like Strava... the error is minimum, down to rounding of seconds.
The point of power is that it is immediate, you can't calculate your VAM for the last 3 seconds, you can't calculate if you've gone off way too fast on the first 30 seconds of a climb. Power gives an almost immediate reading so you can avoid burning yourself out early.
Training based on VAM relies on riding to a perceived effort level and then calculating the value after the fact. There's no immediate feedback which is what power gives you. Want to ride up that hill at a certain effort level, pick a Watt number and then maintain that all the way up regardless of the gradient.
A power meter is a rather complicated unit, consisting of a strain gauge to measure torque and a unit to measure RPM. Both of them are prone to error, especially the former.
Strain gauges are incredibly sensitive and it doesn't take much to produce a wrong reading. For instance if it is on the crank, then you hit it with your shoe and you produce an incorrect reading for that pedal stroke. I am also unsure what happens when the bottom bracket ages. there is a bit of play and the response of the crank upon the pedal stroke becomes different... there are a number of ways the calibration can be altered.
I wouldn't take those numbers as Gospel... on balance I tend to trust a solid hill and a stop watch.
Luckily there are people like DC Rain Maker that comprehensively test different brands/models of power meters against each other by using different types (i.e. pedal based power meters against crank based, and/or against hub based). The junk ones aside they all tend to sit within their published tolerances (1-2% generally) - that's perfectly good enough for the likes of me.
The problems you seem to think might exist within power meters don't really exist in reality (again, except for the rubbish brands).
That said, one day I will probably buy a power meter, but if you know how to collect data, they seem to offer so little for so much money!
And if you do buy a power meter you'll be amazed at how much more data they provide, data that you can't extract using a stopwatch.