I still think the first question for Blodwyn Pig is
why. You mentioned Audax, and like andyoxon that's why I first got one. I've now had several, but I still don't think I'm an expert.
For Audaxes, as you probably know there's now usually a GPX file available from the organiser, or occasionally an enthusiastic fellow-participant. Since Audaxes specialise in twisting, turning routes using obscure back roads then, as others have said, following the box on your bars can be much easier than using a paper map, and more certain than relying on a route sheet when there may be several turnings all signed towards Littleton. Since you can choose the screen zoom, you decide between getting a very close view that gives a few hundred metres' warning of a turn, and a somewhat wider view that gives a pretty good sense that the next turn is more than a mile away. I prefer the former. A line on the map shows your route. Some versions of the file may give turn instructions, such as "go left arrows" that appear as you approach the junction, and/or beeps. You don't usually get the annoying voices that you hear from car sat-navs!
In this mode, you (or whoever set up the GPX) are in control of the routing decisions, and the box is just showing you where you are and where you planned to go next. That's not like a car sat-nav of course, which tends to take you where you want to go, but make the decisions on how you get there. This extends quite easily to your own rides, which you can plan in advance on any of a number of PC-based systems. Then you download a GPX and use it as above. I've occasionally used this for urban riding as well, going across London to suppliers, or to railway stations on the way to Audax events - I get Cyclestreets, which I trust much more than any GPS box, to plan me a route, sense check it, and then upload the GPX to the GPS.
But you absolutely can let the GPX behave more like a sat nav, and choose the route for you. This always comes with a frisson of excitement; just as car sat navs have got lorries stuck down narrow roads, so bike ones, as others have mentioned, may happily take your best road bike down steep, bumpy, muddy tracks that would challenge an MTB. One additional thing that a bike unit may understand, that still seems to perplex car ones, is the idea of going, instead of from A to B, from A to A. So, if I'm in an unfamiliar area, mine will set up a 30-mile circular route for me. To be honest, I never use mine like this; I love paper maps (or, now, electronic representations of paper maps), and would never dream of going anywhere without spending more time than the ride will take, just looking on the map at where I'm going and what's around. So, the idea of not knowing in advance where the thing is going to take me is anathema. But, having decided, I might record my plan on a PC-based tool, generate a GPX, and use it in the same way as above again.
Something I've developed into more recently is recording rides. With the whole social media thing has arisen the idea of sharing the rides you've done with others, either so that they can admire them, or so that they can benefit from your local knowledge when visiting your area. I'm fundamentally of the view that I'm too boring for anyone to care, so I don't share my rides, but two things have happened to me. One is the club starting an annual mileage competition. I'll never come near the top of it, but it's given me impetus to track my mileage, and try at least to beat 10 miles a day (average). The second is a heart bypass, and being encouraged to exercise. OK, I'm a cyclist, so I was never exactly inactive, but the upshot is wearing a heart monitor and an "exercise watch", and tracking what I've done more. All these things connect easily to GPS units - so, on my last Audax, I can see what happened to my heart in the hilly bits, and how it recovered in the cafe stops, and generally be sure that I'm neither over- nor underdoing it. (Anyone riding near me next time needn't worry; I'm no more likely than the next rider to keel over, but a certain amount of checking does no harm. Those doing serious training would want the same information for different reasons. I'm just trying to illustrate what you can do.)
And then that has extended into using the tracking for virtual racing ("Strava segments"). Since everyone's riding is being recorded, it's not difficult to figure out who has done the fastest ride ever up any given hill. All you need is set start and finish points. But I guess that's not your first thought for a use of these things. It's about my last (probably because I'd be more likely to be slowest ever).
And yes, once you've decided that you want to try these dark arts, then the whole issue of which product opens up. And using a phone is an increasing possibility. It used to be that the phone's battery wouldn't last a day ride. That's improving I think. It may be that just trying it on a phone, or buying a basic GPS second hand, or borrowing/asking about one from a friend, may be the best way in. It's like buying a bike really - you probably won't get your ideal one first time, because you won't know what your ideal one would be like anyway.
If none of these things interests you, you maybe don't need a GPS