Yet Another Cycling Forum
General Category => The Knowledge => GPS => Topic started by: quilkin on 31 July, 2021, 09:34:25 pm
-
Well, I tried a Garmin a few years ago and it kept crashing so I returned it. But today I have found a new reason not to like this company. I have built a website which allows people to upload their GPX files for others to share. It's working fine but some files appear to be bigger than necessary. In particular this is from a Garmin Connect file:
......
<trkpt lat="50.25767440907657146453857421875" lon="-4.7910380177199840545654296875">
<ele>82.40000152587890625</ele>
<time>2020-08-30T09:42:07.000Z</time>
<extensions>
<ns3:TrackPointExtension/>
</extensions>
</trkpt>
<trkpt lat="50.257544405758380889892578125" lon="-4.79131378233432769775390625">
<ele>82.8000030517578125</ele>
<time>2020-08-30T09:42:11.000Z</time>
<extensions>
<ns3:TrackPointExtension/>
</extensions>
</trkpt>
.....
Is it really necessary to record the elevation, and the latitude and longitude, to a billionth of a micrometer? This just show a really poor programming practice. It's wasting time and resources dealing with unneccesarily large files.
For comparison purposes, Strava is a lot better:
.....
<trkpt lat="50.258950000000006" lon="-5.058820000000001">
<ele>45.33</ele>
</trkpt>
<trkpt lat="50.25887" lon="-5.058610000000001">
<ele>46.96</ele>
</trkpt>
.....
and RideWithGPS is a 'sensible' one:
.....
<trkpt lat="50.26242" lon="-5.05012">
<ele>6.9</ele>
</trkpt>
<trkpt lat="50.26226" lon="-5.0504">
<ele>6.9</ele>
</trkpt>
.....
-
Here's an extract from a GPX from my eTrex:
<trkpt lat="57.1594102588" lon="-2.1076577157">
<ele>42.44</ele>
<time>2021-07-25T09:05:13Z</time>
</trkpt>
-
Well, I tried a Garmin a few years ago and it kept crashing so I returned it. But today I have found a new reason not to like this company. I have built a website which allows people to upload their GPX files for others to share. It's working fine but some files appear to be bigger than necessary. In particular this is from a Garmin Connect file:
......
<trkpt lat="50.25767440907657146453857421875" lon="-4.7910380177199840545654296875">
<ele>82.40000152587890625</ele>
<time>2020-08-30T09:42:07.000Z</time>
<extensions>
<ns3:TrackPointExtension/>
</extensions>
</trkpt>
<trkpt lat="50.257544405758380889892578125" lon="-4.79131378233432769775390625">
<ele>82.8000030517578125</ele>
<time>2020-08-30T09:42:11.000Z</time>
<extensions>
<ns3:TrackPointExtension/>
</extensions>
</trkpt>
.....
Is it really necessary to record the elevation, and the latitude and longitude, to a billionth of a micrometer? This just show a really poor programming practice. It's wasting time and resources dealing with unneccesarily large files.
For comparison purposes, Strava is a lot better:
.....
<trkpt lat="50.258950000000006" lon="-5.058820000000001">
<ele>45.33</ele>
</trkpt>
<trkpt lat="50.25887" lon="-5.058610000000001">
<ele>46.96</ele>
</trkpt>
.....
and RideWithGPS is a 'sensible' one:
.....
<trkpt lat="50.26242" lon="-5.05012">
<ele>6.9</ele>
</trkpt>
<trkpt lat="50.26226" lon="-5.0504">
<ele>6.9</ele>
</trkpt>
.....
Except in the system itself, it probably just uses a float. so it's going to use 32 bits in memory, whether it's 50.26242 or 50.122563452312381. Does it need to have that precision? no. Does it take up more space when you write the file to disk? yes. But in the grandscheme of things, cpu is cheap, disk is cheap. And 99.9% of people never look at the GPX, so never see it. It's just a 32bit float.
J
-
If you want smaller file sizes, use FIT format.
-
Worrying about the unnecessary size of text files in 2021 is poor programming practice.
I presume you’re reprocessing the files anyway then you can store them at whatever precision you want and throw the original away.
(I mean, I hope you’re not just letting people download each other’s raw files)
-
It’s a piece of piss for you to round or truncate those numbers and drop elements you don’t want when you process the GPX before storing the result.
-
I'm not worried about the size of text files - yes I know I can add easily code to shorten them (now I know they're so big). But first they have to be sent somewhere (waste of bandwidth), and the microprocessors in the devices have to deal with them (process time, battery life, etc). Just seems poor practice to me. And poor practice here suggests poor practive in the devices themselves - no wonder mine (any many others that I've read about in threads) crash so often. Sorry if everyone disagrees!
-
Why do they have to be sent somewhere ? You process the GPX when they are uploaded to your website. You need to process it anyway or you may be accepting malicious executables for download by unsuspecting users.
-
I'm not worried about the size of text files - yes I know I can add easily code to shorten them (now I know they're so big). But first they have to be sent somewhere (waste of bandwidth), and the microprocessors in the devices have to deal with them (process time, battery life, etc). Just seems poor practice to me. And poor practice here suggests poor practive in the devices themselves - no wonder mine (any many others that I've read about in threads) crash so often. Sorry if everyone disagrees!
Why?
You're more likely to introduce a bug into your code by doing extraneous formatting and processing of a string, than you are if you just treat it as a float*. Even on an 8 bit micro at 8mhz, reading a 16 digit value in 1 digit at a time will take 2µs. It would take 500000 16 digit numbers to take 1 second to read in at 1 digit per clock cycle.
If you do the bare minimum you need to a variable, you will end up with more stable code, as you'll have less stuff that can go wrong.
I appreciate that you may consider the Garmin's to be buggy as hell, and they do appear to have some fun bugs to annoy us all, but I think sticking out lots of digits on the lat/long in the is the least of their worries.
As for transmission of these things, bluetooth is 1Mbit per second typical throughput, if you did have a file with 250k[1] 16 digit lat/long, it's going to take 40 seconds longer to transfer. But that's 250k trkpoints. That's a lot. For a more reasonable 10000 trkpoints in a file, the difference is 1.6s.
The numbers are tiny. It's not like it's an issue of scale, if you're doing lots of these, then you're using a bigger system with 10Gbit networking, and modern 64bit CPU's. In which case you're gonna have higher overheads from other areas than you are storing superfluous digits.
There's lots of reasons to be annoyed with Garmin, this one isn't really worth worrying about.
J
*Tho there is an argument for acting like you would with currency, and treating it as an integer number of microdegrees or nanodegrees, stored as a 32bit signed int. But that's still gonna be 32 bits in memory.
[1] 500k numbers, 2 per coordinate.
-
Where are these GPX files coming from?
-
Where are these GPX files coming from?
from club members' phones or PCs, - routes they have actually ridden or designed on a route-planning site.
Why do they have to be sent somewhere ? You process the GPX when they are uploaded to your website. You need to process it anyway or you may be accepting malicious executables for download by unsuspecting users.
Processing is done by the server, not in the browser, so they need to be uploaded first, before being processed and stored in the db. The fact that some of these files are so big means I've had to change the setting for max file size of upload - which in itself possibly allows bigger chunks of malware.
Anyway, thanks to my 'discovery' and all your comments I can now move to improving the site.
Regardless of storage space or transfer rates (which I agree aren't really an issue) I still think it's messy of Garmin to create files like this.
-
Where are these GPX files coming from?
from club members' phones or PCs, - routes they have actually ridden or designed on a route-planning site.
Why do they have to be sent somewhere ? You process the GPX when they are uploaded to your website. You need to process it anyway or you may be accepting malicious executables for download by unsuspecting users.
Processing is done by the server, not in the browser, so they need to be uploaded first, before being processed and stored in the db. The fact that some of these files are so big means I've had to change the setting for max file size of upload - which in itself possibly allows bigger chunks of malware.
Anyway, thanks to my 'discovery' and all your comments I can now move to improving the site.
Regardless of storage space or transfer rates (which I agree aren't really an issue) I still think it's messy of Garmin to create files like this.
Well of course they have to be uploaded to your server. Otherwise you wouldn’t have a website service for sharing GPX! Processing the GPX is no extra bandwidth on top of that. You could in fact pre process in the browser if bandwidth was an issue. Though it’s unlikely even with the largest GPX file.
-
I note that Strava's API accepts GPX (and whatever other things it accepts) in gzipped form. While testing my strava uploader script (on a desktop PC with a fast network connection), I found that gzipping before upload made the process marginally quicker. I presume the mobile apps use this to save bandwidth, which makes good sense on a slow cellular connection.
It goes without saying that GPX files are eminently compressable.
-
Cycling is a tiny part of the Garmin market (so I'm told) and I'm guessing there are customer segments who need that level of precision.
I'm also guessing that producing separate code for each customer segment would be non profitable for Garmin.
-
It is reassuring that every small plane we have been in across Africa, Far East, etc has had a Garmin flight GPS. Even if it gives a date in the last century on startup.
-
I'm also guessing that producing separate code for each customer segment would be non profitable for Garmin.
The code in question is part of Garmin Connect, which is an entire platform that’s unique to the hiking / fitness / cycling segment.
-
Cycling is a tiny part of the Garmin market (so I'm told) and I'm guessing there are customer segments who need that level of precision.
No-one needs (or gets) that level of precision. Anything beyond the 9th decimal place is meaningless noise. The example quoted in the OP had 29 decimal places, it's offensive to any right-thinking person.
-
Having checked a few of my GPX files (as archived on the device & copied by USB)...
64s = 10 dp
30x = 10 dp
Vista HCx = 6 dp
60Csx = 6 dp
IIRC, it's 5 dp for around 1 m precision (the actual precision will vary depending on whether it's E-W or N-S)
-
I've just taken delivery of a couple of plastic bushes, to refurbish a bike thing.
I see they are catering for the weight weenies who keep track of this kind of thing in a spreadsheet...
(https://live.staticflickr.com/65535/51387934616_11405e970f_z.jpg) (https://flic.kr/p/2mhYACG)
micrograms (https://flic.kr/p/2mhYACG) by Ron Lowe (https://www.flickr.com/photos/62966413@N04/), on Flickr
-
You joke but I think some obsessives would try to pare it down to 1.637816g!
-
From the Garmin website:
ABOUT US
Built to last.
Three simple words that describe our products, our company, our culture, our future. As a leading, worldwide provider of navigation, we are committed to making superior products for automotive, aviation, marine, outdoor and sports. For more information about our company, see the about us (http://www.garmin.com/en-GB/company/about) section.
I don't think so. My (four year-old) Edge Touring Plus doesn't boot up past the start-up screen
(despite me carrying out the suggested reboot). I'll contact Garmin UK tomorrow, to see if they
can help me.
-
But in the grandscheme of things, consumer cpu is cheap, disk is cheap
Bold is my edit.
I've got a colleague who after 15 years still can't get his head around that fact that 1GB of disk space in a data centre is considerably more expensive than desktop.
I also had a recent "shat my self" moment when I found out that our test servers are on enterprise level storage... which means backup service, UPS etc all the sorts of things you don't give a toss about at Test or even UAT level. But that's not my side of things...
Worrying about the unnecessary size of text files in 2021 is poor programming practice.
Not really, if we were talking about old school servers where you paid for the server with a set capacity then fair enough, but cloud changes that.
You can make significant financial savings on cloud by writing code to use services charged at usage level rather than provisioning level, the cost of a SQL server on Azure is considerably more for a small database for example than using Azure DB which is charged by what you're doing in it.
That should be encouraging high efficiency at developer level, but we're pretty much conditioned to the concept of moores law and CD distribution, not where things have gone.
Platform providers are profiting off us not being efficient in what we do.
Which very much includes worrying about inefficiency in text files, because scale it up massively and you're doing a KLF.
But the OP isn't working massively...
If I was looking to optimise the transfer of GPS tracks I would not be using XML, I probably wouldn't even be using JSON either, too many curly brackets, I'd probably do what HL7 does and use a pipe delimited string.
I'd make the browser do the work of converting from GPX to my custom format, including reducing the values to sensible scales and probably turn on transport compression.
Why? Well why would I pay more for CPU, Bandwidth and Storage space when you can put that load on the clients electricity bill and pc/phone life instead?
From the Garmin website:
ABOUT US
Built to last.
Three simple words that describe our products, our company, our culture, our future. As a leading, worldwide provider of navigation, we are committed to making superior products for automotive, aviation, marine, outdoor and sports. For more information about our company, see the about us (http://www.garmin.com/en-GB/company/about) section.
I don't think so. My (four year-old) Edge Touring Plus doesn't boot up past the start-up screen
(despite me carrying out the suggested reboot). I'll contact Garmin UK tomorrow, to see if they
can help me.
More like it, I had 2 510s and a Fenix 2 and intend to avoid Garmin consumer devices.
I've seen their nautical kit on a RIB on the way out to StKilda quite impressive and didn't crash once... mean while my Fenix2 had a fit about timezones.
-
Meanwhile my eTrex works fine after 7 years...
...I've said it now.
-
As it happens, I did have a routesheet and map as a back-up, but I set out with the determination of
using the Garmin to wholly navigate an audax with for the first time.
Online fixes didn't work, neither did the one suggested by Garmin. They said that as my model
(Garmin Edge Touring Plus) is discontinued, they couldn't fix it or offer a refurbished one. They
did offer me a 30% discount on an upgrade, so I took the 30% off an Edge 830 and paid £249.99.
So much for the tagline on their website: Built to last. The Edge Touring Plus was only four years old. ::-)
-
Meanwhile my eTrex works fine after 7 years...
My eTrex 30 similarly, but the back button is starting to go (not sure if it's the rubber membrane or the tactile switch itself, but either means doom for the waterproofing. In light of the ongoing supply chain problems, and not having time to research GPSes I purchased an eTrex 32x as a low-effort replacement.
It's deeply underwhelming: Screen is the same slightly-higher-resolution murky rubbish that's almost unreadable without direct sunlight or backlight illumination as the Edge series are afflicted by. The only meaningful changes are the increased internal storage and that they've re-arranged the options in the reset menu so your previous 'clear the track' muscle-memory performs a factory reset. Still using a mini-USB port, too.
Anecdotally, it seems a bit more inclined to crash when auto-routing. Probably a function of the OSM-derived mapping. I haven't thoroughly tested it though, as it arrived just before barakta's surgery, and I haven't been able to do a long ride since.
-
So much for the tagline on their website: Built to last. The Edge Touring Plus was only four years old. ::-)
Nothing with an internal rechargeable battery is built to last.
-
Or a safe buy on eBay.
-
I'm hoping to get another garmin edge touring gps as my last one escaped returning homeward from nobbys otp along the bcn navigation somewhere. I rather miss the ability for round trip routing when I'm in a new area :).
I have been looking at a few different gps for cycletouring but I suspect that there still isn't a perfect touring gps out there ::-)
-
I'm hoping to get another garmin edge touring gps as my last one escaped returning homeward from nobbys otp along the bcn navigation somewhere. I rather miss the ability for round trip routing when I'm in a new area :-) .
I have been looking at a few different gps for cycletouring but I suspect that there still isn't a perfect touring gps out there ::-)
My Garmin Edge Touring Plus went kaput over the weekend. Non of the online fixes, or the one offered
by Garmin UK solved the issue. They said that as that model was no longer made they couldn't fix it
or offer a refurbished one. However, the did offer me a 30% discount on an upgraded model. I took
them up on that and purchased a Garmin Edge 830 for £249.99.
I think the nearest to an Edge Touring is the Edge Explore. If you're interested, I have one as a
back-up to the Touring Plus, but as Garmin gave me the 30% offer on a better model, it won't used.
-
I'm certainly interested :).
I will send you a pm
-
Similarly, my eTrex 20 is still working fine after about six years (not sure exactly how long I've had it but definitely at least five).
-
I've had three etrexes.
The first lasted 4 years
The second lasted 5 years
The third lasted a month.
My longest running device is my Edge 500, which is now hugely scratched and haa been retired, but does still work.
-
Hmm...
eTrex (bought 2004ish): Sold after a couple of years, in full working order.
eTrex Legend (bought 2007ish): Still have it, still works fine.
eTrex Vista HCx (bought 2009): Still have it, works fine but the rubber band started to go after about 3 years, re-taped to keep it going for another year, but now too slack to line up with the buttons properly.
eTrex 30 (bought 2013): Still working, but the 'back' button is on borrowed time.
eTrex 32x (bought 2021): Shiny and new, but disappointing murky screen (see above).
There seems to be an inverse relationship between the quality of the software/features and the quality of the hardware. The Vista HCx was by far the best software-wise for cycling, on account of its clear turn instructions and ability to make a contemporaneous copy of the recorded track on the SD card that was very hard to accidentally delete.
The eTrex 30 appears to have been the sweet spot, and was certainly the best iteration of the hardware. I reckon 7 years is a good innings for a tactile switch behind a rubber membrane. But it's clear that Garmin can't be arsed with developing the eTrex series any more - the real money is in short-lived fitness devices, and they concentrate their innovation on the bulkier higher-end outdoor units.
-
Certainly agree the old eTrex worked well (lost one, other one still ticking maybe 12 years on albeit with a couple of lost columns of pixels).
Recent devices though (e.g. 530) have a painful user interface, and crash a lot. So unpleasant I was actually not upset when a really minor bump opening a gate resulted in a cracked screen and a dead device.
-
Recent devices though (e.g. 530) have a painful user interface, and crash a lot.
That's not been my experience using the Edge series.
-
2005 - Legend - OK though lack of memory meant loading map segments along with routes, replaced by
2010 - Oregon 450t Probably my favourite, still works though had to bodge the power button with the eraser from the end of a pencil, cut down and taped in. SD card no longer readable
? - Edge Touring Plus - hateful device, very glad when it died after a random software glitch
2020 - Etrex 32x - OK, had some improvements and I hope whoever found it on the Mineral Line near Watchet is making good use of it :(
2021 - Oregon 700 probably the best so far though the battery life is nothing like Garmin claim
I agree with comments over the clunky UIs though after a (steep) learning curve you learn to live with it, ditto Garmin's schizophrenia over routes, tracks, activities - still working on that one
-
My first Garmin was a GPS 50 (http://retro-gps.info/Garmin/Garmin-GPS-50/index.html) bought in 1994 (ish). I have had various ones since then and am currently using an eTrex 30X bought in 2015. None of them have died in use although they have all had the occasional glitch. For some reason the 30X dislikes going through Barrow upon Soar and Alrewas ::-)
They all have their quirks and I have just found the work flows and menu selections that allow me to do what I want to do.