Author Topic: Slim down your bloated GPX track files  (Read 9879 times)

Phil W

Slim down your bloated GPX track files
« on: June 09, 2016, 04:27:41 pm »
Are you feeling bloated, or at least are your GPX track files feeling bloated? Then read on.

Sian Lambert posted on the Audax FB group a few days back about her GPX file being rejected by the AUK mandatory route form because it was too large / bloated. It became clear not many know how to reduce the size of their GPX or even if they do they find it quite hard to get right.

I've taken something I've been doing for years to slim down GPX files and converted it into a simple one page web site / application.

All you need to do is select a GPX file on your computer, and a tolerance. A new slimmed down GPX based on the tolerance will be returned to you in zip format .

Some sample outputs for a 200km track

1m tolerance - .209KB (58KB zipped) about 3200 track points
 5m tolerance - 93KB (27KB zipped) about 1400 track points
 10m tolerance - 61KB (19KB zipped) and about 900 track points
 20m tolerance - 42KB (13KB zipped) and about 600 track points
 50m tolerance - 23KB (7KB zipped) and about 300 track points
 100m tolerance - 15KB (4KB zipped) and about 200 track points

I'm hosting it on a free server and it's free to use. Use at your own risk and I certainly recommend putting the slimmed down GPX back into a mapping program to make sure you are happy with the result.

Because it's a free server, performance will be what it is, but unless a great number of you hit it at the same time it should process your GPX in a timely manner.


https://simple-gpx.herokuapp.com

frankly frankie

  • I kid you not
    • Fuchsiaphile
Re: Slim down your bloated GPX track files
« Reply #1 on: June 10, 2016, 09:13:06 am »
Noting of course that '1m tolerance' exceeds the accuracy of the consumer GPS system that we use.

That looks very good and stripping out the timestamps and elevations really makes a big difference.  Of course that's not appropriate when dealing with a ridden tracklog - but for Planner-generated files this would work very well, if your main focus is reducing file size.   Mind, with GPX simply zipping usually achieves a huge reduction in file size - sometimes a tenfold reduction.
If your focus is a specific number of points (for example if you use an older Etrex, you need to target 500 points) - then I think Bikehike is easier to use.

I experimented:
A 200km Track made in Bikehike - contained 3070 trackpoints, 378Kb, 43.4Kb zipped.
Reduced to 1000 points in Bikehike - (actually 976 trackpoints) 120Kb, 15.5Kb zipped.
Reduced to 500 points in Bikehike - (actually 500 ) 62Kb, 8.5Kb zipped.  All these files include elevations and timestamps.

The first file (3070 points / 378Kb) run through your converter at its default setting - reduced to 887 points, 41Kb, 7.7Kb zipped.

Nice, clean minimal utility.  :thumbsup:
"This is a complex subject, with a need for more than one highlighter pen."

Phil W

Re: Slim down your bloated GPX track files
« Reply #2 on: June 10, 2016, 03:31:33 pm »
Yes 1m does exceed accuracy of GPS. I may well remove that option but it does still reduce bloat whilst essentially keeping the track mostly untouched unless it started with a ridiculous number of track points.

Yes this is for the tracks you plan to follow rather than recorded tracklogs where you wouldn't want to strip out timestamp and elevation data.

I'm toying with the idea adding the option to split long tracks into 50/100/200km sections with a suffix etc.  So you could feed it a 600km track, tick the split option and split distance, and it'd reduce your track points and GPX size and also split the track into 3 tracks say BCM-1 BCM-2 BCM-3 where the prefix BCM etc. is of your choosing.

Phil W

Re: Slim down your bloated GPX track files
« Reply #3 on: June 10, 2016, 03:40:57 pm »
Not just file size either, I sense my eTrex 20 is much quicker at processing the slimmed down tracks than the bloated versions that online maps produce. This is of course anecdotal :-)

Baron Brymbo von Pickelhaube

  • Adel auf dem Radel
Re: Slim down your bloated GPX track files
« Reply #4 on: June 11, 2016, 02:49:46 pm »
I'm not sure I understand your tolerance concept - surely a 200km track at 1m tolerance with about 3200 track points has a trackpoint at every 200*1000/3200 metres?  This is a trackpoint 62.5 metres, not every 1 metre.  So what does 1m tolerance mean if not the average distance between trackpoints?  Or have missed/miscalculated something?
The present is a foreign country: they do things differently here.

Phil W

Re: Slim down your bloated GPX track files
« Reply #5 on: June 11, 2016, 04:10:24 pm »
I'm not sure I understand your tolerance concept - surely a 200km track at 1m tolerance with about 3200 track points has a trackpoint at every 200*1000/3200 metres?  This is a trackpoint 62.5 metres, not every 1 metre.  So what does 1m tolerance mean if not the average distance between trackpoints?  Or have missed/miscalculated something?

Yes you've misunderstood. It's not about the number of track points per metre.  It's about removing redundant track points from the track but still having a track that aligns to the original within the specified tolerance.

See https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm and look at the animated diagram on the right hand side.  So lets for simplicity say I have a track with 3 track points.  I draw a line between the first and last track points. If the second track point is less than (or equal) x metres from that line then it is removed, if it further than x metres it is kept.  Thus keeping the aligment of the simplified track to within the tolerance.

Re: Slim down your bloated GPX track files
« Reply #6 on: June 12, 2016, 09:54:07 pm »
GPS Babel does a pretty good job of this.  I used to use it to downsize GMaps Pedometer routes for the eTrex.  It is clever enough to only delete points where there are no bends or turns to be mapped, so on an archetypal stretch of Roman road it would just have a point at either end but it would closely map any wiggly bits on other roads.
Never tell me the odds.

Phil W

Re: Slim down your bloated GPX track files
« Reply #7 on: June 13, 2016, 01:42:34 pm »
GPS Babel does a pretty good job of this.  I used to use it to downsize GMaps Pedometer routes for the eTrex.  It is clever enough to only delete points where there are no bends or turns to be mapped, so on an archetypal stretch of Roman road it would just have a point at either end but it would closely map any wiggly bits on other roads.

It'll be using the same algorithm (or a close variant) my web app uses, but my app provides a much simpler and easier way of doing it, which is the point.

I've now added the option to optionally also split a long track down into either 50km, 100km, or 200km tracks.

Re: Slim down your bloated GPX track files
« Reply #8 on: June 13, 2016, 07:42:07 pm »
Works very  well, 600K route downloaded from RWGPS  1.8MB - put  thro' @5M accuracy comes  out  at an unzipped 132Kb  :) 

Re: Slim down your bloated GPX track files
« Reply #9 on: June 13, 2016, 08:57:00 pm »
Phil.  I  notice that you tag the file, quite right too! -
Quote
creator="Simple GPX https://simple-gpx.herokuapp.com ">
How about using this  to auto-compress zip and tag AUK copyright to GPX tracks when orgs. upload to the new AUKweb-site?

Phil W

Re: Slim down your bloated GPX track files
« Reply #10 on: June 13, 2016, 10:55:17 pm »
Phil.  I  notice that you tag the file, quite right too! -
Quote
creator="Simple GPX https://simple-gpx.herokuapp.com ">
How about using this  to auto-compress zip and tag AUK copyright to GPX tracks when orgs. upload to the new AUKweb-site?

I'm not involved in the new website and I'm sure they've tons of competing priorities to deal with already. No reason you couldn't submit your idea to the new IT manager though and see what he says. It would certainly simplify the workflow for many.

Oaky

  • ACME Fire Safety Officer
  • Audax Club Mid-Essex
    • MEMWNS Map
Re: Slim down your bloated GPX track files
« Reply #11 on: July 27, 2016, 12:07:01 pm »
i think I found a bug (or perhaps there's something pathological in this particular RideWithGPS output).  I have a file that reduces to nothing when I upload it (default 10 metre setting).

input: 831 points, 36 miles
output: 2 pts, 0 miles
You are in a maze of twisty flat droves, all alike.

85.4 miles from Marsh Gibbon

Audax Club Mid-Essex Fire Safety Officer
http://acme.bike

Phil W

Re: Slim down your bloated GPX track files
« Reply #12 on: July 28, 2016, 11:16:22 am »
i think I found a bug (or perhaps there's something pathological in this particular RideWithGPS output).  I have a file that reduces to nothing when I upload it (default 10 metre setting).

input: 831 points, 36 miles
output: 2 pts, 0 miles

Try again, should be fixed. 

Oaky

  • ACME Fire Safety Officer
  • Audax Club Mid-Essex
    • MEMWNS Map
Re: Slim down your bloated GPX track files
« Reply #13 on: July 28, 2016, 11:18:11 am »
i think I found a bug (or perhaps there's something pathological in this particular RideWithGPS output).  I have a file that reduces to nothing when I upload it (default 10 metre setting).

input: 831 points, 36 miles
output: 2 pts, 0 miles

Try again, should be fixed.

Yep - working now!
You are in a maze of twisty flat droves, all alike.

85.4 miles from Marsh Gibbon

Audax Club Mid-Essex Fire Safety Officer
http://acme.bike

Re: Slim down your bloated GPX track files
« Reply #14 on: June 27, 2017, 08:02:10 pm »
Phil.  I  notice that you tag the file, quite right too! -
Quote
creator="Simple GPX https://simple-gpx.herokuapp.com ">
How about using this  to auto-compress zip and tag AUK copyright to GPX tracks when orgs. upload to the new AUKweb-site?

AUK doesn't own the copyright though, it'll most likely belong to the organiser.
"Yes please" said Squirrel "biscuits are our favourite things."

Oaky

  • ACME Fire Safety Officer
  • Audax Club Mid-Essex
    • MEMWNS Map
Re: Slim down your bloated GPX track files
« Reply #15 on: June 27, 2017, 09:54:11 pm »
Phil's tool reduces the track complexity by removing unnecessary points (and stripping superfluous information e.g. elevation).  It's meant for simplification of tracks that are intended for navigational use in GPS units (some of which have  limits on the number of trackpoints for navigation).

zipping a file does a binary compression of the track, without changing the number of points.  you'd use this typically when transferring the track to soemone else, to save on bandwidth and/or storage space.  The track once unzipped would be identical to the original,  with the number of points unchanged etc.

(Incidentally,  Phil's tool does also zip the file, probably for exactly the reasons in the second paragraph above, but only after doing the clever stuff as per the first paragraph above).
You are in a maze of twisty flat droves, all alike.

85.4 miles from Marsh Gibbon

Audax Club Mid-Essex Fire Safety Officer
http://acme.bike

Feanor

  • It's mostly downhill from here.
Re: Slim down your bloated GPX track files
« Reply #16 on: June 27, 2017, 10:12:04 pm »
Because the Garmin units can't use a ZIP file, they need the un-zipped GPX file.
The issue is not so much the size of the file in bytes; it's the number of trackpoints they contain.
That's what some of the older Garmins choke on.

frankly frankie

  • I kid you not
    • Fuchsiaphile
Re: Slim down your bloated GPX track files
« Reply #17 on: June 28, 2017, 10:19:42 am »
If you present a file already zipped does this cause issues? FWIW as a DIY org I can report that the auto zip function seems to have increased successful use of the form significantly.

Hi Andy, I'm glad to hear that the zipping appears to have improved the entry success rate.  That's my observation too, from looking at error logs generated whenever an entry fails.  Basically the entry form accepts files up to 5x larger than before and zips them, if they were not zipped before submission.  Files already zipped before submission are unaffected, that is they don't get double-zipped, and they are still subject to the stated size limit on the form.  Because zipping of GPX is highly efficient (much more than 5x), the best approach with a very large file is still to zip it before submission - which is one part of what Phil's very neat utility does.

I must say I have used Phil's utility a lot, in contexts nothing to do with Audax or DIYs - yes other tools such as GPS Babel can do similar things with more options, but I find this is an incredibly useful 'quick fix' to get to a clean and efficient GPX file. 
"This is a complex subject, with a need for more than one highlighter pen."

Phil W

Re: Slim down your bloated GPX track files
« Reply #18 on: June 28, 2017, 10:44:12 pm »
Because all edge units up to the 800 series and all etrex units up to 10/20/30 will truncate or crash if trying to navigate a track greater than 10,000 points. Ride with GPS and other online tools will typically spit out more than 10,000 points even for a 200km track. For even older units you can change the tolerance to drop below 500 points, or also decide to split a track into multiple tracks based on distance.

So it solves two problems,  10,000 track point  limit and size of files. With just zipping some ride with GPS GPX files are still in the Mb in size.  Plus it also gives the GPS track navigation and auk mandatory route comparison tool a lot less work to do. Efficiency and stability all round.

Phil W

Re: Slim down your bloated GPX track files
« Reply #19 on: June 28, 2017, 10:54:28 pm »
One other point.

I didn't build the tool for mandatory GPX track  submission as I built it long before such a thing existed. I just saw it could be used to solve that particular problem (too large a file submitted) for some, without it needing to be modified in any way.

Re: Slim down your bloated GPX track files
« Reply #20 on: June 28, 2017, 11:01:05 pm »
It is true that Garmins can't use zip files and that older devices have a 500 trackpoint limit. Perhaps Phil's tool could be useful in this area but this isn't the subject of the OP. It is all about file size.

Eh? Phil's original post gives half a dozen examples of how choosing different accuracies affects the number of points, which is the only aspect of his tool I've been interested in.

As a case in point, the RWGPS tracks of days one and two of this weekend's Celtic Knot were about 13000 and 12000 points respectively, leading to them being truncated on my Etrex at about 290km each. Running them through the tool at default settings brought them each down to more like 2500 points - they opened faster on the device, and more importantly showed the track for the full length of the day.

It's definitely useful in this area, no perhaps about it.

Re: Slim down your bloated GPX track files
« Reply #21 on: July 13, 2017, 10:19:24 am »
Because all edge units up to the 800 series and all etrex units up to 10/20/30 will truncate or crash if trying to navigate a track greater than 10,000 points. Ride with GPS and other online tools will typically spit out more than 10,000 points even for a 200km track. For even older units you can change the tolerance to drop below 500 points, or also decide to split a track into multiple tracks based on distance.

So it solves two problems,  10,000 track point  limit and size of files. With just zipping some ride with GPS GPX files are still in the Mb in size.  Plus it also gives the GPS track navigation and auk mandatory route comparison tool a lot less work to do. Efficiency and stability all round.

Do TCX files suffer from the same problem, i.e. too many points for - in my case - an Edge Touring? I've just replaced my 810 (died of old age) with an Edge Touring. I never had an issue with anything up to and including 400km TCX files on the 810 but don't know whether the new device will cope.
R10000 x 2   RRtY x 7    SR x 7    E = 128

αdαmsκι

  • Twitter @Dr_AAY Instagram @ucfaaay Strava @ucfaaay
  • Look haggard. It sells.
Re: Slim down your bloated GPX track files
« Reply #22 on: July 16, 2017, 05:54:33 pm »
I used this yesterday to slim down the tracks for my ride today.  It was very easy to use and at no point was I unsure where I was going.

One feature that would be nice would be an option to maintain the elevation data. On my Etrex 20x I get high and low points, but with the elevation data removed from the track these didn't appear. It isn't too much a deal in Essex, but it's very useful when doing long climbs to know the top of the hill involves another X amount of vertical climbing. Ta.
What on earth am I doing here on this beautiful day?! This is the only life I've got!!

https://tyredandhungry.wordpress.com/

Oaky

  • ACME Fire Safety Officer
  • Audax Club Mid-Essex
    • MEMWNS Map
Re: Slim down your bloated GPX track files
« Reply #23 on: July 16, 2017, 09:13:41 pm »
I used this yesterday to slim down the tracks for my ride today.  It was very easy to use and at no point was I unsure where I was going.

One feature that would be nice would be an option to maintain the elevation data. On my Etrex 20x I get high and low points, but with the elevation data removed from the track these didn't appear. It isn't too much a deal in Essex, but it's very useful when doing long climbs to know the top of the hill involves another X amount of vertical climbing. Ta.

You can get (some) high/low points back if your map has elevation information.  These usually correspond to more significant high/lows compared with the ones from a track with elevation which often seemed to be every little lump.
You are in a maze of twisty flat droves, all alike.

85.4 miles from Marsh Gibbon

Audax Club Mid-Essex Fire Safety Officer
http://acme.bike

αdαmsκι

  • Twitter @Dr_AAY Instagram @ucfaaay Strava @ucfaaay
  • Look haggard. It sells.
Re: Slim down your bloated GPX track files
« Reply #24 on: July 16, 2017, 09:23:28 pm »
You can get (some) high/low points back if your map has elevation information.

I'm not sure if my map has this or not. I'll investigate.



These usually correspond to more significant high/lows compared with the ones from a track with elevation which often seemed to be every little lump.

Even if my map has elevation it won't have found anything significant today as I was in mid Essex.
What on earth am I doing here on this beautiful day?! This is the only life I've got!!

https://tyredandhungry.wordpress.com/