Yet Another Cycling Forum

General Category => The Knowledge => GPS => Topic started by: Phil W on June 09, 2016, 04:27:41 pm

Title: Slim down your bloated GPX track files
Post by: Phil W on June 09, 2016, 04:27:41 pm
Are you feeling bloated, or at least are your GPX track files feeling bloated? Then read on.

Sian Lambert posted on the Audax FB group a few days back about her GPX file being rejected by the AUK mandatory route form because it was too large / bloated. It became clear not many know how to reduce the size of their GPX or even if they do they find it quite hard to get right.

I've taken something I've been doing for years to slim down GPX files and converted it into a simple one page web site / application.

All you need to do is select a GPX file on your computer, and a tolerance. A new slimmed down GPX based on the tolerance will be returned to you in zip format .

Some sample outputs for a 200km track

1m tolerance - .209KB (58KB zipped) about 3200 track points
 5m tolerance - 93KB (27KB zipped) about 1400 track points
 10m tolerance - 61KB (19KB zipped) and about 900 track points
 20m tolerance - 42KB (13KB zipped) and about 600 track points
 50m tolerance - 23KB (7KB zipped) and about 300 track points
 100m tolerance - 15KB (4KB zipped) and about 200 track points

I'm hosting it on a free server and it's free to use. Use at your own risk and I certainly recommend putting the slimmed down GPX back into a mapping program to make sure you are happy with the result.

Because it's a free server, performance will be what it is, but unless a great number of you hit it at the same time it should process your GPX in a timely manner.


https://simple-gpx.herokuapp.com
Title: Re: Slim down your bloated GPX track files
Post by: frankly frankie on June 10, 2016, 09:13:06 am
Noting of course that '1m tolerance' exceeds the accuracy of the consumer GPS system that we use.

That looks very good and stripping out the timestamps and elevations really makes a big difference.  Of course that's not appropriate when dealing with a ridden tracklog - but for Planner-generated files this would work very well, if your main focus is reducing file size.   Mind, with GPX simply zipping usually achieves a huge reduction in file size - sometimes a tenfold reduction.
If your focus is a specific number of points (for example if you use an older Etrex, you need to target 500 points) - then I think Bikehike is easier to use.

I experimented:
A 200km Track made in Bikehike - contained 3070 trackpoints, 378Kb, 43.4Kb zipped.
Reduced to 1000 points in Bikehike - (actually 976 trackpoints) 120Kb, 15.5Kb zipped.
Reduced to 500 points in Bikehike - (actually 500 ) 62Kb, 8.5Kb zipped.  All these files include elevations and timestamps.

The first file (3070 points / 378Kb) run through your converter at its default setting - reduced to 887 points, 41Kb, 7.7Kb zipped.

Nice, clean minimal utility.  :thumbsup:
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on June 10, 2016, 03:31:33 pm
Yes 1m does exceed accuracy of GPS. I may well remove that option but it does still reduce bloat whilst essentially keeping the track mostly untouched unless it started with a ridiculous number of track points.

Yes this is for the tracks you plan to follow rather than recorded tracklogs where you wouldn't want to strip out timestamp and elevation data.

I'm toying with the idea adding the option to split long tracks into 50/100/200km sections with a suffix etc.  So you could feed it a 600km track, tick the split option and split distance, and it'd reduce your track points and GPX size and also split the track into 3 tracks say BCM-1 BCM-2 BCM-3 where the prefix BCM etc. is of your choosing.
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on June 10, 2016, 03:40:57 pm
Not just file size either, I sense my eTrex 20 is much quicker at processing the slimmed down tracks than the bloated versions that online maps produce. This is of course anecdotal :-)
Title: Re: Slim down your bloated GPX track files
Post by: Genosse Brymbo on June 11, 2016, 02:49:46 pm
I'm not sure I understand your tolerance concept - surely a 200km track at 1m tolerance with about 3200 track points has a trackpoint at every 200*1000/3200 metres?  This is a trackpoint 62.5 metres, not every 1 metre.  So what does 1m tolerance mean if not the average distance between trackpoints?  Or have missed/miscalculated something?
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on June 11, 2016, 04:10:24 pm
I'm not sure I understand your tolerance concept - surely a 200km track at 1m tolerance with about 3200 track points has a trackpoint at every 200*1000/3200 metres?  This is a trackpoint 62.5 metres, not every 1 metre.  So what does 1m tolerance mean if not the average distance between trackpoints?  Or have missed/miscalculated something?

Yes you've misunderstood. It's not about the number of track points per metre.  It's about removing redundant track points from the track but still having a track that aligns to the original within the specified tolerance.

See https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm and look at the animated diagram on the right hand side.  So lets for simplicity say I have a track with 3 track points.  I draw a line between the first and last track points. If the second track point is less than (or equal) x metres from that line then it is removed, if it further than x metres it is kept.  Thus keeping the aligment of the simplified track to within the tolerance.
Title: Re: Slim down your bloated GPX track files
Post by: rogerzilla on June 12, 2016, 09:54:07 pm
GPS Babel does a pretty good job of this.  I used to use it to downsize GMaps Pedometer routes for the eTrex.  It is clever enough to only delete points where there are no bends or turns to be mapped, so on an archetypal stretch of Roman road it would just have a point at either end but it would closely map any wiggly bits on other roads.
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on June 13, 2016, 01:42:34 pm
GPS Babel does a pretty good job of this.  I used to use it to downsize GMaps Pedometer routes for the eTrex.  It is clever enough to only delete points where there are no bends or turns to be mapped, so on an archetypal stretch of Roman road it would just have a point at either end but it would closely map any wiggly bits on other roads.

It'll be using the same algorithm (or a close variant) my web app uses, but my app provides a much simpler and easier way of doing it, which is the point.

I've now added the option to optionally also split a long track down into either 50km, 100km, or 200km tracks.
Title: Re: Slim down your bloated GPX track files
Post by: aidan.f on June 13, 2016, 07:42:07 pm
Works very  well, 600K route downloaded from RWGPS  1.8MB - put  thro' @5M accuracy comes  out  at an unzipped 132Kb  :) 
Title: Re: Slim down your bloated GPX track files
Post by: aidan.f on June 13, 2016, 08:57:00 pm
Phil.  I  notice that you tag the file, quite right too! -
Quote
creator="Simple GPX https://simple-gpx.herokuapp.com ">
How about using this  to auto-compress zip and tag AUK copyright to GPX tracks when orgs. upload to the new AUKweb-site?
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on June 13, 2016, 10:55:17 pm
Phil.  I  notice that you tag the file, quite right too! -
Quote
creator="Simple GPX https://simple-gpx.herokuapp.com ">
How about using this  to auto-compress zip and tag AUK copyright to GPX tracks when orgs. upload to the new AUKweb-site?

I'm not involved in the new website and I'm sure they've tons of competing priorities to deal with already. No reason you couldn't submit your idea to the new IT manager though and see what he says. It would certainly simplify the workflow for many.
Title: Re: Slim down your bloated GPX track files
Post by: Oaky on July 27, 2016, 12:07:01 pm
i think I found a bug (or perhaps there's something pathological in this particular RideWithGPS output).  I have a file that reduces to nothing when I upload it (default 10 metre setting).

input: 831 points, 36 miles (https://www.dropbox.com/s/7np2ap4ztory3i3/Fails.gpx?dl=1)
output: 2 pts, 0 miles (https://www.dropbox.com/s/q9sro0r5uyfodzk/Fails.zip?dl=1)
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on July 28, 2016, 11:16:22 am
i think I found a bug (or perhaps there's something pathological in this particular RideWithGPS output).  I have a file that reduces to nothing when I upload it (default 10 metre setting).

input: 831 points, 36 miles (https://www.dropbox.com/s/7np2ap4ztory3i3/Fails.gpx?dl=1)
output: 2 pts, 0 miles (https://www.dropbox.com/s/q9sro0r5uyfodzk/Fails.zip?dl=1)

Try again, should be fixed. 
Title: Re: Slim down your bloated GPX track files
Post by: Oaky on July 28, 2016, 11:18:11 am
i think I found a bug (or perhaps there's something pathological in this particular RideWithGPS output).  I have a file that reduces to nothing when I upload it (default 10 metre setting).

input: 831 points, 36 miles (https://www.dropbox.com/s/7np2ap4ztory3i3/Fails.gpx?dl=1)
output: 2 pts, 0 miles (https://www.dropbox.com/s/q9sro0r5uyfodzk/Fails.zip?dl=1)

Try again, should be fixed.

Yep - working now!
Title: Re: Slim down your bloated GPX track files
Post by: Greenbank on June 27, 2017, 08:02:10 pm
Phil.  I  notice that you tag the file, quite right too! -
Quote
creator="Simple GPX https://simple-gpx.herokuapp.com ">
How about using this  to auto-compress zip and tag AUK copyright to GPX tracks when orgs. upload to the new AUKweb-site?

AUK doesn't own the copyright though, it'll most likely belong to the organiser.
Title: Re: Slim down your bloated GPX track files
Post by: Oaky on June 27, 2017, 09:54:11 pm
Phil's tool reduces the track complexity by removing unnecessary points (and stripping superfluous information e.g. elevation).  It's meant for simplification of tracks that are intended for navigational use in GPS units (some of which have  limits on the number of trackpoints for navigation).

zipping a file does a binary compression of the track, without changing the number of points.  you'd use this typically when transferring the track to soemone else, to save on bandwidth and/or storage space.  The track once unzipped would be identical to the original,  with the number of points unchanged etc.

(Incidentally,  Phil's tool does also zip the file, probably for exactly the reasons in the second paragraph above, but only after doing the clever stuff as per the first paragraph above).
Title: Re: Slim down your bloated GPX track files
Post by: Feanor on June 27, 2017, 10:12:04 pm
Because the Garmin units can't use a ZIP file, they need the un-zipped GPX file.
The issue is not so much the size of the file in bytes; it's the number of trackpoints they contain.
That's what some of the older Garmins choke on.
Title: Re: Slim down your bloated GPX track files
Post by: frankly frankie on June 28, 2017, 10:19:42 am
If you present a file already zipped does this cause issues? FWIW as a DIY org I can report that the auto zip function seems to have increased successful use of the form significantly.

Hi Andy, I'm glad to hear that the zipping appears to have improved the entry success rate.  That's my observation too, from looking at error logs generated whenever an entry fails.  Basically the entry form accepts files up to 5x larger than before and zips them, if they were not zipped before submission.  Files already zipped before submission are unaffected, that is they don't get double-zipped, and they are still subject to the stated size limit on the form.  Because zipping of GPX is highly efficient (much more than 5x), the best approach with a very large file is still to zip it before submission - which is one part of what Phil's very neat utility does.

I must say I have used Phil's utility a lot, in contexts nothing to do with Audax or DIYs - yes other tools such as GPS Babel can do similar things with more options, but I find this is an incredibly useful 'quick fix' to get to a clean and efficient GPX file. 
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on June 28, 2017, 10:44:12 pm
Because all edge units up to the 800 series and all etrex units up to 10/20/30 will truncate or crash if trying to navigate a track greater than 10,000 points. Ride with GPS and other online tools will typically spit out more than 10,000 points even for a 200km track. For even older units you can change the tolerance to drop below 500 points, or also decide to split a track into multiple tracks based on distance.

So it solves two problems,  10,000 track point  limit and size of files. With just zipping some ride with GPS GPX files are still in the Mb in size.  Plus it also gives the GPS track navigation and auk mandatory route comparison tool a lot less work to do. Efficiency and stability all round.
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on June 28, 2017, 10:54:28 pm
One other point.

I didn't build the tool for mandatory GPX track  submission as I built it long before such a thing existed. I just saw it could be used to solve that particular problem (too large a file submitted) for some, without it needing to be modified in any way.
Title: Re: Slim down your bloated GPX track files
Post by: jsabine on June 28, 2017, 11:01:05 pm
It is true that Garmins can't use zip files and that older devices have a 500 trackpoint limit. Perhaps Phil's tool could be useful in this area but this isn't the subject of the OP. It is all about file size.

Eh? Phil's original post gives half a dozen examples of how choosing different accuracies affects the number of points, which is the only aspect of his tool I've been interested in.

As a case in point, the RWGPS tracks of days one and two of this weekend's Celtic Knot were about 13000 and 12000 points respectively, leading to them being truncated on my Etrex at about 290km each. Running them through the tool at default settings brought them each down to more like 2500 points - they opened faster on the device, and more importantly showed the track for the full length of the day.

It's definitely useful in this area, no perhaps about it.
Title: Re: Slim down your bloated GPX track files
Post by: Whitedown Man on July 13, 2017, 10:19:24 am
Because all edge units up to the 800 series and all etrex units up to 10/20/30 will truncate or crash if trying to navigate a track greater than 10,000 points. Ride with GPS and other online tools will typically spit out more than 10,000 points even for a 200km track. For even older units you can change the tolerance to drop below 500 points, or also decide to split a track into multiple tracks based on distance.

So it solves two problems,  10,000 track point  limit and size of files. With just zipping some ride with GPS GPX files are still in the Mb in size.  Plus it also gives the GPS track navigation and auk mandatory route comparison tool a lot less work to do. Efficiency and stability all round.

Do TCX files suffer from the same problem, i.e. too many points for - in my case - an Edge Touring? I've just replaced my 810 (died of old age) with an Edge Touring. I never had an issue with anything up to and including 400km TCX files on the 810 but don't know whether the new device will cope.
Title: Re: Slim down your bloated GPX track files
Post by: αdαmsκι on July 16, 2017, 05:54:33 pm
I used this yesterday to slim down the tracks for my ride today.  It was very easy to use and at no point was I unsure where I was going.

One feature that would be nice would be an option to maintain the elevation data. On my Etrex 20x I get high and low points, but with the elevation data removed from the track these didn't appear. It isn't too much a deal in Essex, but it's very useful when doing long climbs to know the top of the hill involves another X amount of vertical climbing. Ta.
Title: Re: Slim down your bloated GPX track files
Post by: Oaky on July 16, 2017, 09:13:41 pm
I used this yesterday to slim down the tracks for my ride today.  It was very easy to use and at no point was I unsure where I was going.

One feature that would be nice would be an option to maintain the elevation data. On my Etrex 20x I get high and low points, but with the elevation data removed from the track these didn't appear. It isn't too much a deal in Essex, but it's very useful when doing long climbs to know the top of the hill involves another X amount of vertical climbing. Ta.

You can get (some) high/low points back if your map has elevation information.  These usually correspond to more significant high/lows compared with the ones from a track with elevation which often seemed to be every little lump.
Title: Re: Slim down your bloated GPX track files
Post by: αdαmsκι on July 16, 2017, 09:23:28 pm
You can get (some) high/low points back if your map has elevation information.

I'm not sure if my map has this or not. I'll investigate.



These usually correspond to more significant high/lows compared with the ones from a track with elevation which often seemed to be every little lump.

Even if my map has elevation it won't have found anything significant today as I was in mid Essex.
Title: Re: Slim down your bloated GPX track files
Post by: Oaky on July 16, 2017, 09:42:04 pm
These usually correspond to more significant high/lows compared with the ones from a track with elevation which often seemed to be every little lump.

Even if my map has elevation it won't have found anything significant today as I was in mid Essex.

I did say "more" significant. That was taking into account mid-Essex standards.  ;D ;D

I got fed up of the sheer number of high/lows tracks with elevation were generating here in the hallowed ACME heartlands, and so startd stripping the elevation.  The ones I get now, from the elevation data in my OSM maps, are far fewer, so I have more of my actual waypoints appearing in the "distance to next" field, in place of the minor lumps the Garmin was spamming me with before.
Title: Re: Slim down your bloated GPX track files
Post by: Manotea on July 18, 2017, 07:31:35 pm
What would be useful is a utility that strips out all the extraneous data leaving just location, timing and elevation info in place, i.e., just those elements required for validation.
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on October 14, 2017, 02:27:53 pm
Simple GPX updated today.

https://simple-gpx.herokuapp.com/

It will round the lat / lon coordinates of the track down to 5 decimal places. I believe this is a precision of about 1 metre, rather than the misleading indicated precision and untold decimal places you find in source GPX tracks these days.

Options to retain the time stamp and / or elevation data in a track. Elevation data will be rounded to the nearest metre. The default is that this data will be stripped out.   If you intend to navigate a track then please don't choose to retain time stamp data. Some GPS units do not like it.
Title: Re: Slim down your bloated GPX track files
Post by: citoyen on October 14, 2017, 02:56:14 pm
I've been using GPS Visualiser but will give this a go next time I need to reduce a GPX file. Sounds good.  :thumbsup:
Title: Re: Slim down your bloated GPX track files
Post by: frankly frankie on October 14, 2017, 11:37:32 pm
That looks really, really, good.  All options very useful.  Thanks!  :thumbsup:
Title: Re: Slim down your bloated GPX track files
Post by: citoyen on October 15, 2017, 09:20:56 pm
Just had a go, using a file exported from RideWithGPS. 525kb reduced to 36kb, using 10m tolerance and removing elevation data. Have checked the new file by uploading it back into RWGPS and it looks fine to me. Really simple and quick to use. Top work, Phil!  :thumbsup:
Title: Re: Slim down your bloated GPX track files
Post by: frankly frankie on October 16, 2017, 09:01:14 am
It's excellent for de-bloating large track files (I've used it a lot over the last year or so, thanks Phil  :thumbsup: ), but not for reducing the point count to below a specified number - it doesn't offer that option.  For that specific purpose (which admittedly isn't needed much any more) I find BikeHike works better than most.
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on October 16, 2017, 03:17:44 pm
but not for reducing the point count to below a specified number - it doesn't offer that option. 

It doesn't but to get below the 500 track point limit you can use these rough guidelines for getting there.

100km route - set to 20m tolerance (suspect 15m tolerance would be near enough spot on but I don't offer that option)
200km route - set 50m tolerance
400km route - set 100m tolerance
Title: Re: Slim down your bloated GPX track files
Post by: Greenbank on October 18, 2017, 08:29:20 am
A "below n trackpoints" option would still be nice (I know this wasn't the primary intention of the tool) but...

(The usual method is to use exactly the same algorithm it currently does but binary search based on the threshold until you find a threshold value that gives the required number of trackpoints in the output. More processing but it should be able to handle it).
Title: Re: Slim down your bloated GPX track files
Post by: frankly frankie on October 18, 2017, 09:03:23 am
Most tools return a track with substantially fewer points than the target figure (for fairly obvious reasons) but BikeHike succeeds in meeting the target almost exactly.  Not sure if that's better or worse but I like it.
Title: Re: Slim down your bloated GPX track files
Post by: Somnolent on October 18, 2017, 11:44:08 am
Just got around to trying Simple GPX on a recorded track (with timestamp & elevation retained) to check all is kosher before updating the advice in http://www.aukweb.net/resources/filesize/

Highly successful in reducing the number of trackpoints and file size ... but there is something not quite right with the file that means routevalidator not liking it.  Nor is Mapsource for that matter - another one that is known to be sensitive to exact compliance with the schema.

FF - number of failed uploads to PL01 this morning.  You may be able to see the problem?
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on October 18, 2017, 12:26:08 pm
Highly successful in reducing the number of trackpoints and file size ... but there is something not quite right with the file that means routevalidator not liking it.  Nor is Mapsource for that matter - another one that is known to be sensitive to exact compliance with the schema.

I had reversed the elevation and timestamp within each trackpoint which according to the GPX schema are in the sequence <ele>..</ele> then <time>...</time>.  Just corrected it and it now outputs them in the same sequence the GPX 1.1 schema states
Title: Re: Slim down your bloated GPX track files
Post by: Somnolent on October 18, 2017, 12:57:38 pm
Highly successful in reducing the number of trackpoints and file size ... but there is something not quite right with the file that means routevalidator not liking it.  Nor is Mapsource for that matter - another one that is known to be sensitive to exact compliance with the schema.

I had reversed the elevation and timestamp within each trackpoint which according to the GPX schema are in the sequence <ele>..</ele> then <time>...</time>.  Just corrected it and it now outputs them in the same sequence the GPX 1.1 schema states

Works now  :thumbsup:
Title: Re: Slim down your bloated GPX track files
Post by: Greenbank on October 18, 2017, 01:18:13 pm
The joy of XML is that you have to be very exact/correct in what you produce, but very lax in what you accept in.
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on October 18, 2017, 06:16:52 pm
I have now run my Wild Atlantic Way Audax tracklog through it at default settings keeping timestamo and elevation.  File comes out at 78kB zipped and 452kB un compressed.  That is a 2016km tracklog, which retained 9958 trackpoints.
Title: Re: Slim down your bloated GPX track files
Post by: frankly frankie on October 18, 2017, 06:57:45 pm
Impressive.  5 points per kilometre I think is unnecessarily sparse though - maybe for a ultra ride like this it makes sense but in general I think around 20 points per km is a better target to aim at.

Otherwise its a far cry from the 'gold standard' of a raw recorded tracklog - even worse if you then start to add interpolation as suggested elsewhere.  It's bad enough that virtually all tracklogs are post-processed before submission anyway (up to Strava and down again, topping and tailing, joining two parts of a 600, etc etc).  Talk about fake data.  If I had written the checker software the very first thing it would red-flag would be a <creator> tag that indicated 'not a GPS'.

</rant>
Title: Re: Slim down your bloated GPX track files
Post by: Greenbank on October 18, 2017, 07:43:46 pm
Until there are cryptographically signed trackpoints there's little point in worrying about whether data is real or fake.
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on October 28, 2017, 06:55:19 pm
The algorithm is now 3 dimensional in determining which points to keep.  So if you decide to retain elevation it will use that in the algorithm.

As a little extra I thought I'd get it to create a HMAC of its work

<gpx version="1.1" creator="Simple GPX https://simple-gpx.herokuapp.com" ..... >
  <trk>
    <name>2017-10-27 05:43:05</name>
    <cmt>HMAC Digest: 8d336dc36e1fbd0d1bfdb6f93fbd09ecdabb25f5ce0850438e935d6b04e5fafd</cmt>
    <trkseg>
....
Title: Re: Slim down your bloated GPX track files
Post by: Genosse Brymbo on October 28, 2017, 07:03:03 pm
As a little extra I thought I'd get it to create a HMAC of its work
What's the hashing algorithm and what data is being hashed?
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on October 28, 2017, 07:32:11 pm
As a little extra I thought I'd get it to create a HMAC of its work
What's the hashing algorithm and what data is being hashed?

SHA-256 and the set of track points belonging to the track are being hashed.
Title: Re: Slim down your bloated GPX track files
Post by: BobCatMax on July 08, 2018, 08:55:36 pm
This is a wonderful tool!

Does it maintain cuepoints/way points?

Thank you for taking the time to create this. Forgive my ignorance, is there any way of checking the spat out files new number of points?
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on July 09, 2018, 05:22:16 pm
It only touches tracks and track points. So if you have a route or waypoints in the same GPX file, they will come out the other side unchanged. 

You can see the number of track points in things like Basecamp.  I did run some tests at various tolerances to see what the number track points roughly came out at for different distances and settings.  But have long since forgotten what the results were.  I may add a text file into the returned zip file with the stats as that would easy enough to do.  Something for when it finally rains. I have bike riding to be done.
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on July 09, 2018, 05:39:04 pm
As for below 500 points, below 10,000 points I may add these in as tick box options.  The relationship between tolerance and the number of points retained for a distance is surprisingly linear. So it would be relatively easy to get close to the target based on the relationship then a small bit of binary searching to refine.  Again something for a rainy day when the weather finally breaks.
Title: Re: Slim down your bloated GPX track files
Post by: BobCatMax on July 10, 2018, 06:26:33 am
Fantastic, thanks for your replies.

I won't wish on the rain just yet!
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on July 23, 2018, 02:15:45 pm
Hi,

Updated today, got bored waiting for a rainy day.

Simplify a track down to 500 or 10,000 points

You can now select 500 or 10,000 points as the tolerance in the drop down.

It will aim to get within 1% of the chosen number of points. 

For 500 points it will return a track with between 495 and 500 points.
For 10,000 points it will return a track with between 9,900 and 10,000 points.

If the track is already below the required number of points, it will just return your original track but with the crap cleared out of it. Retain elevation and / or timestamps is respected when filtering for 500 or 10,000 points.

Statistics

In the zip file returned will be a stats.txt file containing statistics on the simplification carried out

Simple GPX
https://simple-gpx.herokuapp.com
Tolerance chosen: 500
Original trackpoints: 4009, simplified trackpoints: 498

Cheers

Phil
Title: Re: Slim down your bloated GPX track files
Post by: JulesP on July 23, 2018, 02:20:39 pm
You can now select 500 or 10,000 points as the tolerance in the drop down.

What's the rationale for these two vastly different counts? Do they correspond to the ideal number of track points to two particular types of GPS device, or something else..?

I use your excellent tool all the time, but I'm never sure I need to with a Garmin 810 or 1000 (though I've had past problems with RwGPS single GPX files for 200km routes).
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on July 23, 2018, 02:39:04 pm
500 track points is the limit for older GPS units.
10,000 track points is the limit for for GPS units of the generation that saw the Edge 205, Edge 305, Edge 500, Edge 605, Edge 705, Etrex 20, 30 come out etc.  Not sure of the track point limit of Edge 810 but anecdotally I believe it has the same limits.

A 200km track generated online is unlikely to exceed 10,000 points but it has certainly been seen on a 300km track.  I have seen some RWGPS tracks with a trackpoint every 7m even on straight roads.  This is about 28,000 track points for a 200km ride. An unnecessary number. By default I run all the tracks I intend to navigate through my own tool, this includes tracks I have generated myself for DIYs etc.   

But it is not just about these upper limits for tracks.  A clean track with fewer trackpoints is also less likely to glitch the unit.  An example is tracks with timestamp data in them.  If some Edge units encounter timestamps in a track they are trying to navigate then they will glitch / fail.  Etrex are more tolerant than Edge units and less likely to glitch but they will encounter problems if you try and navigate a track above its track point limits.

Certainly since I started only using cleaned / filtered tracks I have not had a problem with glitches / failed navigation.


Title: Re: Slim down your bloated GPX track files
Post by: citoyen on July 23, 2018, 07:49:50 pm
Top work, Phil, very nice. I especially like having the stats file to compare.

I normally use the 5m tolerance setting, which produces files that are small enough for my Edge 510 to handle. Using the 500pt setting on a 200km route, they're about half the size again. The 500pt tracks seem detailed enough for navigating country roads, although there were a couple of places on my test routes where I would want a few more track points (mainly in towns).

Certainly since I started only using cleaned / filtered tracks I have not had a problem with glitches / failed navigation.

I've had a few problems in the past caused by excessively large GPX files - mainly ones downloaded from RWGPS. Worst example was a ride where the file I was using was so bloated, it took 15 minutes to load. By the time I got going, I was well behind the field, hence when my device crashed less than 20km later, I was without any kind of navigation nor even a wheel to follow. Luckily I knew roughly where the first control was but I still got lost a couple of times and added a good 10km to my distance. Eventually I got it working, luckily, otherwise I'd have had to abandon the ride. I learnt two lessons on that ride: one, always carry a printed routesheet as backup; two, always check that your GPX files work on your device before setting off.

Not that I've had any such problems again since learning how to make files smaller and strip out the bloat.
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on July 23, 2018, 09:11:56 pm
I have never ridden on a 500 point track for a 200km. I normally use the 10m setting which returns around 1100-1500 points depending on how twisty the route.  I have an overnight DIY 200 this Friday to beat the heatwave during the current RRTY run.  I might just try out a 500 point version.  Certainly tracks with less track points and bloat load quickly  and require less processing by the low power GPS CPUs
Title: Re: Slim down your bloated GPX track files
Post by: Ben T on July 23, 2018, 09:25:58 pm
If I want to reduce to 500 it should reduce to exactly 500, not 498. ;) B minus Phil  ;)
Title: Re: Slim down your bloated GPX track files
Post by: frankly frankie on July 24, 2018, 11:23:53 am
I normally use the 5m tolerance setting, which produces files that are small enough for my Edge 510 to handle. Using the 500pt setting on a 200km route, they're about half the size again. The 500pt tracks seem detailed enough for navigating country roads, although there were a couple of places on my test routes where I would want a few more track points (mainly in towns).

There was a 400km event in the N-E a couple of years ago where several people were reporting trouble with the organiser-supplied GPX file - I had a look at it out of interest and it was about 13,000 points (sourced RWGPS).  I downsampled it to 500 and although there were a couple of towns where it did look a bit rough, even so every junction in 400km that would have involved a 'turn' was still properly described by the track shape.  An extreme example - obviously in practice the best approach would have been to divide the Track into 3 or 4 and then optionally downsample.
Title: Re: Slim down your bloated GPX track files
Post by: citoyen on July 24, 2018, 11:51:50 am
I had a look at it out of interest and it was about 13,000 points (sourced RWGPS).

As you've mentioned before, it's not just the number of points but the extraneous data - eg co-ordinates given to microscopic levels of precision. One of the things I like about Phil's app is that it makes files much smaller without necessarily reducing the number of trackpoints significantly. 13,000 trackpoints does seem somewhat excessive though.

You have to wonder why the authors of the RWGPS app don't do something about this themselves, as surely it would be easy enough to fix.

Quote
I downsampled it to 500 and although there were a couple of towns where it did look a bit rough, even so every junction in 400km that would have involved a 'turn' was still properly described by the track shape.

The practicality of 500pt tracks possibly depends on what device you're using - on a device without maps, more trackpoints are beneficial in towns where a junction might offer two or more roads going in approximately the right direction. Without a map for reference, it's sometimes hard to know which one to choose.
Title: Re: Slim down your bloated GPX track files
Post by: frankly frankie on July 24, 2018, 12:10:27 pm
It's not usually critical in a town.  Unless you're heading for an actual control location, as a randonneur all you need to do is get across the town and out on the right road, there may be 100 ways of doing that with not much time/distance penalty between the best and the worst.

13,000 trackpoints does seem somewhat excessive though.

Excessive as in, most Garmins used by cyclists have a limit of 10,000.  To make it worse, they often won't report a 'too many points' error (or if they do, it's just an irritating little pop-up you click through during boot-up) but instead simply truncate and the first you know about it is in the middle of nowhere.  That makes either Garmin or RWGPS or both somewhat liable for a situation that could conceivably be worse than just a bit uncomfortable.
Title: Re: Slim down your bloated GPX track files
Post by: JulesP on July 24, 2018, 03:47:18 pm
To make it worse, they often won't report a 'too many points' error (or if they do, it's just an irritating little pop-up you click through during boot-up) but instead simply truncate and the first you know about it is in the middle of nowhere.

That's exactly what happened to me on a RwGPS-plotted 200 route on a Garmin 810. At around 130km, it suddenly switched to a straight line that went who-knows-where (the destination, presumably). This was before I knew of such glitches but, after following someone to the next control and then restarting the route, all was well.

It's not happened since I started using Phil's tool, nor since I started splitting files to ~100km/inter-control tracks (the latter using a paid RwGPS account).
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on July 24, 2018, 03:56:52 pm
I have reversed the order it will split and simplify a track.

So (if requested) it will split the track first and then it will simplify each track.  Makes no odds if using the metre tolerances.  But if using the 500 point or 10,000 point tolerances and splitting as well you will get a different result. 

Let's say you decide to split a PBP GPX track into lengths of 300km with a tolerance of 500 points.  Then you will get 5 tracks (PBP is longer than 1200km), and each track will have 500 or less points. Under the old method it would have reduced the original 1200km track to 500 points and then split it, ending up will quite a bit less points in each track.   

The stats.txt file returned has been updated to include track splitting data. Below are figures for the WAWA 2016 2100km GPX track split into 200km chunks.

Simple GPX
https://simple-gpx.herokuapp.com
Tolerance chosen: 500
Split track: yes, split distance: 200km
Original tracks: 1, simplified tracks: 11
Original trackpoints: 21734, simplified trackpoints: 5390, average trackpoints per track: 490
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on July 24, 2018, 04:27:00 pm

It's not happened since I started using Phil's tool, nor since I started splitting files to ~100km/inter-control tracks (the latter using a paid RwGPS account).

I have thought about adding the option to split tracks at controls but it only works if

The GPX file contains waypoints for every control and the control waypoints are within a reasonable proximity of the actual track (say within 50m).  You do not have extra waypoints in the GPX file being used for other purposes. 

I am not a fan of splitting tracks at controls as it gives me something extra to do when there is plenty to do at the control anyway.  Generally I run long tracks and only split them where they cross over themselves. It is the number of points causing truncation not the length of the track in km.

But certainly adding split by control would not be that hard with the above provisos.
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on July 25, 2018, 04:11:36 pm
If there are no routes in the GPX then additionally it will now.

Generate a route from the track, with an average of one route point for every 5km.  Obviously the spread of route points will vary, with more where twisty and less when not. If it finds waypoints (it assumes they are controls / places you wish to pass through) then it will try and insert them into the route in the appropriate place.  It will then split the route into 50 route point chunks so it can be auto routed in GPS that support it.

Below is one generated through the process with flags representing via route points.  Use at your own risk etc. etc. or just stick with the simplified tracks which it already outputs.

(https://www.dropbox.com/s/yktafu01x6g8so3/Capture.JPG?raw=1)



Title: Re: Slim down your bloated GPX track files
Post by: frankly frankie on July 25, 2018, 06:35:37 pm
Possibly this (and/or possibly the stats.txt file) might break the zip file for the purpose in post #1 of this thread, that is, submission to AUK and subsequently running it through the RouteValidator tool.  I don't know that it does - just speculating, I know that RV did have some difficulties initially with zip files having unexpected contents, and I know that was fixed to some extent but I don't know how bomb-proof the fix is.

However this project has maybe gone way beyond that, and from that angle, it's all good  :thumbsup:

edit: no I've tested this now in RV, and it still seems to work OK, ignoring both the added Route and the added stats.txt file.
Title: Re: Slim down your bloated GPX track files
Post by: citoyen on July 26, 2018, 02:59:29 pm
Phil, I've tried to use the app a couple of times this morning and got an application error. Problem at my end or yours?

Slightly OT question regarding routes: my device doesn't support routes so they're extraneous data for me personally. However, is it worth keeping them in the GPX files that I supply to riders for use on my events? What is the benefit of routes if your GPX file also contains a track?
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on July 26, 2018, 06:25:46 pm
Some prefer tracks, others prefer routes for the turn by turn, some like to use both.  I generally have't used routes as my experience has been that they did not go where I wanted them. When creating routes I do not click every junction. So when it gets to the gps it rarely looked like what I had in Basecamp. But with a route filtered from a track it seems from what I can see that Basecamp and my gps concur.  So I may just revisit routes again for my own nav though I am quite used to tracks. I would leave the route in the GPX as an optional extra!

Regards other point will see if I can see in logs what it did not like about your GPX
Title: Re: Slim down your bloated GPX track files
Post by: citoyen on July 26, 2018, 06:32:05 pm
Thanks, Phil.

Just been reading up on routes, in relation to the other thread. I think I get the general idea now. It seems to me that for following a prescribed route (as on an audax), tracks are probably a better option.
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on July 26, 2018, 07:07:13 pm
The only thing I can see in the logs are timeouts at 8:51 and 9:06am.  Was that when you were using Simple GPX?   Is it a GPX with a particularly large number of track points or waypoints and what settings were you using? Mind I have run it against a track with 21,000 track points without issue.
Title: Re: Slim down your bloated GPX track files
Post by: citoyen on July 26, 2018, 10:48:51 pm
The only thing I can see in the logs are timeouts at 8:51 and 9:06am.  Was that when you were using Simple GPX?   Is it a GPX with a particularly large number of track points or waypoints and what settings were you using? Mind I have run it against a track with 21,000 track points without issue.

Sounds about right, time-wise. If the problem is timing out, that could be due to my flaky internet connection.
Title: Re: Slim down your bloated GPX track files
Post by: citoyen on July 27, 2018, 08:54:06 am
Forgot to add: it was an RWGPS file, 585KB, 7161 points, set to 5m tolerance
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on July 27, 2018, 10:39:28 am
Your GPX works fine now.  Trying to hit too fine a points tolerance when generating a route from such a short track meant it hit the timeouts.  I have relaxed the tolerances for route generation.

Route generation is now also a seperate action in the tolerance drop down rather than an automatic operation every time.  If you now select generate route in the dropdown then that is all it will do, leaving the original track alone.



Title: Re: Slim down your bloated GPX track files
Post by: citoyen on July 27, 2018, 10:49:32 am
Top work, Phil. :thumbsup:
Title: Re: Slim down your bloated GPX track files
Post by: frankly frankie on July 27, 2018, 11:10:07 am
Very good.  I would only comment that if you now want a Track and Route, you end up with two GPX files of the same name.  It would be nice (I know  ::-)) if the route filename were suffixed -r or similar?
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on July 27, 2018, 11:27:15 am
There you go it will now add   _route to the end of the zip file and GPX names if you are generating a route. Be aware the original track file will still be in a GPX where you have generated a route.  Could not decide whether to leave the original track in there or delete it when generating a route.
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on July 27, 2018, 11:36:11 am
Strike that, if generationg a route the returned GPX will only contain the route and any original waypoints. It will now delete the original track.  That way you will not accidently load an unfiltered bloatware track if you load the GPX route file.
Title: Re: Slim down your bloated GPX track files
Post by: frankly frankie on July 27, 2018, 05:56:14 pm
Very useful utility.  :thumbsup:
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on July 28, 2018, 02:36:11 pm
Thanks I cannot see me adding anything more to it unless someone has a killer idea. The Heroku platform and plan I have  is free for life but should they ever shut down I can shunt the application across to other servers I run stuff on.
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on August 13, 2018, 05:34:50 pm
So much for not adding anything more.

It can now split tracks by control.  It is a new option in the split track dropdown.  Track naming works the same way as split by distance.  It looks for waypoints in the GPX, finds the nearest trackpoint for each one. Then splits the track accordingly.  It ignores any waypoints close to the start or end of the original track; to stop any stupidly short tracks being created.
Title: Re: Slim down your bloated GPX track files
Post by: Phil W on September 25, 2018, 04:34:00 pm
It can now add proximity alarms to all the waypoints in a GPX file. New dropdown to select from 50 metres to 400 meters.

https://simple-gpx.herokuapp.com/

Title: Re: Slim down your bloated GPX track files
Post by: De Sisti on September 25, 2018, 05:38:59 pm
Slightly OT, but I find that when my Garmin Edge Touring is connected to my computer
I can drop any gpx (or tcx) file that I have on my machine into the Touring's 'NewFile'
folder. The gpx/tcx are then automatically converted into a .fit file (which the unit prefers)
and moved to the 'Courses' folder.

It works on some of the smaller rides I've done recently, but will try it on a long ride
soon (which will be the Upper Thames 200 in November).
Title: Re: Slim down your bloated GPX track files
Post by: phil w on June 17, 2019, 02:03:42 pm
Will now split out waypoints, tracks, routes into seperate GPX within the Zip file.  If you have multiple tracks or routes they will each be within their own GPX file.  Naming of GPX files is self evident hopefully.
Title: Re: Slim down your bloated GPX track files
Post by: JulesP on June 17, 2019, 09:07:10 pm
I'm now getting "undefined0" appended to downloaded file names, but the contents appear to be OK. Just an FYI.
Title: Re: Slim down your bloated GPX track files
Post by: phil w on June 17, 2019, 11:12:18 pm
I'm now getting "undefined0" appended to downloaded file names, but the contents appear to be OK. Just an FYI.

Ah, I half started making some changes with the GPX file names then changed my mind. Probably left a bit of that in for certain actions. That won't affect contents. Will sort tomorrow.
Title: Re: Slim down your bloated GPX track files
Post by: phil w on June 18, 2019, 12:36:36 pm
Sorted
Title: Re: Slim down your bloated GPX track files
Post by: el on November 01, 2019, 06:09:17 pm
Hi- I'm new to this forum.
Do I understand correctly that if I put gpx files through this website to be made smaller, my Garmin 800 will no longer crash leaving me to rely on my paper cue sheets or other participants?
Title: Re: Slim down your bloated GPX track files
Post by: phil w on November 01, 2019, 08:14:25 pm
Hi- I'm new to this forum.
Do I understand correctly that if I put gpx files through this website to be made smaller, my Garmin 800 will no longer crash leaving me to rely on my paper cue sheets or other participants?

No guarantees about the quirks of your Garmin Edge 800 but it'll certainly reduce the frequency / likelihood of it crashing when navigating the GPX. Note it slims GPX tracks only, not GPX  routes which are a different beast.
Title: Re: Slim down your bloated GPX track files
Post by: fhills on November 01, 2019, 09:37:46 pm

No guarantees about the quirks of your Garmin Edge 800 but it'll certainly reduce the frequency / likelihood of it crashing when navigating the GPX. Note it slims GPX tracks only, not GPX  routes which are a different beast.
They aren't much different at all I don't think if you use point to point navigation, riding between marks of the turns.

I slim down GPX tracks/routes semi manually for use on an etrex.
Get gpsies.com to reduce the points automatically if needed (not really needed with my own routes from cycletravel as they only mark turns to begin with) then fine tune on the gpsies map, then semi automatically strip stuff out using a simple text editor.
There's an awful lot of not needed junk in most gpx files.
Title: Re: Slim down your bloated GPX track files
Post by: grams on November 01, 2019, 09:46:59 pm
Hi- I'm new to this forum.
Do I understand correctly that if I put gpx files through this website to be made smaller, my Garmin 800 will no longer crash leaving me to rely on my paper cue sheets or other participants?

I believe the only way to stop a Garmin Edge of that era crashing is the War Games solution.