Author Topic: Slim down your bloated GPX track files  (Read 7242 times)

Oaky

  • ACME Fire Safety Officer
  • Audax Club Mid-Essex
    • MEMWNS Map
Re: Slim down your bloated GPX track files
« Reply #25 on: July 16, 2017, 09:42:04 pm »
These usually correspond to more significant high/lows compared with the ones from a track with elevation which often seemed to be every little lump.

Even if my map has elevation it won't have found anything significant today as I was in mid Essex.

I did say "more" significant. That was taking into account mid-Essex standards.  ;D ;D

I got fed up of the sheer number of high/lows tracks with elevation were generating here in the hallowed ACME heartlands, and so startd stripping the elevation.  The ones I get now, from the elevation data in my OSM maps, are far fewer, so I have more of my actual waypoints appearing in the "distance to next" field, in place of the minor lumps the Garmin was spamming me with before.
You are in a maze of twisty flat droves, all alike.

85.4 miles from Marsh Gibbon

Audax Club Mid-Essex Fire Safety Officer
http://acme.bike

Re: Slim down your bloated GPX track files
« Reply #26 on: July 18, 2017, 07:31:35 pm »
What would be useful is a utility that strips out all the extraneous data leaving just location, timing and elevation info in place, i.e., just those elements required for validation.
Anfractuous & Nyctophobic Oct 13       Willy Warmer Jan 19    Winter Boat Ride Feb 10

Phil W

Re: Slim down your bloated GPX track files
« Reply #27 on: October 14, 2017, 02:27:53 pm »
Simple GPX updated today.

https://simple-gpx.herokuapp.com/

It will round the lat / lon coordinates of the track down to 5 decimal places. I believe this is a precision of about 1 metre, rather than the misleading indicated precision and untold decimal places you find in source GPX tracks these days.

Options to retain the time stamp and / or elevation data in a track. Elevation data will be rounded to the nearest metre. The default is that this data will be stripped out.   If you intend to navigate a track then please don't choose to retain time stamp data. Some GPS units do not like it.

citoyen

  • Cat 6 Racer
Re: Slim down your bloated GPX track files
« Reply #28 on: October 14, 2017, 02:56:14 pm »
I've been using GPS Visualiser but will give this a go next time I need to reduce a GPX file. Sounds good.  :thumbsup:

frankly frankie

  • I kid you not
    • Virtual Alps
Re: Slim down your bloated GPX track files
« Reply #29 on: October 14, 2017, 11:37:32 pm »
That looks really, really, good.  All options very useful.  Thanks!  :thumbsup:
"This is a complex subject, with a need for more than one highlighter pen."

citoyen

  • Cat 6 Racer
Re: Slim down your bloated GPX track files
« Reply #30 on: October 15, 2017, 09:20:56 pm »
Just had a go, using a file exported from RideWithGPS. 525kb reduced to 36kb, using 10m tolerance and removing elevation data. Have checked the new file by uploading it back into RWGPS and it looks fine to me. Really simple and quick to use. Top work, Phil!  :thumbsup:

frankly frankie

  • I kid you not
    • Virtual Alps
Re: Slim down your bloated GPX track files
« Reply #31 on: October 16, 2017, 09:01:14 am »
It's excellent for de-bloating large track files (I've used it a lot over the last year or so, thanks Phil  :thumbsup: ), but not for reducing the point count to below a specified number - it doesn't offer that option.  For that specific purpose (which admittedly isn't needed much any more) I find BikeHike works better than most.
"This is a complex subject, with a need for more than one highlighter pen."

Phil W

Re: Slim down your bloated GPX track files
« Reply #32 on: October 16, 2017, 03:17:44 pm »
but not for reducing the point count to below a specified number - it doesn't offer that option. 

It doesn't but to get below the 500 track point limit you can use these rough guidelines for getting there.

100km route - set to 20m tolerance (suspect 15m tolerance would be near enough spot on but I don't offer that option)
200km route - set 50m tolerance
400km route - set 100m tolerance

Re: Slim down your bloated GPX track files
« Reply #33 on: October 18, 2017, 08:29:20 am »
A "below n trackpoints" option would still be nice (I know this wasn't the primary intention of the tool) but...

(The usual method is to use exactly the same algorithm it currently does but binary search based on the threshold until you find a threshold value that gives the required number of trackpoints in the output. More processing but it should be able to handle it).
"Yes please" said Squirrel "biscuits are our favourite things."

frankly frankie

  • I kid you not
    • Virtual Alps
Re: Slim down your bloated GPX track files
« Reply #34 on: October 18, 2017, 09:03:23 am »
Most tools return a track with substantially fewer points than the target figure (for fairly obvious reasons) but BikeHike succeeds in meeting the target almost exactly.  Not sure if that's better or worse but I like it.
"This is a complex subject, with a need for more than one highlighter pen."

Re: Slim down your bloated GPX track files
« Reply #35 on: October 18, 2017, 11:44:08 am »
Just got around to trying Simple GPX on a recorded track (with timestamp & elevation retained) to check all is kosher before updating the advice in http://www.aukweb.net/resources/filesize/

Highly successful in reducing the number of trackpoints and file size ... but there is something not quite right with the file that means routevalidator not liking it.  Nor is Mapsource for that matter - another one that is known to be sensitive to exact compliance with the schema.

FF - number of failed uploads to PL01 this morning.  You may be able to see the problem?

Phil W

Re: Slim down your bloated GPX track files
« Reply #36 on: October 18, 2017, 12:26:08 pm »
Highly successful in reducing the number of trackpoints and file size ... but there is something not quite right with the file that means routevalidator not liking it.  Nor is Mapsource for that matter - another one that is known to be sensitive to exact compliance with the schema.

I had reversed the elevation and timestamp within each trackpoint which according to the GPX schema are in the sequence <ele>..</ele> then <time>...</time>.  Just corrected it and it now outputs them in the same sequence the GPX 1.1 schema states

Re: Slim down your bloated GPX track files
« Reply #37 on: October 18, 2017, 12:57:38 pm »
Highly successful in reducing the number of trackpoints and file size ... but there is something not quite right with the file that means routevalidator not liking it.  Nor is Mapsource for that matter - another one that is known to be sensitive to exact compliance with the schema.

I had reversed the elevation and timestamp within each trackpoint which according to the GPX schema are in the sequence <ele>..</ele> then <time>...</time>.  Just corrected it and it now outputs them in the same sequence the GPX 1.1 schema states

Works now  :thumbsup:

Re: Slim down your bloated GPX track files
« Reply #38 on: October 18, 2017, 01:18:13 pm »
The joy of XML is that you have to be very exact/correct in what you produce, but very lax in what you accept in.
"Yes please" said Squirrel "biscuits are our favourite things."

Phil W

Re: Slim down your bloated GPX track files
« Reply #39 on: October 18, 2017, 06:16:52 pm »
I have now run my Wild Atlantic Way Audax tracklog through it at default settings keeping timestamo and elevation.  File comes out at 78kB zipped and 452kB un compressed.  That is a 2016km tracklog, which retained 9958 trackpoints.

frankly frankie

  • I kid you not
    • Virtual Alps
Re: Slim down your bloated GPX track files
« Reply #40 on: October 18, 2017, 06:57:45 pm »
Impressive.  5 points per kilometre I think is unnecessarily sparse though - maybe for a ultra ride like this it makes sense but in general I think around 20 points per km is a better target to aim at.

Otherwise its a far cry from the 'gold standard' of a raw recorded tracklog - even worse if you then start to add interpolation as suggested elsewhere.  It's bad enough that virtually all tracklogs are post-processed before submission anyway (up to Strava and down again, topping and tailing, joining two parts of a 600, etc etc).  Talk about fake data.  If I had written the checker software the very first thing it would red-flag would be a <creator> tag that indicated 'not a GPS'.

</rant>
"This is a complex subject, with a need for more than one highlighter pen."

Re: Slim down your bloated GPX track files
« Reply #41 on: October 18, 2017, 07:43:46 pm »
Until there are cryptographically signed trackpoints there's little point in worrying about whether data is real or fake.
"Yes please" said Squirrel "biscuits are our favourite things."

Phil W

Re: Slim down your bloated GPX track files
« Reply #42 on: October 28, 2017, 06:55:19 pm »
The algorithm is now 3 dimensional in determining which points to keep.  So if you decide to retain elevation it will use that in the algorithm.

As a little extra I thought I'd get it to create a HMAC of its work

<gpx version="1.1" creator="Simple GPX https://simple-gpx.herokuapp.com" ..... >
  <trk>
    <name>2017-10-27 05:43:05</name>
    <cmt>HMAC Digest: 8d336dc36e1fbd0d1bfdb6f93fbd09ecdabb25f5ce0850438e935d6b04e5fafd</cmt>
    <trkseg>
....

Baron Brymbo von Pickelhaube

  • Adel auf dem Radel
Re: Slim down your bloated GPX track files
« Reply #43 on: October 28, 2017, 07:03:03 pm »
As a little extra I thought I'd get it to create a HMAC of its work
What's the hashing algorithm and what data is being hashed?
The present is a foreign country: they do things differently here.

Phil W

Re: Slim down your bloated GPX track files
« Reply #44 on: October 28, 2017, 07:32:11 pm »
As a little extra I thought I'd get it to create a HMAC of its work
What's the hashing algorithm and what data is being hashed?

SHA-256 and the set of track points belonging to the track are being hashed.

Re: Slim down your bloated GPX track files
« Reply #45 on: July 08, 2018, 08:55:36 pm »
This is a wonderful tool!

Does it maintain cuepoints/way points?

Thank you for taking the time to create this. Forgive my ignorance, is there any way of checking the spat out files new number of points?

Re: Slim down your bloated GPX track files
« Reply #46 on: July 09, 2018, 05:22:16 pm »
It only touches tracks and track points. So if you have a route or waypoints in the same GPX file, they will come out the other side unchanged. 

You can see the number of track points in things like Basecamp.  I did run some tests at various tolerances to see what the number track points roughly came out at for different distances and settings.  But have long since forgotten what the results were.  I may add a text file into the returned zip file with the stats as that would easy enough to do.  Something for when it finally rains. I have bike riding to be done.

Re: Slim down your bloated GPX track files
« Reply #47 on: July 09, 2018, 05:39:04 pm »
As for below 500 points, below 10,000 points I may add these in as tick box options.  The relationship between tolerance and the number of points retained for a distance is surprisingly linear. So it would be relatively easy to get close to the target based on the relationship then a small bit of binary searching to refine.  Again something for a rainy day when the weather finally breaks.

Re: Slim down your bloated GPX track files
« Reply #48 on: July 10, 2018, 06:26:33 am »
Fantastic, thanks for your replies.

I won't wish on the rain just yet!

Re: Slim down your bloated GPX track files
« Reply #49 on: July 23, 2018, 02:15:45 pm »
Hi,

Updated today, got bored waiting for a rainy day.

Simplify a track down to 500 or 10,000 points

You can now select 500 or 10,000 points as the tolerance in the drop down.

It will aim to get within 1% of the chosen number of points. 

For 500 points it will return a track with between 495 and 500 points.
For 10,000 points it will return a track with between 9,900 and 10,000 points.

If the track is already below the required number of points, it will just return your original track but with the crap cleared out of it. Retain elevation and / or timestamps is respected when filtering for 500 or 10,000 points.

Statistics

In the zip file returned will be a stats.txt file containing statistics on the simplification carried out

Simple GPX
https://simple-gpx.herokuapp.com
Tolerance chosen: 500
Original trackpoints: 4009, simplified trackpoints: 498

Cheers

Phil