Author Topic: Slim down your bloated GPX track files  (Read 43639 times)

Re: Slim down your bloated GPX track files
« Reply #50 on: 23 July, 2018, 02:20:39 pm »
You can now select 500 or 10,000 points as the tolerance in the drop down.

What's the rationale for these two vastly different counts? Do they correspond to the ideal number of track points to two particular types of GPS device, or something else..?

I use your excellent tool all the time, but I'm never sure I need to with a Garmin 810 or 1000 (though I've had past problems with RwGPS single GPX files for 200km routes).

Phil W

Re: Slim down your bloated GPX track files
« Reply #51 on: 23 July, 2018, 02:39:04 pm »
500 track points is the limit for older GPS units.
10,000 track points is the limit for for GPS units of the generation that saw the Edge 205, Edge 305, Edge 500, Edge 605, Edge 705, Etrex 20, 30 come out etc.  Not sure of the track point limit of Edge 810 but anecdotally I believe it has the same limits.

A 200km track generated online is unlikely to exceed 10,000 points but it has certainly been seen on a 300km track.  I have seen some RWGPS tracks with a trackpoint every 7m even on straight roads.  This is about 28,000 track points for a 200km ride. An unnecessary number. By default I run all the tracks I intend to navigate through my own tool, this includes tracks I have generated myself for DIYs etc.   

But it is not just about these upper limits for tracks.  A clean track with fewer trackpoints is also less likely to glitch the unit.  An example is tracks with timestamp data in them.  If some Edge units encounter timestamps in a track they are trying to navigate then they will glitch / fail.  Etrex are more tolerant than Edge units and less likely to glitch but they will encounter problems if you try and navigate a track above its track point limits.

Certainly since I started only using cleaned / filtered tracks I have not had a problem with glitches / failed navigation.



citoyen

  • Occasionally rides a bike
Re: Slim down your bloated GPX track files
« Reply #52 on: 23 July, 2018, 07:49:50 pm »
Top work, Phil, very nice. I especially like having the stats file to compare.

I normally use the 5m tolerance setting, which produces files that are small enough for my Edge 510 to handle. Using the 500pt setting on a 200km route, they're about half the size again. The 500pt tracks seem detailed enough for navigating country roads, although there were a couple of places on my test routes where I would want a few more track points (mainly in towns).

Certainly since I started only using cleaned / filtered tracks I have not had a problem with glitches / failed navigation.

I've had a few problems in the past caused by excessively large GPX files - mainly ones downloaded from RWGPS. Worst example was a ride where the file I was using was so bloated, it took 15 minutes to load. By the time I got going, I was well behind the field, hence when my device crashed less than 20km later, I was without any kind of navigation nor even a wheel to follow. Luckily I knew roughly where the first control was but I still got lost a couple of times and added a good 10km to my distance. Eventually I got it working, luckily, otherwise I'd have had to abandon the ride. I learnt two lessons on that ride: one, always carry a printed routesheet as backup; two, always check that your GPX files work on your device before setting off.

Not that I've had any such problems again since learning how to make files smaller and strip out the bloat.
"The future's all yours, you lousy bicycles."

Phil W

Re: Slim down your bloated GPX track files
« Reply #53 on: 23 July, 2018, 09:11:56 pm »
I have never ridden on a 500 point track for a 200km. I normally use the 10m setting which returns around 1100-1500 points depending on how twisty the route.  I have an overnight DIY 200 this Friday to beat the heatwave during the current RRTY run.  I might just try out a 500 point version.  Certainly tracks with less track points and bloat load quickly  and require less processing by the low power GPS CPUs

Ben T

Re: Slim down your bloated GPX track files
« Reply #54 on: 23 July, 2018, 09:25:58 pm »
If I want to reduce to 500 it should reduce to exactly 500, not 498. ;) B minus Phil  ;)

frankly frankie

  • I kid you not
    • Fuchsiaphile
Re: Slim down your bloated GPX track files
« Reply #55 on: 24 July, 2018, 11:23:53 am »
I normally use the 5m tolerance setting, which produces files that are small enough for my Edge 510 to handle. Using the 500pt setting on a 200km route, they're about half the size again. The 500pt tracks seem detailed enough for navigating country roads, although there were a couple of places on my test routes where I would want a few more track points (mainly in towns).

There was a 400km event in the N-E a couple of years ago where several people were reporting trouble with the organiser-supplied GPX file - I had a look at it out of interest and it was about 13,000 points (sourced RWGPS).  I downsampled it to 500 and although there were a couple of towns where it did look a bit rough, even so every junction in 400km that would have involved a 'turn' was still properly described by the track shape.  An extreme example - obviously in practice the best approach would have been to divide the Track into 3 or 4 and then optionally downsample.
when you're dead you're done, so let the good times roll

citoyen

  • Occasionally rides a bike
Re: Slim down your bloated GPX track files
« Reply #56 on: 24 July, 2018, 11:51:50 am »
I had a look at it out of interest and it was about 13,000 points (sourced RWGPS).

As you've mentioned before, it's not just the number of points but the extraneous data - eg co-ordinates given to microscopic levels of precision. One of the things I like about Phil's app is that it makes files much smaller without necessarily reducing the number of trackpoints significantly. 13,000 trackpoints does seem somewhat excessive though.

You have to wonder why the authors of the RWGPS app don't do something about this themselves, as surely it would be easy enough to fix.

Quote
I downsampled it to 500 and although there were a couple of towns where it did look a bit rough, even so every junction in 400km that would have involved a 'turn' was still properly described by the track shape.

The practicality of 500pt tracks possibly depends on what device you're using - on a device without maps, more trackpoints are beneficial in towns where a junction might offer two or more roads going in approximately the right direction. Without a map for reference, it's sometimes hard to know which one to choose.
"The future's all yours, you lousy bicycles."

frankly frankie

  • I kid you not
    • Fuchsiaphile
Re: Slim down your bloated GPX track files
« Reply #57 on: 24 July, 2018, 12:10:27 pm »
It's not usually critical in a town.  Unless you're heading for an actual control location, as a randonneur all you need to do is get across the town and out on the right road, there may be 100 ways of doing that with not much time/distance penalty between the best and the worst.

13,000 trackpoints does seem somewhat excessive though.

Excessive as in, most Garmins used by cyclists have a limit of 10,000.  To make it worse, they often won't report a 'too many points' error (or if they do, it's just an irritating little pop-up you click through during boot-up) but instead simply truncate and the first you know about it is in the middle of nowhere.  That makes either Garmin or RWGPS or both somewhat liable for a situation that could conceivably be worse than just a bit uncomfortable.
when you're dead you're done, so let the good times roll

Re: Slim down your bloated GPX track files
« Reply #58 on: 24 July, 2018, 03:47:18 pm »
To make it worse, they often won't report a 'too many points' error (or if they do, it's just an irritating little pop-up you click through during boot-up) but instead simply truncate and the first you know about it is in the middle of nowhere.

That's exactly what happened to me on a RwGPS-plotted 200 route on a Garmin 810. At around 130km, it suddenly switched to a straight line that went who-knows-where (the destination, presumably). This was before I knew of such glitches but, after following someone to the next control and then restarting the route, all was well.

It's not happened since I started using Phil's tool, nor since I started splitting files to ~100km/inter-control tracks (the latter using a paid RwGPS account).

Phil W

Re: Slim down your bloated GPX track files
« Reply #59 on: 24 July, 2018, 03:56:52 pm »
I have reversed the order it will split and simplify a track.

So (if requested) it will split the track first and then it will simplify each track.  Makes no odds if using the metre tolerances.  But if using the 500 point or 10,000 point tolerances and splitting as well you will get a different result. 

Let's say you decide to split a PBP GPX track into lengths of 300km with a tolerance of 500 points.  Then you will get 5 tracks (PBP is longer than 1200km), and each track will have 500 or less points. Under the old method it would have reduced the original 1200km track to 500 points and then split it, ending up will quite a bit less points in each track.   

The stats.txt file returned has been updated to include track splitting data. Below are figures for the WAWA 2016 2100km GPX track split into 200km chunks.

Simple GPX
https://simple-gpx.herokuapp.com
Tolerance chosen: 500
Split track: yes, split distance: 200km
Original tracks: 1, simplified tracks: 11
Original trackpoints: 21734, simplified trackpoints: 5390, average trackpoints per track: 490

Phil W

Re: Slim down your bloated GPX track files
« Reply #60 on: 24 July, 2018, 04:27:00 pm »

It's not happened since I started using Phil's tool, nor since I started splitting files to ~100km/inter-control tracks (the latter using a paid RwGPS account).

I have thought about adding the option to split tracks at controls but it only works if

The GPX file contains waypoints for every control and the control waypoints are within a reasonable proximity of the actual track (say within 50m).  You do not have extra waypoints in the GPX file being used for other purposes. 

I am not a fan of splitting tracks at controls as it gives me something extra to do when there is plenty to do at the control anyway.  Generally I run long tracks and only split them where they cross over themselves. It is the number of points causing truncation not the length of the track in km.

But certainly adding split by control would not be that hard with the above provisos.

Phil W

Re: Slim down your bloated GPX track files
« Reply #61 on: 25 July, 2018, 04:11:36 pm »
If there are no routes in the GPX then additionally it will now.

Generate a route from the track, with an average of one route point for every 5km.  Obviously the spread of route points will vary, with more where twisty and less when not. If it finds waypoints (it assumes they are controls / places you wish to pass through) then it will try and insert them into the route in the appropriate place.  It will then split the route into 50 route point chunks so it can be auto routed in GPS that support it.

Below is one generated through the process with flags representing via route points.  Use at your own risk etc. etc. or just stick with the simplified tracks which it already outputs.






frankly frankie

  • I kid you not
    • Fuchsiaphile
Re: Slim down your bloated GPX track files
« Reply #62 on: 25 July, 2018, 06:35:37 pm »
Possibly this (and/or possibly the stats.txt file) might break the zip file for the purpose in post #1 of this thread, that is, submission to AUK and subsequently running it through the RouteValidator tool.  I don't know that it does - just speculating, I know that RV did have some difficulties initially with zip files having unexpected contents, and I know that was fixed to some extent but I don't know how bomb-proof the fix is.

However this project has maybe gone way beyond that, and from that angle, it's all good  :thumbsup:

edit: no I've tested this now in RV, and it still seems to work OK, ignoring both the added Route and the added stats.txt file.
when you're dead you're done, so let the good times roll

citoyen

  • Occasionally rides a bike
Re: Slim down your bloated GPX track files
« Reply #63 on: 26 July, 2018, 02:59:29 pm »
Phil, I've tried to use the app a couple of times this morning and got an application error. Problem at my end or yours?

Slightly OT question regarding routes: my device doesn't support routes so they're extraneous data for me personally. However, is it worth keeping them in the GPX files that I supply to riders for use on my events? What is the benefit of routes if your GPX file also contains a track?
"The future's all yours, you lousy bicycles."

Phil W

Re: Slim down your bloated GPX track files
« Reply #64 on: 26 July, 2018, 06:25:46 pm »
Some prefer tracks, others prefer routes for the turn by turn, some like to use both.  I generally have't used routes as my experience has been that they did not go where I wanted them. When creating routes I do not click every junction. So when it gets to the gps it rarely looked like what I had in Basecamp. But with a route filtered from a track it seems from what I can see that Basecamp and my gps concur.  So I may just revisit routes again for my own nav though I am quite used to tracks. I would leave the route in the GPX as an optional extra!

Regards other point will see if I can see in logs what it did not like about your GPX

citoyen

  • Occasionally rides a bike
Re: Slim down your bloated GPX track files
« Reply #65 on: 26 July, 2018, 06:32:05 pm »
Thanks, Phil.

Just been reading up on routes, in relation to the other thread. I think I get the general idea now. It seems to me that for following a prescribed route (as on an audax), tracks are probably a better option.
"The future's all yours, you lousy bicycles."

Phil W

Re: Slim down your bloated GPX track files
« Reply #66 on: 26 July, 2018, 07:07:13 pm »
The only thing I can see in the logs are timeouts at 8:51 and 9:06am.  Was that when you were using Simple GPX?   Is it a GPX with a particularly large number of track points or waypoints and what settings were you using? Mind I have run it against a track with 21,000 track points without issue.

citoyen

  • Occasionally rides a bike
Re: Slim down your bloated GPX track files
« Reply #67 on: 26 July, 2018, 10:48:51 pm »
The only thing I can see in the logs are timeouts at 8:51 and 9:06am.  Was that when you were using Simple GPX?   Is it a GPX with a particularly large number of track points or waypoints and what settings were you using? Mind I have run it against a track with 21,000 track points without issue.

Sounds about right, time-wise. If the problem is timing out, that could be due to my flaky internet connection.
"The future's all yours, you lousy bicycles."

citoyen

  • Occasionally rides a bike
Re: Slim down your bloated GPX track files
« Reply #68 on: 27 July, 2018, 08:54:06 am »
Forgot to add: it was an RWGPS file, 585KB, 7161 points, set to 5m tolerance
"The future's all yours, you lousy bicycles."

Phil W

Re: Slim down your bloated GPX track files
« Reply #69 on: 27 July, 2018, 10:39:28 am »
Your GPX works fine now.  Trying to hit too fine a points tolerance when generating a route from such a short track meant it hit the timeouts.  I have relaxed the tolerances for route generation.

Route generation is now also a seperate action in the tolerance drop down rather than an automatic operation every time.  If you now select generate route in the dropdown then that is all it will do, leaving the original track alone.




citoyen

  • Occasionally rides a bike
Re: Slim down your bloated GPX track files
« Reply #70 on: 27 July, 2018, 10:49:32 am »
Top work, Phil. :thumbsup:
"The future's all yours, you lousy bicycles."

frankly frankie

  • I kid you not
    • Fuchsiaphile
Re: Slim down your bloated GPX track files
« Reply #71 on: 27 July, 2018, 11:10:07 am »
Very good.  I would only comment that if you now want a Track and Route, you end up with two GPX files of the same name.  It would be nice (I know  ::-)) if the route filename were suffixed -r or similar?
when you're dead you're done, so let the good times roll

Phil W

Re: Slim down your bloated GPX track files
« Reply #72 on: 27 July, 2018, 11:27:15 am »
There you go it will now add   _route to the end of the zip file and GPX names if you are generating a route. Be aware the original track file will still be in a GPX where you have generated a route.  Could not decide whether to leave the original track in there or delete it when generating a route.

Phil W

Re: Slim down your bloated GPX track files
« Reply #73 on: 27 July, 2018, 11:36:11 am »
Strike that, if generationg a route the returned GPX will only contain the route and any original waypoints. It will now delete the original track.  That way you will not accidently load an unfiltered bloatware track if you load the GPX route file.

frankly frankie

  • I kid you not
    • Fuchsiaphile
Re: Slim down your bloated GPX track files
« Reply #74 on: 27 July, 2018, 05:56:14 pm »
Very useful utility.  :thumbsup:
when you're dead you're done, so let the good times roll