edge 200 for AAA points

[Greenbank]

Yes FIT does look like a good format and it would be great to develop the software to accept it.

My own stuff accepts fit files and uses fit2tcx and a script to convert it to GPX before doing any work on it. I'm slowly replacing that with my own program that reads FIT directly.

4.7Mb would zip down to an acceptable size, 9.5Mb probably wouldn't (about 8.5 would probably be the max).

Where does the limit come from? 10MB isn't a big file in today's world (as Phil's utility shows since his tools accept it with no problems).

I would suggest using the '5 metre' setting for tracklogs - this retains the most data. And of course retain the time stamps and elevations.  Out of interest, what does your 9.5Mb file end up as if you run it through this?

137KB (5m, preserve timestamps and elevation). Down to 1074 points. (He could shave 30KB off that by not indenting lines with spaces.)

Last year I helped someone salvage and coalesce a ride they had done into one file (it was saved as 8 or 9 individual FIT files corresponding roughly to individual days and some had some mild corruption). The result was a single 25MB FIT file. Oof.

[frankly frankie]

Pretty good, but that has lost too many points really.  Could do with a '2 metre' option, or just a single tickbox to optimise everything for tracklogs.

Where does the limit come from? 10MB isn't a big file in today's world (as Phil's utility shows since his tools accept it with no problems).

The limit is an arbitrary one set by myself, since I regard anything over 1Mb as overkill for a tracklog.  Zipping before upload gets you about 8Mb as I say.  Although there does also seem to be some server-related upload limit as well, of around 5Mb for a file, I haven't been able to track down what causes that.
It's a question of storage, as much as bandwidth - Phil's utility doesn't store the files - AUK's server retains uploaded files for a few months, there are currently 360 tracklogs and over 1,500 'proposed' GPX files in store.
Also the routevalidator software slows to a crawl if either or both files submitted are overstacked with points.  Again some development here to make it more sophisticated (eg, sampling every nth point) would be on my wish list.

[Greenbank]

Pretty good, but that has lost too many points really.  Could do with a '2 metre' option, or just a single tickbox to optimise everything for tracklogs.

Just looked at it in various tools and it is perfectly good to me, more than enough info to determine climbing.

My own old rudimentary script gives:-

TOT: points=1073 climb=1262 dist=108697.03m

It's a question of storage, as much as bandwidth - Phil's utility doesn't store the files - AUK's server retains uploaded files for a few months, there are currently 360 tracklogs and over 1,500 'proposed' GPX files in store.

Sure, but you don't need to store the original file either. Receive it (even if it is huge), convert it, remove unnecessary data/attributes, and then downsample it. Compare it to make sure you haven't damaged it beyond recognition and, if ok, throw away the old huge file and keep the much simpler smaller one. [if not ok, throw away both and provide a suitable error message]. There's no need to store anything bigger than you already are. (Although my stuff does but I have something to download the original large files every day and store them on my own machines at home where storage is cheap as online storage is not required for them).

My own stuff runs on a cheap VPS from OVH. 2GB RAM is enough for the OS, httpd server and my scripts to run on the data. 10GB storage is enough for a minimal OS install and the rest (6GB+) is left for data/logs. All that for £3/month. (Bandwidth is a non-issue, you get 100Mbps unlimited traffic.)

[frankly frankie]

Bandwidth is always an issue with me 

First up, Fit2Gpx looks useful and I'll try to install that so that the uploader can accept FIT files (and convert them to TCX before storage - zipped TCX is an acceptable format for RouteValidator).

Just looked at it in various tools and it is perfectly good to me, more than enough info to determine climbing.

Maybe you're right.  I don't know how the AAA checking currently works.  (Gosh, we're almost back on topic!)
An average of 1 point per 100 metres seems a bit 'rough' to me, for AAA purposes.  But if the checking goes back to a DEM, it's neither here nor there.

Sure, but you don't need to store the original file either. Receive it (even if it is huge), convert it, remove unnecessary data/attributes, and then downsample it.

Hmm yes.  Well this side of the operation (validation by GPS) is in its infancy but obviously a huge growth area, so it probably is a good idea to put some effort into that sort of thing.  Thanks for the ideas.  I'm kinda hoping and expecting that when we take this stuff to the next level, it won't be me at the sharp end any more. 
Meanwhile I will raise the size thresholds *a bit* so that at least you or anyone else recording a point per second can get a 100km tracklog up in zipped form.

[Somnolent]

Bandwidth is always an issue with me 

Sure, but you don't need to store the original file either. Receive it (even if it is huge), convert it, remove unnecessary data/attributes, and then downsample it.

Hmm yes. 

I've nothing against creating filtering tools so that only just sufficient information is stored and it's only the folks who choose to upload large files that have to sit watching an egg timer (and not the volunteers as with the current prototypical tools for track comparison and AAA determination) - but what baffles me is why folks choose to record a trackpoint (and the extension data) every second?
What does all that data do for them ?
I've just looked at my most recent track overlaid on mapping and satellite imagery - it only rarely deviates from the road by more than the width of the road itself, and that's recording at less than one tenth the frequency.   Much better than adequate for validation of a mandatory route DIY and quite good enough, except in the rarest of cases, determination of AAA points.

[Greenbank]

but what baffles me is why folks choose to record a trackpoint (and the extension data) every second?
What does all that data do for them ?

Power and HR data.

Sure it could record those at, say, 5 second intervals (as averages of the last 5 seconds) but all of the tools that analyse power expect to see per second data.

If you don't have those then 'smart recording' will be good enough to record a good enough track and be far from 1 data point per second.

[frankly frankie]

Power and HR data.

Still alive aren't you?  What more do you need to know?   

[Somnolent]

but what baffles me is why folks choose to record a trackpoint (and the extension data) every second?
What does all that data do for them ?

Power and HR data.

Sure it could record those at, say, 5 second intervals (as averages of the last 5 seconds) but all of the tools that analyse power expect to see per second data.

So what do these tools that need so much data actually tell you?
That you were producing less power on datapoint 30 000 than you were at datapoint 1000 ?   A cynic (moi?) might say "Well, duh"
I can see detailed analytics might be useful if you were training to keep up with the pro peloton but if you are interested in AAA points then I kind of think that may not be where you are at?
- - - - - -
Edit: beaten to it by FF (and for succinctness too)   

[Greenbank]

Data help keep me motivated.

Do you submit all of your brevet cards for rides you do? Why? You're still alive without those AUK/AAA/FWC/etc points aren't you? etc.

I suppose the other answer is, because I can. I record one data point per second because the GPS has more than enough storage to be able to do that without causing complications for me (and I'm more than happy to downscale it if required, and I'm even able to write my own code to do exactly that). As a computers/maths geek I abhor the idea of throwing away data, even if I may not have an immediate (or even future) need for it.

[fuaran]

Recording 1 point per second is good for Strava segments. It could make a difference to your time / ranking.
Or surveying for OpenStreetMap etc, it is useful to have lots of detail.
Also if lower frequency, I do sometimes notice tracks cutting corners. Not really a problem for on-road cycling, but it can be noticeable for twisty mountain biking or running.

Better to record too much detail than too little. It is easy to simplify afterwards if required.

[Phil W]

An average of 1 point per 100 metres seems a bit 'rough' to me, for AAA purposes.  But if the checking goes back to a DEM, it's neither here nor there

The algorithm's aim within my simple-gpx tool is to retain the shape of track to within the tolerance you choose, discarding points that do not impact that.  So the average point per 100m is slightly skewed.  Where a track is twisting and turning there will be more points, where it goes dead straight (like in the Fens) there will be less.  So perfect for mandatory route validation as the filtered track will still align.  For AAA if you have a dead straight road across the top you may not have a point on the summit and mis some metres. But AAA ahould be consistent and not depend on points in the track. So electronic  contour counting using the OS data or some other DEM model. 

Adding finer or coarser grained tolerances to Simple-GPX would be trivial, just more options in the drop down, the back end just uses the tolerance submitted with the request.

[Revellinho]

I don’t know how to change the sample frequency of my garmin- is it possible with a edge?

[Greenbank]

But AAA ahould be consistent and not depend on points in the track. So electronic  contour counting using the OS data or some other DEM model. 

That's the key point. If relying upon a DEM then elevation counting algorithms should be able to take a sparse track and interpolate points along the route to use values from the DEM.

Adding finer or coarser grained tolerances to Simple-GPX would be trivial, just more options in the drop down, the back end just uses the tolerance submitted with the request.

If the elevation data in the track was going to be kept then you might want to apply the Ramer–Douglas–Peucker algorithm in 3 dimensions, i.e. a point is not considered for removal if it is an inflexion point in the elevation (i.e. the points either side are both lower than it, or both higher than it, except within some small tolerance).

[Greenbank]

I don’t know how to change the sample frequency of my garmin- is it possible with a edge?

With my Edge 705:

Menu -> Settings -> Data Recording -> Record Data Points

It gives me two options: "Every Second" or "Smart Recording".

"Smart Recording maximizes storage by recording data when it changes. For more detailed Every Second can be used."

However, with a power meter, cadence sensor and HRM strap, Smart Recording pretty much becomes "Every Second" unless you put in a very consistent and even effort.

[frankly frankie]

The algorithm's aim within my simple-gpx tool is to retain the shape of track to within the tolerance you choose, discarding points that do not impact that.  So the average point per 100m is slightly skewed.  Where a track is twisting and turning there will be more points, where it goes dead straight (like in the Fens) there will be less.  So perfect for mandatory route validation as the filtered track will still align.

The checker tool takes each point in the 'proposed' track and searches the tracklog for a point within a specified distance of that point (the default setting is admittedly 200m) - so having a tracklog that is too sparse (say on a long straight road) could conceivably raise red flags. 
Other than that, it looks as though your tool could now be used to simplify tracklogs before submission, which I'm sure some people might find very useful.  It's great for stripping out all that metrics cruft.  But it would need to be at its lowest setting or as I say, some explicit 'tracklogs only' optimised combination of settings.

[grams]

For my GPX app the first thing it does when you add a route is interpolate it so there's a point at least every 20 metres. Otherwise various things misbehave (climb calculations, distance along route calculations, etc) any time there's a straight section of road with no points. I'd hope any AUK tools working with GPX files do something similar.

On the topic of thinning files, when uploading proposed Mandatory Route DIYs, one problem I've had is that shedding points down to the required size can easily lose a percent or two of distance, which can take e.g. a 606 km route under distance, which is somewhat annoying given all the other stresses of route planning.

[Phil W]

The algorithm's aim within my simple-gpx tool is to retain the shape of track to within the tolerance you choose, discarding points that do not impact that.  So the average point per 100m is slightly skewed.  Where a track is twisting and turning there will be more points, where it goes dead straight (like in the Fens) there will be less.  So perfect for mandatory route validation as the filtered track will still align.

The checker tool takes each point in the 'proposed' track and searches the tracklog for a point within a specified distance of that point (the default setting is admittedly 200m) - so having a tracklog that is too sparse (say on a long straight road) could conceivably raise red flags. 

You put both the proposed and the actual track log through the simplification.  Then you check the simplified track / tracklog seeing if any trackpoints are more than 200 metres apart. If they are, you interpolate track points on those sections to ensure there is one at least every 200 m.  Then your existing validation could continue as it is now. 

[Phil W]

If the elevation data in the track was going to be kept then you might want to apply the Ramer–Douglas–Peucker algorithm in 3 dimensions.

Not that hard to do either.  I would just need to change two functions, one for the distance between two points and one for the distance of a point from a segment from 2D to 3D. i.e introduce a z coordinate, the elevation into the equations. The rest of the algorithm wouldn't care or need change.  In fact the functions could be updated to 3D with a default of 0 for the z (elevation) coordination , so the default continues to work in 2D. If elevation data set to be retained then it uses the elevation data and we get a 3D calculation.

[fuaran]

I don’t know how to change the sample frequency of my garmin- is it possible with a edge?

Depends on the model. The Edge 200 will only do smart recording. Most of the other models have the option of 1 point per second.

[frankly frankie]

As mooted (and I hope this is still more or less on topic) the Tracklog uploader for DIY(M) events will now accept .FIT files, up to the size limit stated on the form.  They are converted to TCX and zipped before storage.  My understanding is that zipped TCX is an acceptable format for the checker tool, but NB this final step (checking) hasn't been tested.

As we are on the topic. The Edge series churn out a FIT file. The organisers want a GPX track file... how do I convert one into the other retaining the time information? Do I need GPSbabel?

No need any more - file size permitting.

[Revellinho]

I don’t know how to change the sample frequency of my garmin- is it possible with a edge?

Depends on the model. The Edge 200 will only do smart recording. Most of the other models have the option of 1 point per second.

The 500 has the choice in settings. I ended up getting a second 500 for less than £50.