Author Topic: what 3 words  (Read 4285 times)

Re: what 3 words
« Reply #75 on: November 30, 2019, 11:42:56 am »
Storage is cheap and 10Mb is neither here nor there in the days of 128Gb SD cards. Some things are worth optimising for space, some for speed.  Data of 10Mb isn't worth the time to optimise in this day and age.  Different to the 80's when I first became a professional programmer and had to optimise for programs and their data to fit into 32kB; least you wanted to invoke the joy of dynamic loading and memory swapping

Mind I'm not sure where this DB sits as I believe the translation between Lat, lon and the three words take place on their servers. So the app doesn't really need a copy.

Re: what 3 words
« Reply #76 on: November 30, 2019, 11:58:30 am »
I would have though a few tens of kilobytes would be all you would need to store the list of words.

I went back to look at https://whatfreewords.org/about.html to see how many words were in use and it's gone. Lawyers I presume.

But if we take 40,000 words as a given:-

40,000 words * average 6 letters per word = 240,000 bytes.

Compression of english text (with no words repeated) is going to be hard to get better than 50% so you're going to have 120,000 bytes minimum for storage of the words. That's way more than "a few tens of kilobytes" but still way less than 10MB.

If they ship one database with multiple languages then I can see how they can get to 10MB for the database.

Mind I'm not sure where this DB sits as I believe the translation between Lat, lon and the three words take place on their servers. So the app doesn't really need a copy.

The whole point is that the app works where you have no data connection, so the app must have everything it needs to do the conversion.
"Yes please" said Squirrel "biscuits are our favourite things."

Re: what 3 words
« Reply #77 on: November 30, 2019, 12:32:10 pm »
Ah, I thought the app didn't have the DB as the company was worried about someone reverse engineering the code to get to what is proprietary data. If it requires a data connection it's a lot less useful in remote areas where you can often get a SMS out but nothing more.

Re: what 3 words
« Reply #78 on: November 30, 2019, 01:35:46 pm »
There are only 26 letters so 1 byte per letter seems a little inefficient, both in terms of storage and search speed. Many letter pairs and triplets are frequent and a sorted listed of words have common stems. I believe it will be a few tens of kB rather than hundreds of kB and not mb. As it is produced as a one off and extra few minutes generating it seems worthwhile.

Storing data efficiently is making a bit of a comeback as datasets are getting huge and also the move to battery powered devices and transmission. Open streetmap for example uses variable bit length integers, so for example you may only use 11 bits to store a particular number and 35 bits for another. Garmin ant+ protocol is very compact because it means the sensor batteries last longer.


Sent from my iPad using Tapatalk

Kim

  • Timelord
Re: what 3 words
« Reply #79 on: November 30, 2019, 01:50:33 pm »
Yeah, but if it's not being transmitted, it makes sense to optimise for CPU cycles required to do the lookup, rather than storage space.  A few more megabytes of storage might be a reasonable tradeoff for higher speed and a bit more battery life.
Careful, Kim. Your sarcasm's showing...

Re: what 3 words
« Reply #80 on: November 30, 2019, 03:01:29 pm »
If you are looking up a string that is half as many bytes it will be more efficient, even though for 40,000 entries it is only 16 or so comparisons to find an entry so not a lot of saving.


Sent from my iPad using Tapatalk