When a Raspberry Pi just isn't enough

[David Martin]

My birdfeeder is hung on a weighing scale which, via a pi zero, triggers a photo to be taken with an entry of the weight change and other data stored in a MongoDB database. I have ignored this over the summer but on returning to it (may be building a red squirrel cam like it) I discovered that the MongoDB database is full. It can only handle 2Gb data as Raspbian is a 32 bit OS.

So I will move the mongodb server to a 64bit machine. I'm havering on a recycled fanless thin client box, but was wondering if the crowd had any wisdom they would care to impart on a suitable box. Obviously it would be good to use that for the photos as well, so could do an NFS export of the appropriate drives for photo capture.
I could just recycle an old desktop but that is noisy with the PSU and overkill for power.

[Gattopardo]

Pi 3 b is 64bit.

[David Martin]

But the OS isn't. It is deliberately kept at 32bit

[tiermat]

Change the OS, then. Fedora have a 64 bit image.

[David Martin]

It's currently on a Pi Zero. I'll take a look at Fedora. I think I'm still after a fanless PC rather than another Pi. I do have a Pi 3 somewhere that I could use for starters.

[Greenbank]

It can only handle 2Gb data as Raspbian is a 32 bit OS.

Running on a 32-bit OS doesn't automatically limit things to 2GB.

The limit is because of something in your toolchain, not because of the operating system.

[philip]

Debian's 32-bit i386 has had large file support for years, but I suppose the arm architecture might have an additional limitation.

Another possibility is that the system supports large files but you have "ulimit -f" set to limit the size of files that can be written.

[fuaran]

Isn't there some other sort of database you could use? Does it need to load the whole database into memory?
Or split it into shards, or archive part of it into a separate database.

2GB must be quite a lot of bird weights.

[Pickled Onion]

It can only handle 2Gb data as Raspbian is a 32 bit OS.

Running on a 32-bit OS doesn't automatically limit things to 2GB.

The limit is because of something in your toolchain, not because of the operating system.

The 2GB is a MongoDB limit. It's to do with the way it uses memory-mapped files for performance. They made a design decision not to worry too much about 32-bit OSs as the intended use of MongoDB as a superfast "Big Data" document store kind of expects most people to be using slightly bigger hardware than a rPi.

[David Martin]

Isn't there some other sort of database you could use? Does it need to load the whole database into memory?
Or split it into shards, or archive part of it into a separate database.

2GB must be quite a lot of bird weights.

It is. A significant number (3 digits) of visits per day over 6 months. Does seem to be rather inefficient though for space. If each record is a few kb then that adds up rather rapidly to a few Gb.

I could look at using a different DB, but will move it off the Pi in the first instance once I can find a suitable machine to use. Might be able to salvage one from the lab.

[pdm]

Postgres and MySQl (aka MariaDB for open source version) both run on linux systems and are part of the installation repositories for almost all systems and ,if not, are easily installed or compiled from repositories. Database limits are pretty much limited by attached storage provided, of course, the appropriate database storage directories are on that media.

Data limits for Postgres:
    Maximum size for a database?             unlimited (4 TB databases exist)
    Maximum size for a table?                16 TB
    Maximum size for a row?                  1.6TB
    Maximum size for a field?                1 GB
    Maximum number of rows in a table?       unlimited
    Maximum number of columns in a table?    250-1600 depending on column
types
    Maximum number of indexes on a table?    unlimited

Data limits for MySQL (bit more complicated and depends a bit on  what architecture it is compiled for and if tables are memory based or not and so forth)
See http://forums.mysql.com/read.php?21,25724,224521 and http://mysql.rjweb.org/doc.php/ricksrots

Both are usable by most programming languages with APIs provided. (e.g perl, python, php, etc.)

FWIW, when I did some dabbling with databases a few years ago, I preferred Postgres over MySQL/MariaDB....

[David Martin]

I've used Oracle, PostgreSQL (older versions pre and MySQL in the past from many languages starting with P (and a bit of J). My rationale for using MongoDB was the ease of refactoring. And a chance to get some familiarity with it.

[tiermat]

Are you sure about the document sizes? A few kb seems rather large to store a weight and time.

Scratch that, I have reread and noticed the requirement for storing a photo. If you are hitting a hard stop with the collection size, I would look at how to keep the photos out of the documents but reference them therein. Either that or do some image processing to reduce the picture resolution / size.

Whilst I am laid up with a dodgy leg, I'll mull it over a bit

[David Martin]

Photos aren't in there. Not sure why it hits so much, maybe it is the indexing as well? There are a few hundred records per day. Each contains a handful of timestamps and associated metadata , filenames, annotations (whether a bird has been recorded) . Probably approaching a k or so per record when all the random text is taken into account.
I hadn't expected it to be so large. My workaround is to create a new DB every month which should keep the file size down to very manageable.

[Afasoas]

If you want a server for storing the database/files (..which makes a lot of sense as it can be kept indoors and can serve as monitoring for your Pi zeros), then an Atom D525 board is probably a good budget choice.
Paired with a Pico PSU and an SSD, you are looking at power consumption around 10W:

https://www.ebay.co.uk/itm/Intel-D525MW-Intel-Atom-processor-and-1GB-DDR3-RAM-Motherboard-Mini-ITX-Bundle/252901312253?hash=item3ae217c6fd:g:oiwAAOSwdjNZAmXX


I have a single core Intel Atom NUC which sips 5W

It's a more expensive option, took a firmware upgrade to get wake-on-LAN to work but it's been utterly reliable. It sits in my DMZ as a jenkins build server, demo web server and IRC client.

I also have a Brix based on the Celeron N2807 CPU which is less asthmatic in CPU department (dual core) for a similar power draw. It's a bit less stable under linux than the NUC and doesn't feel as solidly put together.

The Brix BXBT-1900 is quad core, only a smidge dearer than the above and tramples all over the above in terms of CPU grunt, but will draw ~10W.

[David Martin]

Thanks, that looks like a plan. A small SSD and it should be ideal.

..d