Interesting that these words are problematic in other languages too. The Polish words stadion, datum, medium, konsorcjum, are strictly speaking neutral because that is their Latin gender and form plurals stadia, data, media, konsorcja, but in practice they are often treated as masculine, in line with the general rule that words ending in consonants are masculine, which gives them problematic plurals stadiumy, datumy, etc. These just sound wrong and produce some strange declensions.
So, you're in international society with this one.