Nahh, it's the power-optimised CPU that makes scrolling slow. I vaguely recall that the USB interface is limited to USB 1 speeds, too; if you're transferring map data to/from the card it will go a lot quicker if you remove the card and put it in a reader.
Even with a really slow one, the card isn't really the bottleneck.