If you really want to get your hands dirty, try Gentoo. Be warned though, it's not for the timid, most things are installed by compiling them from the source, although you can install most of the initial chunk as a binary. This can make installing things a bit slow on older machines, but most modern machines can hammer through all but the most complex installations pretty fast.
I use it on some of our machines, since there servers, and headless, so I don't need all the X11 junk to be installed. I configure and run everything using the command line, so SSH is fine for that. X11 can be installed, and will run fine though.
The flip side is that since almost nothing is installed by default, you probably want a network connection until you are sure that everything you need or want is installed.
The installation is also significantly more complex than most distros like Ubuntu, where you only have to make the occasional mouse click, either print out the installation guide, or have a second machine to hand to check things on.