I installed Gentoo 3 months ago, I still remember the first (and only) time I installed it, it took me three days to be able to put everything together and have a working system, including the GUI (KDE).
I've tried other distros before I start using Gentoo (maily Mandriva and Kubuntu), I can't say that I didn't see a difference in speed between Mandriva and Gentoo (as a matter of fact, you can see the difference between Mandriva and any other distro), but there was almost no difference in speed between Kubuntu and Gentoo.
As hard to use as it is, and as hard to install it is, I liked Gentoo, may be because of the geeky-ness factor :-/ I don't know.
Eventually, I got fed up with Gentoo, I reached the point that I don't want to install from source anymore.
That's when I decided to try a new distro, since I was planning to take the RHCE exam, I decided to give Fedora a shot, since it is the closest to RHEL.
So I did, and my only regret is that I didn't find it earlier.