If you haven't found your optimal distro yet, Lubuntu (using LXDE) will definitely be faster than Xubuntu (using XFCE) as it's more lightweight.
Also, it's not just about the Desktop Environment and so on, a lot depends on the applications you install. Everything else might be really light but if the programs you use use up a lot of RAM it won't help much. See ArchWiki for a list of
lightweight applications! For example, I consider Firefox to be heavy, especially when compared to Links. The least you could do is use Midori or Epiphany. There are plenty of lightweight alternatives.
When you're feeling awesome you can try something like Arch Linux and start from scratch installing only the things you want to have until you have something completely personalised to how you work. When you're too awesome you'll probably try Gentoo…