I think Archive from 1959 is a good place to start. It's kind of a best of kinda thing from his career, though of course tons of great songs were left off. You could see which bands you are most interested in from this, and then start working on getting the albums.
Really though, he has so many bands and they have released so many albums I don't think there is wrong place to start.
I won't try to list every project, but check out:
Pop Rivets, his early punk band
Milkshakes, great garage rock with a 60s flavor
Mighty Caesars, more chaotic garage with punk elements
Thee Headcoats, see above
Buff Medways, garage rock with some nice melodic stuff mixed in
Headcoatees, great girl version of the Headcoats, he plays guitar and writes the songs
There are more of course.. Maybe you already know about this stuff, but figured I'd list it just in case.
If you want to hear his current stuff, check out The Musicians of the British Empire, it's great. Also they are easier to get your hands on, since a lot of his older stuff is out of print right now.
Check out Damaged Goods Records, Interpunk or ebay and you should find what you need.