Having now heard all three of his albums, I can safely make an informed judgement:

Five Leaves Left is miles better than the other two. Bryter Later is obviously, a million times worse, and is sometimes offensively poor.

Pink Moon, whilst being better than BL, doesn't seem to have the same effect FLL has on me. You can tell he's very bitter and pissed off at everyone for being nasty to him, and he had a right to be, since he's so good. But I think it gets in the way of creating pretty songs. The scores on Five Leaves Left, and the weird time signatures and stuff... its almost classical. Its perfect.

So yeah. Five Leaves Left. The best?