My theory was that they had kept him tucked away somewhere in Vorlon space all that time - I forget if they said he worked over others, but if so then he could have been kept either in stasis or merely with no work to be done in-between, which might be heaven for some but for his personality type seemed like hell.

But either way, like you said about Lyta, he was an absolute tool - their absolute tool.