Was I right to end this "emotional affair?"?

Ending it is the right thing to do. If he wishes to end his marriage and then pursue a relationship with you, that's different, but having what amounts to an emotional affair is hardly working on his marriage. By staying away and having nothing more to do with him you are behaving as well as you can, it's really the only thing to do. Of course you seem like a lovely fantasy to a married man 20 years older than you, but there's no future in it. Stop wasting your youth on this faithless man and let him get on with fixing his marriage.