Thought on this...
You seem like a more gentle and non-confrontational personality type. The fact that you're struggling with this character is proof that you are outside your zone of comfort in dealing with his remarks. It also makes you a prime target for such remarks. You're getting upset about it is exactly the positive feedback this character wants.
Of course, we're adults, so hurting him is out of the question. If your own self-esteem is in check, than the words of this guy should slide right off your shoulders.
Are you really going to let some wise guy mess with your head over comments? Perhaps there is some truth in his comments or perhaps you feel that you are not witty enough to take him on in a battle of words.
In either case, in this game, it's best NOT to play.
The next time he makes a comment, take a deep breath, mentally note what was said, and decide if the remark requires a response.
Don't go climbing any clock towers yet.
He'll get in over his head sooner or later and you can stand and watch as he drowns. Or, you can help him out of the water. I prefer the latter.
Good luck and best regards.
Doc Ouch