I'm frightened...

Save-the-dates are out (yay!) and this means that the people I've invited from work are now talking about it.  But there's one problem with this.  I didn't invite everyone from work.  I only invited people that I'm friends with.  Now, that shouldn't be an issue with most people because I really don't think the people who aren't invited would expect to be invited.  But I've learned that when it comes to weddings, people will surprise me.

You see, as with any typical workplace, there's this one guy at work who is extremely... well, let's just say difficult.  He's very abrasive and manipulative, and we just don't get along.  At. All.  So when it came time to draw up a guest list, there was no way he was going to make the cut.  Ever.

Source

But... this is where the problem comes in.  Eventually, he's going to figure it out.  He's going to realize that almost everyone else from work is invited, save for a few random people, and he's not.  And the trouble with this particular guy is that in addition to his abrasive and manipulative behavior, he's just tactless.  I honestly wouldn't put it past him to ask me where his invitation is or why he's not invited.  And the worst part of it is that he has to know he wouldn't be invited and he can't possibly expect an invitation.  So his only reason for asking would be to put me in an uncomfortable position. 

Being the relatively non-confrontational person that I am, the prospect of having this conversation frightens me.  I can't imagine what I'd say if this happened.  I mean, the reality is that we're not in kindergarten anymore where all the kids in the class get in invite.

Source

I really shouldn't feel bad, but I do.  Even though he treats me so poorly and obviously has little to no respect for me, I still feel bad.  Not bad enough to invite him, but still.  The guilt is there.

So if you were me and he asked, what would you say?  Have you ever had anyone not invited who got confrontational about it?  How did you handle it?