@crusader8463: Yeah I say confront him about it and if he continues you have to go to your supervisor. A couple times it's funny, if it's an ongoing thing you have to report him or something.
I just let it slide for the longest time because for some reason gay guys love me and always hit on me, so I'm used to it, but the bit at the end of my OP about what he said in the lunchroom is what made me decide otherwise. If he makes an effort to keep coming at me after we get split up and assigned to the floor I will start looking into confronting him and then going from there. Hopefully it will just blow over.