I'm not sure if i should contact him?

If you want to get back in touch with him then yes. You should tell him that you are so sorry, and he is an awesome person, but you were just unsure how to respond when you were suddenly and unexpectedly put on the spot about it. And tell him what you said, that the rumors suggested that you hated him, but you really don't and that maybe you can be friends and that you have been thinking about it and really wanted to clear it up. Good luck :)