This is an uncomfortable situation. Try to notice if he does look around. I remember thinking my teachers/professors would only be looking at me, but that would just be when I noticed I caught their eye, yet they would look around the room throughout class. Try to speak to him, or another teacher that you have a close relationship with, to discuss your concerns if you do discover that he is, in fact, only looking at you.
Teaching In Real Life 