You can be firm without being mean. It helps when you talk to him to actually speak quieter than normal. Then he'll know you are serious but not vindictive.