Katrinka,
Funny you should mention thinking about contacting his agent, because that's exactly what I would be tempted to do in that situation. I would think that his agent/publisher should be - and would want to be - made aware that the author has used your likeness/emails without permission. It may or may not change anything, but even if the material is/was innocent and harmless, I'd stll do everything I could to protect my privacy and myself from any negative impact that publishing this material could have later on down the road. I definitely don't think that's going over the line or unwarranted.

What do others think?