Personally I would at this stage in the game tell him, them or me? Seew how happy the ex would be with her kids dropped on her doorstep. Does he have legal custody or is he just stupid?
You either need him to understand that this treatment of you is not okay. DAH!!! If he doesn't care then why waste your time, all you are now is a sitter for those unappreciative brats....is that okay with you?