Help me figure out how to make this work!?

Talk to him! Its not good keeping this all inside to build up and up. Give him an ultimatum. If you really love him, tell him that this is the last straw. He has children and is in a relationship with you, and deep one and that should be his first priorities. He can have fun every once in a while with his buddies but not everyday. You need him. If he doesn't shape up, leave. You have been doing it on your own already.