I really didn't want to post in this thread but felt compelled to when you refer to the 'physical relationship' you guys are having. I'm assuming that by physical you mean 'sexual'.
If thats the case, then I'll bet my bottom dollar that God is not pleased with your relationship, regardless of what this guy has and is doing for you and your kids.
I'm not good with bible verses but I will tell you from revelation of He Himself:
My closest friend lost his sister from a sudden and misdiagnosed illness.
She was living with her boyfriend, had a child and was unmarried.
God showed him her when he was wide awake: no symbolic dream, no confusing feelings or hallucination, but a divine image of her- I'd prefer to not go into detail as to what was shown to him...She's now Hades and lost eternally.
Like I said before, I didn't want to say it but its the truth.
I have to wonder how many people are going to lose out on eternal life because of ungodly relationships...Is he really worth it?? He wont follow Christ?? Whats he so attached to in this world that he rejects Christ?
I suggest you leave the guy and ask for forgiveness.