left with no choice...?

I just don't think I have a choice anymore. I think the only thing that's left to do is admit defeat. He was right... from day one.. that I couldn't do this. I argued. I fought it. I really thought I could be strong enough for both of us. Even after finding out that "she" is pregnant... I thought I could handle it. My heart hadn't changed... and I really don't think it ever will.... but I've started to realize that I have to let him go. This isn't a case where a partner in a marriage cheats, then realizes what he's done wrong and begs for forgiveness until the wife is able to either accept him or not. That's not it at all. I pushed him away through very poor actions of my own and he ran to another woman. Yes, he hurt me badly during that time he was with her... more than I could say I'll ever forget, though not enough for me not to forgive. My strength was there at his darkest time where I really thought he might end his life after finding out about her pregnancy. I was able to reassure him that it's not the end of the world and that it didn't change the way I felt about him. But.... in the week that's passed since then... we've shared some very big highs and some very low lows. My heart was filled for love with him and I'd feel like it was enough. The only problem is that it's not being returned.... and I'm discovering that it's clearly not enough. He admits that something between us just feels off. Not right. I ask "which part exactly doesn't feel right? when you're looking me in the eyes.. kissing and touching me? or when you're spending the afternoon with your son and I laughing and playing?" He says he enjoys all of that. Then what more.. the silence that ensues after we make love? Or when I make an attempt at holding his hand in the car when we drive somewhere and he's not able to reciprocate? You see.. it's not to say I haven't noticed those times, but I try so hard not to focus on them. I feel he's lost in thought.. perhaps in guilt... and that it will fade in time. The only problem is that I feel as though time is not on our side anymore. There will be so many changes in the coming weeks. A move.. a new baby... while he enjoys being in our lives for a few hours a day, he can't bring himself to talk or think about anything past that. While that was ok in the beginning, I'm slowly coming to terms with the fact that I DO have a time limit in my head. I need to feel like a decision has been made regarding my marriage before I deal with the stress of caring for a newborn. I don't know if I can physically or mentally handle both at the same time. If I were to suddenly have a newborn right now.. in the middle of what's going on.. I really don't think it would receive the proper care and attention that's required. I'm NOT ok with that.

I know he's incredibly sorry (as am I) and I have forgiven him, but it's not enough. He's not running back into my arms like you might expect. My deepest fear is that I will forever love someone who cannot return the same emotions. I don't know what to do.