I wouldn’t be so quick to let guilt shoot myself in the foot if I were you.
Do you love him? Does he love you? That is, is it mutual? That’s the baseline here. If either one of those ever works out to ‘no’, then it’s time to end the relationship.
If the relationship stops working, that is, if it doesn’t meet the needs of one or both of you, then it’s time to either work on the relationship or end it.
If enough time has passed and one or both of you can’t commit to the relationship, then it’s time to break up.
Everything else can be worked on. Those three things can’t. It sounds like you want to throw away a working relationship over feelings of guilt. Well, if you really can’t get over the guilt, then the relationship isn’t working for you and you need to break it off. But it sounds to me like someone finding a winning lottery ticket and then just throwing it away because you don’t think you deserve to be rich.