I'm mostly OK. I'm happy at work, happy at home. But I feel like I'm on the precipice of royally f*cking things up. My relationship with a colleague is dangero

I'm mostly OK. I'm happy at work, happy at home. But I feel like I'm on the precipice of royally f*cking things up. My relationship with a colleague is dangerously close to becoming inappropriate. I'm happily married, with kids, and I don't have the need for anything extra marital, be it on a social or physical level. Genuinely happy. Yet, this work relationships seems to be progressing into something that, at some point, can be seen as an affair. We're both fairly isolated in the company, and (mostly) only have each other to speak to about issues and pressure within the company. We also work on a lot of things together. This results in us spending a ton of time together, either working or talking about work. There's some social compatibility as well, but nothing mind blowing. We've developed an easy going relationship with a lot of shared experiences and vocabulary, resulting in an intimacy which, although I enjoy it, I'm not always comfortable with. Especially since he seems to be so easy going with casual touches and interactions. I've made a point of prioritizing my spouse and kids, and not to allow my work relationship to interfere. I've set up sanity checks for myself to assess if I'm over stepping in any way (through looks, touch, prioritizing, day dreaming, whatever), yet the relationship keeps on growing deeper, and I have no way of stopping it. Short of quitting. I don't want to quit. I love my job. I've worked towards a job like this for years. It took years of hard work, learning, networking, grinding and hustling to get to this point. I've invested so much in this job that it will take me years to build up to something like this again. The job allows me to live how I want to live, spend time with my family, provide for my family. Still, I know that I'll loose pretty much all of it if I step over the line. I won't be able to trust myself, I won't be able to ask my spouse to trust me. I just can not allow that to happen. So that's me, mostly OK, but stuck between a rock and a hard place.