I have Comcast Gigabit Pro which comes with a 6 Gbps symmetrical fiber connection and a separate 1 Gbps symmetrical ethernet connection, each connection having

I have Comcast Gigabit Pro which comes with a 6 Gbps symmetrical fiber connection and a separate 1 Gbps symmetrical ethernet connection, each connection having their own block of 5 IP addresses. I've been considering moving my colocated server back home and putting it on the 1 Gbps line, my only hesitation is that the IP addresses between the two connections are so similar. If I was only serving https traffic to the public that wouldn't be too much of a concern, since I could just stick Cloudflare in front of it. However, I'm also hosting game servers on there, and those don't seem as straightforward in masking my IP address. Should I be this paranoid about that? They would be completely separate networks, there's no route from the publicly accessible server into my home network.