Unless your product can really benefit from a strong network effect in the open-source community without compromising your competitive edge, I wouldn't do it.

Unless your product can really benefit from a strong network effect in the open-source community without compromising your competitive edge, I wouldn't do it. However, if you feel the potential to gain traction outweighs the threat of losing an edge (or a paying customer, since they can just use your code), open-source the engine, but hold back on the parts that are unique to your offering and/or make your engine easy to work with (like a GUI, simplified wrappers or builds for different platforms), though this depends on the nature of your product.