Well they could patent it. This is literally the purpose of patents. You can't patent it if you don't publish, and if you do patent it, you have every incentive

Well they could patent it. This is literally the purpose of patents. You can't patent it if you don't publish, and if you do patent it, you have every incentive to publish it. To get other people to use it and pay you licensing fees, or just to brag and attract better scientists to work at your company. I do hate patents, and I think they are often given to ideas that seems simple, trivial, or iterative, rather than ground breaking. Stuff that probably would have been invented by someone else a few years, or even months, later. But for your hypothetical "revolutionary AI breakthrough", that's the ideal use case for them.