5 ms·

Personally my theory is that Gemini benefits from being able to train on Googles massive internal code base and because Rust has been very low on uptake interna

Personally my theory is that Gemini benefits from being able to train on Googles massive internal code base and because Rust has been very low on uptake internally at Google, especially since they have some really nice C++ tooling, Gemini is comparatively bad at Rust.

dilap 1y ago
That's interesting. I've tried Gemini 2.5 Pro from time to time because of the rave reviews I've seen, on C# + Unity code, and I've always been disappointed (compared to ChatGPT o3 and o4-high-mini and even Grok). This would support that theory.
danielbln 1y ago
Interesting, Gemini must be a monster when it comes to Go code then. I gotta try it for that
- Unroasted6154 1y ago
  There is way more Java and C++ than Go at Google.
- jordanbeiber 1y ago
  As go feels like a straight-jacket compared to many other popular languages, it’s probably very suitable for an LLM in general. Thinking about it - was this not the idea of go from the start? Nothing fancy to keep non-rocket scientist away from foot-guns, and have everyone produce code that everyone else can understand. Diving in to a go project you almost always know what to expect, which is a great thing for a business.
- chewz 1y ago
  Reasonsbly small Go codebase works well almost with any LLM I had always designed very large projects as few medium sized independent Go tools and that strategy pays in times of AI assisted coding.
thimabi 1y ago
> Personally my theory is that Gemini benefits from being able to train on Googles massive internal code base But does Google actually train its models on its internal codebase? Considering that there’s always the risk of the models leaking proprietary information and security architecture details, I hardly believe they would run that risk.
- kridsdale3 1y ago
  Googler here. We have a second, isolated model that has trained on internal code. The public Gemini AFAIK has never seen that content. The lawyers would explode.
  - thimabi 1y ago
    Oh, you’re right, there are the legal issues as well. Just out of curiosity, do you see much difference in quality between the isolated model and the public-facing ones?
    - kridsdale3 1y ago
      We actually only got the “2.5” version of the internal one a few days ago so I don’t have an opinion yet. But when I had to choose between “2.0 with Google internal knowledge” and “2.5 that knows nothing” the latter was always superior. The bitter lesson indeed.
      - deleted 1y ago
        [deleted]
  - blurrybird 1y ago
    What model do your lawyers run on?
leoh 1y ago
>Personally my theory is that Gemini benefits from being able to train on Googles massive internal code base and because Rust has been very low on uptake internally at Google, especially since they have some really nice C++ tooling, Gemini is comparatively bad at Rust. Were they to train it on their C++ codebase, it would not be effective on account of the fact that they don't use boost or cmake or any major stuff that C++ in the wider world use. It would also suggest that the user make use of all kinds of non-available C++ libraries. So no, they are not training on their own C++ corpus nor would it be particularly useful.
- leoh 1y ago
  Excuse me why was this downvoted so aggressively??
  - simianwords 1y ago
    How can they train on internal codebase without leaking specifics?
    - leoh 1y ago
      They can’t, which is a good point. Also it would be basically useless for the reasons I mention.
data-ottawa 1y ago
Tangental, but I worry that LLMs will cause a great stagnation in programming language evolution, and possibly a bunch of tech. I've tried using a few new languages and the LLMs would all swap the code for syntactically similar languages, even after telling them to read the doc pages. Whether that's for better or worse I don't know, but it does feel like new languages are genuinely solving hard problems as their raison d'etre.
- breakingcups 1y ago
  Not just that, I think this will happen on multiple levels too. Think de-facto ossified libraries, tools, etc. LLMs thrive because they had a wealth of high-quality corpus in the form os Stack Overflow, Github, etc. and ironically their uptake is causing a strangulation of that source of training data.
- sillystu04 1y ago
  Perhaps the next big programming language will be designed specifically for LLM friendliness. Some things which are human friendly like long keywords are just a waste of tokens for LLMs, and there could be other optimisations too.

dilap 1y ago

That's interesting. I've tried Gemini 2.5 Pro from time to time because of the rave reviews I've seen, on C# + Unity code, and I've always been disappointed (compared to ChatGPT o3 and o4-high-mini and even Grok). This would support that theory.

danielbln 1y ago

Interesting, Gemini must be a monster when it comes to Go code then. I gotta try it for that

Unroasted6154 1y ago
There is way more Java and C++ than Go at Google.
jordanbeiber 1y ago
As go feels like a straight-jacket compared to many other popular languages, it’s probably very suitable for an LLM in general. Thinking about it - was this not the idea of go from the start? Nothing fancy to keep non-rocket scientist away from foot-guns, and have everyone produce code that everyone else can understand. Diving in to a go project you almost always know what to expect, which is a great thing for a business.
chewz 1y ago
Reasonsbly small Go codebase works well almost with any LLM I had always designed very large projects as few medium sized independent Go tools and that strategy pays in times of AI assisted coding.

thimabi 1y ago

> Personally my theory is that Gemini benefits from being able to train on Googles massive internal code base But does Google actually train its models on its internal codebase? Considering that there’s always the risk of the models leaking proprietary information and security architecture details, I hardly believe they would run that risk.

kridsdale3 1y ago
Googler here. We have a second, isolated model that has trained on internal code. The public Gemini AFAIK has never seen that content. The lawyers would explode.
- thimabi 1y ago
  Oh, you’re right, there are the legal issues as well. Just out of curiosity, do you see much difference in quality between the isolated model and the public-facing ones?
  - kridsdale3 1y ago
    We actually only got the “2.5” version of the internal one a few days ago so I don’t have an opinion yet. But when I had to choose between “2.0 with Google internal knowledge” and “2.5 that knows nothing” the latter was always superior. The bitter lesson indeed.
    - deleted 1y ago
      [deleted]
- blurrybird 1y ago
  What model do your lawyers run on?

leoh 1y ago

>Personally my theory is that Gemini benefits from being able to train on Googles massive internal code base and because Rust has been very low on uptake internally at Google, especially since they have some really nice C++ tooling, Gemini is comparatively bad at Rust. Were they to train it on their C++ codebase, it would not be effective on account of the fact that they don't use boost or cmake or any major stuff that C++ in the wider world use. It would also suggest that the user make use of all kinds of non-available C++ libraries. So no, they are not training on their own C++ corpus nor would it be particularly useful.

leoh 1y ago
Excuse me why was this downvoted so aggressively??
- simianwords 1y ago
  How can they train on internal codebase without leaking specifics?
  - leoh 1y ago
    They can’t, which is a good point. Also it would be basically useless for the reasons I mention.

data-ottawa 1y ago

Tangental, but I worry that LLMs will cause a great stagnation in programming language evolution, and possibly a bunch of tech. I've tried using a few new languages and the LLMs would all swap the code for syntactically similar languages, even after telling them to read the doc pages. Whether that's for better or worse I don't know, but it does feel like new languages are genuinely solving hard problems as their raison d'etre.

breakingcups 1y ago
Not just that, I think this will happen on multiple levels too. Think de-facto ossified libraries, tools, etc. LLMs thrive because they had a wealth of high-quality corpus in the form os Stack Overflow, Github, etc. and ironically their uptake is causing a strangulation of that source of training data.
sillystu04 1y ago
Perhaps the next big programming language will be designed specifically for LLM friendliness. Some things which are human friendly like long keywords are just a waste of tokens for LLMs, and there could be other optimisations too.