6 ms·
> We work very closely with Google DeepMind to adapt Gemini models for Google-scale coding and other Software Engineering usecases. Considering how terrible an
by Twirrim 2y ago
> We work very closely with Google DeepMind to adapt Gemini models for Google-scale coding and other Software Engineering usecases.
Considering how terrible and frequently broken the code that the public facing Gemini produces, I'll have to be honest that that kind of scares me.
Gemini frequently fails at some fairly basic stuff, even in popular languages where it would have had a lot of source material to work from; where other public models (even free ones) sail through.
To give a fun, fairly recent example, here's a prime factorisation algorithm it produce for python:
# Find the prime factorization of n
prime_factors = []
while n > 1:
p = 2
while n % p == 0:
prime_factors.append(p)
n //= p
p += 1
prime_factors.append(n)
Can you spot all the problems?
- senko 2y agoWe collectively deride leetcoding interviews yet ask AI to flawlessly solve leetcode questions. I bet I'd make more errors on my first try at it.
- AnimalMuppet 2y agoWriting a prime-number factorization function is hardly "leetcode".
- atomic128 2y agoEmpirical testing (for example: https://news.ycombinator.com/item?id=33293522 https://news.ycombinator.com/item?id=33293522) has established that the people on Hacker News tend to be junior in their skills. Understanding this fact can help you understand why certain opinions and reactions are more likely here. Surprisingly, the more skilled individuals tend to be found on Reddit (same testing performed there).
- louthy 2y agoI’m not sure that’s evidence; I looked at that and saw it was written in Go and just didn’t bother. As someone with 40 years of coding experience and a fundamental dislike of Go, I didn’t feel the need to even try. So the numbers can easily be skewed, surely.
- atomic128 2y agoOnly individuals who submitted multiple bad solutions before giving up were counted as failing. If you look but don't bother, or submit a single bad solution, you aren't counted. Thousands of individuals were tested on Hacker News and Reddit, and surprisingly, it's not even close: Reddit is where the hackers are. I mean, at the time of the testing, years ago.
- louthy 2y agoThat doesn’t change my point. It didn’t test every dev on all platforms, it tested a subset. That subset may well have different attributes to the ones that didn’t engage. So, it says nothing about the audience for the forums as a whole, just the few thousand that engaged. Perhaps even, there could be fewer Go programmers here and some just took a stab at it even though they don’t know the language. So it could just select for which forum has the most Go programmers. Hardly rigourous. So I’d take that with a pinch of salt personally
- atomic128 2y agoAgreed. But remember, this isn't the only time the population has been tested. This is just the test (from two years ago, in 2022) that I happen to have a link to.
- louthy 2y agoThe population hasn’t been tested. A subset has.
- 59nadir 2y agoIt's also fine to be an outlier. I've been programming for 24 years and have been hanging out on HackerNews on and off for 11. HN was way more relevant to me 11 years ago than it is now, and I don't think that's necessarily only because the subject matter changed, but probably also because I have.
- Izikiel43 2y agoHow is that thing testing? Is it expecting a specific solution or actually running the code? I tried some solutions and it complained anyways
- atomic128 2y agoThe way the site works is explained in the first puzzle, "Hack This Site". TLDR, it builds and runs your code against a test suite. If your solutions weren't accepted, it's because they're wrong.
- 0xDEAFBEAD 2y agoWhere is the data?
- freilanzer 2y agoYeah, this is useless.
- senko 2y agoI didn't say it's hard, but it's most definitely leetcode, as in "pointless algorithmic exercise that will only show you if the candidate recently worked on a similar question". If that doesn't satisfy, here's a similar one at leetcode.com: https://leetcode.com/problems/distinct-prime-factors-of-product-of-array/ https://leetcode.com/problems/distinct-prime-factors-of-prod... I would not expect a programmer of any seniority to churn stuff like that and have it working without testing.
- AnimalMuppet 2y ago> "pointless algorithmic exercise that will only show you if the candidate recently worked on a similar question". I've been able to write one, not from memory but from first principles, any time in the last 40 years.
- senko 2y agoCurious, I would expect a programmer of your age to remember Knuth's "beware of the bugs in above code, I have only proven it's correct but haven't actually run it". I'm happy you know math, but my point before this thread got derailed was that we're holding (coding) AI to a higher standard than actual humans, namely to expect to write bug-free code.
- Jensson 2y agoHumans can actually run the code and knows what it should output. the LLM can't, and putting it in a loop against code output doesn't work well either since the LLM can't navigate that well.
- smrq 2y agoI hold myself to a higher standard than AI tools are capable of, from my experience. (Maybe some people don't, and that's where the disconnect is between the apologists and the naysayers?)
- 0points 2y ago> my point before this thread got derailed was that we're holding (coding) AI to a higher standard than actual humans, namely to expect to write bug-free code This seems like a very layman attitude and I would be surprised to find many devs adhering to this idea. Comments in this thread alone suggests that many devs on HN do not agree.
- gerash 2y agoI believe most people use AI to help them quickly figure out how to use a library or an API without having to read all their (often out dated) documentation instead of helping them solve some mathematical challenge
- taeric 2y agoIf the documentation is out of date, such that it doesn't help, this doesn't bode well for the training data of the AI helping it get it right, either?
- macintux 2y agoAI can presumably integrate all of the forum discussions talking about how people really use the code. Assuming discussions don't happen in Slack, or Discord, or...
- randomNumber7 2y agoAnd all the code on which it was trained...
- woodson 2y agoUnfortunately, it often hallucinates wrong parameters (or gets their order wrong) if there are multiple different APIs for similar packages. For example, there are plenty ML model inference packages, and the code suggestions for NVIDIA Triton Inference Server Python code are pretty much always wrong, as it generates code that’s probably correct for other Python ML inference packages with slightly different API.
- jon_richards 2y agoI often find the opposite. Documentation can be up to date, but AI suggests deprecated or removed functions because there’s more old code than new code. Pgx v5 is a particularly consistent example.
- Capricorn2481 2y agoForum posts can also be out of date.
- kgeist 2y agoThey probably use AI for writing tests, small internal tools/scripts, building generic frontends and quick prototypes/demos/proofs of concept. That could easily be that 25% of the code. And modern LLMs are pretty okayish with that.
- calf 2y agoWe are sorely lacking a "Make Computer Science a Science" movement, the tech lead's blurb is par for the course, talking about "SWE productivity" with no reference to scientific inquiry and a foundational understanding of safety, correctness, verification, validation of these new LLM technologies.
- almostgotcaught 2y agoDid you know that Google is a for-profit business and not a university? Did you know that most places where people work on software are the same?
- calf 2y agoDid you know that Software Engineering is a university level degree? That it is a field of scientific study, with professors who dedicate their lives to it? What happens when companies ignore science and worse yet cause harm like pollution or medical malpractice, or in this case, spread Silicon Valley lies and bullshit??? Did you know? How WEIRD. How about you not harass other commenters with such arrogantly ignorant sarcastic questions?? Or is that part of corporate "for-profit" culture too????
- almostgotcaught 2y ago> Did you know that Software Engineering is a university level degree? That it is a field of scientific study, with professors who dedicate their lives to it? So is marketing? So finance? So is petroleum engineering?
- zifpanachr23 2y agoSo are most medical facilities. Somehow, the vibes are massively different.
- almostgotcaught 2y agoThat's rich? Never heard of the opioid crisis? Or the over-prescription of imaging tests?
- justinpombrio 2y ago> Can you spot all the problems? You were probably being rhetorical, but there are two problems: - `p = 2` should be outside the loop - `prime_factors.append(n)` appends `1` onto the end of the list for no reason With those two changes I'm pretty sure it's correct.
- deleted 2y ago[deleted]
- dangsux 2y ago[dead]
- rmbyrro 2y ago`n` isn't defined
- justinpombrio 2y agoThe implicit context that the poster removed (as you can tell from the indentation) was a function definition: def factorize(n): ... return prime_factors
- kenjackson 2y agoYou don't need to append 'p' in the inner while loop more than once. Maybe instead of an array for keeping the list of prime factors do it in a set.
- zeroonetwothree 2y agoIt’s valid to return the multiplicity of each prime, depending on the goal of this.
- justinpombrio 2y agohttps://www.wolframalpha.com/input?i=prime+factors+of+12 https://www.wolframalpha.com/input?i=prime+factors+of+12
- ijidak 2y agoI'm the first to say that AI will not replace human coders. But I don't understand this attempt to tell companies/persons that are successfully using AI that no they really aren't. In my opinion, if they feel they're using AI successfully, the goal should be to learn from that. I don't understand this need to tell individuals who say they are successfully using AI that, "no you aren't." It feels like a form of denial. Like someone saying, "I refuse to accept that this could work for you, no matter what you say."
- deleted 2y ago[deleted]