7 ms·

I've contemplated this a bit, and I think I have a bit of an unconventional take: First, this is really impressive. Second, with that out of the way, these mo

• paladin314159 1y ago
  > I think that, if you gave me the ability to search the pre-contest Internet and a week to prepare my submissions, I would be kind of embarrassed if I didn't get gold, and I'd find the contest to be rather less interesting than I would find the real thing. I don't know what your personal experience with competitive programming is, so your statement may be true for yourself, but I can confidently state that this is not true for the VAST majority of programmers and software engineers. Much like trying to do IMO problems without tons of training/practice, the mid-to-hard problems in the ICPC are completely unapproachable to the average computer science student (who already has a better chance than the average software engineer) in the course of a week. In the same way that LLMs have memorized tons of stuff, the top competitors capable of achieving a gold medal at the ICPC know algorithms, data structures, and how to pattern match them to problems to an extreme degree.
  • amluto 1y ago
    > I can confidently state that this is not true for the VAST majority of programmers and software engineers. That may well be true. I think it's even more true in cases where the user is not a programmer by profession. I once watched someone present their graduate-level research in a different field and explain how they had solved a real-world problem in their field by writing a complicated computer program full of complicated heuristics to get it to run fast enough and thinking "hmm, I'm pretty sure that a standard algorithm from computer graphics could be adapted to directly solve your problem in O(n log n) time". If users can get usable algorithms that approximately match the state of the art out of a chatbot (or a fancy "agent") without needing to know the magic words, then that would be amazing, regardless of whether those chatbots/agents ever become creative enough to actually advance the state of the art. (I sometimes dream of an AI producing a piece of actual code that comes even close to state of the art for solving mixed-integer optimization problems. That's a whole field of wonderful computer science / math that is mostly usable via a couple of extraordinarily expensive closed-source offerings.)
    • avcxz 1y ago
      > That's a whole field of wonderful computer science / math that is mostly usable via a couple of extraordinarily expensive closed-source offerings. Take a look at Google OR-Tools: https://developers.google.com/optimization/ https://developers.google.com/optimization/
      • amluto 1y ago
        OR-Tools is a whole grab-bag of tools, most of which are wrappers around various solvers, including Gurobi and CPLEX. It seems like CP-SAT is under the OR-Tools umbrella, and CP-SAT may well be state-of-the-art for the specific sets of problems that it's well-suited for.
  • bubblyworld 1y ago
    Yeah, absolutely, I spent months preparing for the ICPC with my team and ended up scoring a paltry 3/12. A week would likely not have helped us at all - we simply had no idea how to approach the rest! Top teams are on another level. Compute and such is a fair point but that AI is here at all is mind-blowing to me.
• modeless 1y ago
  It doesn't matter how many instances were running. All that matters is the wall clock time and the cost. The fact that they don't disclose the cost is a clue that it's probably outrageous today. But costs are coming down fast. And hiring a team of these guys isn't exactly cheap either.
  • zeroonetwothree 1y ago
    Human teams are limited to three people. So why doesn’t it matter how many instances they used?
    • modeless 1y ago
      Human brains and cloud instances are not remotely equivalent. What you can compare on an equivalent basis is cost.
      • LunaSea 1y ago
        Or energy usage.
        • modeless 1y ago
          Cost is inclusive of energy and a lot more. It's a better metric.
    • ben_w 1y ago
      All instances of any given model are kinda the same, for lack of a better word, "person": same knowledge, same skills, same failings.
    • warkdarrior 1y ago
      I bet with human teams it'll take longer to solve a problem the more people you have on the team.
    • kenjackson 1y ago
      This is what the argument is? 10 years ago if you said you could do this with every computer on the planet and every computer scientist focused on trying to create the code to do this I would’ve given you absurd odds against it getting 12 problems right on ICPC. 10 years ago it couldn’t even reliably parse the question statement.
• dist-epoch 1y ago
  If you want to play that game, let's compute how much energy was spent to grow, house and educate one team since they were born, over 20 years against how much was spent training the model.
  • somewholeother 1y ago
    This is a fair analogy, but let's also consider that these human beings weren't designed with the express purpose of becoming experts in their field and performing in this way for this specific purpose (albeit in a generalist manner). We are most definitely in agreement about the folly of comparing the abilities of LLMs to humans, since LLMs are to a greater extent the product of much collective human endeavour. "Living memories" would perhaps be a better description of their current state, and their resultant impact on the human psyche.
  • vrighter 1y ago
    a human being uses about 1 gWh in a lifetime on average. Training one model nowadays uses more than that. So the model also cost more to train than to raise a team member (possibly all three of them)
    • vrighter 1y ago
      whoops, got the numbers wrong. 1gWh is how much electricity a person uses, not how much energy the body consumes. That works out to about 72 years365 days2000 kcal = approx 61 gWh
• _diyar 1y ago
  I think your assessment is spot on. But I also think there's a bigger picture that's getting lost in the sauce, not just in your comment but in the general discourse around AI progress: - We're currently unlocking capabilities to solve many tasks which could previously only be solved by the top-1% of the experts in the field. - Almost all of that progress is coming from large scale deep learning. Turns out transformers with autoregression + RL are mighty generalists (tho yet far from AGI). Once it becomes cheap enough so the average joe can tinker with models of this scale, every engineering field can apply it to their niche interest. And ultimately nobody cares if you're playing by the same rules as humans outside of these competitions, they only care that you make them wealthy, healthy and comfy.
• theragra 1y ago
  I think your analogy is lacking. Human brain is much more efficient, so it is not right to say "giving a human team a week instead of five hours". Most likely, the whole OpenAI compute cannot match one brain in terms of connections and relations and computation power.
  • stevenhuang 1y ago
    As always with these comparisons you neglect to account for the eons necessary for evolution to create human brains.
    • GoatInGrey 1y ago
      But as a product of evolved organisms, LLMs are also a product of evolution. They also came several hundreds of thousands of years later.
  • vrighter 1y ago
    so in that case, don't compare them to humans in the first place
• asboans 1y ago
  Firstly, automobiles are really impressive. Second, with that out the way, these cars are not playing the same game as horses… first, and quite obviously they have massive amounts of horsepower, which is kind of like giving a team of horses… many more horses. But also cars have an absolutely massive fuel capacity. Petrol is such an efficient store of chemical energy compared to hay and cars can store gallons of it. I think if you give my horse the ability of 300 horses and fed it pure gasoline, I would be kind of embarrassed if it wasn’t able to win a horse race.
  • huflungdung 1y ago
    [dead]
  • LaffertyDev 1y ago
    I don’t think you’ll find many race tracks that permit horses and cars to compete together. (I did enjoy the sarcasm, though!)
  • lbrandy 1y ago
    I was struck how the argument is also isomorphic to how we talked about computers and chess. We're at the stage where we are arguing the computer isn't _really_ understanding chess, though. It's just doing huge amounts of dumb computation with huge amounts of opening book and end tables and no real understanding, strategy or sense of whats going on. Even though all the criticism were, in a sense, valid, in the end none of it amounted to a serious challenge to getting good at the task at hand.
  • Swizec 1y ago
    > Firstly, automobiles are really impressive. Second, with that out the way, these cars are not playing the same game as horses Yes. That’s why cars don’t compete in equestrian events and horses don’t go to F1 races. This non-controversial surely? You want different events for humans, humans + computers, and just computers. Notice that self driving cars have separate race events from both horses and human-driven cars.
    • gxs 1y ago
• th0ma5 1y ago
  Yes yes given this why didn't it do better and isn't it embarrassing to have done it through statistical brute force and not intelligence.
• OtherShrezzing 1y ago
  The human teams also get limited to one computer shared between 3 people. The models have access to an effectively unbounded number of computers. My argument does feel a bit like the “Watson doesn’t need to physically push the button” equivalents from when that system beat Jeopardy for the first time. I assume 5 hours on a single high-end Mac would probably still be enough compute in the near future.
  • amluto 1y ago
    I found the Watson match to be rather absurd. It would have been much more interesting if the rules had been modified so that all contestants had, say, two seconds two press the buzzer and that the contestant who got to answer first would be chosen by random selection among those who pressed the button. This would at least have made the competition be about who could come up with the most correct answers (questions).
  • Darmani 1y ago
    The people saying that were wrong, BTW -- Watson did have to physically press a button. https://www.wired.com/2011/02/ibm-watson-speed/ https://www.wired.com/2011/02/ibm-watson-speed/ > __Brown: __ Watson has a mechanical button-presser. It uses the same signaling device [the button] that the human competitors use in the game. Once Watson has decided that it wants to ring in because it has found an answer with a high-enough confidence, and it receives the signal that the buzzers are open and you can ring in, it then has to trigger the mechanical button presser and mechanically press the button.
    • OtherShrezzing 1y ago
      Watson originally just sent an electronic signal. The physical button pushing was introduced later, to stave off the criticism that it had an unfair advantage.
• roadside_picnic 1y ago
  > whereas the teams are allowed to bring a 25-page PDF This is where I see the biggest issue. LLMs are first-and-foremost text compression algorithms. They have a compressed version of a very good chunk of human writing. After being text compression engines, LLMs are really good at interpolating text based on the generalization induced by the lossy compression. What this result really tells us is that, given a reasonably well compressed corpus of human knowledge, the ICPC can be view as an interpolation task.
  • Rudybega 1y ago
    If we develop a system that can: - compress (in a relatively recoverable way) the entire domain of human knowledge - interpolate across the entire domain of human knowledge - draw connections or conclusions that haven't previously been stated explicitly - verify or disprove those conclusions or connections - update its internal model based on that (further expanding the domain it can interpolate within) Then I think we're cooking with gasoline. I guess the question becomes whether those new conclusions or connections result in a convergent or divergent increase in the number of new conclusions and connections the model can draw (e.g. do we understand better the domains we already know or does updating the model with these new conclusions/connections allow us to expand the scope of knowledge we understand to new domains).
  • andoando 1y ago
    You can use the same framing for human reasoning except its over visual/auditority/spatial data and not just text. You don't remember every detail of what you've seen correct? You store some lossy compression like "I went to a park"
• tdb7893 1y ago
  As someone who has been to the ICPC finals around a decade ago I agree that the limited time is really the big problem that these machine learning models don't really experience in the same way. Though that being said these problems are hard, the actual coding of the algorithms is pretty easy (most of the questions use one of a handful of algorithms that you've implemented a hundred times by the time you're in the finals) but recognizing which one will actually solve the problem correctly is not obvious at all. I know a lot of people that struggled in their undergrad algorithms class and I think a lot of those people given the ICPC finals problems would struggle even with being able to research.
• dddgghhbbfblk 1y ago
  I think that's because the framing around this (and similar stories about eg IMO performances) is imo slightly wrong. It's not interesting that they can get a gold medal in the sense of trying to rank them against human competitors. As you say, the direct comparisons are, while not entirely meaningless, at least very hard to interpret in the best of cases. It's very much an apples to oranges situation. Rather, the impressive thing is simply that an AI is capable of solving these problems at all. These are novel (ie not in training set) problems that are really hard and beyond the ability of most professional programmers. The "gold medal" part is informative more in the sense that it gives an indication of how many problems the AI was able to solve & how well it was able to do them. When talking with some friends about chatgpt just a couple years ago I remember being very confident that there was no way this technology would be able to solve this kind of novel, very challenging reasoning problem, and that there was no way it would be able to solve IMO problems. It's remarkable how quickly I've been proven wrong.
  • ragequittah 1y ago
    It feels like half of the people I see talk about AI are still under the impression it's a spicy autocomplete. If you use a SOTA model for a week and still feel this way your bias must be very strong.
    • dddgghhbbfblk 1y ago
      I must be missing something because I don't understand how this is related to my comment.
• m3kw9 1y ago
  the end game is that running similar tasks at any moment time and place.
• theywillnvrknw 1y ago
  The more important question I'd like answered is did the models have the same number of attempts (submissions) as human contestants. If not were penalities applied for multiple solutions.