3 ms·

Gemini 3 is top of the leaderboard: https://andonlabs.com/evals/vending-bench-2 https://andonlabs.com/evals/vending-bench-2

• seizethecheese 11mo ago
  > Models are tasked with running a simulated vending machine business over a year and scored on their bank account balance at the end. The article being discussed here is about how AI couldn't run a real world vending machine. There was no issue in the components that would be in a standard simulation.
  • dinfinity 11mo ago
    To be fair, most vending machine operators do not allow suggestions from customers on what products to stock, let alone extensive ongoing and intentional adversarial psychological manipulation and deception. If it had just made stocking decisions autonomously and based changes in strategy on what products were bought most, it wouldn't have any of the issues reported.
  • xnx 11mo ago
    Right. Just interesting that this has become one of the benchmarks.
• UncleMeat 11mo ago
  "It works in the simulation" is the new "it works on my machine."