3 ms·

Cool, and marginally informative on the current state of things. but kind of a waste of energy given everything is re-done every minute to compare. We'd probab

by ugh123 11mo ago

Cool, and marginally informative on the current state of things. but kind of a waste of energy given everything is re-done every minute to compare. We'd probably only need a handful of each to see the meaningful differences.

whoisjuan 11mo ago
It's actually quite fascinating if you watch it for 5 minutes. Some models are overall bad, but others nail it in one minute and butcher it in the next. It's perhaps the best example I have seen of model drift driven by just small, seemingly unimportant changes to the prompt.
- alister 11mo ago
  > model drift driven by just small, seemingly unimportant changes to the prompt What changes to the prompt are you referring to? According the comment on the site, the prompt is the following: Create HTML/CSS of an analog clock showing ${time}. Include numbers (or numerals) if you wish, and have a CSS animated second hand. Make it responsive and use a white background. Return ONLY the HTML/CSS code with no markdown formatting. The prompt doesn't seem to change.
  - sambaumann 11mo ago
    presumably the time is replaced with the actual current time at each generation. I wonder if they are actually generated every minute or if all 6480 permutations (720 minutes in a day * 9 llms) were generated and just show on a schedule
  - whoisjuan 11mo ago
    The time given to the model. So the difference between two generations is just somethng trivially different like: "12:35" vs 12:36"
- moffkalast 11mo ago
  Kimi seems the only reliable one which is a bit surprising, and GPT 4o is consistently better than GPT 5 which on the other hand is unfortunately not surprising at all.
- nbaugh1 11mo ago
  It is really interesting to watch them for a while. QWEN keeps outputting some really abstract interpretations of a clock, KIMI is consistently very good, GPT5's results line up exactly with my experience with its code output (overly complex and never working correctly)
- bglusman 11mo ago
  We can't know how much is about the prompt though and how much is just stochastic randomness in the behavior of that model on that prompt, right? I mean, even given identical prompts, even at temp 0, models don't always behave identically.... at least, as far as I know? Some of the reasons why are I think still a research question, but I think its a fact nonetheless.
ascorbic 11mo ago
The energy usage is minuscule.
- jdiff 11mo ago
  It's wasteful. If someone built a clock out of 47 microservices that called out to 193 APIs to check the current time, location, time zone, and preferred display format we'd rightfully criticize it for similar reasons. In a world where Javascript and Electron are still getting (again, rightfully) skewered for inefficiency despite often exceeding the performance of many compiled languages, we should not dismiss the discussion around efficiency so easily.
  - Arisaka1 11mo ago
    What I find amusing with this argument is that, no one ever brought power savings when e.g. used "let me google that for you" instead of giving someone the answer to their question, because we saw the utility of teaching others how to Google. But apparently we can't see the utility of measuring the oversold competence of current AI models, given sufficiently large sampling size.
  - saulpw 11mo ago
    Let's do some math. 60x24x30 = 40k AI calls per month per model. Let's suppose there are 1000 output tokens (might it be 10k tokens? Seems like a lot for this task). So 40m tokens per model. The price for 1m output tokens[0] ranges from $.10 (qwen-2.5) to $60 (GPT-4). So $4/mo for the cheapest, and $2.5k/mo for the most expensive. So this might cost several thousand dollars a month? Something smells funny. But you're right, throttling it to once an hour would achieve a similar goal and likely cost less than $100/mo (which is still more than I would spend on a project like this). [0] https://pricepertoken.com/ https://pricepertoken.com/
    - qwe----3 11mo ago
      They use 4o (maybe a mini version?)(
  - 11mo ago
energy123 11mo ago
I sort of assumed they cached like 30 inferences and just repeat them, but maybe I'm being too cynical.

by ugh123 11mo ago

whoisjuan 11mo ago

It's actually quite fascinating if you watch it for 5 minutes. Some models are overall bad, but others nail it in one minute and butcher it in the next. It's perhaps the best example I have seen of model drift driven by just small, seemingly unimportant changes to the prompt.

alister 11mo ago
> model drift driven by just small, seemingly unimportant changes to the prompt What changes to the prompt are you referring to? According the comment on the site, the prompt is the following: Create HTML/CSS of an analog clock showing ${time}. Include numbers (or numerals) if you wish, and have a CSS animated second hand. Make it responsive and use a white background. Return ONLY the HTML/CSS code with no markdown formatting. The prompt doesn't seem to change.
- sambaumann 11mo ago
  presumably the time is replaced with the actual current time at each generation. I wonder if they are actually generated every minute or if all 6480 permutations (720 minutes in a day * 9 llms) were generated and just show on a schedule
- whoisjuan 11mo ago
  The time given to the model. So the difference between two generations is just somethng trivially different like: "12:35" vs 12:36"
moffkalast 11mo ago
Kimi seems the only reliable one which is a bit surprising, and GPT 4o is consistently better than GPT 5 which on the other hand is unfortunately not surprising at all.
nbaugh1 11mo ago
It is really interesting to watch them for a while. QWEN keeps outputting some really abstract interpretations of a clock, KIMI is consistently very good, GPT5's results line up exactly with my experience with its code output (overly complex and never working correctly)
bglusman 11mo ago
We can't know how much is about the prompt though and how much is just stochastic randomness in the behavior of that model on that prompt, right? I mean, even given identical prompts, even at temp 0, models don't always behave identically.... at least, as far as I know? Some of the reasons why are I think still a research question, but I think its a fact nonetheless.

ascorbic 11mo ago

The energy usage is minuscule.

jdiff 11mo ago
It's wasteful. If someone built a clock out of 47 microservices that called out to 193 APIs to check the current time, location, time zone, and preferred display format we'd rightfully criticize it for similar reasons. In a world where Javascript and Electron are still getting (again, rightfully) skewered for inefficiency despite often exceeding the performance of many compiled languages, we should not dismiss the discussion around efficiency so easily.
- Arisaka1 11mo ago
  What I find amusing with this argument is that, no one ever brought power savings when e.g. used "let me google that for you" instead of giving someone the answer to their question, because we saw the utility of teaching others how to Google. But apparently we can't see the utility of measuring the oversold competence of current AI models, given sufficiently large sampling size.
- saulpw 11mo ago
  Let's do some math. 60x24x30 = 40k AI calls per month per model. Let's suppose there are 1000 output tokens (might it be 10k tokens? Seems like a lot for this task). So 40m tokens per model. The price for 1m output tokens[0] ranges from $.10 (qwen-2.5) to $60 (GPT-4). So $4/mo for the cheapest, and $2.5k/mo for the most expensive. So this might cost several thousand dollars a month? Something smells funny. But you're right, throttling it to once an hour would achieve a similar goal and likely cost less than $100/mo (which is still more than I would spend on a project like this). [0] https://pricepertoken.com/ https://pricepertoken.com/
  - qwe----3 11mo ago
    They use 4o (maybe a mini version?)(
- 11mo ago

energy123 11mo ago

I sort of assumed they cached like 30 inferences and just repeat them, but maybe I'm being too cynical.