4 ms·
I am not convinced either of these are good test prompts for generic complexity tasks. Many solutions have already been included in the training data! You can
by fruitworks 9mo ago
I am not convinced either of these are good test prompts for generic complexity tasks. Many solutions have already been included in the training data!
You can trivially produce a web browser by copying and compiling the code for firefox, no transformer needed.
- Choco31415 9mo agoBut that would produce Firefox. The goal with these tests is to see if the models can make something new, not just copy an existing solution. That is the goal, at least.
- aix1 9mo agoBut how do you define, or indeed assess, novelty? It's not that difficult to take an existing mature codebase and morph it such that it looks quite different but is functionally unchanged. This is a very different task than building something that's not been built before.
- chihuahua 9mo agoObviously Microsoft felt bad when they had to kill the old Edge browser that was based on their own HTML rendering engine. Must feel like a second-rate tech company when you can't write some code to render HTML with sufficient quality. Now they can get back in the game with a 3-word prompt! And then every time there's some change to web standards, it's just one more prompt where you say "Hey Copilot, take a look at this page that describes the change, and update our browser code to add this!" /s
- baxtr 9mo agoCan still be a good capability test. Building a car is a real world equivalent. It’s highly complex and has been done billions of times. Still hard to pull off if you ask me.
- Retr0id 9mo agoFamously, one cannot download a car