4 ms·
"and timed how long it took them to write an HTTP server in JavaScript" OK ... interesting choice. This could mean almost anything. I can "write a web server"
by jeffbee 4y ago
"and timed how long it took them to write an HTTP server in JavaScript"
OK ... interesting choice. This could mean almost anything. I can "write a web server" in a python 1-liner in about 2 seconds (ctrl-r SimpleHttpServer, enter).
- mgkimsal 4y agoThere was a test suite to work against, so some standard notion of things that needed to be in place to be 'working'.
- idan 4y agoWe'll be sharing some more about how we conducted this in an upcoming post! But no, the task was actually to write a little engine to process a limited subset of HTTP requests and return a valid response or relevant error.
- verdverm 4y agoWould you pit it against my non-ai solution for the same problem? As a baseline to measure against? It would seem prudent to compare against other tools trying to alleviate the same problem, rather than just humans writing the code by hand https://github.com/hofstadter-io/demos/tree/main/full-stack-app https://github.com/hofstadter-io/demos/tree/main/full-stack-... I've since shortened that demo to be even faster for a user to complete
- blibble 4y agoit's a profoundly stupid task to measure to measure developer productivity too the hard bit of being a developer is deciding in precise, pedantic, utterly exact detail on what needs building to solve the business problem not bashing out the code in the editor now if they had given out a vague problem that ended up with the dev eventually producing a webserver in Javascript, then fine but then the difference copilot makes would be a much smaller percentage of the development effort I swear we're going back to the age of "lines of code written per hour", just now with arTiFiCiAL InTellIgEncE
- mgkimsal 4y ago> the hard bit of being a developer is deciding in precise, pedantic, utterly exact detail on what needs building to solve the business problem The process seemed to involve have a test suite to write against. So, essentially, you had some/most of your "business problem" defined by acceptance tests (produce these outputs given these inputs). Some of that process may normally stretch out for days or weeks during meetings - condensed down, provided for you ahead of time takes some of that variability out. I was possibly a bit more surprised that less than 80% finished. Did they not commit code? Did they submit code but it didn't pass? Did they take too long? Was the 'did not finish' an explicit call on the dev side ("I'm done") or did they just stop taking input after X days?
- mcluck 4y agoBased on this task, I could write a snippet that would beat Copilot every single time.