7 ms·

I know this is a little controversial but the lack of performance on SWE-bench is hugely disappointing I think economically. These models don’t have any viable

by catigula 11mo ago

I know this is a little controversial but the lack of performance on SWE-bench is hugely disappointing I think economically. These models don’t have any viable path to profitability if they can’t take engineering jobs.

martinald 11mo ago
I thought that but it does do a lot better on other benchmarks. Perhaps SWE bench just doesn't capture a lot of the improvement? If the web design improvements people have been posting on twitter, I suspect this will be a huge boon for developers. SWE benchmark is really testing bugfixing/feature dev more. Anyway let's see. I'm still hyped!
- catigula 11mo ago
  That would be great! But AI is a bubble if these models can’t do serious engineering work.
- rfoo 11mo ago
  SWE Bench doesn't even test bugfixing / feature dev properly after you achieve roughly 70% if you don't benchmaxx it .
- camdenreslink 11mo ago
  It seems the benchmarks that had a big jump had to do with visual capabilities. I wonder how that will translate to improvements to the workloads LLMs are currently used for (or maybe it will introduce new workloads).
api 11mo ago
Really? If they can make an engineer more productive, that's worth a lot. Naive napkin math: 1.5X productivity on one $200k/year engineer is worth $100k/year.
- mikert89 11mo ago
  People generally dont understand what these models are doing to engineering salaries. The skill level required to produce working software is going way down
Workaccount2 11mo ago
People here, and in tech in general, are so lost in the sauce. According to at least OpenAI, who probably produces the most tokens (if we don't count google AI overviews and other unrequested AI bolt-ons) out of all the labs, programming tokens account for ~4% of total generations. That's nothing. The returns will come from everyone and their grandma paying $30-100/mo to use the services, just like everyone pays for a cell phone and electricity. Don't be fooled, we are still in the "Open hands" start-up business phase of LLMs. The "enshitification" will follow.

martinald 11mo ago

I thought that but it does do a lot better on other benchmarks. Perhaps SWE bench just doesn't capture a lot of the improvement? If the web design improvements people have been posting on twitter, I suspect this will be a huge boon for developers. SWE benchmark is really testing bugfixing/feature dev more. Anyway let's see. I'm still hyped!

catigula 11mo ago
That would be great! But AI is a bubble if these models can’t do serious engineering work.
rfoo 11mo ago
SWE Bench doesn't even test bugfixing / feature dev properly after you achieve roughly 70% if you don't benchmaxx it .
camdenreslink 11mo ago
It seems the benchmarks that had a big jump had to do with visual capabilities. I wonder how that will translate to improvements to the workloads LLMs are currently used for (or maybe it will introduce new workloads).

api 11mo ago

Really? If they can make an engineer more productive, that's worth a lot. Naive napkin math: 1.5X productivity on one $200k/year engineer is worth $100k/year.

mikert89 11mo ago
People generally dont understand what these models are doing to engineering salaries. The skill level required to produce working software is going way down

Workaccount2 11mo ago

People here, and in tech in general, are so lost in the sauce. According to at least OpenAI, who probably produces the most tokens (if we don't count google AI overviews and other unrequested AI bolt-ons) out of all the labs, programming tokens account for ~4% of total generations. That's nothing. The returns will come from everyone and their grandma paying $30-100/mo to use the services, just like everyone pays for a cell phone and electricity. Don't be fooled, we are still in the "Open hands" start-up business phase of LLMs. The "enshitification" will follow.