3 ms·
"Say, for instance, that you could assemble a dataset of hundreds of thousands—even millions—of English language descriptions of the features of a software prod
by kromem 3y ago
"Say, for instance, that you could assemble a dataset of hundreds of thousands—even millions—of English language descriptions of the features of a software product, as written by a product manager, as well as the corresponding source code developed by a team of engineers to meet these requirements. Even with this data, you could not train a deep learning model to simply read a product description and generate the appropriate codebase."
I would love a centralized resource of all the predictions of what the technology would not be able to do in the future and then the dates at which those capabilities were first demonstrated on a giant timeline.
Maybe in the next few years I'll be able to ask a model to generate that for me. It will be pretty fun to look at.
- DeathArrow 3y agoYou still can not "could not train a deep learning model to simply read a product description and generate the appropriate codebase", unless the product is very, very, very simple or trivial.
- kromem 3y agoPretty sure the text there didn't have an exception added in. And the complexity threshold for the product where writing a description and getting working code is successful increases each year with no real reason to doubt that it will plateau, particularly as models hook into linters and static compilation in multiple rounds of correction over the next eighteen months. "It won't ever do this thing" and "ok, it kind of does it in simple cases and may do it in increasingly complex cases as time goes on" are two very different statements with a gulf between them.
- Tao3300 3y ago> "It won't ever do this thing" and "ok, it kind of does it in simple cases and may do it in increasingly complex cases as time goes on" are two very different statements with a gulf between them. Until it's perfect, the amount of trust you can have in the latter case makes it equivalent to the former. There's no gulf really — you can't count on what the code "may do".
- kromem 3y ago"It will never win a chess game" "Ok, it can win a chess game, but if we don't actually keep track of the moves it is making we'll have no idea if it can win a chess game, and therefore it's the same thing as if it can't win a chess game." You can always read code you didn't write, by the way.
- danjc 3y agoHe's not actually wrong yet though. You can now generate what appears to be a codebase but the chance of it functioning as specified as a whole is basically 0.
- kromem 3y agoThis is already over a year old at this point: https://andrewmayne.com/2022/03/17/building-games-and-apps-entirely-through-natural-language-using-openais-davinci-code-model/ https://andrewmayne.com/2022/03/17/building-games-and-apps-e...
- blharr 3y agoThey're great demos, but the tasks are not really at the scale of larger projects and codebases.
- kromem 3y agoAgain - is the copy quoted saying that paragraphs of simple tasks and demos will be able to be coded and only larger complex software can't be written by description? Because it really seems to be saying that no software at any level of complexity can be written from paragraph to working code by deep learning. We should be careful not to move goalposts post facto. And given the trend so far, it is likely that paragraphs of software with greater complexity will be able to be successfully coded as time goes on and models and infrastructure around those models improve. So let's also be careful not to assume the status quo is the end of the process to be evaluated rather than a stepping stone between the beginning and the theoretical end.