3 ms·

I'm not well versed in this but IIUC the less context/data that exists, the less it should afford the creation of a succinct response, right? This would make se

I'm not well versed in this but IIUC the less context/data that exists, the less it should afford the creation of a succinct response, right? This would make sense for the gap between "frontend js api" responses, and "aerospace thesis" responses. Would it be safe to assume the limiting factor with this or any algo of this type might always be availability of context/data, and these niche questions might always be an edge case? Are there alglrithms that can reliably extrapolate "new" data/context that does not exist yet and is accurate IRL? Sorry if my terminology or understanding is way off, I don't know much about this field.

mk_stjames 4y ago
No I think that is pretty much it- it is deep learning after all, the scope of the training data limits the model. It's so good with code because not only is there a TON of code in the data set, there is a TON of data of tutorials and examples and documentation EXPLAINING the code! I've long been amazed at the computer programming world at the amount of content programmers write about... programming. When it comes to other engineering fields, people don't write nearly as much on the internet (and, ofter, are not allowed to write as much, publicly) about the topics. So of course these large language models are simply not going to be able to produce super detailed analysis of those fields. There is something a bit more needed - Meta did a deep model recently with training specifically more on scientific papers, and it is able to 'talk the talk' better but unfortunately the 'understanding' just isn't there. I think at the end of the day, programming languages are languages after all, thus work well with these tools, but there is something harder about scientific reasoning and interdisciplinary planning that is going to take something a bit more. This is the main goal of various research into bring symbolic methods back into interface with deep learning models.

mk_stjames 4y ago

No I think that is pretty much it- it is deep learning after all, the scope of the training data limits the model. It's so good with code because not only is there a TON of code in the data set, there is a TON of data of tutorials and examples and documentation EXPLAINING the code! I've long been amazed at the computer programming world at the amount of content programmers write about... programming. When it comes to other engineering fields, people don't write nearly as much on the internet (and, ofter, are not allowed to write as much, publicly) about the topics. So of course these large language models are simply not going to be able to produce super detailed analysis of those fields. There is something a bit more needed - Meta did a deep model recently with training specifically more on scientific papers, and it is able to 'talk the talk' better but unfortunately the 'understanding' just isn't there. I think at the end of the day, programming languages are languages after all, thus work well with these tools, but there is something harder about scientific reasoning and interdisciplinary planning that is going to take something a bit more. This is the main goal of various research into bring symbolic methods back into interface with deep learning models.