4 ms·

> Purportedly working strategies found by trial and error and discussed in a language that is only intelligible to the in-group This really does sound like Com

by dysoco 1y ago

> Purportedly working strategies found by trial and error and discussed in a language that is only intelligible to the in-group This really does sound like Computer Science since it's very beginnings. The only difference is that now it's a much more popular field, and not restricted to a few nerds sharing tips over e-mail or bbs.

dawnofdusk 1y ago
>This really does sound like Computer Science since it's very beginnings. Except in actual computer science you can prove that your strategies, discovered by trial and error, are actually good. Even though Dijkstra invented his eponymous algorithm by writing on a napkin, it's phrased in the language of mathematics and one can analyze quantitatively its effectiveness and trade-offs, and one can prove if it's optimal (as was done recently).
- pbhjpbhj 1y ago
  Surely claims about context engineering can also be tested using scientific methodology?
  - fleischhauf 1y ago
    except the area is so hugely popular with people who unfortunately lack the rigor or curiosity to ask for this and blindly believe claims. for example this hugely popular repository https://github.com/x1xhlol/system-prompts-and-models-of-ai-tools https://github.com/x1xhlol/system-prompts-and-models-of-ai-t... where the authors fail to explain how the prompts are obtained and how they prove that they are valid and not a hallucination.
  - slightwinder 1y ago
    Yes, in theory. But it's testing against highly complex, ever-changing systems, where small changes can have big impact on the outcome. So it's more akin to "weak" science like psychology. And weak here means that most finding have a weak standing, because of each variable having little individual contribution in the complex setup researched, making it harder to reproduce results. Even more problematic is that too many "researchers" are just laymen, lacking a proper scientific background, and they are often just playing around with third-party-services, while delivering too much noise to the community. So in general, AI has also something like the replication crisis in its own way. But on the other side, the latest wave of AI is just some years old (3 years now?), which is not much in real scientific progress-rates.
  - parpfish 1y ago
    yeah, but it's a different type of science. the move from "software engineering" to "AI engineering" is basically a switch from a hard science to a soft science. rather than being chemists and physicists making very precise theory-driven predictions that are verified by experiment, we're sociologists and psychologists randomly changing variables and then doing a t-test afterward and asking "did that change anything?"
    - bwfan123 1y ago
      the difference is between having a "model" and a "theory". A theory tries to explain the "why" based on some givens, and a model tell you the "how". For engineering, we want why and not how. ie, for bugs, we want to root-cause, and fix - not fix by trial-and-error. the hard sciences have theories. and soft sciences have models. computer science is built on theory (turing machine/lambda calc/logic). AI models are well "models" - we dont know why it works but it seems to - thats how models are.
- BoiledCabbage 1y ago
  > >This really does sound like Computer Science since it's very beginnings. > Except in actual computer science you can prove that your strategies, discovered by trial and error, are actually good. Maybe it's true for computer science - but most people on here aren't doing computer science. They're doing software engineering. And it sure as heck isn't true for software engineering. If it were, I wouldn't be hearing arguments about programming languages for years, or static vs dynamic typing, or functional vs OOP... So what you're arguing about AI isn't exactly anything new to software development.
bigfishrunning 1y ago
> This really does sound like Computer Science since it's very beginnings. Dijkstra is rolling in his grave. Computer Science was a rigorous sub-field of mathematics before the tech bros showed up and started moving fast and breaking things. The endless supply of VC money has destroyed this field.

by dysoco 1y ago

dawnofdusk 1y ago

>This really does sound like Computer Science since it's very beginnings. Except in actual computer science you can prove that your strategies, discovered by trial and error, are actually good. Even though Dijkstra invented his eponymous algorithm by writing on a napkin, it's phrased in the language of mathematics and one can analyze quantitatively its effectiveness and trade-offs, and one can prove if it's optimal (as was done recently).

pbhjpbhj 1y ago
Surely claims about context engineering can also be tested using scientific methodology?
- fleischhauf 1y ago
  except the area is so hugely popular with people who unfortunately lack the rigor or curiosity to ask for this and blindly believe claims. for example this hugely popular repository https://github.com/x1xhlol/system-prompts-and-models-of-ai-tools https://github.com/x1xhlol/system-prompts-and-models-of-ai-t... where the authors fail to explain how the prompts are obtained and how they prove that they are valid and not a hallucination.
- slightwinder 1y ago
  Yes, in theory. But it's testing against highly complex, ever-changing systems, where small changes can have big impact on the outcome. So it's more akin to "weak" science like psychology. And weak here means that most finding have a weak standing, because of each variable having little individual contribution in the complex setup researched, making it harder to reproduce results. Even more problematic is that too many "researchers" are just laymen, lacking a proper scientific background, and they are often just playing around with third-party-services, while delivering too much noise to the community. So in general, AI has also something like the replication crisis in its own way. But on the other side, the latest wave of AI is just some years old (3 years now?), which is not much in real scientific progress-rates.
- parpfish 1y ago
  yeah, but it's a different type of science. the move from "software engineering" to "AI engineering" is basically a switch from a hard science to a soft science. rather than being chemists and physicists making very precise theory-driven predictions that are verified by experiment, we're sociologists and psychologists randomly changing variables and then doing a t-test afterward and asking "did that change anything?"
  - bwfan123 1y ago
    the difference is between having a "model" and a "theory". A theory tries to explain the "why" based on some givens, and a model tell you the "how". For engineering, we want why and not how. ie, for bugs, we want to root-cause, and fix - not fix by trial-and-error. the hard sciences have theories. and soft sciences have models. computer science is built on theory (turing machine/lambda calc/logic). AI models are well "models" - we dont know why it works but it seems to - thats how models are.
BoiledCabbage 1y ago
> >This really does sound like Computer Science since it's very beginnings. > Except in actual computer science you can prove that your strategies, discovered by trial and error, are actually good. Maybe it's true for computer science - but most people on here aren't doing computer science. They're doing software engineering. And it sure as heck isn't true for software engineering. If it were, I wouldn't be hearing arguments about programming languages for years, or static vs dynamic typing, or functional vs OOP... So what you're arguing about AI isn't exactly anything new to software development.

bigfishrunning 1y ago

> This really does sound like Computer Science since it's very beginnings. Dijkstra is rolling in his grave. Computer Science was a rigorous sub-field of mathematics before the tech bros showed up and started moving fast and breaking things. The endless supply of VC money has destroyed this field.