36 ms·

Which fields have they completely transformed? How was it before and how is it now? I won't pretend like it hasn't impacted my field, but I would say the impact

Which fields have they completely transformed? How was it before and how is it now? I won't pretend like it hasn't impacted my field, but I would say the impact is almost entirely negative.

Profan 1y ago
hah well, transformative doesn't necessarily mean positive!
- econ 1y ago
  All we get is distraction.
dekhn 1y ago
Genomics, protein structure prediction, various forms of small molecule and large molecule drug discovery.
- thesz 1y ago
  No neural protein structure prediction papers I read have compared transformers to SAT solvers. As if this approach [1] does not exist. [1] https://pmc.ncbi.nlm.nih.gov/articles/PMC7197060/ https://pmc.ncbi.nlm.nih.gov/articles/PMC7197060/
  - dekhn 1y ago
    What exactly are you suggesting- that the SAT solver example given in the paper (applied the HP model of protein structure), or an improvement on it, could produce protein structure prediction at the level of AlphaFold? This seems extremely, extremely unlikely for many reasons. The HP model is a simplification of true protein folding/structure adoption, while AlphaFold (and the open source equivalents) works with real proteins. The SAT approach uses little to no prior knowledge about protein structures, unlike AlphaFold (which has basically memorized and generalized the PDB). To express all the necessary details would likely exceed the capabilities of the best SAT solvers. (don't get me wrong- SAT and other constraint approaches are powerful tools. But I do not think they are the best approach for protein structure prediction).
    - YeGoblynQueenne 1y ago
      (Not the OP) If we've learned one thing from the ascent of neural nets is that you have no idea whether something works or not until you have tried it. And by "tried it" I mean really, really gave it a go as hard as possible, with all resources you can muster. The industry has thrown everything it's got on Transformers, but there are many other approaches that have at least as promising empirical results and much better theoretical support and have not been pursued with the same fervor, so we have no idea how well or bad they'd do against neural nets, if they were given the same treatment. Like the OP says, it's as if such approaches don't even exist.
      - chermi 1y ago
        Do you understand the relevant fundamental difference between SAT and neural net approaches? One is a machine learning approach, the other is not. We know the computational complexity of SAT solvers, they're fixed algorithms. SAT doesn't learn with more data. It has performance limits and that's the end of the story. BTW, as I mentioned in my other comment, people have been trying SAT solvers in the CASP competition for decades. They got blown away by transformer approach. Such approaches exist, and they've been found wanting, and no amount of compute is going to improve their performance limits, because it isn't an ML approach with scaling laws. This is definitely not some unfair conspiracy against SAT, and probably not against the majority of pre-transformer approaches. I am sympathetic to the concern that transformer based research is getting too much attention at the expense of other approaches. However, I'd think the success of transformer makes it more likely than ever that proven-promising alternative approaches would get funding as investors try to beat everyone to the next big thing. See quantum computing funding or funding for way out there ASIC startups. TL;DR I don't know what is meant by the "same treatment" for SAT solvers. Funding is finite and goes toward promising approaches. If there "at least as promising" approaches, go show clear evidence of that to a VC and I promise you'll get funding.
  - chermi 1y ago
    Hey man, CASP is an open playing field. If it was better, they would've showed it by now.
    - YeGoblynQueenne 1y ago
      Said somebody about neural nets in the 1980's.
      - chermi 1y ago
        I don't really understand what point you or parent are trying to make. SAT approaches have been used in CASP, an open competition for protein structure prediction. They have been trying for decades with SAT. The transformer based models blew every approach out of the water to the point of approaching experimental resolution. Why am supposed to pretend SAT is being treated unfairly or whatever you guys are expounding? Based on your response and the parent's, don't think you'd be happy if SAT approaches WERE cited. Maybe you and parent think every preexisting approach hasn't been proven to be inferior to the transformer approach until some equivalent amount of compute has been thrown at them compared to the transformer approach? That's the best I can come up with. There is no room for 'scaling' gains with SAT solvers that will be found with more compute, it's not an ML approach. That is, it doesn't learn with more data. If you mean something else more specific I'd be interested to know.
jimmyl02 1y ago
in the super public consumer space, search engines / answer engines (like chatgpt) are the big ones. on the other hand it's also led to improvements in many places hidden behind the scenes. for example, vision transformers are much more powerful and scalable than many of the other computer vision models which has probably led to new capabilities. in general, transformers aren't just "generate text" but it's a new foundational model architecture which enables a leap step in many things which require modeling!
- ACCount37 1y ago
  Transformers also make for a damn good base to graft just about any other architecture onto. Like, vision transformers? They seem to work best when they still have a CNN backbone, but the "transformer" component is very good at focusing on relevant information, and doing different things depending on what you want to be done with those images. And if you bolt that hybrid vision transformer to an even larger language-oriented transformer? That also imbues it with basic problem-solving, world knowledge and commonsense reasoning capabilities - which, in things like advanced OCR systems, are very welcome.
CamperBob2 1y ago
Which fields have they completely transformed? Simultaneously discovering and leveraging the functional nature of language seems like kind of a big deal.
- jimbo808 1y ago
  Can you explain what this means?
  - CamperBob2 1y ago
    Given that we can train a transformer model by shoveling large amounts of inert text at it, and then use it to compose original works and solve original problems with the addition of nothing more than generic computing power, we can conclude that there's nothing special about what the human brain does. All that remains is to come up with a way to integrate short-term experience into long-term memory, and we can call the job of emulating our brains done, at least in principle. Everything after that will amount to detail work.
    - jimbo808 1y ago
      > we can conclude that there's nothing special about what the human brain does ...lol. Yikes. I do not accept your premise. At all. > use it to compose original works and solve original problems Which original works and original problems have LLMs solved, exactly? You might find a random article or stealth marketing paper that claims to have solved some novel problem, but if what you're saying were actually true, we'd be flooded with original works and new problems being solved. So where are all these original works? > All that remains is to come up with a way to integrate short-term experience into long-term memory, and we can call the job of emulating our brains done, at least in principle What experience do you have that caused you to believe these things?
      - CamperBob2 1y ago
        Which is fine, but it's now clear where the burden of proof lies, and IMHO we have transformer-based language models to thank for that. If anyone still insists on hidden magical components ranging from immortal souls to Penrose's quantum woo, well... let's see what you've got.
isoprophlex 1y ago
Everyone who did NLP research or product discovery in the past 5 years had to pivot real hard to salvage their shit post-transformers. They're very disruptively good at most NLP task edit: post-transformers meaning "in the era after transformers were widely adopted" not some mystical new wave of hypothetical tech to disrupt transformers themselves.
- rootnod3 1y ago
  So, unless this went r/woosh over my head....how is current AI better than shit post-transformers? If all....old shit post-transformers are at least deterministic or open and not a randomized shitbox. Unless I misinterpreted the post, render me confused.
  - dgacmu 1y ago
    I think you're misinterpreting: "with the advent of transformers, (many) people doing NLP with pre-transformers techniques had to salvage their shit"
    - rootnod3 1y ago
      I guess. That's why I added the "unless I am mis-interpreting", still got downvoted for it because I guess it was against AI. The wording was confusing but so was my understanding of it as a non-native speaker. Shit happens.
      - dgacmu 1y ago
        I agree that the wording was a bit confusing (as a native speaker).
  - isoprophlex 1y ago
    I wasn't too clear, I think. Apologies if the wording was confusing. People who started their NLP work (PhDs etc; industry research projects) before the LLM / transformer craze had to adapt to the new world. (Hence 'post-mass-uptake-of-transformers')
warkdarrior 1y ago
Spam detection and phishing detection are completely different than 5 years ago, as one cannot rely on typos and grammar mistakes to identify bad content.
- onlyrealcuzzo 1y ago
  The signals might be different, but the underlying mechanism is still incredibly efficient, no?
- walkabout 1y ago
  Spam, scams, propaganda, and astroturfing are easily the largest beneficiaries of LLM automation, so far. LLMs are exactly the 100x rocket-boots their boosters are promising for other areas (without such results outside a few tiny, but sometimes important, niches, so far) when what you're doing is producing throw-away content at enormous scale and have a high tolerance for mistakes, as long as the volume is high.
  - visarga 1y ago
    It seems unfair to call out LLMs for "spam, scams, propaganda, and astroturfing." These problems are largely the result of platform optimization for engagement and SEO competition for attention. This isn't unique to models; even we, humans, when operating without feedback, generate mostly slop. Curation is performed by the environment and the passage of time, which reveals consequences. LLMs taken in isolation from their environment are just as sloppy as brains in a similar situation. Therefore, the correct attitude to take regarding LLMs is to create ways for them to receive useful feedback on their outputs. When using a coding agent, have the agent work against tests. Scaffold constraints and feedback around it. AlphaZero, for example, had abundant environmental feedback and achieved amazing (superhuman) results. Other Alpha models (for math, coding, etc.) that operated within validation loops reached olympic levels in specific types of problem-solving. The limitation of LLMs is actually a limitation of their incomplete coupling with the external world. In fact you don't even need a super intelligent agent to make progress, it is sufficient to have copying and competition, evolution shows it can create all life, including us and our culture and technology without a very smart learning algorithm. Instead what it has is plenty of feedback. Intelligence is not in the brain or the LLM, it is in the ecosystem, the society of agents, and the world. Intelligence is the result of having to pay the cost of our execution to continue to exist, a strategy to balance the cost of life. What I mean by feedback is exploration, when you execute novel actions or actions in novel environment configurations, and observe the outcomes. And adjust, and iterate. So the feedback becomes part of the model, and the model part of the action-feedback process. They co-create each other.
jonas21 1y ago
Out of curiosity, what field are you in?
EGreg 1y ago
AI fan (type 1 -- AI made a big breakthrough) meets AI defender (type 2 -- AI has not fundamentally changed anything that was already a problem). Defenders are supposed to defend against attacks on AI, but here it misfired, so the conversation should be interesting. That's because the defender is actually a skeptic of AI. But the first sentence sounded like a typical "nothing to see here" defense of AI.
mountainriver 1y ago
Software, and it’s wildly positive. Takes like this are utterly insane to me
- Silamoth 1y ago
  It’s had an impact on software for sure. Now I have to fix my coworker’s AI slop code all the time. I guess it could be a positive for my job security. But acting like “AI” has had a wildly positive impact on software seems, at best, a simplification and, at worst, the opposite of reality.
  - mountainriver 1y ago
    oh heavens, do you not review code?
- sponnath 1y ago
  Wouldn't say it's transformative.
  - mrieck 1y ago
    My workflow is transformed. If yours isn’t you’re missing out. Days that I’d normally feel overwhelmed from requests by management are just Claude Code and chill days now.
    - sponnath 1y ago
      I've tried to make AI work but a lot of times the overall productivity gains I do get are so negligible that I wouldn't say it's been transformative for me. I think the fact that so many of us here on HN have such different experiences with AI goes to show that it is indeed not as transformative as we think it is (for the field at least). I'm not trying to invalidate your experience.
    - jimbo808 1y ago
      If you're being honest, I bet your codebase is going to shit and your skills are in rapid decline. I bet you have a ton of redundant code, broken patterns, shit tests, and your coworkers are noticing the slop and getting tired of fixing it. Eventually, your code be such shit that Claude Code will struggle to even do basic CRUD because there are four redundant functions and it keeps editing the wrong ones. Your colleagues will go to edit your code, only to realize that it's such utter garbage that they have to rewrite the whole thing because that's easier than trying to make sense of the slop you produced under your own name. If you were feeling overwhelmed by management, and Claude Code is alleviating that, I fear you aren't cut out for the work.
blibble 1y ago
> but I would say the impact is almost entirely negative. quite the transformer innovation was to bring down the cost of producing incorrect, but plausible looking content (slop) in any modality to near zero not a positive thing for anyone other than spammers
CHY872 1y ago
In computer vision transformers have basically taken over most perception fields. If you look at paperswithcode benchmarks it’s common to find like 10/10 recent winners being transformer based against common CV problems. Note, I’m not talking about VLMs here, just small ViTs with a few million parameters. YOLOs and other CNNs are still hanging around for detection but it’s only a matter of time.
- thesz 1y ago
  Can it be that transformer-based solutions come from the well-funded organizations that can spend vast amount of money on training expensive (O(n^3)) models? Are there any papers that compare predictive power against compute needed?
  - nickpsecurity 1y ago
    You're onto something. BabyLM competition had caps. Many LLM's were using 1TB training data for some time. In many cases, I can't even see how many GPU hours or what size cluster of what GPU's the pretraining required. If I can't afford it, then it doesn't matter what it achieved. What I can afford is what I have to choose from.

Which fields have they completely transformed? How was it before and how is it now? I won't pretend like it hasn't impacted my field, but I would say the impact is almost entirely negative.

Profan 1y ago

hah well, transformative doesn't necessarily mean positive!

econ 1y ago
All we get is distraction.

dekhn 1y ago

Genomics, protein structure prediction, various forms of small molecule and large molecule drug discovery.

thesz 1y ago
No neural protein structure prediction papers I read have compared transformers to SAT solvers. As if this approach [1] does not exist. [1] https://pmc.ncbi.nlm.nih.gov/articles/PMC7197060/ https://pmc.ncbi.nlm.nih.gov/articles/PMC7197060/
- dekhn 1y ago
  What exactly are you suggesting- that the SAT solver example given in the paper (applied the HP model of protein structure), or an improvement on it, could produce protein structure prediction at the level of AlphaFold? This seems extremely, extremely unlikely for many reasons. The HP model is a simplification of true protein folding/structure adoption, while AlphaFold (and the open source equivalents) works with real proteins. The SAT approach uses little to no prior knowledge about protein structures, unlike AlphaFold (which has basically memorized and generalized the PDB). To express all the necessary details would likely exceed the capabilities of the best SAT solvers. (don't get me wrong- SAT and other constraint approaches are powerful tools. But I do not think they are the best approach for protein structure prediction).
  - YeGoblynQueenne 1y ago
    (Not the OP) If we've learned one thing from the ascent of neural nets is that you have no idea whether something works or not until you have tried it. And by "tried it" I mean really, really gave it a go as hard as possible, with all resources you can muster. The industry has thrown everything it's got on Transformers, but there are many other approaches that have at least as promising empirical results and much better theoretical support and have not been pursued with the same fervor, so we have no idea how well or bad they'd do against neural nets, if they were given the same treatment. Like the OP says, it's as if such approaches don't even exist.
    - chermi 1y ago
      Do you understand the relevant fundamental difference between SAT and neural net approaches? One is a machine learning approach, the other is not. We know the computational complexity of SAT solvers, they're fixed algorithms. SAT doesn't learn with more data. It has performance limits and that's the end of the story. BTW, as I mentioned in my other comment, people have been trying SAT solvers in the CASP competition for decades. They got blown away by transformer approach. Such approaches exist, and they've been found wanting, and no amount of compute is going to improve their performance limits, because it isn't an ML approach with scaling laws. This is definitely not some unfair conspiracy against SAT, and probably not against the majority of pre-transformer approaches. I am sympathetic to the concern that transformer based research is getting too much attention at the expense of other approaches. However, I'd think the success of transformer makes it more likely than ever that proven-promising alternative approaches would get funding as investors try to beat everyone to the next big thing. See quantum computing funding or funding for way out there ASIC startups. TL;DR I don't know what is meant by the "same treatment" for SAT solvers. Funding is finite and goes toward promising approaches. If there "at least as promising" approaches, go show clear evidence of that to a VC and I promise you'll get funding.
- chermi 1y ago
  Hey man, CASP is an open playing field. If it was better, they would've showed it by now.
  - YeGoblynQueenne 1y ago
    Said somebody about neural nets in the 1980's.
    - chermi 1y ago
      I don't really understand what point you or parent are trying to make. SAT approaches have been used in CASP, an open competition for protein structure prediction. They have been trying for decades with SAT. The transformer based models blew every approach out of the water to the point of approaching experimental resolution. Why am supposed to pretend SAT is being treated unfairly or whatever you guys are expounding? Based on your response and the parent's, don't think you'd be happy if SAT approaches WERE cited. Maybe you and parent think every preexisting approach hasn't been proven to be inferior to the transformer approach until some equivalent amount of compute has been thrown at them compared to the transformer approach? That's the best I can come up with. There is no room for 'scaling' gains with SAT solvers that will be found with more compute, it's not an ML approach. That is, it doesn't learn with more data. If you mean something else more specific I'd be interested to know.

jimmyl02 1y ago

in the super public consumer space, search engines / answer engines (like chatgpt) are the big ones. on the other hand it's also led to improvements in many places hidden behind the scenes. for example, vision transformers are much more powerful and scalable than many of the other computer vision models which has probably led to new capabilities. in general, transformers aren't just "generate text" but it's a new foundational model architecture which enables a leap step in many things which require modeling!

ACCount37 1y ago
Transformers also make for a damn good base to graft just about any other architecture onto. Like, vision transformers? They seem to work best when they still have a CNN backbone, but the "transformer" component is very good at focusing on relevant information, and doing different things depending on what you want to be done with those images. And if you bolt that hybrid vision transformer to an even larger language-oriented transformer? That also imbues it with basic problem-solving, world knowledge and commonsense reasoning capabilities - which, in things like advanced OCR systems, are very welcome.

CamperBob2 1y ago

Which fields have they completely transformed? Simultaneously discovering and leveraging the functional nature of language seems like kind of a big deal.

jimbo808 1y ago
Can you explain what this means?
- CamperBob2 1y ago
  Given that we can train a transformer model by shoveling large amounts of inert text at it, and then use it to compose original works and solve original problems with the addition of nothing more than generic computing power, we can conclude that there's nothing special about what the human brain does. All that remains is to come up with a way to integrate short-term experience into long-term memory, and we can call the job of emulating our brains done, at least in principle. Everything after that will amount to detail work.
  - jimbo808 1y ago
    > we can conclude that there's nothing special about what the human brain does ...lol. Yikes. I do not accept your premise. At all. > use it to compose original works and solve original problems Which original works and original problems have LLMs solved, exactly? You might find a random article or stealth marketing paper that claims to have solved some novel problem, but if what you're saying were actually true, we'd be flooded with original works and new problems being solved. So where are all these original works? > All that remains is to come up with a way to integrate short-term experience into long-term memory, and we can call the job of emulating our brains done, at least in principle What experience do you have that caused you to believe these things?
    - CamperBob2 1y ago
      Which is fine, but it's now clear where the burden of proof lies, and IMHO we have transformer-based language models to thank for that. If anyone still insists on hidden magical components ranging from immortal souls to Penrose's quantum woo, well... let's see what you've got.

isoprophlex 1y ago

Everyone who did NLP research or product discovery in the past 5 years had to pivot real hard to salvage their shit post-transformers. They're very disruptively good at most NLP task edit: post-transformers meaning "in the era after transformers were widely adopted" not some mystical new wave of hypothetical tech to disrupt transformers themselves.

rootnod3 1y ago
So, unless this went r/woosh over my head....how is current AI better than shit post-transformers? If all....old shit post-transformers are at least deterministic or open and not a randomized shitbox. Unless I misinterpreted the post, render me confused.
- dgacmu 1y ago
  I think you're misinterpreting: "with the advent of transformers, (many) people doing NLP with pre-transformers techniques had to salvage their shit"
  - rootnod3 1y ago
    I guess. That's why I added the "unless I am mis-interpreting", still got downvoted for it because I guess it was against AI. The wording was confusing but so was my understanding of it as a non-native speaker. Shit happens.
    - dgacmu 1y ago
      I agree that the wording was a bit confusing (as a native speaker).
- isoprophlex 1y ago
  I wasn't too clear, I think. Apologies if the wording was confusing. People who started their NLP work (PhDs etc; industry research projects) before the LLM / transformer craze had to adapt to the new world. (Hence 'post-mass-uptake-of-transformers')

warkdarrior 1y ago

Spam detection and phishing detection are completely different than 5 years ago, as one cannot rely on typos and grammar mistakes to identify bad content.

onlyrealcuzzo 1y ago
The signals might be different, but the underlying mechanism is still incredibly efficient, no?
walkabout 1y ago
Spam, scams, propaganda, and astroturfing are easily the largest beneficiaries of LLM automation, so far. LLMs are exactly the 100x rocket-boots their boosters are promising for other areas (without such results outside a few tiny, but sometimes important, niches, so far) when what you're doing is producing throw-away content at enormous scale and have a high tolerance for mistakes, as long as the volume is high.
- visarga 1y ago
  It seems unfair to call out LLMs for "spam, scams, propaganda, and astroturfing." These problems are largely the result of platform optimization for engagement and SEO competition for attention. This isn't unique to models; even we, humans, when operating without feedback, generate mostly slop. Curation is performed by the environment and the passage of time, which reveals consequences. LLMs taken in isolation from their environment are just as sloppy as brains in a similar situation. Therefore, the correct attitude to take regarding LLMs is to create ways for them to receive useful feedback on their outputs. When using a coding agent, have the agent work against tests. Scaffold constraints and feedback around it. AlphaZero, for example, had abundant environmental feedback and achieved amazing (superhuman) results. Other Alpha models (for math, coding, etc.) that operated within validation loops reached olympic levels in specific types of problem-solving. The limitation of LLMs is actually a limitation of their incomplete coupling with the external world. In fact you don't even need a super intelligent agent to make progress, it is sufficient to have copying and competition, evolution shows it can create all life, including us and our culture and technology without a very smart learning algorithm. Instead what it has is plenty of feedback. Intelligence is not in the brain or the LLM, it is in the ecosystem, the society of agents, and the world. Intelligence is the result of having to pay the cost of our execution to continue to exist, a strategy to balance the cost of life. What I mean by feedback is exploration, when you execute novel actions or actions in novel environment configurations, and observe the outcomes. And adjust, and iterate. So the feedback becomes part of the model, and the model part of the action-feedback process. They co-create each other.

jonas21 1y ago

Out of curiosity, what field are you in?

EGreg 1y ago

AI fan (type 1 -- AI made a big breakthrough) meets AI defender (type 2 -- AI has not fundamentally changed anything that was already a problem). Defenders are supposed to defend against attacks on AI, but here it misfired, so the conversation should be interesting. That's because the defender is actually a skeptic of AI. But the first sentence sounded like a typical "nothing to see here" defense of AI.

mountainriver 1y ago

Software, and it’s wildly positive. Takes like this are utterly insane to me

Silamoth 1y ago
It’s had an impact on software for sure. Now I have to fix my coworker’s AI slop code all the time. I guess it could be a positive for my job security. But acting like “AI” has had a wildly positive impact on software seems, at best, a simplification and, at worst, the opposite of reality.
- mountainriver 1y ago
  oh heavens, do you not review code?
sponnath 1y ago
Wouldn't say it's transformative.
- mrieck 1y ago
  My workflow is transformed. If yours isn’t you’re missing out. Days that I’d normally feel overwhelmed from requests by management are just Claude Code and chill days now.
  - sponnath 1y ago
    I've tried to make AI work but a lot of times the overall productivity gains I do get are so negligible that I wouldn't say it's been transformative for me. I think the fact that so many of us here on HN have such different experiences with AI goes to show that it is indeed not as transformative as we think it is (for the field at least). I'm not trying to invalidate your experience.
  - jimbo808 1y ago
    If you're being honest, I bet your codebase is going to shit and your skills are in rapid decline. I bet you have a ton of redundant code, broken patterns, shit tests, and your coworkers are noticing the slop and getting tired of fixing it. Eventually, your code be such shit that Claude Code will struggle to even do basic CRUD because there are four redundant functions and it keeps editing the wrong ones. Your colleagues will go to edit your code, only to realize that it's such utter garbage that they have to rewrite the whole thing because that's easier than trying to make sense of the slop you produced under your own name. If you were feeling overwhelmed by management, and Claude Code is alleviating that, I fear you aren't cut out for the work.

blibble 1y ago

> but I would say the impact is almost entirely negative. quite the transformer innovation was to bring down the cost of producing incorrect, but plausible looking content (slop) in any modality to near zero not a positive thing for anyone other than spammers

CHY872 1y ago

In computer vision transformers have basically taken over most perception fields. If you look at paperswithcode benchmarks it’s common to find like 10/10 recent winners being transformer based against common CV problems. Note, I’m not talking about VLMs here, just small ViTs with a few million parameters. YOLOs and other CNNs are still hanging around for detection but it’s only a matter of time.

thesz 1y ago
Can it be that transformer-based solutions come from the well-funded organizations that can spend vast amount of money on training expensive (O(n^3)) models? Are there any papers that compare predictive power against compute needed?
- nickpsecurity 1y ago
  You're onto something. BabyLM competition had caps. Many LLM's were using 1TB training data for some time. In many cases, I can't even see how many GPU hours or what size cluster of what GPU's the pretraining required. If I can't afford it, then it doesn't matter what it achieved. What I can afford is what I have to choose from.