5 ms·

How sure are you about that? The basic theories have been around since the 70s, have been proven at scale in the last decade, and the NSA has more data and com

by nr2x 4y ago

How sure are you about that? The basic theories have been around since the 70s, have been proven at scale in the last decade, and the NSA has more data and compute than anybody else. I’d be shocked if they aren’t very far along in solving many problems.

RC_ITR 4y ago
This isn’t really a “throw money at it” problem like the government is good at. Take drones for example. The government got really good at those because they made them jet-powered (lol) and blew a bunch of money on server-grade FPGA’s in each one of them. You can’t really just buy a lot of GPUs to make an LLM work, you need iterative development of architecture and training methods. Like maybe the government invented self-attention before 2017, but if they didn’t, then the constraint is training time, and the government has the same number of seconds as the rest of us.
- nr2x 4y ago
  I wouldn’t rule out NSA being ahead of the curve, but you have a good point re: GPUs. Likely another factor in the CHIPS act.
- rozal 4y ago
  Did you like, forget the the military invented the nuclear bomb, semiconductors, coding, AI, NLP, the INTERNET. Everything you use is from the military. The government is good at lying, and making themselves ‘appear’ incompetent. They secretly probably have a much further advanced quantum computer. Your viewpoint is limited to mainstream technology and mainstream science.
  - RC_ITR 4y ago
    This is such an interesting take. The military invented the nuclear bomb yes. But Fermi did most of his thinking work in Italy before the Manhattan project. He got money thrown at him once he got here. As for semiconductor devices, it was Bell Labs and TI. Coding is an ambiguous concept that wasn’t really invented, but if it were, it would have first appeared in programmable looms. The military likes to take credit for things, but really all they do is throw money at existing inventions. I’m sure they’re throwing a bunch of money at Transformers now, but who are all these uncredited super geniuses who invent things and then let randos at Google take the credit/earn the money?
    - rozal 4y ago
      I used to agree with you. I used to think the military was kinda dumb. But after doing a deeper dive into past military technology, and present - I've come to realize this is just an intelligence ruse. They made some mistakes in the 40's and 50's to where they had nuclear secrets stolen by the Russians. And ever since has been hyper compartmentalized. It would not surprise me if in the 90's or 00's they had an internal working LLM, considering all the puzzle pieces. You will never hear about classified tech unless it's a bomb, gets leaked. (See code breaking machines declassed after 70+ years) In a hypothetical scenario, a military organization might want to conceal its use of a large language model (LLM) for intelligence gathering and analysis. Another scenario is the military's current interest in everything quantum. Quantum computers for example (you wouldn't want another nation being first and pirate baying out our secrets, would you) so there is an extreme national security importance of being first. And to be first, you need to have the smart people, which the military has. There is a reason China struggles with jet engines 80 years after their invention, and still can't make nuclear carriers. While the US navy works on things like this: https://www.navair.navy.mil/foia/sites/g/files/jejdrs566/files/document/%5Bfilename%5D/2022-006587%20FINAL%20VERSION%20AIAA%20EXPL%2002.pdf https://www.navair.navy.mil/foia/sites/g/files/jejdrs566/fil...
      - RC_ITR 4y ago
        I mean, I think you’re again missing the point here. Jet engines can be solved with money. The actual steps of making an LLM require complex math and you can’t just pay people to make better math. And if you could, wouldn’t those people decamp for industry and become literal trillionares?
        rozal 4y ago
        That's not how military intellectual property works. I don't doubt your technical credentials, but I think you lack deeper insight to what the military does, and has done.
        RC_ITR 4y ago
stevenhuang 4y ago
Pretty sure. Even the Manhattan project had nuclear research going on in public universities at the time. Nothing of the sort here for the attention mechanism which underpins LLMs we know today. Fundamental research isn't something you just throw money at and acquire. All we had back then were cleverbot and other expert systems.
- nr2x 4y ago
  More my point is they have as big a research budget as a corporate lab.
- rozal 4y ago
  The military invented AI and NLP which underpins LLMs. The military is responsible for most the technology we use and talk about today. The government may appear incompetent, but we’re living off military hand-me-downs, the entire world is
411111111111111 4y ago
Your argument reminds me of the discussions about the moon landing. If today's hardware was available 20 yrs ago, this would've been possible just like the moon landing could've been faked if it took place 20+ yrs later. The technology wasn't available at the time (GPUs in this case, and generally no experience in doing such advanced trick techniques for movies back then) These models are having such a strong effect now because we've finally got the hardware to run them

by nr2x 4y ago

RC_ITR 4y ago

This isn’t really a “throw money at it” problem like the government is good at. Take drones for example. The government got really good at those because they made them jet-powered (lol) and blew a bunch of money on server-grade FPGA’s in each one of them. You can’t really just buy a lot of GPUs to make an LLM work, you need iterative development of architecture and training methods. Like maybe the government invented self-attention before 2017, but if they didn’t, then the constraint is training time, and the government has the same number of seconds as the rest of us.

nr2x 4y ago
I wouldn’t rule out NSA being ahead of the curve, but you have a good point re: GPUs. Likely another factor in the CHIPS act.
rozal 4y ago
Did you like, forget the the military invented the nuclear bomb, semiconductors, coding, AI, NLP, the INTERNET. Everything you use is from the military. The government is good at lying, and making themselves ‘appear’ incompetent. They secretly probably have a much further advanced quantum computer. Your viewpoint is limited to mainstream technology and mainstream science.
- RC_ITR 4y ago
  This is such an interesting take. The military invented the nuclear bomb yes. But Fermi did most of his thinking work in Italy before the Manhattan project. He got money thrown at him once he got here. As for semiconductor devices, it was Bell Labs and TI. Coding is an ambiguous concept that wasn’t really invented, but if it were, it would have first appeared in programmable looms. The military likes to take credit for things, but really all they do is throw money at existing inventions. I’m sure they’re throwing a bunch of money at Transformers now, but who are all these uncredited super geniuses who invent things and then let randos at Google take the credit/earn the money?
  - rozal 4y ago
    I used to agree with you. I used to think the military was kinda dumb. But after doing a deeper dive into past military technology, and present - I've come to realize this is just an intelligence ruse. They made some mistakes in the 40's and 50's to where they had nuclear secrets stolen by the Russians. And ever since has been hyper compartmentalized. It would not surprise me if in the 90's or 00's they had an internal working LLM, considering all the puzzle pieces. You will never hear about classified tech unless it's a bomb, gets leaked. (See code breaking machines declassed after 70+ years) In a hypothetical scenario, a military organization might want to conceal its use of a large language model (LLM) for intelligence gathering and analysis. Another scenario is the military's current interest in everything quantum. Quantum computers for example (you wouldn't want another nation being first and pirate baying out our secrets, would you) so there is an extreme national security importance of being first. And to be first, you need to have the smart people, which the military has. There is a reason China struggles with jet engines 80 years after their invention, and still can't make nuclear carriers. While the US navy works on things like this: https://www.navair.navy.mil/foia/sites/g/files/jejdrs566/files/document/%5Bfilename%5D/2022-006587%20FINAL%20VERSION%20AIAA%20EXPL%2002.pdf https://www.navair.navy.mil/foia/sites/g/files/jejdrs566/fil...
    - RC_ITR 4y ago
      I mean, I think you’re again missing the point here. Jet engines can be solved with money. The actual steps of making an LLM require complex math and you can’t just pay people to make better math. And if you could, wouldn’t those people decamp for industry and become literal trillionares?
      - rozal 4y ago
        That's not how military intellectual property works. I don't doubt your technical credentials, but I think you lack deeper insight to what the military does, and has done.
        RC_ITR 4y ago

stevenhuang 4y ago

Pretty sure. Even the Manhattan project had nuclear research going on in public universities at the time. Nothing of the sort here for the attention mechanism which underpins LLMs we know today. Fundamental research isn't something you just throw money at and acquire. All we had back then were cleverbot and other expert systems.

nr2x 4y ago
More my point is they have as big a research budget as a corporate lab.
rozal 4y ago
The military invented AI and NLP which underpins LLMs. The military is responsible for most the technology we use and talk about today. The government may appear incompetent, but we’re living off military hand-me-downs, the entire world is

411111111111111 4y ago

Your argument reminds me of the discussions about the moon landing. If today's hardware was available 20 yrs ago, this would've been possible just like the moon landing could've been faked if it took place 20+ yrs later. The technology wasn't available at the time (GPUs in this case, and generally no experience in doing such advanced trick techniques for movies back then) These models are having such a strong effect now because we've finally got the hardware to run them