5 ms·

I sometimes wonder if there’s an academic career hidden in there for an engineer: go to the library and read what the CS folks were publishing on physical paper

I sometimes wonder if there’s an academic career hidden in there for an engineer: go to the library and read what the CS folks were publishing on physical papers, maybe there are some ideas that can actually be implemented now that weren’t practical back then.

findthewords 2y ago
Yes, "read 10 year old papers as a source of ideas ripe for commercialization" IS common advice in universities.
- chrisbrandow 2y ago
  A post-doc in my chemistry lab had the saying, “two weeks in the lab will save you a day in the library”
  - scns 2y ago
    Weeks of work can save hours of planning.
- deleted 2y ago
  [deleted]
pornel 2y ago
Image and video compression are like that. Ideas for mainframes in the '80s are realtime algorithms now.
- eru 2y ago
  I guess they are _soft_ realtime algorithms now?
  - taneq 2y ago
    'Realtime' isn't a specific thing, there's just 'fast enough'. Oldschool "render each frame during the scan line staying ahead of the electron beam" graphics was pretty hardcore though.
    - eru 2y ago
      'Realtime' actually has multiple meanings. At least one of them is very, very specific and is the one that Wikipedia uses in https://en.wikipedia.org/wiki/Real-time_computing https://en.wikipedia.org/wiki/Real-time_computing > Real-time computing (RTC) is the computer science term for hardware and software systems subject to a "real-time constraint", for example from event to system response.[1] Real-time programs must guarantee response within specified time constraints, often referred to as "deadlines".[2] Strictly speaking, this definition doesn't say anything about how tight those deadlines are, as long as you can guarantee some deadlines. There's also 'soft' real time where you try to hit your deadlines often-enough, but there's no guarantees and a missed deadline is not the end of the world. Games are good example of that, including the example of chasing the electron beam. ABS brakes are an example of a 'hard' real time system: the deadlines aren't nearly as tight as for video game frames, but you really, really can't afford to miss them.
deleted 2y ago
[deleted]
Terr_ 2y ago
In a series of books by David Brin [0] there is a galaxy-wide institution known as the library, and civilizations regularly mine its millions of years of data for suddenly-relevant-again techniques and technologies. I remember one bit where a species had launched some tricky fleet-destroying weapon to surprise their enemies with esoteric physics, only to have it reversed against them, possibly because the Librarian that once helped their research-agent wasn't entirely unbiased. [0] https://en.m.wikipedia.org/wiki/Uplift_Universe https://en.m.wikipedia.org/wiki/Uplift_Universe
- AceJohnny2 2y ago
  Also in Vinge's Deepness in the Sky, there aren't really "programmers" as we know them anymore, but "programmer-archeologists" that just search the archives for code components to reuse.
  - Terr_ 2y ago
    Also: In the Destiny mythic sci-fi franchise, the human golden age ended with a mysterious apocalypse, leaving "cryptarchs" (crypto-archeologists) to try to rebuild from arcane fragments of encrypted data or unknown formats.
  - The_Colonel 2y ago
    That's pretty much what any (decent) programmer does today as well. You first search your code base to see if the application already does something like it, if not, whether there is a published library. Where this starts to fail is the idea that connecting those already written peaces together is easy.
    - manojlds 2y ago
      ..And then ask an LLM for code
  - abecedarius 2y ago
    I think that's unfair to the programmer-archaeologists: young Pham wanted to write things from scratch (and took advantage when he got a chance to), and other characters said he was a talented hacker, but as they also said, it was just way more productive most of the time to cobble together ancient code.
kvemkon 2y ago
Academic? Perhaps, applied: "When Soviets Won the Cold War: Wading Through Pyotr Ufimstev's Work on Stealth" (26.03.2024) https://news.ycombinator.com/item?id=39830671 https://news.ycombinator.com/item?id=39830671 > In the early 1970s, Lockheed Martin engineer Denys Overholser discovered the key to stealth technology hidden in a stack of translated Soviet technical papers. Disregarded by the Soviet academic elite, and unheard of in the United States, Pyotr Ufimstev had worked out calculations that would help win ... Not sure of the details of this story, but in general having there enough people to grant them time just to search for something interesting seems not unrealistic.
eru 2y ago
I wonder if current AI training's effort to hover up all the training data they can find will accidentally give us most of the benefits of that? A human can only read so much, so has to discriminate. But our machines are omnivorous readers.
fragmede 2y ago
The whole AI trend is in part due to things that are now possible on GPU supercomputers with gigabytes of RAM backed by petabytes of data and at the top speed of GPUs. Some of the algorithms date back to before the ai winter and it's just that we can now do the same thing with a ton more data and faster.
- zaptrem 2y ago
  All of the main algorithms do (multi-layer perceptrons and stochastic gradient descent are from the 50s and 60s!). Basically the only thing that changed is we decided to multiply some of the outputs of the multi-layer perceptrons by each other and softmax it (attention) before passing them back into more layers. Almost all of the other stuff is just gravy to make it converge faster (and run faster on modern hardware).
  - eru 2y ago
    Is there any indication that people had figured out that simpler activation functions like ReLU are worth bothering?
    - zaptrem 2y ago
      Oh, forgot about that one. Wikipedia says ReLU was used in NNs in 1969 but not widely until 2011. Idk if anyone has ever trained a transformer with sigmoid activations, but I don’t immediately see why it wouldn’t work?
      - eru 2y ago
        I remember some experiments of using modern day training and data on some old style networks, eg with sigmoid activation. That worked eventually and worked quite well, but took way more compute and training data that anyone back in the olden days would have thought feasible. The two main problems with sigmoid activation compared to ReLU are: (a) harder to compute (both the value itself and the gradient), and (b) vanishing gradients, especially in deeper networks.
toast0 2y ago
Heck. Look at 10 year old product launch PRs from big tech. Anything that Google launched 10 years ago and killed, but seems like a good idea today is probably also easier to do. And if you look 5-10 years before that, you can find the Yahoo launch PR where they did the same thing ;p
- dehrmann 2y ago
  I always like to point out Wordle for nailing the timing. Some version of it has been viable since 2000, mass smart phone adoption helped with how it spread, but it could have been viable in 2010. What it did was landed at the right moment.
lispwitch 2y ago
this seems to be how PEG parsing became popular during the last couple of decades, for example; see https://bford.info/pub/lang/peg/ https://bford.info/pub/lang/peg/ (peg.pdf p11: "This work is inspired by and heavily based on Birman’s TS/TDPL and gTS/GTDPL systems [from the 1970s ...] Unfortunately it appears TDPL and GTDPL have not seen much practical use, perhaps in large measure because they were originally developed and presented as formal models [...]")

findthewords 2y ago

Yes, "read 10 year old papers as a source of ideas ripe for commercialization" IS common advice in universities.

chrisbrandow 2y ago
A post-doc in my chemistry lab had the saying, “two weeks in the lab will save you a day in the library”
- scns 2y ago
  Weeks of work can save hours of planning.
deleted 2y ago
[deleted]

pornel 2y ago

Image and video compression are like that. Ideas for mainframes in the '80s are realtime algorithms now.

eru 2y ago
I guess they are _soft_ realtime algorithms now?
- taneq 2y ago
  'Realtime' isn't a specific thing, there's just 'fast enough'. Oldschool "render each frame during the scan line staying ahead of the electron beam" graphics was pretty hardcore though.
  - eru 2y ago
    'Realtime' actually has multiple meanings. At least one of them is very, very specific and is the one that Wikipedia uses in https://en.wikipedia.org/wiki/Real-time_computing https://en.wikipedia.org/wiki/Real-time_computing > Real-time computing (RTC) is the computer science term for hardware and software systems subject to a "real-time constraint", for example from event to system response.[1] Real-time programs must guarantee response within specified time constraints, often referred to as "deadlines".[2] Strictly speaking, this definition doesn't say anything about how tight those deadlines are, as long as you can guarantee some deadlines. There's also 'soft' real time where you try to hit your deadlines often-enough, but there's no guarantees and a missed deadline is not the end of the world. Games are good example of that, including the example of chasing the electron beam. ABS brakes are an example of a 'hard' real time system: the deadlines aren't nearly as tight as for video game frames, but you really, really can't afford to miss them.

deleted 2y ago

[deleted]

Terr_ 2y ago

In a series of books by David Brin [0] there is a galaxy-wide institution known as the library, and civilizations regularly mine its millions of years of data for suddenly-relevant-again techniques and technologies. I remember one bit where a species had launched some tricky fleet-destroying weapon to surprise their enemies with esoteric physics, only to have it reversed against them, possibly because the Librarian that once helped their research-agent wasn't entirely unbiased. [0] https://en.m.wikipedia.org/wiki/Uplift_Universe https://en.m.wikipedia.org/wiki/Uplift_Universe

AceJohnny2 2y ago
Also in Vinge's Deepness in the Sky, there aren't really "programmers" as we know them anymore, but "programmer-archeologists" that just search the archives for code components to reuse.
- Terr_ 2y ago
  Also: In the Destiny mythic sci-fi franchise, the human golden age ended with a mysterious apocalypse, leaving "cryptarchs" (crypto-archeologists) to try to rebuild from arcane fragments of encrypted data or unknown formats.
- The_Colonel 2y ago
  That's pretty much what any (decent) programmer does today as well. You first search your code base to see if the application already does something like it, if not, whether there is a published library. Where this starts to fail is the idea that connecting those already written peaces together is easy.
  - manojlds 2y ago
    ..And then ask an LLM for code
- abecedarius 2y ago
  I think that's unfair to the programmer-archaeologists: young Pham wanted to write things from scratch (and took advantage when he got a chance to), and other characters said he was a talented hacker, but as they also said, it was just way more productive most of the time to cobble together ancient code.

kvemkon 2y ago

Academic? Perhaps, applied: "When Soviets Won the Cold War: Wading Through Pyotr Ufimstev's Work on Stealth" (26.03.2024) https://news.ycombinator.com/item?id=39830671 https://news.ycombinator.com/item?id=39830671 > In the early 1970s, Lockheed Martin engineer Denys Overholser discovered the key to stealth technology hidden in a stack of translated Soviet technical papers. Disregarded by the Soviet academic elite, and unheard of in the United States, Pyotr Ufimstev had worked out calculations that would help win ... Not sure of the details of this story, but in general having there enough people to grant them time just to search for something interesting seems not unrealistic.

eru 2y ago

I wonder if current AI training's effort to hover up all the training data they can find will accidentally give us most of the benefits of that? A human can only read so much, so has to discriminate. But our machines are omnivorous readers.

fragmede 2y ago

The whole AI trend is in part due to things that are now possible on GPU supercomputers with gigabytes of RAM backed by petabytes of data and at the top speed of GPUs. Some of the algorithms date back to before the ai winter and it's just that we can now do the same thing with a ton more data and faster.

zaptrem 2y ago
All of the main algorithms do (multi-layer perceptrons and stochastic gradient descent are from the 50s and 60s!). Basically the only thing that changed is we decided to multiply some of the outputs of the multi-layer perceptrons by each other and softmax it (attention) before passing them back into more layers. Almost all of the other stuff is just gravy to make it converge faster (and run faster on modern hardware).
- eru 2y ago
  Is there any indication that people had figured out that simpler activation functions like ReLU are worth bothering?
  - zaptrem 2y ago
    Oh, forgot about that one. Wikipedia says ReLU was used in NNs in 1969 but not widely until 2011. Idk if anyone has ever trained a transformer with sigmoid activations, but I don’t immediately see why it wouldn’t work?
    - eru 2y ago
      I remember some experiments of using modern day training and data on some old style networks, eg with sigmoid activation. That worked eventually and worked quite well, but took way more compute and training data that anyone back in the olden days would have thought feasible. The two main problems with sigmoid activation compared to ReLU are: (a) harder to compute (both the value itself and the gradient), and (b) vanishing gradients, especially in deeper networks.

toast0 2y ago

Heck. Look at 10 year old product launch PRs from big tech. Anything that Google launched 10 years ago and killed, but seems like a good idea today is probably also easier to do. And if you look 5-10 years before that, you can find the Yahoo launch PR where they did the same thing ;p

dehrmann 2y ago
I always like to point out Wordle for nailing the timing. Some version of it has been viable since 2000, mass smart phone adoption helped with how it spread, but it could have been viable in 2010. What it did was landed at the right moment.

lispwitch 2y ago

this seems to be how PEG parsing became popular during the last couple of decades, for example; see https://bford.info/pub/lang/peg/ https://bford.info/pub/lang/peg/ (peg.pdf p11: "This work is inspired by and heavily based on Birman’s TS/TDPL and gTS/GTDPL systems [from the 1970s ...] Unfortunately it appears TDPL and GTDPL have not seen much practical use, perhaps in large measure because they were originally developed and presented as formal models [...]")