6 ms·

The first trained multilayer deep neural network was presented by the Soviets in 1970 in a paper titled “Heuristic Self-Organization in Problems of Engineering

The first trained multilayer deep neural network was presented by the Soviets in 1970 in a paper titled “Heuristic Self-Organization in Problems of Engineering Cybernetics“ http://www.gmdh.net/articles/history/heuristic.pdf http://www.gmdh.net/articles/history/heuristic.pdf

Barrin92 6y ago
I feel it's some kind of unwritten law that every time you think you've solved something, there's some Soviet engineer who already did it a few decades ago I imagine the solution to every problem can be found lying on a shelf, untranslated in a research lab in the middle of nowhere in Russia
- walterbell 6y ago
  They also developed TRIZ, a Russian algorithm for inventing new systems, https://en.wikipedia.org/wiki/TRIZ https://en.wikipedia.org/wiki/TRIZ > a theory which defines generalisable patterns in the nature of inventive solutions and the distinguishing characteristics of the problems that these inventions have overcome It's been used by Samsung and other orgs.
  - eternalban 6y ago
    https://upload.wikimedia.org/wikipedia/commons/b/bf/40_principles_of_TRIZ_method_720dpi.jpg https://upload.wikimedia.org/wikipedia/commons/b/bf/40_princ... Looking at that cheat-sheet, it's interesting how you can recognize elements of one's own process. I bet almost everyone can check off a few of those boxes as "I do this all the time". Also interesting to note there are no interaction principles. "19 Pulsed Action Periodic Action" can be mapped in software systems to PUSH+Timer, but that's it. TRIZ apparently lacks the vocabulary to explore the space of interacting components [as first class "principles"].
- jbay808 6y ago
  If you ever really need to invalidate a patent, a good way is to hire a Russian-speaking agent to sift through old engineering journals and libraries. A lot of prior art turns up that nobody knows about.
- ATsch 6y ago
  Especially interesting with all of the talk about the unique levels of innovation that capitalism brings that supposedly justify all of the other stuff.
  - burntoutfire 6y ago
    Research != innovation. Innovation is more about practical implementation of novel ideas (incl. ones coming from research) - something at which soviets weren't very good at.
    - ATsch 6y ago
      Could you give any examples
      - pegasus 6y ago
        The article commented in this thread is one such example: neural networks were put to actual real-world use by western companies. Another example is the attempt of USSR to build something like the internet before the current one was thought out. They had the vision but the implementation somehow didn't take off.
        ATsch 6y ago
        Both the US and USSR knew about neural networks for decades before fast massively parallel hardware made them feasible long after the end of the USSR, so I'm not sure that's a useful comparison. The development of the internet was driven by the state out of national security concerns, not market forces and was not widely used during the USSRs life, so I'm not sure that one is helpful to illustrate the OPs point either.
      - burntoutfire 6y ago
        Innovation happens mostly in a bottom-up fashion, either on a big scale, the way it's done in Silicon Valley, or in smaller scale, by for example just having thousands of independent grocery stores in a given city, each free to experiment with the way they conduct business. In contrast, in soviet states, everything was centrally organized and ran, which meant zero freedom to innovate at the ground level. This resulted in incredible inefficiency, as better ways of doing thing couldn't possibly be invented (unless the bureaucrat in Moscowwho controlled a given sector of the economy had a genuinely good idea, but that was rare). Research, on the other hand, is much more prone to centralization, so it was doing quite well in the Soviets. Hence they ended up with a lot of research, but not that much innovation.
- tempodox 6y ago
  Like the running gag in Star Trek TOS where Chekov would invariably assert that "Russians invented it first". Only, for real this time.
  - jacobush 6y ago
    For all its faults and imperial tendencies and repression of its minority peoples, the USSR was more than just Russian. It had many peoples who all contributed to discoveries, but "Russia" today has inherited the "creds" to what the USSR accomplished, both good and bad. (For a tiny glimpse of the USSR "feel", watch the BBC movie "Afghan experience" where some USSR soldiers from other republics than Russia proper speak.)
ethn 6y ago
I think this is incorrect, this isn't the first trained multi-layer neural network. Minsky & Rosenblatt's https://en.wikipedia.org/wiki/Perceptrons_(book) https://en.wikipedia.org/wiki/Perceptrons_(book) covers multi-layer neural nets. Rosenblatt's electronic implementation, the Mark I https://apps.dtic.mil/dtic/tr/fulltext/u2/236965.pdf https://apps.dtic.mil/dtic/tr/fulltext/u2/236965.pdf Minsky's electronic implementation, SNAC https://en.wikipedia.org/wiki/Stochastic_neural_analog_reinforcement_calculator https://en.wikipedia.org/wiki/Stochastic_neural_analog_reinf...
- the_decider 6y ago
  I mentioned a “deep” multi-layer neural net, which implies more than just 2-3 layers. As far as I’m aware, this is the first eight-layer trained network mentioned in any publication. http://people.idsia.ch/~juergen/deep-learning-conspiracy.html http://people.idsia.ch/~juergen/deep-learning-conspiracy.htm...
  - ethn 6y ago
    Ah, I wasn’t aware of that distinction.

Barrin92 6y ago

I feel it's some kind of unwritten law that every time you think you've solved something, there's some Soviet engineer who already did it a few decades ago I imagine the solution to every problem can be found lying on a shelf, untranslated in a research lab in the middle of nowhere in Russia

walterbell 6y ago
They also developed TRIZ, a Russian algorithm for inventing new systems, https://en.wikipedia.org/wiki/TRIZ https://en.wikipedia.org/wiki/TRIZ > a theory which defines generalisable patterns in the nature of inventive solutions and the distinguishing characteristics of the problems that these inventions have overcome It's been used by Samsung and other orgs.
- eternalban 6y ago
  https://upload.wikimedia.org/wikipedia/commons/b/bf/40_principles_of_TRIZ_method_720dpi.jpg https://upload.wikimedia.org/wikipedia/commons/b/bf/40_princ... Looking at that cheat-sheet, it's interesting how you can recognize elements of one's own process. I bet almost everyone can check off a few of those boxes as "I do this all the time". Also interesting to note there are no interaction principles. "19 Pulsed Action Periodic Action" can be mapped in software systems to PUSH+Timer, but that's it. TRIZ apparently lacks the vocabulary to explore the space of interacting components [as first class "principles"].
jbay808 6y ago
If you ever really need to invalidate a patent, a good way is to hire a Russian-speaking agent to sift through old engineering journals and libraries. A lot of prior art turns up that nobody knows about.
ATsch 6y ago
Especially interesting with all of the talk about the unique levels of innovation that capitalism brings that supposedly justify all of the other stuff.
- burntoutfire 6y ago
  Research != innovation. Innovation is more about practical implementation of novel ideas (incl. ones coming from research) - something at which soviets weren't very good at.
  - ATsch 6y ago
    Could you give any examples
    - pegasus 6y ago
      The article commented in this thread is one such example: neural networks were put to actual real-world use by western companies. Another example is the attempt of USSR to build something like the internet before the current one was thought out. They had the vision but the implementation somehow didn't take off.
      - ATsch 6y ago
        Both the US and USSR knew about neural networks for decades before fast massively parallel hardware made them feasible long after the end of the USSR, so I'm not sure that's a useful comparison. The development of the internet was driven by the state out of national security concerns, not market forces and was not widely used during the USSRs life, so I'm not sure that one is helpful to illustrate the OPs point either.
    - burntoutfire 6y ago
      Innovation happens mostly in a bottom-up fashion, either on a big scale, the way it's done in Silicon Valley, or in smaller scale, by for example just having thousands of independent grocery stores in a given city, each free to experiment with the way they conduct business. In contrast, in soviet states, everything was centrally organized and ran, which meant zero freedom to innovate at the ground level. This resulted in incredible inefficiency, as better ways of doing thing couldn't possibly be invented (unless the bureaucrat in Moscowwho controlled a given sector of the economy had a genuinely good idea, but that was rare). Research, on the other hand, is much more prone to centralization, so it was doing quite well in the Soviets. Hence they ended up with a lot of research, but not that much innovation.
tempodox 6y ago
Like the running gag in Star Trek TOS where Chekov would invariably assert that "Russians invented it first". Only, for real this time.
- jacobush 6y ago
  For all its faults and imperial tendencies and repression of its minority peoples, the USSR was more than just Russian. It had many peoples who all contributed to discoveries, but "Russia" today has inherited the "creds" to what the USSR accomplished, both good and bad. (For a tiny glimpse of the USSR "feel", watch the BBC movie "Afghan experience" where some USSR soldiers from other republics than Russia proper speak.)

ethn 6y ago

I think this is incorrect, this isn't the first trained multi-layer neural network. Minsky & Rosenblatt's https://en.wikipedia.org/wiki/Perceptrons_(book) https://en.wikipedia.org/wiki/Perceptrons_(book) covers multi-layer neural nets. Rosenblatt's electronic implementation, the Mark I https://apps.dtic.mil/dtic/tr/fulltext/u2/236965.pdf https://apps.dtic.mil/dtic/tr/fulltext/u2/236965.pdf Minsky's electronic implementation, SNAC https://en.wikipedia.org/wiki/Stochastic_neural_analog_reinforcement_calculator https://en.wikipedia.org/wiki/Stochastic_neural_analog_reinf...

the_decider 6y ago
I mentioned a “deep” multi-layer neural net, which implies more than just 2-3 layers. As far as I’m aware, this is the first eight-layer trained network mentioned in any publication. http://people.idsia.ch/~juergen/deep-learning-conspiracy.html http://people.idsia.ch/~juergen/deep-learning-conspiracy.htm...
- ethn 6y ago
  Ah, I wasn’t aware of that distinction.