4 ms·

If it was trained on the entire body of scientific knowledge (or you know, as close as possible), would it be able to spit out useful correlations by interpolat

by parksy 7y ago

If it was trained on the entire body of scientific knowledge (or you know, as close as possible), would it be able to spit out useful correlations by interpolating? If trained on massive sets of source code and briefs (or perhaps a set of unit tests), could it spit out functioning code based on a problem statement? I'm just not well versed enough with ML to know if these things are a possibility or if it will just remix its training data in probabilistically correct, not-guaranteed-to-make sense ways?

burtonator 7y ago
I'm unclear but considering that the text output is nonsense I imagine the code would be nonsense. It MIGHT compile but might not actually do anything. I'm curious about this myself. Maybe it could transcompile or re-implement common algorithms to already solved problems.
- ppod 7y ago
  This has been tried. The code looks kind of plausible, but doesn't quite compile. Some of the other examples (Shakespeare, Latex, wikipedia source code) are pretty good at a structural level. http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/ This had about 10 million parameters though, compared to OpenAi's 1.5B. I don't think their text is nonsense though. There's some very interesting examples. They have since released publicly a 345M parameter version, there are some nice examples from that model pre-trained with modern poetry on this twitter account: https://twitter.com/rossgoodwin https://twitter.com/rossgoodwin
toomuchtodo 7y ago
It seems we have enough computing power to let these ideas run wild and see what weighted values come out?
markedathome 7y ago
> If trained on massive sets of source code and briefs (or perhaps a set of unit tests), could it spit out functioning code based on a problem statement? Microsoft's latest Visual Studio 2019 beta (v3) has intellicode[0] support which is supposed to ML "guess" 'code' based on equivalent github similarities. I don't think it is quite near to what you are asking, but certainly part of the way there. 0. https://visualstudio.microsoft.com/services/intellicode/ https://visualstudio.microsoft.com/services/intellicode/
- yellow_lead 7y ago
  Can't wait for those bugs caused by autocorrect.
  - mercer 7y ago
    (typing) 'Robert (autocomplete) 'Robert'); DROP TABLE students;--
    - LifeLiverTransp 7y ago
      Little bobby tables we used to call him. I hope you learned from this to sanitize your database and your AI Input.
Figs 7y ago
Almost certainly not. It will generate things that look like the text of scientific papers on a casual glance, but which will be dream-logic gibberish if you actually read it. The unicorn example makes no sense if you stop and think about what it's actually saying. e.g. a "unicorn" that has four horns? Solves a 200 year old mystery... about something just discovered? So close you could touch their horns... while viewing them from the air? It doesn't make sense if you actually think about it at all, but if you're not really paying attention, the style is like a news report. It's better at making up bullshit, but it's still bullshit.
- MegaButts 7y ago
  This is a good point, and it's a little concerning to me that we will likely soon be inundated not just with spam from lowly-paid shills, but literal bots generating a near endless supply of confident-sounding nonsense. I am not an expert with machine learning, but it seems (at a very high level) we've done amazingly well at creating models that can recognize and sometimes recreate patterns. But they never seem to have any ability to understand the patterns. I'm sure someone much more knowledgeable could compare it to a child of whatever age (or maybe I'm just completely wrong).
- derefr 7y ago
  I fully believe that this is 90% of what the human brain is doing to solve the same problems, though. Brains just also have a constraint/filter layer, determining which of these bits of "made up bullshit" should be allowed to rise from the layer generating them, to the motor output layer. Or, to put that another way, flipping the perspective around: our minds could consist of an intelligent, analytical, but utterly unimaginative agent, that sits there listening to a stream of suggestions spewed out by a distinct second agent, one that is "creative" but has no idea about the constraints of things like physics. The brain's analytical agent filters this stream of suggestions, taking notice of the suggestions that seem like they'll make the world change in the ways it "wants"†; and then it does those. † Or, according to modern perceptual-control research, the agent attempts to predict the world that will occur a few seconds in the future, with a bias toward predicting world-states the reward-system has annotated as being rewarding; and then it looks at what motor commands it "would have" issued in that hypothetical world, and actually issues those. The stream of suggestions, in this model, serve as input to feed the generative model of potential world-states; the executive agent then must notice whether the potential world-state is a "possible" world or an "impossible" world (and whether the motor commands required of it are "possible" or "impossible" inputs), and filter out the "impossible" worlds. Under this hypothesis, dreaming is the state when your executive responsible for filtering out "impossible" worlds isn't online. So you just get a continuous "impossible" world generated from the streamed suggestions of the creative-but-stupid bullshit-generating agent, with nothing to tear it down—just as seen in these generative AIs. As consciousness returns, the mental predicted world-state is noticed to be impossible by the now-online analytical agent, and is torn down.

by parksy 7y ago

burtonator 7y ago

I'm unclear but considering that the text output is nonsense I imagine the code would be nonsense. It MIGHT compile but might not actually do anything. I'm curious about this myself. Maybe it could transcompile or re-implement common algorithms to already solved problems.

ppod 7y ago
This has been tried. The code looks kind of plausible, but doesn't quite compile. Some of the other examples (Shakespeare, Latex, wikipedia source code) are pretty good at a structural level. http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/ This had about 10 million parameters though, compared to OpenAi's 1.5B. I don't think their text is nonsense though. There's some very interesting examples. They have since released publicly a 345M parameter version, there are some nice examples from that model pre-trained with modern poetry on this twitter account: https://twitter.com/rossgoodwin https://twitter.com/rossgoodwin

toomuchtodo 7y ago

It seems we have enough computing power to let these ideas run wild and see what weighted values come out?

markedathome 7y ago

> If trained on massive sets of source code and briefs (or perhaps a set of unit tests), could it spit out functioning code based on a problem statement? Microsoft's latest Visual Studio 2019 beta (v3) has intellicode[0] support which is supposed to ML "guess" 'code' based on equivalent github similarities. I don't think it is quite near to what you are asking, but certainly part of the way there. 0. https://visualstudio.microsoft.com/services/intellicode/ https://visualstudio.microsoft.com/services/intellicode/

yellow_lead 7y ago
Can't wait for those bugs caused by autocorrect.
- mercer 7y ago
  (typing) 'Robert (autocomplete) 'Robert'); DROP TABLE students;--
  - LifeLiverTransp 7y ago
    Little bobby tables we used to call him. I hope you learned from this to sanitize your database and your AI Input.

Figs 7y ago

Almost certainly not. It will generate things that look like the text of scientific papers on a casual glance, but which will be dream-logic gibberish if you actually read it. The unicorn example makes no sense if you stop and think about what it's actually saying. e.g. a "unicorn" that has four horns? Solves a 200 year old mystery... about something just discovered? So close you could touch their horns... while viewing them from the air? It doesn't make sense if you actually think about it at all, but if you're not really paying attention, the style is like a news report. It's better at making up bullshit, but it's still bullshit.

MegaButts 7y ago
This is a good point, and it's a little concerning to me that we will likely soon be inundated not just with spam from lowly-paid shills, but literal bots generating a near endless supply of confident-sounding nonsense. I am not an expert with machine learning, but it seems (at a very high level) we've done amazingly well at creating models that can recognize and sometimes recreate patterns. But they never seem to have any ability to understand the patterns. I'm sure someone much more knowledgeable could compare it to a child of whatever age (or maybe I'm just completely wrong).
derefr 7y ago
I fully believe that this is 90% of what the human brain is doing to solve the same problems, though. Brains just also have a constraint/filter layer, determining which of these bits of "made up bullshit" should be allowed to rise from the layer generating them, to the motor output layer. Or, to put that another way, flipping the perspective around: our minds could consist of an intelligent, analytical, but utterly unimaginative agent, that sits there listening to a stream of suggestions spewed out by a distinct second agent, one that is "creative" but has no idea about the constraints of things like physics. The brain's analytical agent filters this stream of suggestions, taking notice of the suggestions that seem like they'll make the world change in the ways it "wants"†; and then it does those. † Or, according to modern perceptual-control research, the agent attempts to predict the world that will occur a few seconds in the future, with a bias toward predicting world-states the reward-system has annotated as being rewarding; and then it looks at what motor commands it "would have" issued in that hypothetical world, and actually issues those. The stream of suggestions, in this model, serve as input to feed the generative model of potential world-states; the executive agent then must notice whether the potential world-state is a "possible" world or an "impossible" world (and whether the motor commands required of it are "possible" or "impossible" inputs), and filter out the "impossible" worlds. Under this hypothesis, dreaming is the state when your executive responsible for filtering out "impossible" worlds isn't online. So you just get a continuous "impossible" world generated from the streamed suggestions of the creative-but-stupid bullshit-generating agent, with nothing to tear it down—just as seen in these generative AIs. As consciousness returns, the mental predicted world-state is noticed to be impossible by the now-online analytical agent, and is torn down.