6 ms·

First off, I'm not an expert - right now, I'm an amateur student of the field. I don't have any credentials or anything published. In other words, what I'm goin

by cr0sh 10y ago

First off, I'm not an expert - right now, I'm an amateur student of the field. I don't have any credentials or anything published. In other words, what I'm going to say probably has little to no merit to it. What I have seen in recent weeks, as I've investigated different news and articles about machine learning, neural networks, and self-driving vehicles, etc - is that it seems like, at least maybe for a certain class of problems, there is a convergence on a generalized pattern, if not a solution, for implementing the network. This pattern or solution seems built off of the Lecun LeNet MNIST convolutional neural network; specifically, the pattern seems to be roughly: (input) -> (1:n - conv layers) -> (flatten) -> (1:n - fully connected layers) -> (output) where: (1:n - conv layers) = (conv layer) -> (loss layer) -> (activation layer) and: (1:n - fully connected layers) = (fully connected layer) -> (loss layer) -> (activation layer) The (loss layer) is optional, but given that the (activation layer) seems to have converged (in most cases I've seen, which is probably not representative) to RELU, the (loss layer) is needed (sometimes) to prevent overfitting (simple dropout can work well, for instance). I don't want to say that this pattern is the "be-all-end-all" of deep learning, but I have found it curious just how many different problems it can be successfully applied to. The RELU operation, while not being differentiable at zero - seems to work regardless (and there are other activation functions similar to RELU that are differentiable - like softplus - if needed). Anyhow - these "building blocks" seem like the basics - the "lego" for deep learning; taking it from art to engineering. As a student, I've been playing with TensorFlow - and recently Keras (a framework for TensorFlow). These tools make it quick and easy to build deep learning neural networks like I described. My gut feeling is that I'm talking from my rear; my level of knowledge isn't that great in this field. So - grain 'o salt and all that. /disclaimer: I am currently enrolled and taking the Udacity Self Driving Car Engineer Nanodegree; in the past I've taken the Udacity CS373 course (2012) and the Standford-sponsored ML Class (2011).

argonaut 10y ago
When you say "loss layer" I think you mean output? The notation is a bit mixed up. Those building blocks are precisely the art (it's both art and engineering) that siavosh is referring to. Everyone is mostly using derivatives of the same basic MNIST architecture of LeCun from 20 years ago ('97), because that's what works (no other justification). With tweaks of course (ReLU, batch norm, etc), also because "they just work." It's funny you should mention dropout, because that's falling out of favor (fully-connected layers are also falling out of favor). Why? Because in many cases it doesn't seem to be necessary if you do batch norm. That's art and engineering (basically you try with dropout and batch norm, or with just batch norm, and just go with batch norm if no difference).
- jpfed 10y ago
  DISCLAIMER: I am the noobiest noob that ever noobed a noob. I'm not sure there's "no other justification". A couple reasons I find the structures given above should be expected to work: 1. It roughly parallels the progression of signals entering our brains through our senses, going through more-or-less convolutional sensory areas of cortex before getting mashed around in the more ad-hoc anterior regions. 2. Consider that convolution layers take advantage of structure in their inputs to radically reduce the number of parameters to train. Would the output of a fully-connected layer have the sort of structure that a convolution layer could take advantage of? (Maybe you could train the fully-connected layer to be exploitable by later conv layers, but doesn't that imply both more training time to get it to do that and more redundant output than you'd want?) It makes more sense to put your convolutions up front so they can take advantage of the correlations in your inputs (perhaps indirectly through the outputs of other conv layers).
  - argonaut 10y ago
    1. Is not a justification, it's a motivation / intuition. It proves little since only the first visual layer (there are 6 visual cortex layers) does convolution-esque things. 2. Justifies convolution over fully-connected. It does not justify convolution in the first place. None of these justify ReLU, max pooling, batch norm. For example, Geoffrey Hinton (probably out of all deep learning researchers the most neuroscience motivated) actually believes max-pooling is terrible, the brain doesn't do it, and it's a shame that it actually works.
    - jpfed 10y ago
      Regarding your first point, I agree that it doesn't constitute a justification for "this is the best way" so much as justification for "this is worth trying". When you say > It proves little since only the first visual layer (there are 6 visual cortex layers) does convolution-esque things. I admit I'm not clear on the roles of the various layers of visual cortex. But I should note that there are several convolution-like steps which in an artificial network would be implemented with separate layers: center-surround detection (happens to occur before V1, but that's not really a problem for what I'm saying- it's a distraction that I skipped past for concision's sake) feeds into the convolution-like operations of oriented-edge detection, corner detection, etc. It's not like our visual system in toto just does one convolution-like step. My second point must have been expressed poorly, because I have apparently completely failed to convey its meaning. It is meant to justify the particular choice of ordering convolutions before fully-connected layers. If you have already chosen to include both types, my point is that it makes sense to put those convolutions before the fully-connected layers.
malux85 10y ago
Nice summary! I would like to augment just a couple of your observations with a few explanations --- I think the reason the architectures that you mention work so well is because they exploit hierarchical structure inherent in the data. Lots of things in the universe have hierarchical structure, e.g. vision has spatial hierarchy, video has spatial and temporal hierarchy, audio has temporal and spatial hierarchy. (side note, thats how a baby learns vision "unsupervised" -- the spatial patterns on the retina have temporal proximity, so the supervisor is "time") RELU is good, but you have to remember how you're vectorizing your data - if you're vectorising is normalizing to 0-1 then RELU fits nicely, but if you're scaling to mean 0 std dev. 1, then half your samples are negative and you'll get information loss when RELU does this: http://7pn4yt.com1.z0.glb.clouddn.com/blog-relu-perf.png http://7pn4yt.com1.z0.glb.clouddn.com/blog-relu-perf.png so remember to keep your vectorizer in mind. > My gut feeling is that I'm talking from my rear; Nope not at all, you have good insight - my email address is in my profile, come join the slack group too (email me, let's go from there)
- isoprophlex 10y ago
  Regarding the information loss..: How about taking the dumb approach and adding a bias? (Turning 'rectified scaling' into an affine transform, I guess)
- Capt-RogerOver 10y ago
  Would you care to expand "(side note, thats how a baby learns vision "unsupervised" -- the spatial patterns on the retina have temporal proximity, so the supervisor is "time")" to a couple of paragraphs? It seems like a very deep and profound thought and would be very interesting to read.
ep103 10y ago
Okay, to someone just looking to get started in deep learning / ML, This post reads like hieroglyphics. Can you suggest resources I can read to better understand it?
- cr0sh 10y ago
  TL;DR - read my post's "tag" and take those courses! --- As you can see in my "tag" on my post - most of what I have learned came from these courses: 1. AI Class / ML Class (Stanford-sponsored, Fall 2011) 2. Udacity CS373 (2012) - https://www.udacity.com/course/artificial-intelligence-for-robotics--cs373 https://www.udacity.com/course/artificial-intelligence-for-r... 3. Udacity Self-Driving Car Engineer Nanodegree (currently taking) - https://www.udacity.com/course/self-driving-car-engineer-nanodegree--nd013 https://www.udacity.com/course/self-driving-car-engineer-nan... For the first two (AI and ML Class) - these two MOOCs kicked off the founding of Udacity and Coursera (respectively). The classes are also available from each: Udacity: Intro to AI (What was "AI Class"): https://www.udacity.com/course/intro-to-artificial-intelligence--cs271 https://www.udacity.com/course/intro-to-artificial-intellige... Coursera: Machine Learning (What was "ML Class"): https://www.coursera.org/learn/machine-learning https://www.coursera.org/learn/machine-learning Now - a few notes: For any of these, you'll want a good understanding of linear algebra (mainly matrices/vectors and the math to manipulate them), stats and probabilities, and to a lessor extent, calculus (basic info on derivatives). Khan Academy or other sources can get you there (I think Coursera and Udacity have courses for these, too - plus there are a ton of other MOOCs plus MITs Open Courseware). Also - and this is something I haven't noted before - but the terms "Artificial Intelligence" and "Machine Learning" don't necessarily mean the same thing. Based on what I have learned, it seems like artificial intelligence mainly revolves around modern understandings of artificial neural networks and deep learning - and is a subset of machine learning. Machine learning, though, also encompasses standard "algorithmic" learning techniques, like logistic and linear regression. The reason why neural networks is a subset of ML, is because a trained neural network ultimately implements a form of logistic (categorization, true/false, etc) or linear regression (range) - depending on how the network is set up and trained. The power of a neural network comes from not having to find all of the dependencies (iow, the "function"); instead the network learns them from the data. It ends up being a "black box" algorithm, but it allows the ability to work with datasets that are much larger and more complex than what the algorithmic approaches allow for (that said, the algorithmic approaches are useful, in that they use much less processing power and are easier to understand - no use attempting to drive a tack with a sledgehammer). With that in mind, the sequence to learn this stuff would probably be: 1. Make sure you understand your basics: Linear Algebra, stats and probabilities, and derivatives 2. Take a course or read a book on basic machine learning techniques (linear regression, logistic regression, gradient descent, etc). 3. Delve into simple artificial neural networks (which may be a part of the machine learning curriculum): understand what feed-forward and back-prop are, how a simple network can learn logic (XOR, AND, etc), how a simple network can answer "yes/no" and/or categorical questions (basic MNIST dataset). Understand how they "learn" the various regression algorithms. 4. Jump into artificial intelligence and deep learning - implement a simple neural network library, learn tensorflow and keras, convolutional networks, and so forth... Now - regarding self-driving vehicles - they necessarily use all of the above, and more - including more than a bit of "mechanical" techniques: Use OpenCV or another machine vision library to pick out details of the road and other objects - which might then be able to be processed by a deep learning CNN - ex: Have a system that picks out "road sign" object from a camera, then categorizes them to "read" them and use the information to make decisions on how to drive the car (come to a stop, or keep at a set speed). In essence, you've just made a portion of Tesla's vehicle assist system (first project we did in the course I am taking now was to "follow lane lines" - the main ingredient behind "lane assist" technology - used nothing but OpenCV and Python). You'll also likely learn stuff about Kalman filters, pathfinding algos, sensor fusion, SLAM, PID controllers, etc. I can't really recommend any books to you, given my level of knowledge. I've read more than a few, but most of them would be considered "out of date". One that is still being used in university level courses is this: http://aima.cs.berkeley.edu/ http://aima.cs.berkeley.edu/ https://www.amazon.com/Artificial-Intelligence-Modern-Approach-3rd/dp/0136042597 https://www.amazon.com/Artificial-Intelligence-Modern-Approa... Note that it is a textbook, with textbook pricing... Another one that I have heard is good for learning neural networks with is: https://www.amazon.com/Make-Your-Own-Neural-Network/dp/1530826608/ https://www.amazon.com/Make-Your-Own-Neural-Network/dp/15308... There are tons of other resources online - the problem is separating the wheat from the chaff, because some of the stuff is outdated or even considered non-useful. There are many research papers out there that can be bewildering. I would say if you read them, until you know which is what, take them all with a grain of salt - research papers and web-sites alike. There's also the problem of finding diamonds in the rough (for instance, LeNet was created in the 1990s - but that was also in the middle of an AI winter, and some of the stuff written at the time isn't considered as useful today - but LeNet is a foundational work of today's ML/AI practices). Now - history: You would do yourself good to understand the history of AI and ML, the debates, the arguments, etc. The base foundational work come from McCulloch and Pitts concept of an artificial neuron, and where that led: https://en.wikipedia.org/wiki/Artificial_neuron https://en.wikipedia.org/wiki/Artificial_neuron Also - Alan Turing anticipated neural networks of the kind that wasn't seen until much later: http://www.alanturing.net/turing_archive/pages/reference%20articles/connectionism/Turing's%20neural%20networks.html http://www.alanturing.net/turing_archive/pages/reference%20a... ...I don't know if he was aware of McCulloch and Pitts work which came prior, as they were coming at the problem from the physiological side of things; a classic case where inter-disciplinary work might have benefitted all (?). You might want to also look into the philosophical side of things - theory of mind stuff, and some of the "greats" there (Minsky, Searle, etc); also look into the books written and edited by Douglas Hofstadter: https://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach https://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach There's also the "lesser known" or "controversial" historical people: * Hugo De Garis (CAM-Brain Machine) * Igor Aleksander * Donald Michie (MENACE) ...among others. It's interesting - De Garis was a very controversial figure, and most of his work, for whatever it is worth - has kinda been swept under the rug. He built a few computers that were FPGA based hardware neural network machines that used cellular automata a-life to "evolve" neural networks. There were only a handful of these machines made; aesthetically, their designs were as "sexy" as the old Cray computers (seriously). Donald Michie's MENACE - interestingly enough - was a "learning computer" made of matchboxes and beads. It essentially implemented a simple neural network that learned how to play (and win at) naughts and crosses (TIC-TAC-TOE). All in a physically (by hand) manipulated "machine". Then there is one guy, who is "reviled" in the old-school AI community on the internet (take a look at some of the old comp.ai newsgroup archives, among others). His nom-de-plume is "Mentifex" and he wrote something called "MIND.Forth" (and translated it to a ton of other languages), that he claimed was a real learning system/program/whatever. His real name is "Arthur T. Murray" - and he is widely considered to be one of the earliest "cranks" on the internet: http://www.nothingisreal.com/mentifex_faq.html http://www.nothingisreal.com/mentifex_faq.html Heck - just by posting this I might be summoning him here! Seriously - this guy gets around. Even so - I'm of the opinion that it might be useful for people to know about him, so they don't go to far down his rabbit-hole; at the same time, I have a small feeling that there might be a gem or two hidden inside his system or elsewhere. Maybe not, but I like to keep a somewhat open mind about these kinds of things, and not just dismiss them out of hand (but I still keep in mind the opinions of those more learned and experienced than me). EDIT: formatting

by cr0sh 10y ago

argonaut 10y ago

When you say "loss layer" I think you mean output? The notation is a bit mixed up. Those building blocks are precisely the art (it's both art and engineering) that siavosh is referring to. Everyone is mostly using derivatives of the same basic MNIST architecture of LeCun from 20 years ago ('97), because that's what works (no other justification). With tweaks of course (ReLU, batch norm, etc), also because "they just work." It's funny you should mention dropout, because that's falling out of favor (fully-connected layers are also falling out of favor). Why? Because in many cases it doesn't seem to be necessary if you do batch norm. That's art and engineering (basically you try with dropout and batch norm, or with just batch norm, and just go with batch norm if no difference).

jpfed 10y ago
DISCLAIMER: I am the noobiest noob that ever noobed a noob. I'm not sure there's "no other justification". A couple reasons I find the structures given above should be expected to work: 1. It roughly parallels the progression of signals entering our brains through our senses, going through more-or-less convolutional sensory areas of cortex before getting mashed around in the more ad-hoc anterior regions. 2. Consider that convolution layers take advantage of structure in their inputs to radically reduce the number of parameters to train. Would the output of a fully-connected layer have the sort of structure that a convolution layer could take advantage of? (Maybe you could train the fully-connected layer to be exploitable by later conv layers, but doesn't that imply both more training time to get it to do that and more redundant output than you'd want?) It makes more sense to put your convolutions up front so they can take advantage of the correlations in your inputs (perhaps indirectly through the outputs of other conv layers).
- argonaut 10y ago
  1. Is not a justification, it's a motivation / intuition. It proves little since only the first visual layer (there are 6 visual cortex layers) does convolution-esque things. 2. Justifies convolution over fully-connected. It does not justify convolution in the first place. None of these justify ReLU, max pooling, batch norm. For example, Geoffrey Hinton (probably out of all deep learning researchers the most neuroscience motivated) actually believes max-pooling is terrible, the brain doesn't do it, and it's a shame that it actually works.
  - jpfed 10y ago
    Regarding your first point, I agree that it doesn't constitute a justification for "this is the best way" so much as justification for "this is worth trying". When you say > It proves little since only the first visual layer (there are 6 visual cortex layers) does convolution-esque things. I admit I'm not clear on the roles of the various layers of visual cortex. But I should note that there are several convolution-like steps which in an artificial network would be implemented with separate layers: center-surround detection (happens to occur before V1, but that's not really a problem for what I'm saying- it's a distraction that I skipped past for concision's sake) feeds into the convolution-like operations of oriented-edge detection, corner detection, etc. It's not like our visual system in toto just does one convolution-like step. My second point must have been expressed poorly, because I have apparently completely failed to convey its meaning. It is meant to justify the particular choice of ordering convolutions before fully-connected layers. If you have already chosen to include both types, my point is that it makes sense to put those convolutions before the fully-connected layers.

malux85 10y ago

Nice summary! I would like to augment just a couple of your observations with a few explanations --- I think the reason the architectures that you mention work so well is because they exploit hierarchical structure inherent in the data. Lots of things in the universe have hierarchical structure, e.g. vision has spatial hierarchy, video has spatial and temporal hierarchy, audio has temporal and spatial hierarchy. (side note, thats how a baby learns vision "unsupervised" -- the spatial patterns on the retina have temporal proximity, so the supervisor is "time") RELU is good, but you have to remember how you're vectorizing your data - if you're vectorising is normalizing to 0-1 then RELU fits nicely, but if you're scaling to mean 0 std dev. 1, then half your samples are negative and you'll get information loss when RELU does this: http://7pn4yt.com1.z0.glb.clouddn.com/blog-relu-perf.png http://7pn4yt.com1.z0.glb.clouddn.com/blog-relu-perf.png so remember to keep your vectorizer in mind. > My gut feeling is that I'm talking from my rear; Nope not at all, you have good insight - my email address is in my profile, come join the slack group too (email me, let's go from there)

isoprophlex 10y ago
Regarding the information loss..: How about taking the dumb approach and adding a bias? (Turning 'rectified scaling' into an affine transform, I guess)
Capt-RogerOver 10y ago
Would you care to expand "(side note, thats how a baby learns vision "unsupervised" -- the spatial patterns on the retina have temporal proximity, so the supervisor is "time")" to a couple of paragraphs? It seems like a very deep and profound thought and would be very interesting to read.

ep103 10y ago

Okay, to someone just looking to get started in deep learning / ML, This post reads like hieroglyphics. Can you suggest resources I can read to better understand it?

cr0sh 10y ago
TL;DR - read my post's "tag" and take those courses! --- As you can see in my "tag" on my post - most of what I have learned came from these courses: 1. AI Class / ML Class (Stanford-sponsored, Fall 2011) 2. Udacity CS373 (2012) - https://www.udacity.com/course/artificial-intelligence-for-robotics--cs373 https://www.udacity.com/course/artificial-intelligence-for-r... 3. Udacity Self-Driving Car Engineer Nanodegree (currently taking) - https://www.udacity.com/course/self-driving-car-engineer-nanodegree--nd013 https://www.udacity.com/course/self-driving-car-engineer-nan... For the first two (AI and ML Class) - these two MOOCs kicked off the founding of Udacity and Coursera (respectively). The classes are also available from each: Udacity: Intro to AI (What was "AI Class"): https://www.udacity.com/course/intro-to-artificial-intelligence--cs271 https://www.udacity.com/course/intro-to-artificial-intellige... Coursera: Machine Learning (What was "ML Class"): https://www.coursera.org/learn/machine-learning https://www.coursera.org/learn/machine-learning Now - a few notes: For any of these, you'll want a good understanding of linear algebra (mainly matrices/vectors and the math to manipulate them), stats and probabilities, and to a lessor extent, calculus (basic info on derivatives). Khan Academy or other sources can get you there (I think Coursera and Udacity have courses for these, too - plus there are a ton of other MOOCs plus MITs Open Courseware). Also - and this is something I haven't noted before - but the terms "Artificial Intelligence" and "Machine Learning" don't necessarily mean the same thing. Based on what I have learned, it seems like artificial intelligence mainly revolves around modern understandings of artificial neural networks and deep learning - and is a subset of machine learning. Machine learning, though, also encompasses standard "algorithmic" learning techniques, like logistic and linear regression. The reason why neural networks is a subset of ML, is because a trained neural network ultimately implements a form of logistic (categorization, true/false, etc) or linear regression (range) - depending on how the network is set up and trained. The power of a neural network comes from not having to find all of the dependencies (iow, the "function"); instead the network learns them from the data. It ends up being a "black box" algorithm, but it allows the ability to work with datasets that are much larger and more complex than what the algorithmic approaches allow for (that said, the algorithmic approaches are useful, in that they use much less processing power and are easier to understand - no use attempting to drive a tack with a sledgehammer). With that in mind, the sequence to learn this stuff would probably be: 1. Make sure you understand your basics: Linear Algebra, stats and probabilities, and derivatives 2. Take a course or read a book on basic machine learning techniques (linear regression, logistic regression, gradient descent, etc). 3. Delve into simple artificial neural networks (which may be a part of the machine learning curriculum): understand what feed-forward and back-prop are, how a simple network can learn logic (XOR, AND, etc), how a simple network can answer "yes/no" and/or categorical questions (basic MNIST dataset). Understand how they "learn" the various regression algorithms. 4. Jump into artificial intelligence and deep learning - implement a simple neural network library, learn tensorflow and keras, convolutional networks, and so forth... Now - regarding self-driving vehicles - they necessarily use all of the above, and more - including more than a bit of "mechanical" techniques: Use OpenCV or another machine vision library to pick out details of the road and other objects - which might then be able to be processed by a deep learning CNN - ex: Have a system that picks out "road sign" object from a camera, then categorizes them to "read" them and use the information to make decisions on how to drive the car (come to a stop, or keep at a set speed). In essence, you've just made a portion of Tesla's vehicle assist system (first project we did in the course I am taking now was to "follow lane lines" - the main ingredient behind "lane assist" technology - used nothing but OpenCV and Python). You'll also likely learn stuff about Kalman filters, pathfinding algos, sensor fusion, SLAM, PID controllers, etc. I can't really recommend any books to you, given my level of knowledge. I've read more than a few, but most of them would be considered "out of date". One that is still being used in university level courses is this: http://aima.cs.berkeley.edu/ http://aima.cs.berkeley.edu/ https://www.amazon.com/Artificial-Intelligence-Modern-Approach-3rd/dp/0136042597 https://www.amazon.com/Artificial-Intelligence-Modern-Approa... Note that it is a textbook, with textbook pricing... Another one that I have heard is good for learning neural networks with is: https://www.amazon.com/Make-Your-Own-Neural-Network/dp/1530826608/ https://www.amazon.com/Make-Your-Own-Neural-Network/dp/15308... There are tons of other resources online - the problem is separating the wheat from the chaff, because some of the stuff is outdated or even considered non-useful. There are many research papers out there that can be bewildering. I would say if you read them, until you know which is what, take them all with a grain of salt - research papers and web-sites alike. There's also the problem of finding diamonds in the rough (for instance, LeNet was created in the 1990s - but that was also in the middle of an AI winter, and some of the stuff written at the time isn't considered as useful today - but LeNet is a foundational work of today's ML/AI practices). Now - history: You would do yourself good to understand the history of AI and ML, the debates, the arguments, etc. The base foundational work come from McCulloch and Pitts concept of an artificial neuron, and where that led: https://en.wikipedia.org/wiki/Artificial_neuron https://en.wikipedia.org/wiki/Artificial_neuron Also - Alan Turing anticipated neural networks of the kind that wasn't seen until much later: http://www.alanturing.net/turing_archive/pages/reference%20articles/connectionism/Turing's%20neural%20networks.html http://www.alanturing.net/turing_archive/pages/reference%20a... ...I don't know if he was aware of McCulloch and Pitts work which came prior, as they were coming at the problem from the physiological side of things; a classic case where inter-disciplinary work might have benefitted all (?). You might want to also look into the philosophical side of things - theory of mind stuff, and some of the "greats" there (Minsky, Searle, etc); also look into the books written and edited by Douglas Hofstadter: https://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach https://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach There's also the "lesser known" or "controversial" historical people: * Hugo De Garis (CAM-Brain Machine) * Igor Aleksander * Donald Michie (MENACE) ...among others. It's interesting - De Garis was a very controversial figure, and most of his work, for whatever it is worth - has kinda been swept under the rug. He built a few computers that were FPGA based hardware neural network machines that used cellular automata a-life to "evolve" neural networks. There were only a handful of these machines made; aesthetically, their designs were as "sexy" as the old Cray computers (seriously). Donald Michie's MENACE - interestingly enough - was a "learning computer" made of matchboxes and beads. It essentially implemented a simple neural network that learned how to play (and win at) naughts and crosses (TIC-TAC-TOE). All in a physically (by hand) manipulated "machine". Then there is one guy, who is "reviled" in the old-school AI community on the internet (take a look at some of the old comp.ai newsgroup archives, among others). His nom-de-plume is "Mentifex" and he wrote something called "MIND.Forth" (and translated it to a ton of other languages), that he claimed was a real learning system/program/whatever. His real name is "Arthur T. Murray" - and he is widely considered to be one of the earliest "cranks" on the internet: http://www.nothingisreal.com/mentifex_faq.html http://www.nothingisreal.com/mentifex_faq.html Heck - just by posting this I might be summoning him here! Seriously - this guy gets around. Even so - I'm of the opinion that it might be useful for people to know about him, so they don't go to far down his rabbit-hole; at the same time, I have a small feeling that there might be a gem or two hidden inside his system or elsewhere. Maybe not, but I like to keep a somewhat open mind about these kinds of things, and not just dismiss them out of hand (but I still keep in mind the opinions of those more learned and experienced than me). EDIT: formatting