4 ms·
What you’re describing would be an Expert System, part of GOFAI. Neural nets work by generalizing. This is important because GOFAI systems are brittle. We are
by bckr 3y ago
What you’re describing would be an Expert System, part of GOFAI.
Neural nets work by generalizing. This is important because GOFAI systems are brittle. We are never able to encode all the relevant rules for real world situations by hand.
Also, note that the synthetic data is generated from rules about how the world looks, not rules about what the model should do.
There may still be some direction for zapping the rules-of-the-world right into the circuitry of, say, a subnet of the model, without doing backprop. Something like a one-shot initialization of the network to approximate the given distribution.
- moffkalast 3y agoI think what he's saying is that currently we can get one of these expert systems to create 1000 examples with the encoded rules, which a NN might then generalize correctly or it may come up with a completely batshit insane ruleset that overfits on that data. There's no way for us to actually feed the NN the rules directly, like one would give them to a human brain.
- varjag 3y agoThe main issue with expert systems appeared to be not the amount of necessary rules but pathway dependency on representation model for the problem domain. If it mapped well, it could perform exceptionally well. If not, you got into all sort of combinatory troubles real quick and had to wing it (or as they used to say in the 1980s, to use heuristic rules). The combinatory bit due to its branching nature lends itself poorly to leveraging SIMD hardware as statistical solutions do. And getting a right representation model is a substantial challenge. Everyone knew the importance of that ('knowledge engineering') by the onset of AI Winter but nobody had any good methods.
- bckr 3y agoThanks for going one deeper. Would love to learn more about this representation model problem.