6 ms·
Neural networks need data to learn, even if it’s fake
- carlmr 3y agoSounds like we've invented dreaming now.
- williamtrask 3y agoFake isn’t the right word here. The synthetic images contain real structure useful for learning.
- ilyt 3y agoHallucinated then
- sim7c00 3y agoagree. fake sounds prone to erroneous data, while some systems are fairly easy to simulate, generating real data from a simulation. for more complex scenarios this is trickier, and a faulty simulation moght leave subtle issues in the networks functioning. if u dont have real.data to verify it with u will be in the dark about real accuracy. depending on the networks purpose, this moght be really dangerous.
- lettergram 3y agoLol we actually showed synthetic data often produced better models depending what you’re trying to do https://www.capitalone.com/tech/software-engineering/why-you-dont-necessarily-need-data-for-data-science/ https://www.capitalone.com/tech/software-engineering/why-you... Part of the reason was models generating synthetic data hone in on more key differentiable features
- mabbo 3y agoWhat's weird about this to me is the flow of information, knowledge from one place to another. We're using software that has encoded the rules we want to use to generate data. The ML is using the data to infer the rules. It seems like we need a better way to give our ML algorithms the rules that we already know rather than this processing-intensive process. But perhaps that's the cost of using neural networks- they can learn any function, but they can only learn from data, examples.
- Timon3 3y agoI think "giving the rules we already know" is only a good idea if we are 100% sure that we know 100% of the rules necessary with 100% accuracy. The fuzzyness of NNs is what in many cases allows them to work as well as they do, and training on close-to-perfect data is going to preserve this aspect better than close-to-perfect rules could.
- bckr 3y agoWhat you’re describing would be an Expert System, part of GOFAI. Neural nets work by generalizing. This is important because GOFAI systems are brittle. We are never able to encode all the relevant rules for real world situations by hand. Also, note that the synthetic data is generated from rules about how the world looks, not rules about what the model should do. There may still be some direction for zapping the rules-of-the-world right into the circuitry of, say, a subnet of the model, without doing backprop. Something like a one-shot initialization of the network to approximate the given distribution.
- moffkalast 3y agoI think what he's saying is that currently we can get one of these expert systems to create 1000 examples with the encoded rules, which a NN might then generalize correctly or it may come up with a completely batshit insane ruleset that overfits on that data. There's no way for us to actually feed the NN the rules directly, like one would give them to a human brain.
- varjag 3y agoThe main issue with expert systems appeared to be not the amount of necessary rules but pathway dependency on representation model for the problem domain. If it mapped well, it could perform exceptionally well. If not, you got into all sort of combinatory troubles real quick and had to wing it (or as they used to say in the 1980s, to use heuristic rules). The combinatory bit due to its branching nature lends itself poorly to leveraging SIMD hardware as statistical solutions do. And getting a right representation model is a substantial challenge. Everyone knew the importance of that ('knowledge engineering') by the onset of AI Winter but nobody had any good methods.
- orangepurple 3y agoCMU tried to build a self-learning expert system called NELL but it failed. Never-Ending Language Learning system (NELL) is a semantic machine learning system developed by a research team at Carnegie Mellon University, and supported by grants from DARPA, Google, NSF, and CNPq with portions of the system running on a supercomputing cluster provided by Yahoo! In his 2019 book "Human Compatible", Stuart Russell commented that 'Unfortunately NELL has confidence in only 3 percent of its beliefs and relies on human experts to clean out false or meaningless beliefs on a regular basis—such as its beliefs that “Nepal is a country also known as United States” and "value is an agricultural product that is usually cut into basis."'
- Buttons840 3y agoI know this is different than what the article is talking about, but I've wondered if something like a GPTZero would be possible. Train a model to reproduce random sequences, and see if it's able to learn any patterns that are also useful for language or other real-world tasks. If there's nothing to be gained from training on pure noise, what is the most generic and generally useful sequence we could train on?
- mycall 3y agoIs there not an algebraic function approach to neural networks instead of discrete datasets? Instead of multidimensional vectors, have balanced parametric equations or similar.
- informal007 3y agowhy the machine can’t learn that something is fake like person Person learn thing base on thing that we see, listen, we know something is fake , why can’t machine. It’s a wired question