4 ms·
Fourier analysis may help to quickly train more accurate neural networks
- rkhleung 4y agoAhhh. Good old Fourier analysis on the change in model parameters: "low-pass filters that screen out background noise, high-pass filters that help analyze background signals, and Gabor filters that are often used in image processing. The Fourier analysis of the kernels revealed the neural network’s parameters were behaving like a combination of low-pass, high-pass, and Gabor filters."
- deleted 4y ago[deleted]
- galangalalgol 4y agoI can't imagine the architects of the model didn't design the layers and kernels without some idea in mind of what sorts of operations they would be wanting to simulate right? Do people just throw random layers and activation functions together until something works? I'm only on the periphery of dl but those I know that work in it either have a general idea of what equuations they would want to use to do it without dl, or have a biological inspiration in mind. This doesn't seem like a biologically inspired sort of domain.
- fumeux_fume 4y agoConvolutions are learned by the neural network. In general, how layers are composed is more of an educated trial and error process than of an intentional design producing expected results.
- bobbylarrybobby 4y ago> Do people just throw random layers and activation functions together until something works? In a lot of cases, yes. You can start with a reasonable baseline architectural guess, like “convolution should be good for vision” or “attention should be good for language”, but after that it's a lot of guess and check.
- uoaei 4y agoThis may be the case in certain shops or disciplines, but I can assure you, if you have the right insights, you can use statistical and ML principles to achieve better accuracy with orders of magnitude less compute. I consider it a tragedy that we throw huge hogs of models into datacenters and let them churn without giving much thought to improving performance. The climate impacts relative to the real benefits are measurable and depressing for a field so focused on innovation. Usually the 100x model built by a team of "data scientists" over a month is no better than a 1x model built by a couple SMEs over a couple weeks.
- nerdponx 4y agoJust because there are a lot of under-qualified bootcamp data scientists out there who think Pytorch can solve all their problems, that doesn't give you the right to sneer at the cutting edge of ML research on the grounds that you've built some good models in Excel in the past. There are plenty of times when deep learning is the wrong solution, and probably most businesses out there should not ever expect to need any kind of highly compute-intensive model training. But there are also plenty cases where neural networks are the right solution as well (even if not "deep learning" as such). Maybe I'll be convinced if you share some specific examples. Otherwise I'll assume that your comment is just arrogant derision.
- uoaei 4y agoI have a lot to say on this and related topics, but I'll try to keep it on topic to this thread. > sneer at the cutting edge of ML research I was reacting specifically to mis-specifications of what "good" means in these contexts, e.g., "attention should be good for language" from GGP comment. The major advances of these models du jour are not the outputs, but rather the infrastructure that it takes to run them. I salute all those involved in implementation. I hope (and see!) these innovations spill into things that help people and generate real value, for instance, all the things businesses use accurate climate forecasts for. My issue mostly revolves around the discrepancy between value-created-per-FLOPS and cost-per-FLOPS. If we run huge models for what serves for us the same purpose as the TV walls in Fahrenheit 451 do for their owners, what are we adding? If we are constantly running pixel-level segmentation networks on the cameras of automobiles that are already being driven by humans, how is that helping? Stable Diffusion is a real innovation in optimization techniques. This is the only part of recent news that can be considered "cutting edge ML research", as it is an important step in unifying optimization theory with that of non-equilibrium statistical mechanics. The parallels have been noticed there for decades, and the cutting edge of ML research today is unifying them by cribbing aspects of each into the other. The ideas (models) themselves are quite naive. The perceptual space of these models is essentially one-dimensional. This, in the broad strokes, is a terrible model for reality, and so these casual discussions of "intelligence" seem mis-informed, not to mention under-determined. It seems like a lot of mis-informed people speak quite confidently about the things they don't know that they don't know, especially on a site where people who value their autodidactive skills tend to congregate. Inductive bias is the primary term of art that is missing from these discussions. The models du jour do not have any architectural components which represent language per se, only flat representations thereof. The reason LLMs work so well is because of the Transformers inside; multi-head attention is a clever trick to vastly expand the possible data processing techniques as information passes and is distilled through the network into the outputs, but ultimately it is just a fancy way of doing a technique we could call "quasi-random projections" under the domain of linear algebra. The notion of intelligence we entertain in these discussions is hopelessly anthropocentric, and, furthermore, incredibly arrogant. The recent achievements say much more about how easy it is to simulate human-comprehensible language and images than it does about how we are birthing new intelligences. I believe the hype and deployment of these tools is doing more to hurt society in the long term than it is to help it, and I welcome more candid discussions about how simple the rules that make us "intelligent" might be. That seems to me, during my studies, to be the real question.
- cushychicken 4y ago“You’re a nice young man, but it’s Fourier analysis all the way down.” - DSP engineer to ML engineer (probably)
- wnkrshm 4y agoEach time I talk about any problem with our signal theory expert he says: "we could use a Kalman filter" (in jest). That's like saying "hey we could use a matrix".
- aDfbrtVt 4y agoI mean yes and no? Kalman filters fuses information about known system behaviour along with observations in an optimal predictor (for certain types of noise). It uses a matrix in the same way that almost every operation on a computer uses a matrix. Edit: I missed the quip about the Kalman comment being in jest, my reply probably wasn’t necessary but I’ll leave it up.
- blululu 4y agoRichard Hamming has a story where he was shocked/appalled that some folks wanted to include him on a paper after he suggested using Linear Regression in an obvious application of linear regression. The upshot is, if it were obvious it would have been done already. So many times people have done everything they can to avoid using a matrix decomposition or a proper filter.
- cushychicken 4y agoA lot of expertise is really: 1. Knowing when to use the right tool, and 2. Being in the right place at the right time to offer the insight. What’s obvious to the expert is frequently intractable to the outsider. In many cases, I wouldn’t call it so much “avoidance” as “not knowing the solution exists at all”.
- thearn4 4y agoIn my area, every controls engineer says "we can just use a PID". 99% of the time it seems they're not wrong, despite control engineering being a pretty large area of research.
- pfdietz 4y agoFourier analysis generalizes to harmonic analysis, where more general results can be obtained on locally compact topological groups. I wonder if there's some generalization of these ML observations to this more general setting.
- laci37 4y agoWhat a terrible article. This could be potentially interesting in my research but no citation to any original study, just vague descriptions of findings.
- wnkrshm 4y agoThe last link in the article links to the paper [0] the article talks about, I think. [0] https://academic.oup.com/pnasnexus/advance-article/doi/10.1093/pnasnexus/pgad015/6998042 https://academic.oup.com/pnasnexus/advance-article/doi/10.10...
- srcreigh 4y agoIt already is.. can’t find the paper, but some image related deep learning model used sin(x) type of inputs to their model. This was a core improvement. Not unlike how you can linear regress on x1,x1^2,x1^3,x2,x2^2. Encoding prior knowledge about input data as additional inputs helps with machine learning. 1980 called it wants its accomplishments back.
- syrrim 4y agoGpt does this iirc
- srcreigh 4y agoI doubt you’ll find any decent audio or image related models which don’t use sine inputs. It’s ubiquitous. It’s why jpgs and mp3s work.
- leereeves 4y agoI think most transformer models use sin and cos for positional encoding. They were recommended in the original transformer paper, Attention is All You Need. But in that case it's not quite Fourier analysis because those functions aren't applied to the input; they are just constants that are added to the input because transformers are permutation invariant (without some positional encoding). Was there a model that applied sin and cos to the input?
- gessha 4y agoIn a recent-ish paper, there was an analysis of positional encodings using Neural Tangent Kernels which showed positional encodings are very similar to Fourier transformations of the inputs: Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, https://arxiv.org/abs/2006.10739 https://arxiv.org/abs/2006.10739
- gessha 4y agoSin(x)-style encoding is called positional encoding and it’s used in neural radiance fields (NeRFs) and also in transformer-based models.
- jschveibinz 4y agoSomething to ponder… The world was all model-based analysis, and then neural networks came along. Model-based analysis-which requires a thorough understanding of the model mathematics-went out the window. Now, we let the network develop its own model, but it’s based on the data that is used to train it. But what if that data is biased? What if the data doesn’t contain some necessary “truth?” I feel that I t’s like driving a car looking out of the rear window to see where you have been. This can’t be good.
- PeterisP 4y agoIt's not news in ML (machine learning) that whenever you do have a reasonable understanding/model of the underlying task and an algorithm for solving it, you should use that and not try to apply ML to learn a model from data. If an acceptable non-ML solution exists, then usually nobody reasonable would suggest using ML (unless they just want to pad their resume) because the known algorithm will usually do the same thing with much less computation and more reliability and interpretability. However, there are many, many useful real world tasks where we haven't been able to do that (e.g. almost all of image analysis, image generation, a significant part of of text analysis and generation tasks), so for these tasks the (valid) criticism doesn't really matter, as we don't have any reasonable alternatives - if let the network develop its own model, we get a somewhat working solution (with all the risks you mention), and if we don't, then we best we can make is unusably bad. If all the windows in your car are painted opaque black except the rear window, then driving the car by looking out of the rear window to see where you have been can be as good as it gets.
- willy_k 4y agoHowever, for the sake of argument, it would probably be better to wait until you can see out the front, otherwise you’re pretty likely to cause an accident. (My personal opinion is that of yours to, that the good from what we have now outweighs the potential bad).
- nerdponx 4y agoIt's also not a binary choice between "ML" or "hand-crafted model" either. The most successful applications of ML in my own career have always been judiciously building small task-specific models that fit inside a broader system. And there is also usually quite a lot of consideration being given to the model input data as well. Similarly, I have never seen a "just throw a model at it" project succeed in a finite amount of time.
- l33tman 4y agoWith todays networks and training corpus sizes and compute resources available, it's generally better to let the network learn the best transform rather than try to guess what it needs and do it manually first.. unless you have analyzed the problem thoroughly (like I guess they did in this paper). You have several dozens of layers and if you need to "waste" a few low-level layers to learn some transform, the increased training time might offset the man-hours and risk of your meat-based feature-designers to throw something sub-optimal in front of the network..