6 ms·
Understanding Neural Network, Visually
- brudgers 9mo agoThe original Show HN, https://news.ycombinator.com/item?id=44633725 https://news.ycombinator.com/item?id=44633725
- 4fterd4rk 9mo agoGreat explanation, but the last question is quite simple. You determine the weights via brute force. Simply running a large amount of data where you have the input as well as the correct output (handwriting to text in this case).
- ggambetta 9mo ago"Brute force" would be trying random weights and keeping the best performing model. Backpropagation is compute-intensive but I wouldn't call it "brute force".
- Ygg2 9mo ago"Brute force" here is about the amount of data you're ingesting. It's no Alpha Zero, that will learn from scratch.
- jazzpush2 9mo agoWhat? Either option requires sufficient data. Brute force implies iterating over all combinations until you find the best weights. Back-prop is an optimization technique.
- Ygg2 9mo agoIn context of grandparents post. > You determine the weights via brute force. Simply running a large amount of data where you have the input as well as the correct output Brute force just means guessing all possible combinations. A dataset containing most human knowledge is about as brute force as you can get. I'm fairly sure that Alpha Zero data is generated by Alpha Zero. But it's not an LLM.
- fc417fc802 9mo agoNo, a large dataset does not make something brute force. Rather than backprop, an example of brute force might be taking a single input output pair then systematically sampling the model parameter space to search for a sufficiently close match. The sampling stage of Evolution Strategies at least bears a resemblance but even that is still a strategic gradient descent algorithm. Meanwhile backprop is about as far from brute force as you can get.
- esafak 9mo agoThis is just scratching the surface -- where neural networks were thirty years ago: https://en.wikipedia.org/wiki/MNIST_database https://en.wikipedia.org/wiki/MNIST_database If you want to understand neural networks, keep going.
- abrookewood 9mo agoWhich, if you are trying to learn the basics, is actually a great place to start ...
- helloplanets 9mo agoFor the visual learners, here's a classic intro to how LLMs work: https://bbycroft.net/llm https://bbycroft.net/llm
- tpdly 9mo agoLovely visualization. I like the very concrete depiction of middle layers "recognizing features", that make the whole machine feel more plausible. I'm also a fan of visualizing things, but I think its important to appreciate that some things (like 10,000 dimension vector as the input, or even a 100 dimension vector as an output) can't be concretely visualized, and you have to develop intuitions in more roundabout ways. I hope make more of these, I'd love to see a transformer presented more clearly.
- ge96 9mo agoI like the style of the site it has a "vintage" look Don't think it's moire effect but yeah looking at the pattern
- Bengalilol 9mo agoLucky you! <https://visualrambling.space/dithering-part-1/ https://visualrambling.space/dithering-part-1/> <https://visualrambling.space/dithering-part-2/ https://visualrambling.space/dithering-part-2/>
- ge96 9mo agoOh god my eyes! As it zooms in (ha) That's cool, rendering shades in the old days Man those graphics are so good damn
- javaskrrt 9mo agovery cool stuff
- cwt137 9mo agoThis visualizations reminds me of the 3blue1brown videos.
- giancarlostoro 9mo agoI was thinking the same thing. Its at least the same description.
- pks016 9mo agoGreat visualization!
- anon291 9mo agoNice visuals, but misses the mark. Neural networks transform vector spaces, and collect points into bins. This visualization shows the structure of the computation. This is akin to displaying a Matrix vector multiplication in Wx + b notation, except W,x,and b have more exciting displays. It completely misses the mark on what it means to 'weight' (linearly transform), bias (affine transform) and then non-linearly transform (i.e, 'collect') points into bins
- titzer 9mo ago> but misses the mark It doesn't match the pictures in your head, but it nevertheless does present a mental representation the author (and presumably some readers) find useful. Instead of nitpicking, perhaps pointing to a better visualization (like maybe this video: https://www.youtube.com/watch?v=ChfEO8l-fas https://www.youtube.com/watch?v=ChfEO8l-fas) could help others learn. Otherwise it's just frustrating to read comments like this.
- fc417fc802 9mo agoIt's not nitpicking to point out major missing pieces. Comments like this might tend to come across as critical but they are incredibly valuable for any reader that doesn't know what he doesn't know.
- titzer 9mo agoIt just sucks to put in a ton of work into something and then show it off to people but the first reaction is someone comes out of the woodwork to loudly crow that it "misses the mark" and is somehow crap. It's a completely avoidable experience when the community has a more generally positive attitude. All it takes is a little different phrasing of exactly the same feedback, but with a positive emotional and encouraging tone. For example, instead of writing: > Nice visuals, but misses the mark. Neural networks transform vector spaces, and collect points into bins. This visualization shows the structure of the computation. This is akin to displaying a Matrix vector multiplication in Wx + b notation, except W,x,and b have more exciting displays. > It completely misses the mark on what it means to 'weight' (linearly transform), bias (affine transform) and then non-linearly transform (i.e, 'collect') points into bins Here's more or less the same comment but with a completely different attitude: > Oh wow, that's cool! That must have been a ton of work to put together. That got me thinking as to how it's akin to Matrix vector multiplication in Wx + b notation, except W,x,and b have more exciting displays. > An idea I am wondering about but don't know how to solve is what it means to 'weight' (linearly transform), bias (affine transform) and then non-linearly transform (i.e, 'collect') points into bins. > Here's some other links that are related and cool: ... > Cheers, nice work! Let's not crap on people's work so readily. After all, we have no idea about who the author is. Maybe it's a teenager or a university student and this was their first project. It's really a jarring and demoralizing experience to have your first visualization immediately crapped on.
- artemonster 9mo agoI get 3fps on my chrome, most likely due to disabled HW acceleration
- nerdsniper 9mo agoHigh FPS on Safari M2 MBP.
- 8cvor6j844qw_d6 9mo agoOh wow, this looks like a 3d render of a perceptron when I started reading about neural networks. I guess essentially neural networks are built based on that idea? Inputs > weight function to to adjust the final output to desired values?
- adammarples 9mo agoYes, vanilla neural networks are just lots of perceptrons
- sva_ 9mo agoA neural network is basically a multilayer perceptron https://en.wikipedia.org/wiki/Multilayer_perceptron https://en.wikipedia.org/wiki/Multilayer_perceptron
- mr_toad 9mo agoThe layers themselves are basically perceptrons, not really any different to a generalized linear model. The ‘secret sauce’ in a deep network is the hidden layer with a non-linear activation function. Without that you could simplify all the layers to a linear model.
- jazzpush2 9mo agoI love this visual article as well: https://mlu-explain.github.io/neural-networks/ https://mlu-explain.github.io/neural-networks/
- deleted 9mo ago[deleted]
- jetfire_1711 9mo agoSpent 10 minutes on the site and I think this is where I'll start my day from next week! I just love visual based learning.
- shrekmas 9mo agoAs someone who does not use Twitter, I suggest adding RSS to your site.
- swframe2 9mo agoThis Welch Labs video is very helpful: https://www.youtube.com/watch?v=qx7hirqgfuU https://www.youtube.com/watch?v=qx7hirqgfuU
- chan1 9mo agoSuper cool visualization Found this vid by 3Blue1Brown super helpful for visualizing transformers as well. https://www.youtube.com/watch?v=wjZofJX0v4M&t=1198s https://www.youtube.com/watch?v=wjZofJX0v4M&t=1198s
- bilbo-b-baggins 9mo agoTheir series on LLMs, neural nets, etc., is amazing.
- atultw 9mo agoNice work
- vivzkestrel 9mo ago- while impressive, it still doesnt tell me why a neural network is architected the way it is and that my bois is where this guy comes in https://threads.championswimmer.in/p/why-are-neural-networks-architected https://threads.championswimmer.in/p/why-are-neural-networks... - make a visualization of the article above and it would be the biggest aha moment in tech
- stuxnet79 9mo agoRegarding architecture, I don't believe a satisfying "why" is in the cards. Conceptually neural networks are quite simple. You can think of each neural net as a daisy chain of functions that can be efficiently tuned to fulfill some objective via backpropagation. Their effectiveness (in the dimensions we care about) are more a consequence of the explosion of compute and data that occured in the 2010s. In my view, every hyped architecture was what yielded the best accuracy given the compute resources available at the time. It's not a given that these architectures are the most optimal and we certainly don't always fully understand why they work. Most of the innovations in this space over the past 15 years have come from private companies that have lacked a strong research focus but are resource rich (endless compute and data capacity).
- vicentwu 9mo agoI like the CRT-like filter effect.
- droidist2 9mo agoReally cool. The animations within a frame work well.
- KYRRO 9mo agoI have a question. With the logic of neural networks, and pattern recognition, is it not then possible to "predict" everything in everything? Like predicting the future to an exact "thing"? Is this not a tool to manipulate for instace the stock market?
- stuxnet79 9mo agoWell nothing is stopping you from attempting to predict everything with neural networks but that doesn't mean your predictions will be (1) good (2) consistently useful or (3) economical. Transformer models for example suffer from (2) and especially (3) in their current iteration.
- TuringTest 9mo agoIt is possible to try it, and some people do (high speed trading is just that, plus taking advantage of privileged information that speed provides to react before anyone else). However there are two fundamental problems to computational predictions. The first one obviously is accuracy. A model is a compressed memorization of everything observed so far; a prediction with it is just projecting into the future the observed patterns. In a chaotic system, that goes only so far; the most regular, predictable patterns are obvious to everybody and give less return, and the chaotic system states where prediction would be more valuable are the less reliable. You cannot build a perfect oracle that would fix that. The second problem is more insidious. Even if you were able to build a perfect oracle, acting on its predictions would become part of the system itself. That would change the outcomes, making the system behave in a different way as it was trained, and thus less reliable. If several people do it at the same time, there's no way to retrain the model to take into account the new behaviour. There's the possibility (but not a guarantee) to reach a fixed point, that a Nash equilibrium would appear where such system becomes into a stable cycle, but that's not likely in a changing environment where everybody tries to outdo everyone else.
- deleted 9mo ago[deleted]
- KYRRO 9mo ago
- smoode 9mo agoam looking for pengasus trying to heck in to a phone can you help me with that
- harel 9mo agoThis reminds me of a "web site" (remember those) I used to visit a lot years ago, trying to understand Neural Networks and genetic algorithms: http://www.ai-junkie.com/ann/evolved/nnt1.html http://www.ai-junkie.com/ann/evolved/nnt1.html This is old. Perhaps late 90s or early 00. The top domain still uses Flash. But the same OCR example is used to teach the concept. For some reason, that site made it all click for me.
- whilefalse 9mo agoI made a similar thing recently: https://lighthousesoftware.co.uk/projects/neural-network/ https://lighthousesoftware.co.uk/projects/neural-network/ I wanted to get a feel for what specific neurons are actually looking at, and how disabling/enabling them affects the final output. It runs a little MNIST model in the browser, but lets you turn pixels and neurons on/off, and examine the weight and activation patterns of each neuron and how it contributes to each prediction. Helped me get more of an intuitive sense of what is going on inside.
- minaguib 9mo agoCool intro and visuals. Where it ends "how do we calculate the weights ?" is fairly simple. Start completely randomly and compare output to known truth. When it's incorrect, you beat the model up pretty badly and repeat again. Eventually you get the correct answer pretty consistently. ... and by "beat it up" I mean tweak the weights - totally randomly will work but will take a long time (brute force), so we add a bit of intelligence to see which direction to tweak via some algorithms (backpropagation, gradient descent).
- dandelionv1bes 9mo agoIf you’re interested in an easy explainer for backprop, then highly recommend math for deep learning by Kneusel. Finished it recently and you can apply the chain rule backwards by hand with a tiny NN, and code one too.