6 ms·
The Smallest Brain You Can Build: A Perceptron in Python
- deleted 4mo ago[deleted]
- esafak 4mo agoIf you want to learn the fundamentals of ML I recommend a book, such as Deep Learning: Foundations and Concepts by Chris Bishop. If you insist on staying online, one option is https://course.fast.ai/ https://course.fast.ai/ If you don't know ML I don't think you're going to learn much through ad hoc demos.
- rishabhaiover 4mo agoThis book equipped me with the right intuition and tools to visualize machine learning. I wish I was smart enough to hold it all together.
- andai 4mo ago>I wish I was smart enough to hold it all together. I used to have a wife, but they took her in the divorce! The human mind isn't very good at correlating its contents[0]. You can "know" something for years without realizing its implications. The human mind traverses its knowledge like a man with a small flashlight in total darkness. Our beam of attention is small and narrow, so you need to put the right things in it, or the magic doesn't happen. This has important implications for learning. I don't know what they are though. Probably something like, "you can know something without knowing what it means." You haven't connected it to the things it's supposed to be connected to yet. I don't know how to fix that though. (Something involving the Feynman technique, maybe?) [0] H.P. Lovecraft quote - https://www.goodreads.com/quotes/193944-the-most-merciful-thing-in-the-world-i-think-is https://www.goodreads.com/quotes/193944-the-most-merciful-th...
- mysterydip 4mo agoChecked out the book on your recommendation, and they even have a free online option on their site! Very generous: https://www.bishopbook.com/ https://www.bishopbook.com/
- stuxnet79 4mo agoI didn't know Bishop had released a new textbook. I will have to take a look at it. I wasn't the biggest fan of his Pattern Recognition book as I found it overly dense. I much preferred the Murphy and Alpaydin books. EDIT: His son is co-author?
- zxexz 4mo agoI still find his pattern recognition book useful and informative. It may be dense, but some of us consider that a positive for 'reference' literature. That book was one of very few that still holds up well fr when it was published - truly in on of the last "dark ages" of ML. I think those down voting you are perhaps overly eager. I upvoted. Grab "Deep Learning" - you'll find it useful, imteresting, and likely less 'dense' in the negative sense!
- stuxnet79 4mo agoAppreciate your comment. I skimmed the online version and it covers all the 2010s era developments all the way to Transformers which is enough to earn it a spot on my bookshelf. > Grab "Deep Learning" - you'll find it useful, imteresting, and likely less 'dense' in the negative sense! Absolutely! I just ordered it and it's enroute :)
- DevarshRanpara 4mo agoThis fast AI course looks soo good man! Definitely I will start learning soon. Thank you!
- llimllib 4mo agoI remember sitting in the senior study lounge reading the previous Bishop book and implementing the perceptron from it, 22 years ago: https://github.com/llimllib/personal_code/blob/945b017b2915ccd148bb09a0f93d0ab9bdb703a9/python/perceptron/perceptron_old.py#L34 https://github.com/llimllib/personal_code/blob/945b017b2915c... (before numarray and numpy merged!)
- vismit2000 4mo agoEven 'The Welch Labs Illustrated Guide to AI' is pretty good: https://www.welchlabs.com/store/illustrated-guide-to-ai https://www.welchlabs.com/store/illustrated-guide-to-ai
- b33j0r 4mo agoOkay, it’s conscious. But can it run doom? I rest my case.
- Waterluvian 4mo agoCan you run Doom? Let’s find out!
- andblac 4mo agoI know you're joking, but if you really wanted to, you can if you have a network of these, since you can build NAND gates from perceptrons. If you have NAND gates, then you can build any other gates from these and then you can build a computer [1]. [1] https://www.nand2tetris.org/ https://www.nand2tetris.org/
- hbwang2076 4mo ago[dead]
- knightops_dev 4mo ago[flagged]
- karinatran 4mo ago[dead]
- EvanXue 4mo ago[flagged]
- trekhleb 4mo agoNice and minimalistic I played with similar approach in JavaScript and built a NanoNeuron https://github.com/trekhleb/nano-neuron https://github.com/trekhleb/nano-neuron (it is more verbose than Python though)
- jkwang 4mo ago[flagged]
- ankit84 4mo agoI learnt a lot today from the interactive demo. You have the best clarity and right skill to educate
- DevarshRanpara 4mo agoThank you, I will try to make more demo on other concepts.
- charcircuit 4mo agoI can build a smaller brain. f(x) = 0.
- rippeltippel 4mo agoThat's great, now make it learn something :)
- jeffwass 4mo agoIt's the simplest AI nihilist!
- moffkalast 4mo agoYeah, this is small brain time.
- lioeters 4mo agoI take your brain, add a couple more rules, and presto! It can perform any computation. Ix = x Kxy = x Sxyz = xz(yz)
- opem 4mo agoThis brain is interesting. Basically you get a no for everything you ask, right?
- charcircuit 4mo agoYou get whatever you assign 0 to. You can combine it with other simple building blocks like f(x) = ~x to do large calculations to answer more complicated questions.
- haeseong 4mo ago[dead]
- Bimos 4mo ago> A perceptron *is* the smallest brain you can build. > In 1958, a researcher named Frank Rosenblatt built a machine *he called* the perceptron. > It was *inspired* by a single brain cell, a neuron.
- lmf4lol 4mo agoYes . But at least the post seems to be written by OP himself! and its an a great learning resource - which is arguably more important :-)
- Lerc 4mo agoI did a lecture once which included a 5 minute whirlwind tour of neural net history. I included a remark about how time travellers would find Rosenblatt a better target than Miles Dyson. I was never quite sure on how close, or over, the line that was on appropriateness. It was definitely thought provoking.
- zkmon 4mo agoThe IF statement is the root creator of software programming. It has the ability to compare two values against each other and branch out to blocks of instructions. So it is perceiving (reading), decision making and routing - all that which differentiate life from inanimate objects. The AI agents perform the exact same loop, by delegating the first two steps to a model. Going further backwards, the transistor (or a PNP junction) is the hardware level enabler of the IF statement. The action (switching) driven by the current which in turn controls other switches, is the first manifestation of "observe and act" by inanimate things at the speed of electricity. Mechanical equivalents existed ofcourse - speed of a governer which controls the flow of fuel which in turn controls the speed of the governer.
- utopiah 4mo agoYou might enjoy playing with Turing Tumble.
- gpderetta 4mo agoit not really an if statement here in a perceptron though. It is more akin a logic gate. A transistor (driven to saturation) is a much better model.
- BatteryMountain 4mo agoSo, what if, we build a stack/set of transistors in same shape as a trained model? It would eliminate most of the software stack too and should run very fast. No memory/gpu required, the chip acts as both storage and processing device, purpose built to be physical model of a trained model.
- tomtom1337 4mo agoThis is literally what talaas has done with chatjimmy.ai. Try it, it's llama 3.1 8B at 16000 tokens per second. chatjimmy.ai https://taalas.com/the-path-to-ubiquitous-ai/ https://taalas.com/the-path-to-ubiquitous-ai/
- jupr 4mo ago
- rahen 4mo agoIn the early days of machine learning (before the first AI winter), networks like this were often implemented and trained in hardware: https://en.wikipedia.org/wiki/ADALINE https://en.wikipedia.org/wiki/ADALINE That was the first thing that came to mind when I read "the smallest brain you can build". Nowadays, that "small brain" would likely be built on a breadboard using op-amps instead.
- Schlagbohrer 4mo agoAmazing and anachronistic to see something like that from 1960. And then it makes me wonder why there wasn't more progress on neural nets being used for many things prior to the 21st century. (I haven't read the history of the AI winters but I have heard of them)
- j_bum 4mo agoThis doc on Ilya Sutskever & Geoffrey Hinton gives a great background on the progression of deep learning over the past decades [0]. Tl;dr - compute was the bottleneck. I am not associated with this channel/video, just love it. I’ve shared it here before. [0] https://youtu.be/glWvwvhZkQ8?si=XjcwWWy43305tl6O https://youtu.be/glWvwvhZkQ8?si=XjcwWWy43305tl6O
- mr_toad 4mo ago> why there wasn't more progress on neural nets being used for many things prior to the 21st century They were simply too computationally expensive to train for the limited things they could do. It wasn’t until we had the ability to train large neural networks on commodity hardware that things really took off.
- rahen 4mo agoThe first AI winter was largely triggered by Minsky in a book he published in 1969, which mathematically proved that single-layer perceptrons couldn't solve non-linear problems. Favorite quote: "Our intuitive judgment is that the extension [to multilayer systems] is sterile." Yet we had the computational power to run backpropagation in the 1960s and small Transformers in the 1970s (I'm the author of both): https://github.com/dbrll/Xortran https://github.com/dbrll/Xortran (backprop on IBM 1130, 60s) https://github.com/dbrll/ATTN-11 https://github.com/dbrll/ATTN-11 (Transformer on PDP-11, 70s) What was missing wasn't the raw processing power, but the ideas and algorithms themselves. Because funding and research were completely discouraged during the AI winter, neural networks research was left dormant and we lost two decades.
- warengonzaga 4mo agoThis is amazing insight, thanks for sharing!
- CyberDildonics 4mo agoThis is your first comment in six years, what is amazing about it?
- deleted 4mo ago[deleted]
- infoinlet 4mo ago[flagged]
- opem 4mo agoI have still so many questions left, but regardless of that it was a great read. Thanks for sharing!
- DevarshRanpara 4mo agoPlease throw them here, I like to play around with questions, simplify them, and will try to write next part of this!
- kzrdude 4mo agoI think Karpathy's microgpt blogpost is the best in this genre in a long time, and it also includes a multi layer perceptron. It's a step up in the hierarchy, so reading both is helpful, of course. https://karpathy.github.io/2026/02/12/microgpt/ https://karpathy.github.io/2026/02/12/microgpt/
- Lerc 4mo agoI'm not sure if I'd like to declare a best. There are so many different approaches and I think their ability to inform is cumulative, I like the ability of this article to do the tiny training runs in browser. It makes the point of a bias clear. Too many tutorials get sucked into the proof of zero times anything is zero. Everyone knows that. What you should show is where that mstters in the problem at hand. 3blue1brown does one of the best depictions of why we need an activation function. Karpathy's videos are a little tougher for a beginner to grasp, but excel at solving a complete problem. I knew all of the theory behind what it takes to make micrograd before I made my own by following the video, but what you get from doing it can't be understated. It's hard to describe but it what you learn is more of a feel than pure knowledge. It gives you a better sense of knowing when the principles apply in other circumstances. Perhaps it's the distinction of understanding how springs and gears work, then looking at a clock and understanding how the gears and springs move the hands. There's still more needed if you want to make a clock. And that stuff is what let's you also make a wind up toy.
- DevarshRanpara 4mo agoI can't agree more with you, It took me many days to understand the "By we need bias?" I know maths, I know programming, but why was not clear. I love 3blue1brown.
- virajk_31 4mo agoNot a ML expert, but ML tutorials shall start with something like this... Good read. Thanks.
- romaniv 4mo agoI think it should be quite obvious that perceptrons are far from the smallest units that are capable of learning. They store many bytes of information, require a non-local update process, need numeric (i.e. symbolic) inputs and involve relatively complex computations. You can go much simpler. For example: https://medium.com/@VictorBanev/the-simplest-learning-machine-pt-2-e735367f546 https://medium.com/@VictorBanev/the-simplest-learning-machin... This is a description of a 5-line algorithm that learns and stores approximate probability of an event using just 1 byte of persistent memory.
- a1o 4mo agoThat is a cool algorithm, indeed very interesting 5 lines. Also fun to see things in C#. :)
- DevarshRanpara 4mo agoTrue, there can be simpler versions compared to perceptron, just like you made. I have learned something new from that, Thanks for sharing.
- ninalanyon 4mo agoIs this something that could be scaled up and used, for instance, to recognize features in images? Or to put it another way are there any local only tools that can be trained on my own set of images to automatically tag new images? Tools that do not already have built in classes of image. I take a lot of photographs and it would be handy to reduce the drudgery of tagging them so to say broadly what the subject was so that they are easier to find later.
- sspoisk 4mo ago[flagged]
- techteach00 4mo agoIs this too high level for my 8th grade comp sci class?
- utopiah 4mo agoOne day I'll write about my 1-liner physics engine... let gravity = setInterval( _ => { if (projectile.object3D.position.y > 0) projectile.object3D.position.y \*= .99 }, 100) Jokes aside I find that providing ridiculously short toy examples that provide the very limited foundation of a concept are extremely empowering in pedagogy. You "get" it right away because it "fits" in your mind, then you dare tinker with it and quickly see how limited it is, then get excited again. It's a powerful trick to learn more IMHO.
- DevarshRanpara 4mo agoYeah, I will try to make more of these, I like to lean things from core, and I like to keep everything as simple as possible.
- vain 4mo agoShameless plug of my own interactive version of this (ai assisted, but probably not slop) https://sourceobscure.com/perceptron/ https://sourceobscure.com/perceptron/
- DevarshRanpara 4mo agoHey, I am sorry if you felt that way, I watched https://www.youtube.com/watch?v=l-9ALe3U-Fg https://www.youtube.com/watch?v=l-9ALe3U-Fg and got inspiration, I found your content to be similar to that video as well. I don't understand these concepts as deeply as you can, but I have tried to make it simple and toy like. I have to say, you have really good content, if I use any of your resources in my future blogs, surely I will give acknowledgement to you or any other creator!
- SubiculumCode 4mo agoI wish that the tutorial went just one more step. It presents a one dimensional perceptron. But most perceptrons are multi-input. Adapting the article's 1D perceptron to three-input, for example: import random learning_rate = 0.1 EPOCHS = 50 NUM_INPUTS = 3 weights = [random.uniform(-1, 1) for _ in range(NUM_INPUTS)] bias = random.uniform(-1, 1) data = [] for _ in range(100): inputs = [random.uniform(-1, 1) for _ in range(NUM_INPUTS)] result = sum(inputs) > 0 data.append((inputs, result)) for epoch in range(EPOCHS): for inputs, result in data: weighted_sum = bias for i in range(NUM_INPUTS): weighted_sum += inputs[i] * weights[i] prediction = weighted_sum > 0 if prediction != result: error = int(result) - int(prediction) for i in range(NUM_INPUTS): weights[i] += learning_rate * error * inputs[i] bias += learning_rate * error print(f"Final weights: {[round(w, 3) for w in weights]}") print(f"Final bias: {round(bias, 3)}")
- DevarshRanpara 4mo agoMy next article would be on multi dimensional problem statements.
- moi2388 4mo agoI’m sorry, but this entire article is written by AI and I’m just so done with this.
- deleted 4mo ago[deleted]
- kgwxd 4mo ago"Brains" are the most interesting thing in the universe. This is NOT that. The AI industry is taking all the words, this has to stop.