14 ms·
Tinygrad
- tartakovsky 5y agoFor a novice in this space, can anyone provide a link to an article providing a comparison of: Pytorch, Pytorch lightning, tinygrad, micrograd.....? I would like to point my clients to a reference here. It's no longer enough to say "PyTorch" I guess.
- upbeat_general 5y agoPyTorch lightning is just a convenience wrapper for PyTorch. The rest are toy re-implementations of a very small subset of PyTorch features, likely much slower and certainly less optimized overall. PyTorch (or Tensorflow or Keras) are the real options.
- ausbah 5y agoalso Jax, hopefully
- jonahx 5y ago>tinygrad will always be below 1000 lines. If it isn't, we will revert commits until tinygrad becomes smaller. Love this ethos so much. Wish more projects would follow suit.
- deleted 5y ago[deleted]
- eternityforest 5y agoI think it's a bit unnecessary. A small set of operations should be enough to make it easy to port to new accelerators, and there's lots of nice to haves in many programs that don't affect compatibility. If you don't plan on ever doing anything but the core it does seem pretty reasonable though.
- erwincoumans 5y agoAgreed. It still leaves the loop hole of writing infinitely long lines of code (or is there a line length limit?) I also enjoyed watching the life streaming videos https://www.youtube.com/watch?v=Xtws3-Pk69o https://www.youtube.com/watch?v=Xtws3-Pk69o and the videos videos documenting the neural network ANE M1 chip reverse engineering efforts for this project. See video links here: https://news.ycombinator.com/item?id=30852818 https://news.ycombinator.com/item?id=30852818
- babelfish 5y agoCompletely disagree - limiting lines of code as an ethos makes as much sense as using lines of code written as a business metric. What if they want to add a new feature that takes another ~1000 LoC? Can they just write it as a separate library and include it as a dependency? IMO trying to minimize LoC creates a perverse incentive to split up your package when it might not need it (in the same way maximizing LoC creates a perverse incentive to write the most verbose code possible, not necessarily the most readable/maintainable)
- dleslie 5y ago> What if they want to add a new feature that takes another ~1000 LoC? They don't. The LoC limit exists as much to avoid scope creep and maintain focus as it is about anything else.
- blueblob 5y agoIt trades off utility for managing scope creep. Keeping this code base tiny necessarily means that anything that uses it would need to write more code. If the point is a demonstration instead of a usable library, then that is good because it is much easier to follow without excess complexity.
- dleslie 5y agoNecessarily is a strong word; there are assuredly many users of this tool who find it sufficient in its scope as it is, based on the project activity. If a user needs significant feature additions they can write them as a separate tool that interacts with this one.
- mhh__ 5y agoThis one arguably does have a bit of scope creep it's just in a separate directory.
- alwaysjoe 5y agoMaybe don’t add that feature then? Not every library is meant to solve all problems
- deleted 5y ago[deleted]
- hoosieree 5y agoBonus challenge: total number of bytes monotonically decreases over time.
- whimsicalism 5y agoReminds me of https://github.com/minitorch/minitorch https://github.com/minitorch/minitorch
- mdp2021 5y agoMinitorch is intended to be both an engine and high quality didactic material, from Cornell University, for the course 'Machine Learning Engineering'. > the full student code for minitorch. It is designed as a single repo that can be completed part by part following the guide book > Basic Neural Networks and Modules ; Autodifferentiation for Scalars ; Tensors, Views, and Strides ; Parallel Tensor Operations ; GPU / CUDA Programming in NUMBA ; Convolutions and Pooling ; Advanced NN Functions See: http://minitorch.github.io/ http://minitorch.github.io/
- mwexler 5y agoI am reminded of Andrew Tanenbaum's Minix os: before Linux became the belle of the ball, minix implemented many unix functionalities in a less efficient but clearer way than existing unix builds (many of which were not open source) and the still nascent bsd/linux/gnuos/etc, and it meshed nicely with his textbooks on OS design. It was functional on it's own, and by Minix 3 was a full os in it's own right (Intel still uses it in various ways) but balanced pedagogy with performance. https://en.wikipedia.org/wiki/Minix https://en.wikipedia.org/wiki/Minix gives more details.
- leeoniya 5y agothere is also https://github.com/photopea/UNN.js https://github.com/photopea/UNN.js
- yt-sdb 5y agoAlso: https://github.com/karpathy/micrograd https://github.com/karpathy/micrograd
- hmate9 5y agoThis is the guy that did some iOS jailbreaks, reverse engineered the PS3 and now runs a self driving car startup.
- Teknoman117 5y agoLong enough ago that it was iPhone OS jailbreaks.
- pixelpoet 5y agoAnd the shady cheapETH business last year: https://oldreddit.com/r/cheapETH/comments/lkzkso/george_hotz_who_wrote_the_geth_fork_to_create/ https://oldreddit.com/r/cheapETH/comments/lkzkso/george_hotz...
- tromp 5y agoI had to replace oldreddit by reddit to make that link work...
- sydthrowaway 5y agoSo he's a two-bit scammer too.
- fortydegrees 5y agoAdding a comment as I don't think this is a fair representation of George. He livestreamed the creation of cheapETH as a technical demonstration of web3 development[0] and continually talks about it being worthless while developing it. He's also on record as being a serial 'no-coiner'. [0] https://www.youtube.com/watch?v=9LaIezgiUmw https://www.youtube.com/watch?v=9LaIezgiUmw
- hwers 5y agoI like this guy mostly for preserving a hacker-ish culture in the atmosphere and for just being a real human being in public. Too often do I feel like I'm just interacting with interfaces to corporations or people who self censor themselves so much they barely feel human. I'm not sure this library matters much to me but I maintain positive perspective on this guy for these reasons.
- ramblerman 5y agoI wish Dang would just kill this sub-thread. What a mess. A whole amateur dissection of this guy's personality. It's so bloody awkward, just comment on the work and move on. But it's great to know HN readers are all such paragons of virtue.
- hwers 5y agoWell you got your wish. Gonna be honest and say I’m not sure I’m enjoying HNs move to follow twitter and facebook to micromanage discussion to this degree. I really don’t think we were being uncivil in having a discussion about what we appreciate about this guy.
- travisporter 5y agoIts funny you say this because the repo mentions Andrej Karpathy's repo. I absolutely love his work especially his ML tutorials, but there was one tesla automation day where he shuts down this guy asking a question with robotic efficiency. I wish I could find it.
- thspimpolds 5y agoHaving gone to college and knowing this guy personally from then, you would have a much different opinion... Maybe he has grown up, but back then... yeeesh
- hwers 5y agoI have gone to college with similar overachievers/braggers, and now that you mention it if they were in the spotlight as much as him I'd roll my eyes pretty hard. Still though, having not gone to college with this particular one somehow I can stand it.
- modeless 5y agoI find this related page more interesting: A Breakdown of AI Chip Companies https://geohot.github.io/blog/jekyll/update/2021/06/13/a-breakdown-of-ai-chip-companies.html https://geohot.github.io/blog/jekyll/update/2021/06/13/a-bre... I especially like that he outlines an actual plan for an AI chip startup that he thinks will work, and has an update explaining why he was subsequently convinced that it wouldn't work.
- buildbot 5y agoI read this article vs. the actually posted one, it has a lot of good points but also get a fair amount wrong, and the correction on the power usage is just the tip of the iceberg. That's why WaferScale do what they do, the power cost of off die vs. on die is massive, and enabling everything to be on chip means you can feed the design more easily and with less power. For example, Nvidia's compute and consumer GPU line diverged a long time ago. Modern A100s have literally only one SM capable of doing normal GPU tasks, probably to support running a display on whatever Quadro version they end up increasing. They diverged in really specific ways, for example the P100 has hardware scheduling, where as the 1080 does not (in the same way at least). Another issue is the author spends a long time talking about how important software and ecosystem is, then completely misses that point when talking about their own CHIP - just because it is RISCV and compilers exist for that arch does not equal CUDA. Also, big re-order buffers cost area and heat that could be spent on more SMs. That's why in order to beat Nvidia you must get more specialized, they've picked their niche on the CPU-GPU-ASIC continuum, beating them at the same process node requires ditching some stuff of the stuff an Nvidia GPU. Which is why they've been specializing their arch with tensor cores. It just also turned out those are useful for gaming with deep learning to upres the graphics, as that's easy to accelerate than driving quadraticlly higher resolutions.
- rrss 5y ago> for example the P100 has hardware scheduling, where as the 1080 does not (in the same way at least). what does hardware scheduling mean in this context?
- gary_0 5y ago
- londons_explore 5y ago> tinygrad will always be below 1000 lines. If it isn't, we will revert commits until tinygrad becomes smaller. I applaud this. Committing to keeping a project small and simple. So many projects start small and simple, and before long they've been extended in many different directions and now have thousands of options and things to understand before you can get started.
- johnamata 5y agoAre there other examples of famous projects that do that, limiting themselves to an X-amount of LOC? Last one I remember was TempleOS, although Terry went a bit over his limit (100k LOC).
- seritools 5y ago"dwm is only a single binary, and its source code is intended to never exceed 2000 SLOC." https://dwm.suckless.org/ https://dwm.suckless.org/
- asicsp 5y agohttps://github.com/aosabook/500lines https://github.com/aosabook/500lines >Each chapter consists of a walkthrough of a program that solves a canonical problem in software engineering in at most 500 source lines of code. We hope that the material in this book will help readers understand the varied approaches that engineers take when solving problems in different domains, and will serve as a basis for projects that extend or modify the contributions here.
- throwawaybner3 5y agoI like his honest streams. Also this geohot can code for 12 hours straight. That's just amazing. If you take in to account his speed of thought, its more like 36 hours.
- Ninjinka 5y agoUpvoted for geohot
- d915 5y agoHow about we show simple admiration and appreciation? We could learn way more if we try to understand the actions rather than criticize. He's a nerd. If we are here we are nerd too. Keep up the nerd sht geohot!
- deleted 5y ago[deleted]
- Buttons840 5y agoI'd like to ship a video game that does machine learning and trains on its experiences with the player. Yes, I know there are many potential problems with this. What is the best way to ship training code in a game? Do I embed Python and PyTorch or something? Do I code my own NN training algorithm? Do I use a library such as Tinygrad?
- alar44 5y agoDo you know how to program? I have no idea what you're asking.
- Buttons840 5y agoYes, I know how to program and know a good amount about machine learning in Python. The problem is these machine learning frameworks are rather heavy to be shipping around, although I guess you could do it. I was wondering if there's any lighter weight machine learning libraries, which might not have all the bells and whistles, but would be better to ship and train models on consumers computers. Most of the time people ship pretrained models, but don't actually update that model on the end users computer.
- upbeat_general 5y agoI think the real issue here is with the training speed. Presumably a GPU is required to run the game, but is it restricted to just NVIDIA GPUs/a subset of those? I don’t know of any such libraries offhand and my guess would be that’s because the size of the library generally matters less if there’s a requirement to have a powerful GPU.
- CornCobs 5y agoClient side training with the usual python stack sounds like a configuration nightmare. Would it try to use the users GPU if it exists? Would it fallback to CPU? Does your user need a specific GPU card to play?
- phillypham 5y ago
- bmc7505 5y agoEven more tiny: https://github.com/breandan/picograd/blob/main/picograd.py https://github.com/breandan/picograd/blob/main/picograd.py
- recuter 5y agoLook man, this is just too tiny. People are upset in this thread about the excessively short line count of OP as it is. What is this, some sort of autograd for ants? It is freaking me out, remove this link.
- tartakovsky 5y agoLol
- katal90 5y ago[flagged]
- katal90 5y ago[flagged]
- dukeofdoom 5y agoI watched a few of George's live streams, and I'm pretty impressed with his coding skills and determination to solve a problem. In the spirit of learning, anyone else on his level do live streams or has a youtube channel? Here's his last 7 hour stream coding Tinygrad. https://www.youtube.com/watch?v=MeE4Y2862FY https://www.youtube.com/watch?v=MeE4Y2862FY
- trompetenaccoun 5y agoHe takes a random IQ test in the middle of it, lol.
- viperoso312 5y agodo you have the minute of that?
- dukeofdoom 5y ago4th hour, 1 minute
- dhanna 5y agoHe inspired me to look up the research on Raven Progressive Matrices: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.686.3979&rep=rep1&type=pdf https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.68...
- chaxor 5y agoWhat kind of performance can you get with micrograd? It looks to be a few python files (and no external library calls?), so I imagine it's insanely slow. At least with tensirflow/pytorch there are some calls to C in the dependencies - so the speed decreases there are from the loops and such written on top of that. What's the deal here? Why not just use Julia :)
- EZ-Cheeze 5y agoAlgos run the world. "In bioinformatics, BLAST (basic local alignment search tool) is an algorithm and program for comparing primary biological sequence information, such as the amino-acid sequences of proteins or the nucleotides of DNA and/or RNA sequences." - could it be used to identify and group similar structures in NNs? (https://en.wikipedia.org/wiki/BLAST_(biotechnology) https://en.wikipedia.org/wiki/BLAST_(biotechnology)) I've been thinking of some kind of visual representation of weights, graphs etc. Images and evolving images as the substrate of the neural network. Pixels on a plane, planes affecting each other, activation recorded as brightness/color. Then we can use visual algorithms like SIFT (https://en.wikipedia.org/wiki/Scale-invariant_feature_transform https://en.wikipedia.org/wiki/Scale-invariant_feature_transf...) to do cool stuff and grow better and better graphic-based NNs.
- inciampati 5y agoUsing bioinformatics tools to understand neural networks. What a fasinating idea! It probably makes sense to go for the direct application of string and pattern matching methods from stringology. BLAST is very much meant for DNA and protein biosequences. edit: It would be cool to see the evolution of neural networks over their training, and im transfer learning. Comparative neural network genomics.
- EZ-Cheeze 5y agoYeah I want extreme cross-disciplinary collaboration and new ideas being tried constantly
- edward 5y agoI fixed some spelling mistakes: https://github.com/geohot/tinygrad/pull/314 https://github.com/geohot/tinygrad/pull/314