smaddox

10 ms·

91.
by smaddox 3y ago
> You should call it d/dx bar. Or put the modifier on the denominator so that the product and chain rules are obvious (the modifier only persists on the dx, not on the dy)
92.
by smaddox 3y ago
So it would be more like 46 hours to train GPT-3 from scratch. (300BT / 1.2BT) * 11 min * (1 hr / 60 min) = 45.8 hr Still pretty incredible. That's an 18.8x speedup over 36 days.
93.
by smaddox 3y ago
KSUID's are have temporal-lexicographical order plus 128 bits of entropy, which is more than UUIDv4. https://github.com/segmentio/ksuid
94.
by smaddox 3y ago
Based on my reading, under some circumstances the scheduler wasn't dedicating large enough chunks of time to a particular boot process for it actually make progress. The fix appears to have been to change how the time chunk size was ca
95.
by smaddox 3y ago
Even if it didn't damage any equipment (which seems unlikely), it would still likely shut down most of the power grid, which would shut down most of the power plants, since they would have nowhere to sink they're output energy. Co
96.
by smaddox 3y ago
Basically, yes. Or at least it's related. The game is based on (arguably, a misinterpretation of) the writings of Elizer Yudkowski.
97.
by smaddox 3y ago
How so?
98.
by smaddox 3y ago
Easier for quick prototypes, for sure (although I switched to Rust from Haskell due to even worse compile times in Haskell and my disdain for monad transformers). Also, Rust can't currently do nested pattern matching through Boxes, whi
99.
by smaddox 3y ago
Interesting hypothetical. If semiconductors were never discovered, I suspect we would have kept shrinking vacuum tubes, eventually down to the point (which has now been demonstrated) that atmospheric pressure is sufficiently low pressure to
100.
by smaddox 3y ago
Flip flops existed before transistors.
101.
by smaddox 3y ago
I'm not so sure. Last time I followed the field, modern HDDs were nanostructured, such that each magnetic domain was literally a physically distinguishable object, in a sense. I'm not sure if that's still the case, and I&#x27
102.
by smaddox 3y ago
Paperclips?
103.
by smaddox 3y ago
Quite the deluded and hypocritical take. The author focuses exclusively on the negative aspects of technology while ignoring all of its positive benefits. Even the fact that he's using technology in both writing and publishing this ess
104.
by smaddox 3y ago
Technology is just the tip of the iceberg that is knowledge and understanding. The deepest advancements to knowledge and understanding come primarily from self-driven and self-motivated investigation. The closer you get to "technology&
105.
by smaddox 3y ago
Even just watching the first ~20 episodes dramatically altered how I approach coding, for the better. At least, when combined with his blog posts, e.g. https://caseymuratori.com/blog_0015 (you can skip the snarky intro)
106.
by smaddox 3y ago
I don't think that's really true, but you don't need the full context in ML in order to apply it to your field of interest. Just like you don't need the full context in Physics to apply it to an engineering problem. Some
107.
by smaddox 3y ago
The larger the rocket, the less overhead per kg of payload. Also, yes: larger space telescopes. And eventually,manned missions to Mars.
108.
by smaddox 3y ago
Well, it was trained on Stack Overflow responses.
109.
by smaddox 4y ago
RWKV is random chance on the PathX problem. Both of those models are > 95% on it, if I remember correctly. I don't think Hyena can be run in RNN mode, though. But it _might_ scale better than the LRU model (second paper).
110.
by smaddox 4y ago
How would a non-concious entity intelligently plan and act in response to stimuli in situations that require self reflection, such as interaction in a social environment? Consciousness seems necessary for effective goal pursuit in such an e
111.
by smaddox 4y ago
This EfficientVectorRouting paper (and related papers) is fascinating. It seems to have endless potential applications. I'm surprised I had not heard of it before. I'm excited to experiment with novel model structures incorporatin
112.
by smaddox 4y ago
Yeah, just saw that. Sorry for the noise.
113.
by smaddox 4y ago
Other approaches for long sequence lengths: - "Hyena Hierarchy: Towards Larger Convolutional Language Models" https://arxiv.org/pdf/2302.10866.pdf - "Resurrecting Recurrent Neural Networks for Long Seque
114.
by smaddox 4y ago
Yes, one distribution per position. This was a key innovation that allowed training over the entire sequence in parallel, rather than on one token prediction at a time, thereby massively speeding up training. More recently, there are models
115.
by smaddox 4y ago
> An LLM is a function that takes an array of integers and returns a new array of integers. To refine this a bit more, a LLM is a function that takes an array of integers (or really, a batch of arrays of integers), and returns a probabil
116.
by smaddox 4y ago
> 2D materials are clearly the future of the industry, with a lot of momentum pushing the field forward. I've been out of the semiconductor materials and devices game for a while, but I wouldn't be so sure. Just because these m
117.
by smaddox 4y ago
Based on that discussion, it definitely sounds like some sort of bug is hiding. Perhaps run some evaluations to compare perplexity to the standard implementation? Edit : looks like there's now confirmation that running it on a 10GB VM
118.
by smaddox 4y ago
Cerebras is "training compute optimal". Llama appears to be trained far beyond "training compute optimal". The tradeoff is that inference is closer to optimal for Llama, i.e. better performance with a smaller model.
119.
by smaddox 4y ago
There's already at least one company with a product: Mythic AMP https://mythic.ai/products/m1076-analog-matrix-processor/ And there's several research papers about other architectures. For example: http
120.
by smaddox 4y ago
There's already RWKV, if you want a decent performing pre-trained model that's Apache 2.0 licensed: https://twitter.com/BlinkDL_AI/status/1638555109373378560?s=...

More ›