4 ms·
DRµGS: Deep Random Micro-Glitch Sampling
- cranberryturkey 3y agocute :P
- revskill 3y agoWhat does cute mean ?
- firtoz 3y ago[flagged]
- deleted 3y ago[deleted]
- pastacacioepepe 3y agokawaii
- kstrauser 3y agoI appreciate the commitment to the metaphor. Bravo.
- mylons 3y agoit is excellent copy, at the very least
- Log_out_ 3y agoNothing like good old monk humor about sex.. Unless druggy humor by the notoriously drug free..
- golergka 3y agoYes, because HN crowd or AI/software crowd in general is famous for being drug free and never indulging in various psychedelics out in the desert.
- mettamage 3y agoIf they would, they would be burning man. Nothing to ignite some passion than a deep dive in the desert ;)
- BossingAround 3y agoIt's kind of funny, I don't really see psychedelics as "drugs" in the same way that you wouldn't view coffee as a drug. I don't think SW engineers are, for example, known for taking meth or heroin, so I'd definitely call the group fairly "drug free". I'd think (based on nothing but my experience) that your average nurse might have higher likelihood of taking hard drugs than an average SWE :)
- golergka 3y ago> I don't think SW engineers are, for example, known for taking meth Some people think meth means “amphetamine” (although it usually methamphetamine), and there’s a lot of software engineers who take different types of prescribed amphetamines like Adderal and Ritalin. And don’t even get me started on weed and millenials.
- danielbln 3y agoMethylphenidate is not an amphetamine but your points stand.
- knutzui 3y agoFrom Wikipedia: > A drug is any chemical substance that when consumed causes a change in an organism's physiology, including its psychology, if applicable. Coffee, psychedelics and alcohol are drugs just like heroin. Whether you believe they are useful to consume is a different matter.
- stavros 3y agoAhh yes, the old drmgs.
- nyolfen 3y agoi was just wondering yesterday what happened to this guy
- madamelic 3y agoI appreciate the README etc, it's very funny and cute. As a bit of a layperson / just an AI integrator, I want to get some clarification of how this interfaces with a generative AI. I haven't dug too deep in and I have a mostly elementary understanding of neural nets. No need to _entirely_ dumb it down though, just needing a smarter person to confirm or correct my understanding. :) Is this influencing the probabilities of token output? My understanding is that currently it is a static value that is essentially "how wild do you want it to be" where each generation has a different consistent static value of "wildness". So rather than using a static value for an entire generation, this dynamically injects randomness in so rather than the output being 'monotone', it is more dynamic rather than by-the-book that AI tends to be?
- andy99 3y agoTypically the output of an LLM is a classifier that gives scores reflecting the "best" next token, for some measure of best. You can deterministically just take the highest score or you can randomly sample in a way that makes it more likely to get a token with a higher score, according to different schemes. The point is that this all rests in the performance of the output classifier layer. From my skim, this proposes to inject randomness throughout the model (as opposed to just during sampling the classifier output) which according to the author (I'm paraphrasing) lets the full power of the model be used to make the most of that randomness, instead of just jittering the output. So the output is still random but it's a function of random noise being added through the model layers instead of just at the output, supposedly demonstrating better properties. I have no opinion about it as I haven't studied it.
- madamelic 3y ago> You can deterministically just take the highest score or you can randomly sample in a way that makes it more likely to get a token with a higher score, according to different schemes. Ah ok, so it's like the difference between greedily always taking the shortest / highest value vs using A* and exploring alternatives in hopes of finding something better? That probably doesn't line up as a metaphor as it isn't "exploring" or running multiple parallel generations. And it employs the method in the layers so that the benefit can be amplified rather than only employing the method in the final layer / output when it selects the output to take
- Philpax 3y agoIsn’t this the GamerGate guy?
- IshKebab 3y agoApparently so: https://ggwiki.deepfreeze.it/index.php/Eron_Gjoni https://ggwiki.deepfreeze.it/index.php/Eron_Gjoni Unclear how "the gamergate guy" he was Vs just "accidentally triggered gamergate". I'm not going down that rabbit hole though - where's Internet Historian when you need him?
- sleepybrett 3y agoProbably plagiarizing something.
- sureglymop 3y agoInternet Historian recently got cancelled for being a Nazi and hiding "Nazi symbolism" in his videos is what a quick google told me.
- zoklet-enjoyer 3y agoI'm still not really sure what gamer gate was or who the Internet historian is
- GaggiX 3y agoWhere did you find that Internet Historian is a nazi? I only found something when searching "Internet Historian nazi", a reddit post that also claims that Jontron is a white supremacist. This post: https://www.reddit.com/r/196/comments/18erlt0/internet_historian_is_a_nazi/ https://www.reddit.com/r/196/comments/18erlt0/internet_histo... If you actually googled "Internet Historian nazi" for a guy you don't even know who he is, that would be pretty funny.
- sureglymop 3y agoI have seen an Internet Historian video before. I just googled "Internet Historian" and a cancellation style reddit post was the third result. I even mentioned that I only did a quick google; I encourage you to do your own research here and form your own opinion on this. I have heard of Jontron but I am not familiar nor have I read anything about that.
- Der_Einzige 3y agoIt's so funny how little love that work on the decoder/sampler side with LLMs gets. You can take bad LLM and make them decent, or good LLMs and make them garbage with intelligent sampling settings. It's cool to see further innovation in this space.
- RestlessAPI 3y agoI wonder if this works for Stable Diffusion as well...
- samus 3y agoThere are transformer-based approaches to vision models. It should work for those.
- _lvbh 3y agoFrom what I understand, Stable Diffusion starts out with random noise that is then step by step made into the final image. Randomness is there at the beginning. Current LLMs first find the n most likely next token and only then gets randomness on the choice among the top-n. This injects randomness to the initial search for the top-n. So I think, from my ignorance, that Stable Diffusion is already doing this in some sense
- junon 3y agoGreat Readme. Strikes the perfect balance between quickly informative and enjoyably cute.
- bjornsing 3y agoSo instead of the mathematically sound approach of sampling from the categorical output distribution (that the model parameters were fitted to with maximum likelihood) you just inject some noise in the activations, and then pick the highest probability output?
- gwern 3y agoInjecting noise has many mathematically-sound interpretations, like the Bayesian interpretations of dropout for ensembling or posterior sampling.
- zarzavat 3y agoIt’s a biologically inspired approach, hence the name. Biological neurons don’t work based on sampling functions, they have inherent non-determinism distributed throughout the network, and receptors to enable runtime modulation of the network’s function. This dynamic modulation is seemingly at odds with mathematical soundness. Whether mathematical soundness should be a goal or not is a question to be answered empirically.
- madamelic 3y agoThank you for the explaination. I had to chew on this for a minute or two to put it together. This explaination helped a lot.