6 ms·
The Eleuther AI Mafia
- mkl 3y agoTangent: Why do some sites (like this one) disable pinch zooming on mobile? It makes graphs and pictures illegible, which is incredibly frustrating.
- deleted 3y ago[deleted]
- d4rkp4ttern 3y agoAnother tangent — I don’t get why people writing technical blogs use Substack, where support for math and syntax highlighting is nonexistent (math) or minimal (code). Something with Hugo plus PaperPad theme would be far better. I suppose Substack is popular because of the newsletter features.
- nextworddev 3y agoIt's mainly for discovery I believe.
- capableweb 3y agoSame reason anyone would use a hosted service vs setting it up themselves, they don't want to deal with the hassle. For many of us, it's trivial, sometimes because we've done it before, but for others, it's a huge hill to climb, which seems unnecessary if all they want to do is publish some content.
- d4rkp4ttern 3y agoThis makes sense for people who don’t code, but I see plenty of developer oriented bloggers who use Substack, which is puzzling to me. I also don’t know why Substack doesn’t get with the program and just support MathJax and colored syntax highlighting already.
- capableweb 3y ago> This makes sense for people who don’t code, but I see plenty of developer oriented bloggers who use Substack, which is puzzling to me No, it makes sense for any person with any skill set. Just because you are a developer doesn't mean you want to setup everything yourself. We use plenty of tools to make our own experience better and more focused, based on what we want to do. If I'm a programmer, and I want to publish blog posts, sometimes I don't want to care about the setup that makes me able to publish blog posts, I just want to post something. > why Substack doesn’t get with the program and just support MathJax and colored syntax highlighting already This, I don't know. Should have been done ages ago.
- primordialsoup 3y agoI can do it myself and it's not a huge hill to climb, but I don't necessarily have the time to do it. There is also the time that needs to be dedicated towards maintenance (upgrades etc.).
- swyx 3y ago(author here) i have spent years setting up my own blogstack (google swyxkit) so i know how to do it. but comments like this miss that Substack is an *email social network*, and as a newsletter author that is the one currency that loosely/poorly tracks my success and distribution (right down to ability to book quality guests). Substack is the best email network period right now (it is not a very good podcast host but I deal with its many annoyances because it does one thing all the other hosts cannot - auto email my subscribers when a new episode is out). The moment Hashnode improves I'll switch to it.
- d4rkp4ttern 3y agoYes Substack excels at being an email social network. So I’ve settled on the following workflow: 1. Make my blog using mkdocs material plus GitHub pages so it has all the nice code and math rendering features (plus things like admonitions and all the other nice things in mkdocs material). 2. On each post have a Substack signup button. 3. When I make a new post I want to send to subscribers, I manually create a short text-only teaser post on Substack (paragraph or two), with a link to my main blog that says “read more here”. I know this doesn’t auto-push new posts to subscribers but I am ok with that given that I won’t be writing more than once a week at most.
- d4rkp4ttern 3y agoIn fact I really like the look of the Mkdocs Material system for blogs -- https://squidfunk.github.io/mkdocs-material/blog/ https://squidfunk.github.io/mkdocs-material/blog/ And they have integrations with comment systems like giscus.
- blueblisters 3y agoHow are EleutherAI and Carper funded?
- capableweb 3y agoEleuther: > Core Weave, HuggingFace, Google TRC, Nat Friedman, Lambda Labs https://www.eleuther.ai/about https://www.eleuther.ai/about
- _delirium 3y agoA majority of the Carper folks (though not all) are in academia. Cambridge, Brown, Georgia Tech, NYU, etc.
- emadm 3y agoCarper is technically a Stability AI lab and community https://twitter.com/carperai https://twitter.com/carperai Core are full time Stability AI
- ShamelessC 3y agoI realize you say "not authoritative", but just to clarify - LAION isn't the creation of Emad Mostaque or Stability AI. Emad is/was a funder who showed up well after its inception. The same applies for his work with Eleuther. Although some members of both LAION and Eleuther did go on to become Stability employees. LAION was also partially born from _frustrations with_ Eleuther's TPU culture and inability to ship anything (at the time, and excepting Katherine Crowson's great work) for text-to-image work.
- swyx 3y agowill mention, thanks for correction. would love to interview someone from the LAION side to get that part of the story. (its really hard to trace as an outsider)
- ShamelessC 3y agoIt all started originally on lucidrains/dalle-pytorch in the months following the release of DALL-E (1). The group started as `dalle-pytorch-replicate` but was never officially "blessed" by Phil Wang who seems to enjoy being a free agent (can't blame him). https://github.com/lucidrains/DALLE-pytorch/issues/116 https://github.com/lucidrains/DALLE-pytorch/issues/116 is where the discord got kicked off originally. There's a lot of other interactions between us in the github there. You should be able to find when Phil was approached by Jenia Jitsev, Jan Ebert, and Mehdi Cherti (all starting LAION members) who graciously offered the chance to replicate the DALL-E paper using their available compute at the JUWELS and JUWELS Booster HPC system. This all predates Emad's arrival. I believe he showed up around the time guided diffusion and GLIDE, but it may have been a bit earlier. Data work originally focused on amassing several of the bigger datasets of the time. Getting CC12M downloaded and trained on was something of an early milestone (robvanvolt's work). A lot of early work was like that though, shuffling through CC12M, COCO, etc. with the dalle-pytorch codebase until we got an avocado armchair. Christophe Schumann was an early contributor as well and great at organizing and rallying. He focused a lot on the early data scraping work for what would become the "LAION5B" dataset. I don't want to credit him with the coding and I'm ashamed to admit I can't recall who did much of the work there - but a distributed scraping program was developed (the name was something@home... not scraping@home?). I should also mention Romain Beaumont, an original member. His work on clip-retrieval was great and allowed us to eventually ship CLIP embeddings data for LAION5B and do semantic search over the dataset. He worked on a lot there however, including code for training and model architecture. Smart guy! I have heard he's now with Stability? Haven't confirmed though. The discord link on Phil Wang's readme at dalle-pytorch got a lot of traffic and a lot of people who wanted to pitch in with the scraping effort. Eventually a lot of people from Eleuther and many other teams mingled with us, some sort of non-profit org was created in Germany I believe for legal purposes. The dataset continued to grow and the group moved from training DALLE's to finetuning diffusion models. Eventually it was decided by core members that we should get a proper name and LAION was chosen. The `CompVis` team were great inspiration at the time and much of their work on VQGAN and then latent diffusion models basically kept us motivated. As I mentioned a personal motivation was Katherine Crowson's work on a variety of things like CLIP-guided vqgan, diffusion, etc. I believe Emad Mostaque showed up around the time GLIDE was coming out? I want to say he donated money for scrapers to be run on AWS to speed up data collection. I was largely hands off for much of the data scraping process and mostly enjoyed training new models on data we had. As with any online community things got pretty ill-defined, roles changed over, volunteers came/went, etc. I would hardly call this definitive and that's at least partially the reason it's hard to trace as an outsider. That much of the early history is scattered about GitHub issues and PR's can't have helped though.
- tbruckner 3y agoThe arrows in their diagram make no sense. Stability funds everything. All arrows should be coming out of Stability.
- ftxbro 3y agoleast capitalist hacker news comment. do the arrows mean flow of capital? i don't think so. i think it means flow of people or cultural influence or leadership
- swyx 3y agoit was meant to be people :) i obviously have even less insight on flow of funds than i do of people movement. i'm sure Emad could draw a much better chart of the Eleuther mafia than I can (and happy to update my chart to reflect the truth Emad)
- capableweb 3y agoThis doesn't seem to be true, unless you don't trust what Eleuther themselves are saying. > The work we do would not be possible without the generous support of our donors and other sponsors: Core Weave, HuggingFace, Google TRC, Nat Friedman, Lambda Labs https://www.eleuther.ai/about https://www.eleuther.ai/about
- tbruckner 3y agoWhich one of those donors are on their board?
- emadm 3y agoWe fund Eleuther AI too as it says there (but made sure it can run independent when 501(c)3 set up) and provided compute for RWKV. Conjecture was meant to be alignment so can't really support, stability scaled resource provision.
- ftxbro 3y ago> Primarily Discord, pseudonymous, GPU-poor volunteer community somehow coordinating enough to train >10B, OPT/BLOOM-competitive models what are the discords for Eleuther and RWBY are they public?
- hiddencost 3y agoGive it a Google? The discord logo is at the bottom of the page.
- polygamous_bat 3y agoQuestion: I only ever hear how RWKV is threatening the dominance of transformers... from the RWKV folks, while everyone independent is happily using transformers. Is there any substance to the above claim that's been independently verified? Not an NLP person, so apologies in advance if I've missed any obvious papers or references.
- ianbutler 3y agoPersonally, the RNN "mode" runs substantially faster with similar quality to other models of the same size. RWKV is limited by compute and thus model size at present which makes it hard to suggest how it will stack up against models like falcon and llama2 in their larger sizes. There are also trade offs around their specific attention mechanism. It's approximate and consists of an exponential decay across time steps and some constant that represents the magnitude of the attention at that time step. The decay means for long running contexts it in theory will loses some of the ability to recall nuance you see with perfect attention on longer sequences. This isn't a specific problem of RWKV, but really any decay based attention. A perk of the formulation is the attention computation is linear in time complexity and only requires one additional serial scan at the end, so you get nicer scaling properties than quadratic perfect attention for the trade off in lossy attention. The examples in the original paper don't show this to be a problem, but they also show much shorter sequences. In the real world its a toss up, I'd expect it to be faster but more lossy for longer context windows. TLDR: It has some teeth, but I think saying it's threatening is wishful thinking to some degree. Good model, looking forward to seeing larger sizes.
- pico_creator 3y agoAgreed that as a whole, we do need more public benchmarks (both transformers and RWKV), with much longer sequence length. I view this as a case of the "benchmarks" and "datasets" not keeping up with the current progress, as the vast majority of benchmarks presumes the limit of up to 2k tokens. Which was cutting edge last year, but isn't today. Internally we do have some benchmarking for how the decay over time affects lossy memory, and we are making major improvements in this direction for v5, and aim for no loss within the 8k to 100k and beyond context sizes. And hopefully with time, there will be more formalised standard benchmarks for larger context sizes. Edit/PS: I believe until we get a sponsor for the compute required to build on similar scale and dataset compared to LLaMA2, so that we can have that "showdown" - we would be stuck - in its non threatening status. As there will always be that shadow of doubt, if it can scale to the next milestone that transformers have already crossed.
- AndrewKemendo 3y agoI haven't been on the Eleuther Discord in a year or so but the last time I was there it was bustling and thriving on some really hard topics So while I'm not sure that the RWKV movement will see more widespread adoption than the corporate LLM implementation movement - I think there's probably still more goodness to come out of Eleuther community
- bogwog 3y agoWhat a shame that all that discussion is locked behind some app and not accessible on the public internet.
- User23 3y agoDiscord is a cancer. While I don’t understand the appeal, I’ve been around long enough to no longer be surprised by people sacrificing control of their information for trivial short term gain.
- pico_creator 3y agoAs a user from the early internet forum era, I do find it a step back as well. But unfortunately, the people as large has kinda spoken, and voted by their usage. And ultimately for community engagement, their decision is something I have to accept as well. Better to have a messy growing discord with active users, then a forum with little to no engagement. For better search indexing, and making things more accessible, I help constantly move links / items from the discord into the more public github wiki : https://wiki.rwkv.com/ https://wiki.rwkv.com/ (PS: our discord is fully public for anyone to signup/access)
- sverona 3y agoNow that there are comments here I can express the melancholy of not being Stella Biderman, whom I admire from afar. I wish there were something I could do in AI alignment, I really do, but it's beyond me.
- swyx 3y agoStella is goated. we've been dancing around interview scheduling but will definitely schedule one sooner or later.
- astrange 3y agoLuckily AI alignment isn't real and is just a modern millenarian religious movement, so there's no need to do anything. ("Aligning" meaning "making it work at all" is real, but then it's just a synonym for "engineering".)
- sverona 3y agoI think there is something there, it's just maybe not in the Yudsphere. I spent a few months in grad school reading FATML papers, but that is obviously something very different.
- netsharc 3y agoOff-topic: your comment has the item ID 37373737 .. congrats?