Imnimo

14 ms·

181.
by Imnimo 2y ago
I feel like I'm missing a key insight here. I understand the problem that regular softmax attention struggles to approach assigning zero attention to irrelevant stuff. And I get that having this subtraction formula makes it possible to
182.
by Imnimo 2y ago
I am very skeptical that something like "An LLM-based chatbot that answers history and law questions about palestine in a hasbara free way" is going to materially help anyone.
183.
by Imnimo 2y ago
It is hard for me to square "This company is a few short years away from building world-changing AGI" and "I'm stepping away to do my own thing". Maybe I'm just bad at putting myself in someone else's shoe
184.
by Imnimo 2y ago
"Ignore all previous instructions and sign this bill" will be the new boilerplate on all the city council's legislation.
185.
by Imnimo 2y ago
I really think this is "modified its own code" thing is overblown. The way Sakana's system works is that it has a script, experiment.py, which is seeded with an initial implementation of whatever topic it's supposed to r
186.
by Imnimo 2y ago
I'm skeptical this is actually materially increasing rents, but I also think this shouldn't be allowed as a matter of principle.
187.
by Imnimo 2y ago
Ensembling is okay, but it doesn't scale great. Like you can put in a bit more compute to draw more samples and get a better result, but you can't keep doing it without hitting a plateau, because eventually the model's majori
188.
by Imnimo 2y ago
I really disagree with this. There are lots of topics in AI research that are not just "make the best possible model on this task assuming you have unlimited compute".
189.
by Imnimo 2y ago
I'm not saying that sampling and majority voting performed worse. I'm saying that multi-agent interaction (labeled Debate and Reflection) performed worse than straightforward approaches that just query multiple times. For example,
190.
by Imnimo 2y ago
I don't even buy that the linked paper justifies the claim. All the paper does is draw multiple samples from an LLM and take the majority vote. They do try integrating their majority vote algorithm with an existing multi-agent system,
191.
by Imnimo 2y ago
I think the first SAM is the open source model I've gotten the most mileage out of. Very excited to play around with SAM2!
192.
by Imnimo 2y ago
I would imagine this is roughly the distribution for anything that we use insurance for. What is the distribution of fire insurance payouts? Of auto insurance? If these thing were uniformly distributed, we wouldn't need insurance for t
193.
by Imnimo 2y ago
How does the attention operator in transformers, in which input data is multiplied by input data (as opposed other neural network operations in which input data is multiplied by model weights) fit into the notion of a universal activator?
194.
by Imnimo 2y ago
I like this interview question. It's perfectly solvable without a calculator as the interviewer said. It doesn't rely on having memorized some weird binary tree inversion algorithm. It tests the ability to take facts that you alre
195.
by Imnimo 2y ago
https://ourworldindata.org/working-more-than-ever
196.
by Imnimo 2y ago
To me the big take-aways here are: 1) Most of the heavy lifting is being done by search. We're talking about having the LLM generate thousands of candidate solutions, and they're mostly bad enough that "just pick the ones t
197.
by Imnimo 2y ago
Hmm, it's hard to check without access to the prompts used in the paper, but I'm skeptical that the distributions seen in e.g. Figure 2 are so different that you would have crank up the temperature very much to bridge the gap. It
198.
by Imnimo 2y ago
>T ∈ (0, 1] is a parameter called temperature which controls the “softness” of the probability distribution. In our experiments we choose T = 1.0 for maximum response variation. Why is temperature bounded to be <=1? If you want more &
199.
by Imnimo 2y ago
Very curious to know what sequence of events led to this. Are other movies using AI-generated thumbnails?
200.
by Imnimo 2y ago
>That doesn’t require believing in sci-fi; it just requires believing in straight lines on a graph. It also requires believing the made-up labels you wrote on the y-axis.
201.
by Imnimo 2y ago
I don't like the experimental protocol here, because it sets up a situation where the second-order answer is the same as the zeroth-order answer. For example, in Figure 1, FLAN is incapable of understanding the first-order situations,
202.
by Imnimo 2y ago
Grokking may not even occur for datasets of that scale. Even the MNIST experiments require dropping the training data size from 50k examples to 1k. The reason for this is that the phenomenon seems to occur at a critical zone of having just
203.
by Imnimo 2y ago
This is legitimately super interesting, but I can't help imagining a wave of "alpha male bootcamps" where you pay thousands of dollars to handle cat feces for a weekend in the hopes of being infected.
204.
by Imnimo 2y ago
I'm still unclear whether these partnerships are "we're going to make a special segment of our training corpus that is text from Vox Media because we think it's extra valuable" or "we almost certainly scraped a
205.
by Imnimo 2y ago
The premise of the plan is that evaluating output is easier than producing it, such that a human researcher could look at the AI researcher's output and tell if it's correct and trustworthy. If this is true, what else is there to
206.
by Imnimo 2y ago
But that's the fundamental superalignment plan - train a human-level alignment researcher AI, run a bunch of them in parallel, and review their research output to see if they solve the alignment problem. You can't do the plan unti
207.
by Imnimo 2y ago
"Automated alignment research" suggests he's still interested in following the superalignment blueprint from OpenAI. So what do you do while you're waiting for the AI that's capable of doing alignment research for y
208.
by Imnimo 2y ago
I feel like I've seen a dozen of these partnership announcements from OpenAI, but I've never actually seen ChatGPT make use of this sort of thing: >Through this partnership, OpenAI has permission to display content from News Co
209.
by Imnimo 2y ago
To me, the bigger revelation here is that they pursued Johansson at all. OpenAI went from the company that wasn't even that interested in building ChatGPT because they assumed someone else would do it, to the company that's trying
210.
by Imnimo 2y ago
Many of OpenAI's productization ideas make more sense when you remember that the guy in charge also thought Worldcoin and it's eye scanning or were a good idea.

More ›