drob518

16 ms·

451.
by drob518 7mo ago
I’m sure Gemini has done it at some level. Google was pretty much founded on the assumption that more data is better. That has driven them to build or buy data sets that they can mine (Gmail, YouTube, etc.).
452.
by drob518 7mo ago
I think there are post training tweaks that can be done with corporate data to help fit an AI to a specific corporation. But I don’t think that private data will deliver us AGI. The knowledge for AGI is out in the world, not hidden inside c
453.
by drob518 7mo ago
Not true. You can photocopy pages from a book you own for your own use. You can make copies of purchased software as a backup. What you can’t do is make copies and give them to all your friends or sell them to the public.
454.
by drob518 7mo ago
Model parameters are weights, not algorithms. The LLM algorithm is (relatively) fixed: generate the next token according to the existing context, the model weights, and some randomization. That’s it. There is no more algorithm than that. Th
455.
by drob518 7mo ago
You’re missing the point. And we’re not just talking about books, whatever that might mean. We’re talking about all documents ever made. Every magazine article, every blog and web page, every Word doc, etc. I’m pretty sure that whatever is
456.
by drob518 7mo ago
Externalities are a thing, but this isn’t exactly dumping toxic waste into a river.
457.
by drob518 7mo ago
Sure, we could change current law, but I think that only forces an AI company to buy one copy of every book. I don’t think it gives any sort of royalty stream to anyone beyond that. Copyright is literally the right to make copies. Once I ha
458.
by drob518 7mo ago
We know that they do not reason because we know the algorithm behind the curtain. The model is generating the next token via model weights and some randomness. That’s all. It not reasoning. Sometimes it has an appearance of reasoning, but n
459.
by drob518 7mo ago
That’s what I’m said. Breakthroughs happen. No doubt about it, and they are unpredictable. Hence a breakthrough. But right now we’re using up runway with nothing yet identified to take us to the next level. And while sometimes breakthroughs
460.
by drob518 7mo ago
Yes, true. But does that really shift the argument much? An AI is like the most well-read book nerd you’ve ever met. The AI has read everything. They still won’t recite Harry Potter for you at full length and reading what the original autho
461.
by drob518 7mo ago
The test isn’t whether humans also create bullshit, but whether an honest actor knows when they are doing this and doesn’t do it on purpose. As the article points out, LLMs don’t say “I don’t know.” If you demand they do something that neve
462.
by drob518 7mo ago
Right. I think the modern LLMs are quite good at mimicking human words, but we were initially taken in like we were in the 1960s by ELIZA. It’s a (increasingly sophisticated) magic trick, but it’s just a trick.
463.
by drob518 7mo ago
Which is what percent of the world’s content? 0.000000001% or something similar. It’s nothing in the scheme of things. To put it another way, if we were to digitize that continent and train on it, our AIs would not get noticeably better in
464.
by drob518 7mo ago
> "LLMs still can't do X so it's an idiot" Let’s be careful. That’s a straw man. I don’t know anyone who says that. Aphyr says in the article that AIs can do things. But they have been marketed as “intelligent,” and I
465.
by drob518 7mo ago
Right, but we played the scaling card and it worked but is now reaching limits. What is the next card? You can surely argue that we can find a new one at any time. That’s the definition of a breakthrough. I just don’t see one at the moment.
466.
by drob518 7mo ago
A couple thoughts… Mostly, AIs don’t recite back various works. Yes, there a couple of high profile cases where people were able to get an AI to regurgitate pieces of New York Times articles and Harry Potter books, but mostly not. Mostly, i
467.
by drob518 7mo ago
What large caches of undigitized content exists? Surely, not everything has been digitized, but I can’t think it’s much in percentage terms.
468.
by drob518 7mo ago
But does that really help, or do you get distortion? The frequency distribution of human generated content moves slowly over time as new subjects are discussed. What frequency distribution do those “data generating processes” use? And at ro
469.
by drob518 7mo ago
> It remains unclear whether continuing to throw vast quantities of silicon and ever-bigger corpuses at the current generation of models will lead to human-equivalent capabilities. Massive increases in training costs and parameter count
470.
by drob518 7mo ago
I’m curious how this technique works, or not, with unified memory architectures such as Apple’s M series. It seems like it’s relying on using overlapping processes to help speed things up, but I would assume that having everything unified i
471.
by drob518 7mo ago
Nice timing. I was just today needing some of the info that these commands surface. Serendipitous!
472.
by drob518 7mo ago
Nobody does. One person figures it out, then writes a blog post, and we all Google for it. Even git’s man pages are long and sometimes cryptic.
473.
by drob518 7mo ago
Ok, so it’s constrained by specs, but you dialog with the AI and have it create the specs. I should try that. I’ve been creating my own specs and having it work from those and then iterating, but that’s not exactly quick and I find myself t
474.
by drob518 7mo ago
Cool. Are you restricting the AI to be very focused on a function or an architectural blocks that is envisioned, or are you giving it more freedom? I seem to have less slop when I really constrain things, but that takes a lot of work (e.g.,
475.
by drob518 7mo ago
IMO, taste has always been one of the strongest moats because we struggle to define what good taste even is. We know it when we see it, but other than pointing to examples, we can’t really describe it in general terms. I still remember a li
476.
by drob518 7mo ago
Exactly. I thought AI was going to be smarter. I thought AI would give us expert coders. Instead we have idiot savants.
477.
by drob518 7mo ago
Yes, but, that’s a low bar, no? I mean, when they first started talking about AI, were you envisioning a mediocre AI that is just average, or were you imagining an expert?
478.
by drob518 7mo ago
I think the industry-wide hope is that AI manages the AI-written code, but it’s unclear whether that’s actually going to work out in practice. Right now, my experience is that is dicey. I’ve had AI mess up a codebase to the point where I th
479.
by drob518 7mo ago
[flagged]
480.
by drob518 7mo ago
No worries; all good.

More ›