cgearhart

7 ms·

61.
by cgearhart 2y ago
This is an excellent analogy. Aside from “they’re both networks” (which is almost a truism), there’s really nothing in common between an artificial neural network and a brain.
62.
by cgearhart 2y ago
This is often harder at large companies because you very rarely make career progress playing defense, so it becomes very tricky to do it fairly. It can work wonders if you have the right teammates, but it’s almost a prisoners dilemma game
63.
by cgearhart 2y ago
It _is_ all the same task (generating text completions) but that pretraining on that task has seemed to be suitably abstract for the model to work decently well on more narrow problems—certainly they work dramatically better at a collection
64.
by cgearhart 2y ago
This is a pretty common perspective that was introduced to me as “shifting the goalposts” in school. I have always found it a disingenuous argument because it’s applied so narrowly. Humans are intelligent + humans play go => playing go
65.
by cgearhart 2y ago
Interesting, I’ve always run pis from wall power. Is the pi hardware incapable of similar power optimizations to coral, or is this a problem of a lack of software support for power management on pi? (I assume from your mention of external h
66.
by cgearhart 2y ago
Thanks. I needed this today. :-)
67.
by cgearhart 2y ago
A knowledge graph is really just a projection of structured data from disparate sources into a common schema. Take a bunch of tables and covert each row into a tuple (rowkey, columnName, value). Now take the union of all the tables. ^ knowl
68.
by cgearhart 2y ago
I got my kid a raspi 400 a few years ago. She’s enjoyed it tremendously, but I don’t know how it holds up to more recent alternatives. She’s still using it today, though.
69.
by cgearhart 2y ago
Even if they’re plateauing there’s still a lot of value to be had in what they already do. I think the mistake so far has been aiming too high or too low—ie, products that require AGI-like LLMs or unimaginative “low-hanging fruit” product i
70.
by cgearhart 2y ago
This is the same paper (or an extension) — using ternary weights means you can replace multiplication with addition/subtraction.
71.
by cgearhart 2y ago
[1] applied AlphaZero style search with LLMs to achieve performance comparable to GPT-4 Turbo with a llama3-8B base model. However, what's missing entirely from the paper (and the subject article in this thread) is that tree search is
72.
by cgearhart 2y ago
That’s an interesting idea! There’s already a lot of companies that are founded out of the graveyard of startups that failed to scale. (Sound familiar?) ;-) I think the conventional wisdom would say that it’s not worth the effort to put tog
73.
by cgearhart 2y ago
I also found it confusing the first time I saw it. I believe it is sometimes used because the techniques for DL are very similar (in some cases identical) to algorithms that were developed for color palette quantization (in some places shor
74.
by cgearhart 2y ago
> "But for all that I'd love to believe that that was because of the wonderful quality of our product, it realistically wasn't. It was ads." > Why were we able to buy those ads? Because investors gave us a lot of m
75.
by cgearhart 2y ago
Seems like they waited until they could do what Craig said of an iPad calculator app in 2020, “There’s some things that we have not done because we would want to do something really distinctly great in that space,” the Apple VP explained. “
76.
by cgearhart 2y ago
This is always the dividing line in the debate and the place where folks talk past each other. Under normal circumstances people expect “human level” intelligence to be coupled with human level autonomy; but task focused AI often produces s
77.
by cgearhart 2y ago
That’s really where I think the future of these features lives—in figuring out the trust model and security story for providing it. (Maybe it’s even impossible…)
78.
by cgearhart 2y ago
I think Recall is a window into the future of computers—but this version isn’t going to be successful. It may not even be possible to succeed with this model at all. I think a lot of people would consider it a _feature_ if they could simply
79.
by cgearhart 2y ago
Oooooh, I forgot that the self attention layer has a softmax. I thought this was referring to a softmax on the dense forward layer. Thanks! Next question: does the softmax in the SA block cause it to be bandwidth bound—won’t it have to mate
80.
by cgearhart 2y ago
Why does it disproportionately use bandwidth?
81.
by cgearhart 2y ago
This is a neat observation. Slightly terrifying, but still interesting. Seems like there will also be cases where we discover new theories through the uninterpretable models—much easier and faster to experiment endlessly with a computer.
82.
by cgearhart 2y ago
What you’ve described is basically the problem with greedy sampling in the decoder. Many other local optimization sampling strategies exist (e.g., beam search) and there’s been a lot of work on more global sampling (e.g., speculative decodi
83.
by cgearhart 2y ago
It’s silly, but actually not an terrible analogy because nuclear weapons aren’t really that complicated as a technology — there’s even somewhat famous caselaw established when a magazine attempted to publish the basics. [1] While it’s not
84.
by cgearhart 3y ago
TL;DR—couldn’t exactly replicate their results, but broadly confirmed their findings. They agree that the optimal range is 5–40 tokens per parameter, and close to 20 for the “chinchilla” model from the original paper. Very unusual choice to
85.
by cgearhart 3y ago
> Privacy is important. I strongly agree, but it needs to be balanced against being 14 on the open internet. And you can't have it both ways here–it can't be "be a parent: control and limit your kids' on the internet&
86.
by cgearhart 3y ago
I have wondered about that, but we didn’t keep them strictly from it. They had Instagram and a couple other things—with screen time limits and we knew their account handles. That was apparently enough friction for them to find a workaround.
87.
by cgearhart 3y ago
> put parental controls on your kids devices We did that. My kids (twins) pooled their allowance money for a few months and had a friend at school buy them an old iPhone that they shared in a locker at school. They went wild on social
88.
by cgearhart 3y ago
I think it’s evidence that timing is everything. In the 2010s deep learning was still figuring out how to leverage GPUs. The scale of compute required for everything after GPT-2 would have been nearly impossible in 2017/2018–our course
89.
by cgearhart 3y ago
Obviously “Python” was used…and then it used numpy because the image format in the assignment was numpy arrays. However, the 25 lines was basically “sum the rows and sum the columns then compare those vectors” or something like that. This w
90.
by cgearhart 3y ago
The Georgia Tech Knowledge Based AI course involved building a program to answer ravens progressive matrix questions. The course was offered in the online MS program, so thousands of students have taken the course. The most impressive resul

More ›