joaogui1

8 ms·

61.
by joaogui1 3y ago
Programmers excel at it, you don't see that happening much with Physics, Chemistry, Math, Material Sciences etc Funnily enough it has also been observed before, by Alan Kay comparing programming to a pop culture "In the last 25
62.
by joaogui1 3y ago
The attention matrix is computed based on all tokens in the context, so it kind of functions non-parametrically (but over the batch instead of over the whole training dataset)
63.
by joaogui1 3y ago
Sorry, I know what MDL and L2 regularization are, I would like the paper that connects them in the way you mentioned
64.
by joaogui1 3y ago
That looks interesting, do you know what paper talks about the connection between MDL, regret, and weight decay?
65.
by joaogui1 3y ago
I wonder if edge tpus (sold here https://coral.ai/products/ ) could help, though I guess the community would have to optimize for them instead of for standard hardware
66.
by joaogui1 3y ago
Just as a quick source to my claims: 1. The raft paper is titled "In Search of an Understandable Consensus Algorithm" 2. The abstract of this tutorial on Understanding Paxos https://www.ux.uis.no/~meling/paper
67.
by joaogui1 3y ago
Paxos is so mistifyingly hard that Raft was invented as part of a project to understand Paxos (and the advisor and proponent of the project was John Ousterhout, who's pretty badass). There are also I believe a few papers trying to tryi
68.
by joaogui1 3y ago
When we talk about tree search we allow for backtracking, so if a node has 3 children all 3 will be explored generally, or at least a subsample of the children will be, in LLM sampling you generally pick a single token/child and then j
69.
by joaogui1 3y ago
I think the real comparison is "has the proportion of the population that needs to know how to take care of farm animals gone down?"
70.
by joaogui1 3y ago
He doesn't use that much modern tech really, for example email https://www-cs-faculty.stanford.edu/~knuth/email.html Using ChatGPT would require engaging with email and maybe more stuff that he's not inclined
71.
by joaogui1 3y ago
I'll link to my comment in another thread about this https://news.ycombinator.com/item?id=36023814
72.
by joaogui1 3y ago
Before this potentially blows up, I'll say a little on what I hoped this was given its title. I saw many comments on Knuth's lack of curiosity with ChatGPT, and I wanted to give context on it. 1. Knuth's magnum opus is titled
73.
by joaogui1 3y ago
The paper shows the equivalence for specific networks, it doesn't say every GNN (and as such transformers) are Dynamic Programmers. Also the models are explicitly trained on that task, in a regime quite different from ChatGPT. What the
74.
by joaogui1 3y ago
I don't get why the sudden fixation on time, the model is also spending a ton of compute and energy to do it
75.
by joaogui1 3y ago
I'm not sure about any, but a lot of the persistent data structures seem to fit within that limit, and it's what's used in FP languages like Clojure. There's also some compiler research of identifying when something won&
76.
by joaogui1 3y ago
FWIW I work in LLMs and I consistently fail to do simple webdev stuff
77.
by joaogui1 3y ago
You still have one vector per token, that's what they meant, also the fact that the vector associated with each token will ultimately be used to predict the next token, once again showing that it makes sense to talk about other tokens
78.
by joaogui1 4y ago
Kinda, there are GPU kernels that are sped-up by being non-deterministic, so you also gain performance with non-determinism
79.
by joaogui1 4y ago
Someone needs to connect you to Stripe Press (maybe folks in HN can do it?)
80.
by joaogui1 4y ago
How do you know it's pruned?
81.
by joaogui1 4y ago
What's the name of the 3rd one?
82.
by joaogui1 4y ago
2 issues in fact, one is TSMC being the only one capable of making these amazing chips and the second is Apple buying their entire supply. Now if any of that counts as a monopoly I don't think so
83.
by joaogui1 4y ago
Uhm, sorry for asking, but are you sure you're not mistaking Janet and Julia?
84.
by joaogui1 4y ago
Total Recall
85.
by joaogui1 4y ago
It's how the software is known really (I think lots of people, including myself, don't recall the original meaning)
86.
by joaogui1 4y ago
Python has a huge library ecosystem and the average Machine Learning programmer is not a CS geek and so prefers pythonic quasi-OOO then over FP (one of the hurdles for JAX adoption is it's functional paradigm)
87.
by joaogui1 4y ago
You can send money through it (not sure how it's implemented on the backend) but Brazilians still don't pay for whatsapp
88.
by joaogui1 4y ago
A* is more general than Dijkstra, as Dijkstra ia A* with the heuristic set to 0. Also it's not in the style of a psalm
89.
by joaogui1 4y ago
In what way does AlphaZero use evolution?
90.
by joaogui1 4y ago
This initiative is getting some pretty good practical results https://arxiv.org/abs/2209.01188 , taking a look at their citations should lead you to other examples in the field

More ›