williamtrask

9 ms·

31.
by williamtrask 1y ago
This article is meant for a policy audience, so that does keep the technical depth pretty thin. It's rooted in more rigorous deep learning work. Happy to send your way if interested.
32.
by williamtrask 1y ago
I agree with you in a way - that it seems likely that new data will be incorproated in more inference-like ways. RAG is a little extreme... but i think there's going to be middle grounds betweeen full pre-training and RAG. Git-rebasin,
33.
by williamtrask 1y ago
Yeah Zama's stuff is great.
34.
by williamtrask 1y ago
Agree with you on the nuance.
35.
by williamtrask 1y ago
(OP) YOLO
36.
by williamtrask 1y ago
(OP) fwiw I fully agree with the privacywashing you're describing here, and this piece is advocating for a more rigorous standard than input privacy (homomorphic encryption), which is insufficient to enable data owners to actually reta
37.
by williamtrask 1y ago
This is the right question. If full attribution-based control is achieved, then this would be impossible. And the ingredient you've suggested could be a useful way to help achieve it.
38.
by williamtrask 1y ago
This is the magic :)
39.
by williamtrask 1y ago
(OP) the scaling laws / bitter lesson would disagree, but I tend to agree with you with some hedging. If you get copies of the same data, it doesn't help. In a similar fashion, going from 100 TBs of data scraped from the internet
40.
by williamtrask 1y ago
The piece advocates for the opposite of this. Attrbution-based control keeps data holders in control.
41.
by williamtrask 1y ago
Well, we're opining about a statement about the world. Is the universe only 200 terabytes of information? "Biological lifeforms seem to be able to train/develop general intelligence from much, much less." This statement
42.
by williamtrask 1y ago
Fwiw - this post doesn't advocate for trust. It advocates for an enforcement mechanism (attribution-based control).
43.
by williamtrask 1y ago
(OP) 100% and this piece advocates for an enforcement mechanism for that kind of payment (attribution-based control)
44.
by williamtrask 1y ago
(OP Here) This is a fair point. Internal datasets can be deceitful just as public ones can. That said, most propaganda lives in the public domain. :)
45.
by williamtrask 1y ago
(OP here) Homomorphic addition (e.g. aggregation) is very performant, including for the Federated Averaging algorithm used in Federated Learning. Not hand-waivey.
46.
by williamtrask 1y ago
(OP here) — with you on that analysis. This was in an effort to make the piece legible for a (primarily) non-technical, policy audience. Rigorous numbers are in other parts of the piece (and in the sources behind them).
47.
by williamtrask 1y ago
(OP here) I agree with this in spirit, but also it's hard to imagine the world can be fully described with 200 terabytes of data. There's a lot more good stuff out there. But to your point, a crucial question in AI right now is: h
48.
Unlocking a Million Times More Data for AI(ifp.org)
31 points by williamtrask 1y ago|56 comments
49.
by williamtrask 1y ago
IMO - this paper is right about a major contributing factor to hallucinations but wrong about the cause LLM hallucinations are closer to a cache miss . https://x.com/iamtrask/status/1964403351116009671
50.
by williamtrask 1y ago
Nice comment! I had forgotten about metaphorical truth. Sent me on a nice rabbit hole. I think 'metaphorical truth' is correct but slightly too narrow. Pragmatic truth includes metaphorical truth but is slightly wider. And while I
51.
by williamtrask 1y ago
If truth is defined as beliefs which lead one to make decisions that cause you/your society to thrive, this is a good thing (that the Old Testament has similarities to other major works). Implies a kind of evolutionary algorithm for tr
52.
by williamtrask 1y ago
Nit: the author says that supervised fine tuning is a type of RL, but it is not. RL is about delayed reward. Supervised fine tuning is not in any way about delayed reward.
53.
by williamtrask 1y ago
hugged to death?
54.
by williamtrask 1y ago
Claude is down. EDIT: for the moment... it supports 0 tokens of context xD
55.
by williamtrask 1y ago
Breakthroughs usually require a step-function change in data or compute. All the firms have proportional amounts. Next big jump in data is probably private data (either via de-siloing or robotics or both). Next big jump in compute is probab
56.
by williamtrask 1y ago
Well... unless you know how to turn your brain completely off while being awake, you're probably always giving your attention to something . Consequently, attention and time both spend at very similar rates.
57.
Jupyter-dark-detect: Detect dark mode in Jupyter environments(github.com)
5 points by williamtrask 1y ago|0 comments
58.
AI First Hiring, Teamwork and Org Structures, Staying Relevant in an an AI World(madhavajay.com)
6 points by williamtrask 1y ago|0 comments
59.
by williamtrask 1y ago
If this isn’t jumping the shark it’s darn close.
60.
The Cold Start Problem: Using Network Effects to Scale Your Product – A Review(madhavajay.com)
80 points by williamtrask 1y ago|23 comments

More ›