xianshou

10 ms·

61.
by xianshou 2y ago
Turns out the wonderful lucidity of Data Colada posts extends not only to data science, but also descriptions of court proceedings. Looking forward to any and all future updates if the case makes it to discovery.
62.
by xianshou 2y ago
I was going to knock this for incrementally improving performance while massively increasing costs, but it's actually 7x less expensive. Not bad.
63.
by xianshou 2y ago
Reframing inevitably in the direction of AI, this also expresses the difference between, for instance, pretraining and retrieval in LLMs. People expect large models to memorize the dataset and are frustrated when it doesn't, even more
64.
by xianshou 3y ago
Single-GPU, optimal efficiency: unsloth + qlora + mistral-7b on runpod/vast/lambda Blazing fast compared to out-of-the-box transformers, also make sure to use flash attention if you have A100s or better and context length >= 2k
65.
by xianshou 3y ago
"Ability against models with 10x or 100x more parameters" This is a small model optimized for retrieval and function calling. "Reasoning" makes an appearance in the title but no standard benchmarks of general ability, su
66.
by xianshou 3y ago
Claude Opus. GPT-4 gives sensible answers to basic questions, but takes serious persuading to produce useful output on non-trivial work. Opus can be used as an integral part of an engineering workflow and only takes 2-3 tries to get from il
67.
by xianshou 3y ago
While comparing ordinary human activity to language modeling is perhaps the greatest intellectual cliche of 2024, what PG describes in the essay-writing process itself essentially boils down to beam search. Autocomplete, find wrongness, bac
68.
by xianshou 3y ago
Sadly, the main point this post reinforces is the emptiness of modern credentialing. A full degree in "cognitive AI" equipped the author to make exactly none of the correct choices in the network architecture and training. A valia
69.
AI passes 100 IQ (Claude 3)(maximumtruth.org)
5 points by xianshou 3y ago|1 comments
70.
by xianshou 3y ago
From the article: "It is important to note that the scale of investments being made by these cloud providers is dwarfed by actual cloud revenue. For example, Azure from Microsoft generates $25B in revenue a quarter. The ~$10B OpenAI in
71.
by xianshou 3y ago
The path to AGI: 0. Have model A. 1. Use Monte Carlo with A to get supervised data. 2. Train model B with data from A. 3. Use Monte Carlo with B to get supervised data. 4. Train model C with data from B...
72.
by xianshou 3y ago
If Waymo is truly as much better than Cruise as the statistics claim, this is a perfect opportunity to release the full video of the incident. Most collisions are avoidable, but a few are the result of a perfect storm of conditions that wou
73.
by xianshou 3y ago
Last May, Karpathy mentioned that the next step in LLM performance would likely take inspiration from AlphaGo's Monte Carlo-style tree traversal to enable deeper and more self-correcting reasoning: https://youtu.be/bZQu
74.
by xianshou 3y ago
An excellent example of an anti-shibboleth, also known as a "frisco": https://allthingslinguistic.com/post/45448571632/the-opposit... These signal words act as a silent, passive test of competence in tha
75.
by xianshou 3y ago
More excellent resources from Jay and others: The Illustrated Transformer - http://jalammar.github.io/illustrated-transformer/ Beyond the Illustrated Transformer - https://news.ycombinator.com/item?id=3
76.
by xianshou 3y ago
Plagiarism detection will be extremely difficult to create any systematic standard for given the endless creativity of LLMs in generating variations. I foresee a system of enforcement via name and shame, a la Sports Illustrated.
77.
by xianshou 3y ago
Morality and legality aside, there's a substantive difference between use of content and use of a model. Pretraining a GPT 4-class model from raw data requires trillions of tokens and millions of dollars in compute, whereas distilling
78.
by xianshou 3y ago
Not a complete answer, but here are the most helpful resources for understanding transformer basics in particular: Original transformer paper: https://arxiv.org/abs/1706.03762 Illustrated transformer: http:/&#x2F
79.
by xianshou 3y ago
The interesting point here is that the two founders had previously co-founded a similar AI company, Adept, that raised a much larger amount of funding ($415m vs $57m here) to build foundation models: https://www.theinformation.co
80.
by xianshou 3y ago
Marketing: Gemini 90.0% || GPT-4 86.4%, new SotA exceeding human performance on MMLU! Fine print: Gemini 90.0% chain of thought @ 32-shot || GPT-4 86.4% @ 5-shot Technical report: Gemini 83.7% @ 5-shot || GPT-4 86.4% @ 5-shot Granted, this
81.
by xianshou 3y ago
Now that this and other articles have established that the board's actions were based not on a single event but rather on general, long-term, underspecified concerns, no one captures the essence of what happened more perfectly than Mat
82.
by xianshou 3y ago
First reaction: okay, should we be impressed? June: flashy ML-perf demo with CoreWeave using 22k H100s November: actually trained model using 5k H100s June: we claim Inflection-1 is the best model in its compute class and are preparing a fr
83.
by xianshou 3y ago
Emu edit should be exactly what you're looking for: https://ai.meta.com/blog/emu-text-to-video-generation-image-...
84.
by xianshou 3y ago
Whether you side with Sam or Ilya on rollout speed and approach to safety, the breathtaking incompetence of this coup has already incinerated a stunning amount of trust. One could hardly ask for a better demonstration that it matters not on
85.
by xianshou 3y ago
Pour one out for the decline of human exceptionalism! Once you get material-reductionist enough and accept yourself as a pure function of genetics, environment, past experience, and chance, this conclusion becomes inevitable. I also expect
86.
by xianshou 3y ago
The PoW camp example is interesting, but the way this article talks about risk and return makes no sense. The author gives an example of a lottery with a $1m prize, funded by selling 1 million tickets at $1 each, then introduces a "Mr.
87.
by xianshou 3y ago
The actual claim of this paper is that any watermark can be erased, which it justifies by proving the existence of an Erase function that removes the watermark without degrading quality. Therefore, since any watermark can be erased, let&#x2
88.
by xianshou 3y ago
I would agree that summarization is a killer use case, but would actually argue for three: 1. Condense (summarize, synthesize, extract, etc.) 2. Expand (explain, copilot, suggest, etc.) 3. Transform (translation, style transfer, switch mo
89.
by xianshou 3y ago
One expects this to be satire, but it is stated entirely in earnest: Freedom is control. People have always been able to make false or misleading statements, at scale, with either false attribution or none at all. This is a matter of degree
90.
by xianshou 3y ago
Chiming in to confirm that I've tried several of the prompts verbatim and obtained perfectly good answers with sound reasoning from GPT-4. If I add any chain-of-thought markers such as "reason step by step and state your assumptio

More ›