usaar333

9 ms·

91.
by usaar333 2y ago
I don't think that's correct. This looks like a new model. Significant jump in math and gpqa scores.
92.
by usaar333 2y ago
Anthropic isn't really hyping their product that much. It just is really good.
93.
by usaar333 2y ago
https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g... This essay has aged extremely well.
94.
by usaar333 2y ago
> What does it mean to predict the next token correctly though? Arguably (non instruction tuned) models already regurgitate their training data such that it'd complete "Mary had a" with "little lamb" 100% of the
95.
by usaar333 2y ago
Talent? Integrations? Ecosystem? I don't know if this is going to emerge as a monopoly, and likely won't, but for whatever reason, openai and anthropic have been several months ahead of everyone else for quite some time.
96.
by usaar333 2y ago
Something that actually could predict the next token 100% correctly would be omniscient. So I hardly see why this is inherently crazy. At most I think it might not be scalable.
97.
by usaar333 2y ago
https://repository.law.umich.edu/mjlr/vol25/iss3/3/ Most of this is unlikely to be in training data. Doesn't even mention the basics, like ethnic demographics of Fiji today. Confuses history as wel
98.
by usaar333 2y ago
I tried that with a paper. It emphasized the wrong points and 8 out of 10 minutes were just filler. I like the idea of audio based formatting, but this particular implementation is quite inefficient
99.
by usaar333 2y ago
few is not > 3. Literally it's just >= 2, though I think >= 3 is the common definition. 15 is too high to be a "few" except in contexts of a few out of tens of thousands of items. Realistically I interpret this as 3
100.
by usaar333 2y ago
That's not a novel prediction. There were news reports nearly 2 weeks ago about this: https://fortune.com/2024/09/13/sam-altman-openai-non-profit-... Gwern's more novel prediction track record is c
101.
by usaar333 2y ago
That article is from 6 months ago. See more recent ones: https://www.axios.com/2024/08/29/openai-chatgpt-200-million-...
102.
by usaar333 2y ago
Growth
103.
by usaar333 2y ago
The math score exceeds o1-preview (though not mini or o1 full) fwiw.
104.
by usaar333 2y ago
Can't really compare to stuff that requires hardware. The weakly bearish metric is that chatgpt's user growth is slightly slower than Facebook's was at similar user counts (2009 or so)
105.
by usaar333 2y ago
> where I tell gpt-4 exactly what the rule is and it still fails It figured out the rule itself. It has problems applying the rule. In this example btw, asking it to write a program will solve the problem.
106.
by usaar333 2y ago
arc reasoning challenge. I'm going to give you 2 example input/output pairs and then a third bare input. Please produce the correct third output. It used its COT to understand cornering -- then I got it to write a program. But as
107.
by usaar333 2y ago
Both Chat GPT 4o and Claude 3.5 can trivially solve this puzzle if you direct them to do program synthesis to solve it. (that is write a program that solves it - e.g. https://pastebin.com/wDTWYcSx ). Without program synthe
108.
by usaar333 2y ago
> On the other hand, if you were to paste the entire documentation set to a tool it has never seen and ask it to use the tool in a way to accomplish your goals, THEN this model would be likely to produce useful output, despite the fact t
109.
by usaar333 2y ago
The model is really impressive to be fair. It's just how economically relevant it is.
110.
by usaar333 2y ago
I'm not sure how well codeforces percentiles correlate to software engineering ability. Looking at all the data, it still isn't. Key notes: 1. AlphaCode 2 was already at 1650 last year. 2. SWE-bench verified under an agent has jum
111.
by usaar333 2y ago
That's actually kind of slow? Even Facebook in 2008 (at these user counts) was growing faster, doubling monthly active every 8 months
112.
by usaar333 2y ago
> So since March 2023 we haven't seen any increase in quality on their part Huh? On typical benchmarks, the gap between current models and original GPT-4 is as high as original GPT-4 and GPT-3.5 at time of release
113.
by usaar333 2y ago
Isn't spgi the company not the index? I'm seeing 17% projected growth in earnings: https://ycharts.com/indicators/sp_500_earnings_per_share_ttm... P/e is 24. ($493 billion quarterly earnings, $47 trilli
114.
by usaar333 2y ago
EPS is $0.68. forward P/e is 43.5 based on after hours pricing ($118)
115.
by usaar333 2y ago
That's not the correct interpretation. Earnings will likely be higher next quarter. It's just that the rate of growth is slowing and when nvda is at a 45 P/E, you need high growth to justify its market cap.
116.
by usaar333 2y ago
The question is if they are impressive for a company worth $3.1 trillion. Annualizing this quarter's earnings it has a P/E of 45. And earnings "only" grew at 60% annualized, slower than in the past. (Revenue growth
117.
by usaar333 2y ago
LLMs are extremely good at translation, given that the transformer was literally built for that.
118.
by usaar333 2y ago
It lowers the probability. It's well known LLMs have imperfect reliability at following instructions -- part of the reason "agent" projects so far have not succeeded.
119.
by usaar333 2y ago
No, the equality requires the length of a 2 second period pendulum be g / pi^2. Change your definition of length - that no longer holds true. g in imperial units is 32 after all. g has units; pi does not
120.
by usaar333 2y ago
> ZeroEval's Leaderboard on hugging face [0] actually shows that it beats even Claude 3.5 Sonnet on CRUX [1] which is a code reasoning benchmark. The previous version of 4o also beat 3.5 Sonnet on Crux.

More ›