5 ms·
The price really is eye watering. At a glance, my first impression is this is something like Llama 3.1 405B, where the primary value may be realized in generati
by harlanlewis 2y ago
The price really is eye watering. At a glance, my first impression is this is something like Llama 3.1 405B, where the primary value may be realized in generating high quality synthetic data for training rather than direct use.
I keep a little google spreadsheet with some charts to help visualize the landscape at a glance in terms of capability/price/throughput, bringing in the various index scores as they become available. Hope folks find it useful, feel free to copy and claim as your own.
https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddvL0b2a7EuVQw8MoaQlWaDT-w https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddv...
- bennyg 2y agoThis is an amazing spreadsheet - thank you for sharing!
- isoprophlex 2y agoThats... incredibly thorough. Wow. Thanks for sharing this.
- Philpax 2y agoHoly shit, that's incredible. You should publicise this more! That's a fantastic resource.
- beklein 2y agoThey tried a while ago: https://news.ycombinator.com/item?id=40373284 https://news.ycombinator.com/item?id=40373284 Sadly little people noticed...
- throwup238 2y agoSadly few people noticed. I don’t normally cosplay as a grammar Nazi but in this case I feel like someone should stand up for the little people :)
- freehorse 2y agoOr, sadly, little did people notice.
- rebolek 2y agoSo you think that little people didn’t notice? ;)
- dumpsterdiver 2y agoA comma in the original comment would have made it pop even more: "Sadly, little people noticed." (queue a group of little people holding pitch forks (normal forks upon closer inspection))
- beklein 2y agoThanks for the corrections, that’s what I wanted to say!
- adinb 2y agoI cannot overstate how good your shared spreadsheet is. Thanks again!
- jnd0 2y agoThank you so much for sharing this!
- bglusman 2y agovery impressive... also interested in your trip planner, it looks like invite only at the moment, but... would it be rude to ask for an invite?
- gwyllimj 2y agoThat is an amazing resource. Thanks for sharing!
- dumpsterdiver 2y agoNice, thank you for that (upvoted in appreciation). Regarding the absence of o1-Pro from the analysis, is that just because there isn't enough public information available?
- krwiseman 2y agoAwesome spreadsheet. Would a 3D graph of fast, cheap & smart be possible?
- rendist 2y agoAmazing, thank you so much for sharing this.
- kridsdale3 2y ago[flagged]
- senordevnyc 2y agoHey, just FYI, I pasted your url from the spreadsheet title into Safari on macOS and got an SSL warning. Unfortunately I clicked through and now it works, so not sure what the exact cause looked like.
- harlanlewis 2y agoI appreciate the bug report! Unfortunately this is a familiar and sporadically recurring issue with Netlify, which I should really move off of…
- sfink 2y ago> feel free to copy and claim as your own. That's a nice sentiment, but I'd encourage you to add a license or something. The basic "something" would be adding a canonical URL into the spreadsheet itself somewhere, along with a notification that users can do what they want other than removing that URL. (And the URL would be described as "the original source" or something, not a claim that the particular version/incarnation someone is looking at is the same as what is at that URL.) The risk is that someone will accidentally introduce errors or unsupportable claims, and people with the modified spreadsheet won't know that it's not The spreadsheet and so will discount its accuracy or trustability. (If people are trying to deceive others into thinking it's the original, they'll remove the notice, but that's a different problem.) It would be a shame for people to lose faith in your work because of crap that other people do that you have no say in.
- mwigdahl 2y agoWow, what awesome information! Thanks for sharing!
- jwr 2y agoThis is incredibly useful, thank you for sharing!
- 6gvONxR4sf7o 2y agoNot just for training data, but for eval data. If you can spend a few grand on really good labels for benchmarking your attempts at making something feasible work, that’s also super handy.
- swyx 2y ago> https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddv https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddv... how do you do the different size circles and colored sequences like that? this is god tier skills
- world2vec 2y agoBubble charts?
- deleted 2y ago[deleted]
- harlanlewis 2y agohey, thank you! bubble charts, annotated with text and shapes using the Drawing tool. Working with the constraints of Google Sheets is its own challenge. also - love the podcast, one of my favorites. the 3:1 io token price breakdown in my sheet is lifted directly from charts I've seen on latent space.
- swyx 2y agohaha yeah many people might ask you to tweak to 100:1 but at that point you might as well just go by input price
- taurath 2y agoWhat gets me is the whole cost structure is based on practically free services due to all the investor money. They’re not pulling in significant revenue with this pricing relative to what it costs to train the models, so the cost may be completely different if they had to recoup those costs, right?
- crovona 2y agoVery useful