6 ms·
Just signed up for Claude Pro to try out the Opus model. Decided to throw a complex query at it, combining an image with an involved question about SDXL fine tu
by virgildotcodes 3y ago
Just signed up for Claude Pro to try out the Opus model. Decided to throw a complex query at it, combining an image with an involved question about SDXL fine tuning and asking it to do some math comparing the cost of using an RTX 6000 Ada vs an H100.
It made a lot of mistakes. I provided it with a screenshot of Runpod's pricing for their GPUs, and it misread the pricing on an RTX 6000 ADA as $0.114 instead of $1.14.
Then, it tried to do math, and here is the outcome:
-----
>Approach 1: Use the 1x RTX 6000 Ada with a batch size of 4 for 10,000 steps.
>Cost: $0.114/hr * (10,000 steps / (4 images/step * 2.5 steps/sec)) = $19.00
Time: (10,000 steps / (4 images/step * 2.5 steps/sec)) / 3600 = 0.278 hours
>Approach 2: Use the 1x H100 80GB SXMS with a batch size of 8 for 10,000 steps.
>Cost: $4.69/hr * (10,000 steps / (8 images/step * 3 steps/sec)) = $19.54
Time: (10,000 steps / (8 images/step * 3 steps/sec)) / 3600 = 0.116 hours
-----
You will note that .278 * $0.114 (or even the actually correct $1.14) != $19.00, and that .116 * $4.69 != $19.54.
For what it's worth, ChatGPT 4 correctly read the prices off the same screenshot, and did math that was more coherent. Note, it saw that the RTX 6000 Ada was currently unavailable in that same screenshot and on its own decided to substitute a 4090 which is $.74/hr, also it chose the cheaper PCIe version of the H100 Runpod offers @ $3.89/hr:
-----
>The total cost for running 10,000 steps on the RTX 4090 would be approximately $2.06.
>It would take about 2.78 hours to complete 10,000 steps on the RTX 4090.
On the other hand:
>The total cost for running 10,000 steps on the H100 PCIe would be approximately $5.40.
>It would take about 1.39 hours to complete 10,000 steps on the H100 PCIe, which is roughly half the time compared to the RTX 4090 due to the doubled batch size assumption.
-----
- anonymouse008 3y agoI'm convinced GPT is running separate helper functions on input and output tokens to fix the 'tokenization' issues. As in, find items of math, send it to this hand made parser and function, then insert result into output tokens. There's no other way to fix the token issue. For reference, Let's build the GPT Tokenizer https://www.youtube.com/watch?v=zduSFxRajkE https://www.youtube.com/watch?v=zduSFxRajkE
- nine_k 3y agoI personally find approaches like this the correct way forward. An input analyzer that finds out what kinds of tokens the query contains. A bunch of specialized models which handle each type well: image analysis, OCR, math and formal logic, data lookup,sentiment analysis, etc. Then some synthesis steps that produce a coherent answer in the right format.
- michaelt 3y agoThen you might enjoy looking up the "Mixture of Experts" model design.
- numeri 3y agoThat has nothing to do with the idea of ensembling multiple specialized/single-purpose models. Mixture of Experts is an method of splitting the feed-forwards in a model such that only a (hopefully) relevant subset of parameters is run for each token. The model learns how to split them on its own, and usually splits based not on topic or domain, but on grammatical function or category of symbol (e.g., punctuation, counting words, conjunctions, proper nouns, etc.).
- michaelt 3y agoAn ensemble of specialists is different to a mixture of experts? I thought half the point of MoE was to make the training tractable by allowing the different experts to be trained independently?
- CuriouslyC 3y agoYeah. Have a multimodal parser model that can decompose prompts into pieces, generate embeddings for each of them and route those embeddings to the correct model based on the location of the embedding in latent space. Then have a "combiner/resolver" model that is trained to take answer embeddings from multiple models and render it in one of a variety of human readable formats. Eventually there is going to be a model catalog that describes model inputs/outputs in a machine parseable format, all models will use a unified interface (embedding in -> embedding out, with adapters for different latent spaces), and we will have "agent" models designed to be rapidly fine tuned in an online manner that act as glue between all these different models.
- behnamoh 3y agoWhen OpenAI showed that GPT-4 with vision was smarter than GPT-4 without vision, what did they mean really? Does vision capability increase intelligence even in tasks that don't involve vision (no image input)?
- KoolKat23 3y agoYes. They increase the total parameters used in the model and adjust the existing parameters.
- jasondclinton 3y agoHi, CISO of Anthropic here. Thank you for the feedback! If you can share any details about the image, please share in a private message. No LLM has had an emergent calculator yet.
- virgildotcodes 3y agoHey Jason, checked your HN bio and I don't see a contact. Found you on twitter but it seems I'm unable to DM you. Went ahead and uploaded the image here: https://imgur.com/pJlzk6z https://imgur.com/pJlzk6z
- jasondclinton 3y agoThank you!
- samstave 3y agoAn "LLM crawler app" is needed -- in that you should be able to shift Tokenized Workloads between executioners in a BGP routing sort of sense... Least cost routing of prompt response. especially if time-to-respond is not as important as precision... Also, is there a time-series ability in any LLM model (meaning "show me this [thing] based on this [input] but continually updated as I firehose the crap out of it"? -- What if you could get execution estimates for a prompt?
- skoush 3y ago[dead]
- connorgutman 3y agoRegardless of emergence, in the context of "putting safety at the frontier" I would expect Claude 3 to be augmented with very basic tools like calculators to minimize such trivial hallucinations. I say this as someone rooting for Anthropic.
- jasondclinton 3y agoLLMs are building blocks and I’m excited about folks building with a concert of models working together with subagents.
- causal 3y agoI'm guessing the difference is screenshot reading, I'm finding that it's about the same as GPT-4 with text. For example, given this equation: (64−30)−(46−38)+(11+96)+(30+21)+(93+55)−(22×71)/(55/16)+(69/37)+(74+70)−(40/29) Calculator: 22.08555452004 GPT-4 (without Python): 22.3038 Claude 3 Opus: 22.0492
- samstave 3y agoI cant wait until this is the true disruptor in the economy: "Take this $1,000 and maximise my returns and invest it where appropriate. Goal is to make this $1,000 100X" And just let your r/wallStreetBets BOT run rampant with it...
- SubiculumCode 3y agoHow many uses do you get per day of Opus with the pro subscription?
- virgildotcodes 3y agoHmm, not seeing it anywhere on my profile or in the chat interface, but I might be missing it.
- behnamoh 3y ago100 messages per 8 hours: https://support.anthropic.com/en/articles/8324991-about-claude-pro-usage https://support.anthropic.com/en/articles/8324991-about-clau...
- yawnxyz 3y agoInteresting that Opus and Sonnet have the same limits