4 ms·
I discovered yesterday that the “amazing thing that comes out of OpenAI” is Sol, due to its token efficiency. Dollar for tokens, Sol and Fable are the same pri
by aetherspawn 2mo ago
I discovered yesterday that the “amazing thing that comes out of OpenAI” is Sol, due to its token efficiency.
Dollar for tokens, Sol and Fable are the same price.
However, Sol uses (literally: in testing) around 10-100x less output tokens compared to Fable for the same task.
We run our frontier models nearly 24/7, so switching to Sol will save us around $500 per day.
And, due to less guardrails, Sol also performed better, and we lost less tokens due to guardrails shutting down sessions (I feel like it’s illegal to take $50 of someone’s token money and then shut down a session with guardrails before they get an answer, and yet Anthropic do it to us constantly… either take our money and commit, or trigger the guardrails immediately)
- minraws 2mo agoWait isn't Fable like 2x more expensive if we compare under 272k tokens
- ChadMoran 2mo agoThe comment you're replying to almost feels like it was written by a bot or am I crazy?
- w4yai 2mo agoI agree. Weird to use <“> and <”> characters. Maybe written on phone, but certainly not on keyboard.
- gunalx 2mo agoCommon on non us keebs though.
- jrflo 2mo agoOther languages use different characters for quotes, if anything that's an indication that's not written by a LLM because it's not favoring the standard English character. https://en.wikipedia.org/wiki/Quotation_mark#Specific_language_features https://en.wikipedia.org/wiki/Quotation_mark#Specific_langua...
- deleted 2mo ago[deleted]
- aetherspawn 2mo agoWritten “on an iPhone” - yep, seems to automatically switch the quotes.
- jlund-molfese 2mo agoWhat kind of bot would say `less guardrails` instead of `fewer guardrails`? I guess someone could instruct an LLM to deliberately make mistakes, but isn't that too paranoid?
- DrewADesign 2mo ago“include common grammatical imperfections and awkwardness common in casual message board interactions.” I’m not saying that’s what’s happening here, but a high school student told me that’s basically what they do to make papers not sound like AI.
- qgin 2mo agoWe’re reaching transvestigation levels of people trying to spot AI text everywhere they look
- vitorfblima 2mo agoHmm, this sounds like something a bot would say to prevent being caught.
- freeone3000 2mo ago...Does a bot know the word "transvestigation"? Would a bot be allowed to say it by its corporate overlords?
- vitorfblima 2mo agoIt does now that is on hn. Thank you for your input.
- siva7 2mo ago15 years ago i was astounished by the intellectual deepness of this community. Now i'm aware this has always been a cult and their former cult leader Mr. Altman wants to destroy this capitalist society. He isn't even hiding motives. People just stopped listening carefully.
- theplumber 2mo agoPeak under your skin a bit. Something weird is going on. I think we are bots/robots(sic)
- satvikpendem 2mo agoRelated, Under The Skin with Scarlett Johansson is an incredible movie.
- ekabod 2mo ago[flagged]
- archon810 2mo agoPeek*
- aetherspawn 2mo agobeep boop, everyone thinks I’m a robot. :|
- resonious 2mo agoSol is way cheaper than Fable by the token.
- gtree 2mo agoYou could say the token usage is "load-bearing".
- dannyw 2mo agoWe’ve literally saved tens of millions of dollars already (no exaggeration! already 8 digits) by switching to Luna for many workloads at my company. The amount of workloads we can shift with an advisor model pattern continues to grow. It’s seriously amazing.
- jgalt212 2mo agowhat has a single company accomplished with tens of millions of token spend?
- moomoo11 2mo agohigher valuation
- akoboldfrying 2mo agoA pelican on a bike accurate to a subatomic level
- literalAardvark 2mo agoBut the knees still bend the wrong way
- TranAndrewA 2mo agoLuna came out about a month ago, you're saying that the cost saving from switching to Luna has saved your company $20 000 000+ in 1 months spending on API usage?
- woadwarrior01 2mo agoEvidently, Claude's tokenizer vocabulary size is ~15k[1]. On one hand, it's quite mind blowing. On the other hand, Anthropic models' token (in)efficiency makes a lot of sense in that light. [1]: https://xcancel.com/magikarp_tokens/status/2087859173748854983 https://xcancel.com/magikarp_tokens/status/20878591737488549...
- orbital-decay 2mo agoNot just that, they normalize everything into lowercase and use a special character to capitalize words (what about languages with non-trivial normalization/capitalization?) and mark beginning and end of each word, all of that diluting already small vocabulary. That smells like manual tuning of what should be done statistically, I wonder what technical merit they saw in that - I know they mentioned better generalization, but this is pretty counterintuitive.
- namibj 2mo agoSadly it's not too counterintuitive; remember the old "how many r's are in the word strawberry"? Also different tokens for the same named entity/concept if they almost entirely exclusively occur in non-overlapping contexts, and are themselves rare/uncommon in the first place, will result in behavior that's similar to the speech/phrasing/vocabulary registers humans exhibit, where the aspects of the named entity/concept get largely compartmentalized. The most severe case along these lines were the old BERT models that ran over straight UTF-8 bytes (plus a handful special tokens). But for the modern post-GPT2 LLMs such radical simplicity seems to mostly not be considered suitable. Note that CJK (the big one in particular, so Chinese semantic and Japanese Kanji) encodes each one into multiple UTF-8 bytes giving some automatic scaling for semantically dense languages; similar effects also apply to e.g. APL code.
- deleted 2mo ago[deleted]
- focxle 2mo ago[flagged]