5 ms·
This seems like a pricing hack that burns resources, that when the loophole gets closed the price of OCR will have to rise?
by genxy 4mo ago
This seems like a pricing hack that burns resources, that when the loophole gets closed the price of OCR will have to rise?
- ricardobeat 4mo agoIt’s not a loophole, it just happens that encoding information as optical tokens is much more efficient than text.
- guardiangod 4mo agoTruly a picture is worth a thousand words.
- TZubiri 4mo agoOf course it isn't A text encoding uses 8bits per character on average, tokenization further compresses that An image font would be 25 bits if 5x5, and most fonts are 12 pixels high Of course it isn't efficient, this is a pricing inefficiency and a hack to exploit it (even the author describes it as an exploit)
- legel 4mo agoYou are wrong. Text tokens are high-dimensional vectors, not 8 bits per character. Every token has a deep embedding, e.g. 1024 float values per text token. DeepSeek-OCR proved 10x+ compression from visual embedding of text, which was a groundbreaking result. [1] Very cool to see OP's project hacking on this principle. It's still not lossless, as noted in the github, but is a promising research direction. [1] https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSeek_OCR_paper.pdf https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSe...
- deburo 4mo agoA token is probably not a single char, and an image is probably decomposed into tokens as well (and god knows how many tokens an image is decomposed into) which probably map to similar float-hungry vectors. Your counterargument could use a bit more flesh. And we're talking about images of texts, not images that represent complex imagery such as a very detailed scene or what have you.
- Groxx 4mo agoI kinda wonder if it's extracting usable context from 2D proximity between lines? Normal text input wouldn't have that kind of information (though it could, and it's arguably just a lookahead/behind of N characters on average).
- TZubiri 4mo ago>Text tokens are high-dimensional vectors, You are conflating tokens with embeddings. Tokens fit in a single word, modern gpt uses a vocabulary with 200k possible values, which would fit into 18 bits. Have a good one
- gamblor956 4mo agoPeople really need to read their cites and not just the summaries. The paper notes two things: 1) While the compression ratio for visual text is better than it is for regular text, but the absolute space required is still higher for the images. OPs were talking about the space required, not the ratio. 2) The results of the OCR must still be fed into a text-based LLM for linguistic processing. Otherwise, all you have achieved is turning an image into a bunch of text.
- nextaccountic 4mo agoWell, then we could presumably also add lossy compression to texts, without passing through images first
- vineyardmike 4mo ago[dead]
- netsharc 4mo agohuh, what if the image encoding is 8 bits per R, G, B values of the pixel, then one can encode the same amount of text in less pixel dimensions (3 letters would need 1 pixel instead of three 12x12 pixels) The top line can be the OCR-able instruction on how to decode the rest of the image, and the rest of the image would be random-looking colourful palette. It might not even need to use 8 bits per character, since ANSI is 7 bits/character.
- TZubiri 4mo agothen it's no longer an image, as the one in the github repo, you would be encoding the text as characters and sending it as an image. You can achieve this by changing the extension of an image file from .bmp to .txt Guys, not to be mean, but maybe chill with the state of the art research and go back to studying fundamentals.
- netsharc 4mo agoIf you want to be a know-it-all poseur, at least back it up with data.
- TZubiri 3mo agohttps://chatgpt.com/share/6a4a7ea3-65e8-83e9-8871-5041d97bb90a https://chatgpt.com/share/6a4a7ea3-65e8-83e9-8871-5041d97bb9... Download the .bmp file and open it with 'edit' on windows or 'nano' on linux. You should see text. The bytes that precede the text are padding and a header that roughly 'tells how to interpret the data', in the sense that it describes the dimensions of the 'image'. There's python code that shows how the file was generated, note that it just writes a string to the file and then wraps it with a .bmp header so that it can be interpreted by image viewing programs. If you open that image with an image viewer you will see an image with colorful pixels. Whether you send it as an image or as text, the only difference lies in what you declare the file to be. For example if you change the extension, operating systems will "open" it with the appropriate program by default, but the data is the same, there's no difference between ASCII text and the encoding you describe where each subpixel describes one character, it's the same thing. Your idea about using 7 bits instead of 8bits is more space efficient, but you'd need to decompress it so that the text is byte aligned. In terms of costs, you are not charged by the bandwidth consumed in transit, but by the computational cost, which wouldn't change as the text would need to be expanded to 8bits per char, (and then converted to the same token anyways) Again don't want to be mean but these are 1st year comp sci topics, not at all related to LLMs. Hence why I recommend studying fundamentals more than papers that look like bleeding edge LLM research.
- geor9e 4mo agoStep back and think about it another way - ask which scenario is more likely: Some random person discovered a 60% across the board gain in all LLMs, using an extremely simple trick that none of the labs noticed in all these years. That trick being to rasterize 8bit characters into 8x8 pixels in a big image. 60% in a market worth trillions of dollars. or Anthropic's marketing team arbitrarily prices tokens to drive growth, according to vibes and feelings, and didn't think they needed to price images on par with text in their rush to burn cash & drive growth. Some folks take advantage of the trick during the first few days of the model's availability before Anthopic corrects their pricing, to align more proportionally with actual compute costs.
- vineyardmike 4mo ago> Some random person discovered a 60% across the board gain in all LLMs, using an extremely simple trick that none of the labs noticed in all these years of multi-trillion dollar growth DeepSeek published a pretty well circulated paper on exactly this many months ago. It just hasn’t been attempted and shared publicly, asa retrofit, AFAIK. Also, it’s no free lunch, the readme indicates that this “use images” hack is lossy and reduces success rates alongside the reduced cost. Most labs would focus on success increases regardless of price.
- geor9e 4mo agoIf the trick were genuinely useful, and was well circulated months ago, the resource-starved inference providers would have squeezed this trick dry already, instead of wasting 60% of their tokens, waiting for users to implement it themselves in 5 minutes of effort.
- solenoid0937 4mo ago[flagged]
- Klathmon 4mo agoThat's like saying quantization isn't real because the frontier labs aren't using it in their production inference. This is a lossy process, it produces worse results. It might be worth it for some situations, but applying it to everything would just be making your SOTA model worse
- deleted 4mo ago[deleted]
- DaiPlusPlus 4mo ago> encoding information as optical tokens Educate me: what is an "optical token" when dealing with LLMs?
- jrm4 4mo agoAnyone else laugh out loud when they read this? Like, okay so NO, that's entirely impossible. What's really going on?
- Salgat 4mo agoThat's not what is happening. Claude isn't charging for the tokens it generates from the OCR on its side, but it's still processing the same number of tokens as if you had sent the text, just with the extra step of OCR on Claude's side. This is 100% a loophole that's burning extra resources.
- ricardobeat 4mo agoThere is no OCR, in the traditional sense, involved.
- samrus 4mo agoNot really. They arent actually using more resources this way either. This might be a fundamental inefficiency thats being removed It kinda makes sense too. Because while people do read code word by word, we often "glance over" it and do roughly pattern recognition on it to know what it does. Only homing in on something when we need to answer a specific question. I think humans kinda naturally do this exploit anyway