3 ms·
DS being unable to precisely view Playwright screenshots is the only thing I really miss from Sonnet. This is promising. > Images are converted into tokens bas
by ciberado 1mo ago
DS being unable to precisely view Playwright screenshots is the only thing I really miss from Sonnet. This is promising.
> Images are converted into tokens based on their dimensions, and these tokens are billed together with your text tokens.
> Before inference, every image is automatically resized:
> - Images with a total pixel count below roughly 384×384 are scaled up while preserving their aspect ratio.
> - Larger images are scaled down while preserving their aspect ratio so that the total pixel count after resizing is roughly that of an 800×800 image.
> As a result, there is an upper bound of 384 tokens per image: for example, a 2000×2000 image and a 5000×5000 image consume the same number of tokens after resizing. When a request contains multiple images, each image is counted independently under the same rule—there is no separate calculation for multi-image requests.
400 tokens per image results in 2,500 images per dollar, if I’m not mistaken.
edit: format.
- knollimar 1mo agoOof 800 by 800 kills a lot of use cases
- asdfsa32 1mo agoflash vs fine details. Pick one.
- Doohickey-d 1mo agoGemini "flash" models have an option for media resolution, including a high resolution option for screenshots.
- skeledrew 1mo agoAt what price point?
- wongarsu 1mo agoFor most use cases you can fix that in the harness. Just give the model a tool to request a crop of specific coordinates of any image it has in its context. Call the tool "zoom" and it should be intuitive for the model Maybe there are some use cases where you need high detail everywhere at once, but for OCR of small text and the like a zoom ability should be sufficient
- embedding-shape 1mo agoFor really dumb models I've also had success automatically cropping it into a grid of N images with the max size, then processing each cell individually, then once all been processed, do one final call with resized image + all other context previously generated per cell. Basically a workaround to the image dimension restrictions without loosing fidelity. Works well with even dumb 7B models. Can't remember if I stole this idea from some existing public harness though, can't remember. If someone knows of public harnesses that do this already, please share them :)
- dotancohen 1mo agoDoes this not loose context? Especially e.g. in fonts where the character pairs 0O 1I 1l Il may be difficult to differentiate?
- skeledrew 1mo agoThat's what the grid crop should handle. The detail is retained at that level, and then everything is logically stitched together again using the lower-res-full-image as reference. That's going to be 2x token usage at minimum though.
- Aeroi 1mo agoi think the claude cookbook has a file that does this, called tiling.
- johndough 1mo agoMight still be fine. The most recent crop of vLLMs proactively use whichever programs are available on the system (e.g. ImageMagick or PIL) to "zoom in" by cropping subimages if they can't quite make out the details.
- knollimar 1mo agoDownsizing a higher res image to lower res means the zoom will be blurry.
- andai 1mo agoThey process the original image file with Python on the local device. (And I've seen the web chats do this with their "computer use" features too.) The really wild one is even blind models will do this and they'll try to run stats on the pixels to figure out what it looks like... the even wilder thing is that it kind of works!
- knollimar 1mo agoIf the API accepts only 800 by 800, the aegument youre making is "fix it in the harness". I don't think the n by n subgrid fixes this the way most harnesses do, as it'll fail to count things if you have more overlap and fail relatiomships if you have less
- tjoff 1mo agoThat seems weirdly specific? And if you are counting things it should be trivial to note the position of your items and not double-count them, no?
- adastra22 1mo agoThey’re not talking about zooming, hence the quotes.
- knollimar 1mo ago
- shadyr 1mo agoIt might also be due to its experimental status. Wouldn't surprise me if the GA version allows for larger input. Either that or the eventual pro version.
- Chnmy 1mo agowhat are these use cases?
- knollimar 1mo agoAnything where there are symbols representing in space (e.g. schematics). Thats pretty broad
- stronglikedan 1mo agoI don't know about a lot. Probably more like a few. I take a lot of screenshots for various reasons, and over 800 seems like I could have done a better job framing and cropping.
- hedgehog 1mo agoMulti-turn, expose a tool that extracts crops. Works great.
- 13639366668 1mo ago[flagged]