6 ms·
Faking a JPEG
- puttycat 1y ago> compression tends to increase the entropy of a bit stream. Does it? Encryption increases entropy, but not sure about compression.
- JCBird1012 1y agoI can see what was meant with that statement. I do think compression increases Shannon entropy by virtue of it removing repeating patterns of data - Shannon entropy per byte of compressed data increases since it’s now more “random” - all the non-random patterns have been compressed out. Total information entropy - no. The amount of information conveyed remains the same.
- gary_0 1y agoTechnically with lossy compression, the amount of information conveyed will likely change. It could even increase the amount of information of the decompressed image, for instance if you compress a cartoon with simple lines and colors, a lossy algorithm might introduce artifacts that appear as noise.
- gregdeon 1y agoYes: the reason why some data can be compressed is because many of its bits are predictable, meaning that it has low entropy per bit.
- Modified3019 1y agoLove the effort. That said, these seem to be heavily biased towards displaying green, so one “sanity” check would be if your bot is suddenly scraping thousands of green images, something might be up.
- 112233 1y agoSo how do I set up an instance of this beautiful flytrap? Do I need a valid personal blog, or can I plop something on cloudflare to spin on their edge?
- ffsm8 1y agoIt's a flask app, he linked to it https://github.com/gw1urf/spigot/ https://github.com/gw1urf/spigot/
- jandrese 1y agoI wonder if you could mess with AI input scrapers by adding fake captions to each image? I imagine something like: (big green blob) "My cat playing with his new catnip ball". (blue mess of an image) "Robins nesting"
- levzzz 1y ago[dead]
- Dwedit 1y agoA well-written scraper would check the image against a CLIP model or other captioning model to see if the text there actually agrees with the image contents.
- Simran-B 1y agoThen captions that are somewhat believable? "Abstract digital art piece by F. U. Botts resembling wide landscapes in vibrant colors"
- Someone 1y agoDo scrapers actually do such things on every page they download? Sampling a small fraction of a site to check how trustworthy it is, I can see happen, but I would think they’d rather scrape many more pages than spend resources doing such checks on every page. Or is the internet so full of garbage nowadays that it is necessary to do that on every page?
- vintermann 1y agoAin't nobody got the processing time for that! Scraping is about more, more, more. If they do any filtering it'll be afterwards.
- superjan 1y agoThere is a particular pattern (block/tag marker) that is illegal the compressed JPEG stream. If I recall correctly you should insert a 0x00 after a 0xFF byte in the output to avoid it. If there is interest I can followup later (not today).
- ethan_smith 1y agoYou're referring to JPEG's byte stuffing rule: any 0xFF byte in the entropy-coded data must be followed by a 0x00 byte to prevent it from being interpreted as a marker segment.
- superjan 1y agoOk this is correct for traditional JPEG. Other flavors like Jpeg2000 use a similar (but lower overhead) version of this byte-stuffing to avoid JPEG markers from appearing in the compressed stream. Related: https://en.wikipedia.org/wiki/JPEG#Syntax_and_structure https://en.wikipedia.org/wiki/JPEG#Syntax_and_structure
- vintermann 1y agoI remember there was a guy on compression forums who was very annoyed at this waste of coding space. If you're doing compression, shouldn't you make sure every encoded file decodes to a distinct output? He thought so, and made bijective versions of Huffman coding, arithmetic coding, LZ coding and (even more impressive) the BWT transform known from bzip2. He was a bit crazy, but I liked that guy. Rest in peace, David A. Scott. Maybe there will be new uses for making all compression bijective over all byte streams.
- tomsmeding 1y agoThey do have a robots.txt [1] that disallows robot access to the spigot tree (as expected), but removing the /spigot/ part from the URL seems to still lead to Spigot. [2] The /~auj namespace is not disallowed in robots.txt, so even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice. [1]: https://www.ty-penguin.org.uk/robots.txt https://www.ty-penguin.org.uk/robots.txt [2]: https://www.ty-penguin.org.uk https://www.ty-penguin.org.uk concatenated with /~auj/cheese (don't want to create links there)
- josephg 1y ago> even well-intentioned crawlers, if they somehow end up there, can get stuck in the infinite page zoo. That's not very nice. So? What duty do web site operators have to be "nice" to people scraping your website?
- gary_0 1y agoThe Marginalia search engine or archive.org probably don't deserve such treatment--they're performing a public service that benefits everyone, for free. And it's generally not in one's best interests to serve a bunch of garbage to Google or Bing's crawlers, either.
- darkwater 1y agoIf you have such a website, then you will just serve normal data. But it seems perfectly legit to serve fake random gibberish from your website if you want to. A human would just stop reading it.
- marginalia_nu 1y agoIt's not really too big of a problem for a well-implemented crawler. You basically need to define an upper bound both in terms of document count and time for your crawls, since crawler traps are pretty common and have been around since the cretaceous.
- suspended_state 1y ago
- kazinator 1y agoFaking a JPEG is not only less CPU intensive than making one properly, but by doing os you are fuzzing whatever malware is on the other end; if it is decoding the JPEG and isn't robust, it may well crash.
- Szpadel 1y agothe worst offender I saw is meta. they have facebookexternalhit bot (they sometimes use default python request user agent) that (as they documented) explicitly ignores robots.txt it's (as they say) used to validate links if they contain malware. But if someone would like to serve malware the first thing they would do would be to serve innocent page to facebook AS and their user agent. they also re-check every URL every month to validate if this still does not contain malware. the issue is as follows some bad actors spam Facebook with URLs to expensive endpoints (like some search with random filters) and Facebook provides then with free ddos service for your competition. they flood you with > 10 r/s for days every month.
- kuschku 1y agoSince when is 10r/s flooding? That barely registers as a blip even if you're hosting your site on a single server.
- Szpadel 1y agoIn our case this was very heavy specialized endpoint and because each request used different set of parameters could not benefit from caching (actually in this case it thrashed caches with useless entries). This resulted in upscale. When handling such bot cost more than rest of the users and bots, that's an issue. Especially for our customers with smaller traffic. This request rate varied from site to site, but it ranged from half to 75% of whole traffic and was basically saturating many servers for days if not blocked.
- anarazel 1y agoThat depends on what you're hosting. Good luck if it's e.g. a web interface for a bunch of git repositories with a long history. You can't cache effectively because there's too many pages and generating each page isn't cheap.
- oofbey 1y agoIf you're serving static pages through nginx or something, then 10/sec is nothing. But if you're running python code to generate every page, it can add up fast.
- BubbleRings 1y agoIs there reason you couldn’t generate your images by grabbing random rectangles of pixels from one source image and pasting it into a random location in another source image? Then you would have a fully valid jpg that no AI could easily successfully identify as generated junk. I guess that would require much more CPU than your current method huh?
- adgjlsfhk1 1y agoGiven the amount of money AI companies have, you need at least ~100x work amplification for this to begin to be a punishment.
- jekwoooooe 1y agoIt’s our moral imperative to make crawling cost prohibitive and also poison LLM training.
- a-biad 1y agoI am bit confused about the context. What is exactly the point of exposing fake data to webcrawlers?
- jeroenhd 1y agoThis makes me wonder if there are more efficient image formats that one might want to feed botnets. JPEG is highly complex, but PNG uses a relatively simple DEFLATE stream as well as some basic filters. Perhaps one could make a zip-bomb like PNG that only consists of a few bytes?
- john01dav 1y agoThat might be challenging because you can trivially determine the output file sized based on the dimensions in pixels and pixel format, so if the DEFLATE stream goes beyond that you can stop decoding and discard the image as malformed. Of course, some decoders may not do so and thus would be vulnerable.
- _ache_ 1y agoIs it a problem through ? I'm pretty sure that any check is on the weight of the PNG, not the actual dimension of the image. PNG doesn't have size limitation on the image dimensions (4bytes each). So I bet you can break at least one scrap bot with that.
- sltkr 1y agoDEFLATE has a rather low maximum compression ratio of 1:1032, so a file that would take 1 GB of memory uncompressed still needs to be about 1 MB. ZIP bombs rely on recursion or overlapping entries to achieve higher ratios, but the PNG format is too simple to allow such tricks (at least in the usual critical chunks that all decoders are required to support).
- Domainzsite 1y agoThis is pure internet mischief at its finest. Weaponizing fake JPEGs with valid structure and random payloads to burn botnet cycles? Brilliant. Love the tradeoff thinking: maximize crawler cost, minimize CPU. The Huffman bitmask tweak is chef’s kiss. Spigot feels like a spiritual successor to robots.txt flipping you off in binary.
- rfrey 1y agoBut where is my chocolate cupcake recipe?
- time0ut 1y agoJPEG is fascinating and quite complex. Here is a really excellent high level explanation of how it works: https://www.youtube.com/watch?v=0me3guauqOU https://www.youtube.com/watch?v=0me3guauqOU
- larcanio 1y agoHappy to realize real heros does exists.
- deleted 1y ago[deleted]
- ardme 1y agoOld man yells at cloud, then creates a labyrinth of mirrors for the images of the clouds to reflect back on each other.
- bvan 1y agoLove this.
- sim7c00 1y agolove how u speak about pleasing bots and them getting excited :D fun read, fun project. thanks!
- mhuffman 1y agoI don't understand the reasoning behind the "feed them a bunch of trash" option when it seems that if you identify them (for example by ignoring a robots.txt file) you can just keep them hung up on network connections or similar without paying for infinite garbage for crawlers to injest.
- schroeding 1y agoThe "poisoning the data supply" angle seems to be a common motive, similar to tools like nightshade[1] (for actual images and not just garbage data). [1] https://nightshade.cs.uchicago.edu/whatis.html https://nightshade.cs.uchicago.edu/whatis.html
- mhuffman 1y agoI get that, but paying to do it?
- thayne 1y agoI'm curious how the author identifies the crawlers that use random User Agents and and distinct ip addresses per request. Is there some other indicator that can be used to identify them? On a different note, if the goal is to waste resources for the bot, on potential improvement could be to uses very large images with repeating structure that compress extremely well as jpegs for the templates, so that it takes more ram and cpu to decode them with relatively little cpu and ram required to generate them and bandwidth to transfer them.
- lblume 1y agoGiven that current LLMs do not consistently output total garbage, and can be used as judges in a fairly efficient way, I highly doubt this could even in theory have any impact on the capabilities of future models. Once (a) models are capable enough to distinguish between semi-plausible garbage and possibly relevant text and (b) companies are aware of the problem, I do not think data poisoning will be an issue at all.
- Zecc 1y ago> Once (a) models are capable enough to distinguish between semi-plausible garbage and possibly relevant text https://xkcd.com/810/ https://xkcd.com/810/
- jesprenj 1y agoYes, but you still waste their processing power.
- immibis 1y agoThere's no evidence that the current global DDoS is related to AI.
- bschwindHN 1y agoYou should generate fake but believable EXIF data to go along with your JPEGs too.
- russelg 1y agoThey're taking the valid JPEG headers from images already on their site, so it's possible those are already in place.
- electroglyph 1y agothere's no metadata in the example image
- bigiain 1y agoFake exif data with lat/longs showing the image was taken inside Area 51 or The Cheyenne Mountain Complex or Guantanamo Bay...
- derektank 1y agoFrom the headline that's actually what I was expecting the link to discuss
- mrbluecoat 1y ago> I felt sorry for its thankless quest and started thinking about how I could please it. A refreshing (and amusing) attitude versus getting angry and venting on forums about aggressive crawlers.
- ASalazarMX 1y agoHelped without doubt by the capacity to inflict pain and garbage unto those nasty crawlers.
- dheera 1y ago> So the compressed data in a JPEG will look random, right? I don't think JPEG data is compressed enough to be indistinguishable from random. SD VAE with some bits lopped off gets you better compression than JPEG and yet the latents don't "look" random at all. So you might think Huffman encoded JPEG coefficients "look" random when visualized as an image but that's only because they're not intended to be visualized that way.
- maxbond 1y agoEncoded JPEG data is random in the same way cows are spherical.
- BlaDeKke 1y agoCows can be spherical.
- deleted 1y ago[deleted]
- bigiain 1y agoAnd have uniform density.
- anyfoo 1y agoYeah, but in practice you only get that in a perfect vacuum.
- nasretdinov 1y agoI imagine gravity matters more here than atmosphere
- EspadaV9 1y agoI like this one https://www.ty-penguin.org.uk/~auj/spigot/pics/2025/03/25/fake-jpeg/bannrr.jpg https://www.ty-penguin.org.uk/~auj/spigot/pics/2025/03/25/fa... Some kind of statement piece
- myelinsheep 1y agoAnything with Shakespeare in it?
- EspadaV9 1y agoLooks like he didn't get time to finish https://www.ty-penguin.org.uk/~auj/spigot/pics/2025/03/25/fake-jpeg/william%20shakespear.jpg https://www.ty-penguin.org.uk/~auj/spigot/pics/2025/03/25/fa... Terry Pratchett has one I'd like to think he'd approve of. Just a shame I'm unable to see the 8th colour, I'm sure it's in there somewhere. https://www.ty-penguin.org.uk/~auj/spigot/pics/2025/03/25/fake-jpeg/terry%20pratchett.jpg https://www.ty-penguin.org.uk/~auj/spigot/pics/2025/03/25/fa...
- creatonez 1y agoFor the full experience: Firefox: Press F12, go to Network, click No Throttling > change it to GPRS Chromium: Press F12, go to Network, click No Throttling > Custom > Add Profile > Set it to 20kbps and set the profile
- extraduder_ire 1y agoGood mention. There's probably some good art to be made by serving similar jpeg images with the speed limited server-side.
- hashishen 1y agothe hero we needed and deserved
- derefr 1y ago> It seems quite likely that this is being done via a botnet - illegally abusing thousands of people's devices. Sigh. Just because traffic is coming from thousands of devices on residential IPs, doesn't mean it's a botnet in the classical sense. It could just as well be people signing up for a "free VPN service" — or a tool that "generates passive income" for them — where the actual cost of running the software, is that you become an exit node for both other "free VPN service" users' traffic, and the traffic of users of the VPN's sibling commercial brand. (E.g. scrapers like this one.) This scheme is known as "proxyware" — see https://www.trendmicro.com/en_ca/research/23/b/hijacking-your-bandwidth-how-proxyware-apps-open-you-up-to-risk.html https://www.trendmicro.com/en_ca/research/23/b/hijacking-you...
- jeroenhd 1y agoThat's just a variant of a botnet that the users are willingly joining. Someone well-intentioned should probably redirect those IP addresses to a "you are part of a botnet" page just in case they find the website on a site like HN and don't know what their family members are up to. Easiest way to deal with them is just to block them regardless, because the probability that someone who knows what to do about this software and why it's bad will read any particularly botnetted website are close to zero.
- cAtte_ 1y agosounds like a botnet to me
- whatsupdog 1y agoBotnet with extra steps.
- ronsor 1y agobecause it is, but it's a legal botnet
- derefr 1y agoEh. To me, a bot is something users don't know they're running, and would shut off if they knew it was there. Proxyware is more like a crypto miner — the original kind, from back when crypto-mining was something a regular computer could feasibly do with pure CPU power. It's something users intentionally install and run and even maintain, because they see it as providing them some potential amount of value. Not a bot; just a P2P network client. Compare/contrast: https://en.wikipedia.org/wiki/Winny https://en.wikipedia.org/wiki/Winny / https://en.wikipedia.org/wiki/Share_(P2P) https://en.wikipedia.org/wiki/Share_(P2P) / https://en.wikipedia.org/wiki/Perfect_Dark_(P2P) https://en.wikipedia.org/wiki/Perfect_Dark_(P2P) — pieces of software which offer users a similar devil's bargain, but instead of "you get a VPN; we get to use your computer as a VPN", it's "you get to pirate things; we get to use your hard drive as a cache node in our distributed, encrypted-and-striped pirated media cache." (And both of these are different still to something like BitTorrent, where the user only ever seeds what they themselves have previously leeched — which is much less questionable in terms of what sort of activity you're agreeing to play host to.)
- marcod 1y agoReading about Spigot made me remember https://www.projecthoneypot.org/ https://www.projecthoneypot.org/ I was very excited 20 years ago, every time I got emails from them that the scripts and donated MX records on my website had helped catching a harvester > Regardless of how the rest of your day goes, here's something to be happy about -- today one of your donated MXs helped to identify a previously unknown email harvester (IP: 172.180.164.102). The harvester was caught a spam trap email address created with your donated MX:
- notpushkin 1y agoThis is very neat. Honeypot scripts are fairly outdated though (and you can’t modify them according to ToS). The Python one only supports CGI and Zope out of the box, though I think you can make a wrapper to make it work with WSGI apps as well.