23 ms·
20% of requests for Wikimedia Commons are for one image of a flower
- dr_dshiv 6y agoThis happening now! Some suspect a failure in a CV training pipeline. Others suggest an extremely popular app with a hotlinked image.
- magicalhippo 6y agoIf it was my site I'd replace the image with goatse and see who complains, but I guess that's a bit drastic for Wikimedia.
- waheoo 6y agoI'd give them a pass.
- ISL 6y agoEven just serving a giant blinking red X gif to 10^-6 of the requests might be sufficient. Only 10^-6 of the "legitimate" requests would be affected, but a whole lot of the "undesireable" requests would see it...
- viraptor 6y agoServing a giant blinking red gif to unsuspecting internet users is a bad idea. https://en.wikipedia.org/wiki/Photosensitive_epilepsy https://en.wikipedia.org/wiki/Photosensitive_epilepsy There was a better idea posted in comments - serve a picture with a very short explanation and an email to contact.
- Taniwha 6y agoBut he's suggesting that they only be served to 10^-6 people - that's one 1,000,000th of a person - I suspect it will have little effect
- EE84M3i 6y agoThere is no reason it needs to blink quickly.
- viraptor 6y agoIt doesn't have to be quick for bad effects. See https://discussions.apple.com/thread/7908738 https://discussions.apple.com/thread/7908738 for example. I can't find the link now, but there was also an article about one of the older designers (IBM?) making sure the terminal cursor blinks in 1:2 ratio to reduce the problem. Just don't give random unsuspecting people blinking images as a rule.
- EE84M3i 6y agoThe first link says "several flashes per second". I was thinking more like a blink every few seconds, which seems to be annoying but not dangerous.
- hinkley 6y agoTarpit the image and it will take care of itself. Same advice I gave a w3c.org admin who was lamenting how much traffic people generate by not caching xml schemas. Yes, you have to serve the requests. But you don't have to try to serve them in 100 ms. If a human is on the other end, 1-2 seconds is just fine. If a human is not, then the human will surely notice when their batch process goes from 3 minutes to 10 minutes because it fetches the same schema 200 times.
- rictic 6y agoCould you end up executing a slow loris style attack on yourself by doing this? I guess a couple seconds won't matter unless the server is already redlining it and the tarpitted traffic is a small proportion.
- hinkley 6y agoWell any time you start yanking levers and spinning dials you'd better know where the breaking points in your system are. If you care about the traffic because you're already having trouble with that many simultaneous requests, then you are definitely not going to solve that problem by increasing the response time by a factor of 10. But an important property of reverse proxies is that once the proxy sees the last byte of the response, the originating server is no longer involved in the transaction. The proxy server is stuck ferrying bits over a slow connection, and hopefully is designed for that sort of work load. If the payload is a static file, as it is in both of these cases, then it should be cheap for the server to retrieve them.
- toast0 6y agoYes, but slowloris isn't really a big deal if you've got a modern http(s) server with async i/o. It costs nearly nothing to have a idle connection while waiting 3 seconds before sendfiling the schema xml.
- mekkkkkk 6y agoCan you not run out of sockets though? I know it used to be a thing anyway. Maybe it's handled somehow nowadays.
- Clewza313 6y agoIt can't be a training pipeline, because the IPs are all around India. Sample code from Stack Overflow being used by some major app is the most likely candidate. It's also possible that the image fetch call is a vestigial appendix that doesn't even display the image, which will make tracking this down extra challenging.
- macawfish 6y agoTragedy of the commons
- deleted 6y ago[deleted]
- IfOnlyYouKnew 6y agoWikimedia is unique in running some of the most popular websites with open access to almost all systems. As someone who has never been on the inside of FAANG, I found it rather interesting to browse around the backend infrastructure. See, for example, their statistics at https://grafana.wikimedia.org/d/000000102/production-logging?orgId=1&refresh=5m https://grafana.wikimedia.org/d/000000102/production-logging...
- ShakataGaNai 6y agoWikimedia's infrastructure is radically different than most FAANG. In large part because 99% (+/-) of their traffic is read only. While Facebook and Google have to do heavy workloads for every click and action taken on their services, Wikimedia can cache basically everything. Allowing them to operate on a tiny fraction of the number of machines (and infrastructure) that the rest of the players do.
- Retric 6y agoYou can do quite a bit of processing per page load without issue. Facebook and Google just take it rather past that point into near absurdity, while still being highly profitable.
- ianlevesque 6y agoYeah why do they keep spending billions to build new datacenters when they could just stop being absurd instead? The contempt on here is crazy sometimes.
- civilized 6y agoOr maybe you're just reading too much into "absurd" which can just be a colorful word for "an extremely huge amount"
- MereInterest 6y agoI don't think that Facebook/Google developers are foolish or incompetent. That would be contempt. Instead, I think that Facebook and Google as conglomerate entities are fundamentally opposed to my right to privacy. That they make decisions to rationally follow their self-interest does not excuse the absurd lengths to which they go to stalk the general population's activities.
- dmurray 6y ago20% of Wikipedia Commons requests to their Singapore servers (EQSIN), not globally. That's still a lot, of course.
- deleted 6y ago[deleted]
- reaperducer 6y ago90,000,000 requests a day. That's some flower.
- gumby 6y agoGiven that they are coming from India it could be 90M single daily requests!
- ceph_ 6y agoWould an AsterNovi-belgii-flower-1mb by any other name smell as sweet?
- ed25519FUUU 6y agoLet’s all just be grateful it wasn’t AsterNovi-belgii-flower-100mb.
- efreak 6y agoAt one point, I had a very small png file that was large enough to crash Netscape and IE, and later firefox. It had large enough dimensions that browsers couldn't handle it. Today I'm sure it would be fine; instead I'm frustrated by my inability to create webp images larger than 16000 pixels tall (i was trying to write a data-saver proxy for reading webtoons)
- deleted 6y ago[deleted]
- Havoc 6y ago
- BugsJustFindMe 6y agoAnd of course, now that the image is linked in the report, I've just added an additional request for it by clicking.
- Wowfunhappy 6y agoWith the amount of traffic its apparently getting, HN probably won't make a big impact. Plus, most of us aren't in India, and most of us have normal user agents.
- tchalla 6y agoCould it be the "Good Morning" like greetings on WhatsApp gone viral ? https://www.wsj.com/articles/the-internet-is-filling-up-because-indians-are-sending-millions-of-good-morning-texts-1516640068 https://www.wsj.com/articles/the-internet-is-filling-up-beca...
- m463 6y agoThe "traditional" way of fixing this would be a goatse.cx redirect of the image. I'm sure there is a more enlightened fix.
- mey 6y agoA permanent redirect to a non-image page (owned by Wikimedia) may achieve the same thing. Either the calling system can't support a HTML response, or it's a webview in which case you could either report an error or provide a notice. Maybe even ask for donations :)
- ed25519FUUU 6y agoOr just downsample the image to a reasonable size and deal with it. Nothing inherently wrong with having a popular image.
- ehwhyreally 6y agoYes there is when you are hotlinking. Hotlinking in general is considered theft, you are using someone elses bandwidth and could even ddos the host if you are not caching the response.
- arbitrage 6y agoit's so easy to mitigate, though, that the fact that one doesn't sorta implies that one might want randos from the internet to use one's resources to view this image. it's not theft if you leave it out for everyone to use.
- newsclues 6y agoExample code that gets copy pasted into production app somewhere?
- rovr138 6y agoPossibility and linked in a couple places. They found examples
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- gertlex 6y agoAfter realizing "wiki[p|m]edia" and "flower" triggered a specific image in my head I was guessing it would be a yellow flower, this one in the corner of https://www.mediawiki.org/wiki/MediaWiki https://www.mediawiki.org/wiki/MediaWiki but nope, more interesting than that!
- astrea 6y agoMy initial thinking is it's in some flower recognition dataset.
- stevefolta 6y agoOr it's India's canonical non-hotdog.
- bpicolo 6y agoHuh, I worked on a site with a similar issue in ~2019. A massive flood of traffic for a single site from Indian mobile apps (~15kqps at peak iirc). I think it ended up being a sort of mobile-based botnet with a bizarre target, which luckily was deduced from some of the headers sent (they all had a random common header).
- downrightmike 6y agoSaw a story recently that Indians were bringing down the internet because of sending good morning messages: https://www.wsj.com/articles/the-internet-is-filling-up-because-indians-are-sending-millions-of-good-morning-texts-1516640068 https://www.wsj.com/articles/the-internet-is-filling-up-beca... I'd bet that this is the flower of the week for them.
- nso 6y agoSays it's been going since last June.
- downrightmike 6y agoWhat's your point?
- smcl 6y agoWhat's "kqps"? It's obviously kilo-{somethings} per second, but I don't know what the {something} would be edit: queries?
- bpicolo 6y agoYep, queries
- dailypeeker 6y ago15000 queries per second
- m3kw9 6y agoMaybe some sort of social network using it as the default profile pic and isn’t caching
- blunte 6y agoAnd now thanks to this HN post, 21% of requests are for that same flower!
- HotVector 6y agoHN ain't that big
- Mattwmaster58 6y agoI was curious what the actual amount of requests HN would have to muster, and with a frequency of 90,000,000 reqs/day, HN would need to hit it with 4,500,000 requests.
- aaron695 6y agoIt seems like it started on the 2020-06-09 https://pageviews.toolforge.org/mediaviews/?project=commons.wikimedia.org&platform=&agent=spider&referer=all-referers&start=2020-05-04&end=2020-06-20&files=AsterNovi-belgii-flower-1mb.jpg https://pageviews.toolforge.org/mediaviews/?project=commons....
- scrps 6y agoThey didn't mention if the 90M requests were unique, perhaps some app doing background refresh and not caching that image?
- bombcar 6y agoI had some random images on a web server years ago - and noticed that something like 99% of my traffic was one image - and searching through refers I realized I was the #1 hit on google images for robot attack cat. Simpler times.
- Tijdreiziger 6y agoCan we see the image? :D
- ConcreteGidget 6y agoYes, but only 10 million times.
- bombcar 6y agohttps://imgur.com/8MMET5V https://imgur.com/8MMET5V - now there are companies that can host things FOR my servants!
- BlueTemplar 6y agoAnyone knows how imgur is able to afford that ?
- kingnothing 6y agoads
- BlueTemplar 6y agoEven though ad-blockers are ever more common ? (I didn't even think about ads, since I never saw them on imgur !)
- kingnothing 6y agoIt's the same way Google pulled in $180B last year and Facebook made $86B.
- crazygringo 6y agoI'll just be looking forward to the follow-up post on HN announcing when they figure out what the culprit was! Per the comments, right now the top suspect seems to be the app "Josh" or another TikTok clone because of how traffic surged immediately after the TikTon ban: https://twitter.com/bwaber/status/1358915338637873154 https://twitter.com/bwaber/status/1358915338637873154
- nos4A2 6y agoPerhaps they can replace the image with an obviously wrong image (and smaller in size), and then wait for someone to complain
- mlester 6y agoanyone check if it's stegnography?
- dane-pgp 6y agoNice idea, but if it's getting 90 million requests per day, then either there are a lot of people requesting the same message (so it's not very secret), or the few people requesting it are very forgetful (in that they have to keep re-requesting the same message multiple times per day). I suppose the contents of the image/"message" could change every day, but presumably that would be very obvious in the edit history of that file[0], unless Wikimedia Commons were suppressing the fact that the file is constantly changing. If they are part of the conspiracy, though, you'd think they would have taken down the task from Phabricator too. [0] https://commons.wikimedia.org/wiki/File:AsterNovi-belgii-flower-1mb.jpg#filehistory https://commons.wikimedia.org/wiki/File:AsterNovi-belgii-flo...
- eyelidlessness 6y agoYou gotta respect the suggested approach to take preventive measures by banning requests to this individual image without a User Agent header and to try to identify who might be affected. I’m sure I’m not the only one here who would just treat it as abuse and ban without followup.
- deleted 6y ago[deleted]
- andrewmatte 6y ago90M requests daily from India? I wonder if KaiOS is checking whether it's got internet access.
- astrea 6y agoI wonder how much additional traffic this investigation brought said image.
- mark-r 6y agoReminds me of the time Netgear routers were hardcoded with the IP address of a NTP server at the University of Wisconsin. https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse#Netgear_and_the_University_of_Wisconsin%E2%80%93Madison https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse#Ne...
- fomine3 6y agoI also remembered accidental Snapchat's NTP pool abuse. https://community.ntppool.org/t/recent-ntp-pool-traffic-increase/18 https://community.ntppool.org/t/recent-ntp-pool-traffic-incr...
- ramraj07 6y agoI'm in India now, is it possible for me to install some traffic snooper and monitor if any wikimedia requests go out? I can then install some popular apps and see if anything bites!
- Wingy 6y agomitmproxy :)
- batch12 6y agoSuperficial reversing shows that the ravn app mentioned, com.app.rcn may use the file as part of a speedtest: com.app.rcn/smali/com/app/rcn/utils/InternetSpeedCalculator.smali: "hxxps://upload.wikimedia.org/wikipedia/commons/1/16/AsterNovi-belgii-flower-1mb.jpg" edit: defanged the link to maybe save the wikimedia team some bytes
- tomglynch 6y agoJust added this comment on the issue: Hi all, I've been doing a bit of research into possible apps that could be causing this and found two potential culprits that I am currently investigating. The first is Mitron TV, an Indian TikTok alternative which was made available again on the app store June 6th (https://indianexpress.com/article/technology/tech-news-technology/mitron-app-back-on-google-play-store-safe-to-download-6444121/ https://indianexpress.com/article/technology/tech-news-techn...). The second is Say Namaste, an Indian Zoom alternative which was launched on the app stores June 9th (https://indianexpress.com/article/technology/tech-news-technology/say-namaste-india-video-conferencing-app-on-google-play-store-6448332/ https://indianexpress.com/article/technology/tech-news-techn...). Both fall into the timeline of huge increases, have millions of users and may be using '1280px-AsterNovi-belgii-flower-1mb.jpg' to check the users internet connection - especially for Say Namaste to ensure video connectivity. I've reached out to some developers at both companies and will report back. Let me know your thoughts. EDIT: I have also noticed the dates match the reopening after lockdown for the whole of India: "This first phase of reopening was termed as "Unlock 1.0"[13] and permitted shopping malls, religious places, hotels and restaurants to reopen from *8 June*." (https://en.wikipedia.org/wiki/COVID-19_lockdown_in_India#Unlock_1.0_(1%E2%80%9330_June) https://en.wikipedia.org/wiki/COVID-19_lockdown_in_India#Unl... ) Tom
- batch12 6y agoBased on this, I just reversed both Android apps and am not seeing strings related to wikimedia nor asternovi. This doesn't mean it's not obfuscated somehow though. The only app I've found the strings in so far is the "ravn" app proposed by @taviso. As mentioned in the twitter thread though it doesn't seem to have the install base to cause this traffic--
- tomglynch 6y agoThanks batch12. In my edit, it could also be related to a check-in app used at public spaces in India - as it increases from the 8th of June which matches when the India-wide lockdown began to lift. Perhaps a reverse of qr code scan checkin apps in India could be useful?
- dayze 6y agoSukhbir Singh just commented: Thank you everyone for the comments and suggestions. I just wanted to share that we have identified the app and will update this task tomorrow. (And yes, it is a mobile app.)
- NKCSS 6y agoWould be curious to get the full story :-/
- audiometry 6y agoit's at the bottom of the linked case.
- owlninja 6y ago>>Thank you everyone for the comments and suggestions. I just wanted to share that we have identified the app and will update this task tomorrow. (And yes, it is a mobile app.) Looks like we will know soon.
- shp0ngle 6y agoAccording to the comments, it's probably an Indian TikTok clone that checks internet connection by downloading the picture.
- deleted 6y ago[deleted]
- Codesleuth 6y agoNow you just need to draw attention to it further by posting it in hacker news. I'm sure none of us are curious to immediately see the picture.
- Zaheer 6y agoWhat's the right way to solve for this generally? A CORS policy wouldn't be effective since it's not a browser requesting the image.
- Ayesh 6y agoIt's likely that the consumer doesn't even display that image. It's probably a dry 1mb download test. Most indian ISPs, even mobile ones are extremely cheap to not matter a 1mb.
- speedgoose 6y agoIf the image is not displayed and the stack can handle a lot of tcp connections, perhaps a reverse http slowloris attack : you send the image response headers as slow as you can to keep the tcp connection open but to make the receiver waste its time. If it's a speed test, they will eventually use another image.
- jacquesm 6y agoAt the height of the browser wars I once woke up to Microsoft hotlinking a small button for downloading our software from the MSN homepage. I tried to reach someone there for hours but nobody cared enough to do something about it. The image was small (no more than a few K), but the millions of requests that page got were enough to totally kill our server. Finally, I replaced the image on there with a 'Netscape Now' button. Within 15 minutes the matter was resolved.
- snoshy 6y agoThe old school response, weaponized without being inappropriate.
- rattray 6y agoThat's hilarious! Did they continue to link to your software after that? (I'm curious - what was your software?)
- jacquesm 6y agoYes, they did, they actually thought it was quite funny. They even cached the actual download once they realized we wouldn't be able to deal with that either. The software was the first version of the public peer-to-peer webcam software I wrote: http://web.archive.org/web/20000510010712/http://www.camarades.com/ http://web.archive.org/web/20000510010712/http://www.camarad...
- rattray 6y agoBeautiful. The internet was a truly different place back then...
- jacquesm 6y agoWith 100K visitors / day or so we were in the top 30 websites world wide in 1998. The really big boosts came from the Space Shuttle webcasts and an Yves St. Laurent fashion show webcast from Paris. Hard to believe now, a typical blog post will already pick up 30K visitors without too much trouble.
- rburhum 6y agoso what is the app? I felt like I read a full blown novel, and the last sentence with the conclusion is missing!
- Liquid_Fire 6y ago> we have identified the app and will update this task tomorrow I guess we will find out tomorrow.
- deleted 6y ago[deleted]
- z3t4 6y agoJust replace the image... what app/site is this? Email me at my@mail.com
- andi999 6y agoTitle is misleading. It is 20% of requests to the eqsin cluster located at Changi airport.
- wiz21c 6y agoFTB (from the bug) : You could even serve another image in its place to this UA, with some text and an email address to contact. You'd probably find out pretty quickly what it is from users of that mysterious thing. A throwaway email address is probably best Really good idea :-)
- simonebrunozzi 6y agoHere's the flower in question [0]. [0]: https://upload.wikimedia.org/wikipedia/commons/thumb/1/16/AsterNovi-belgii-flower-1mb.jpg/1280px-AsterNovi-belgii-flower-1mb.jpg https://upload.wikimedia.org/wikipedia/commons/thumb/1/16/As...
- danparsonson 6y agoPosting the very link they're trying to reduce traffic on doesn't seem like a very helpful thing to do
- smcl 6y agoThey were trying to figure out the root cause of a sudden uptick of millions of requests being made for a given image with no user agent or referer, presumably with a view to notifying the app responsible or figuring out a workaround. A few thousand requests from clearly identifiable as coming from browsers and with a referer header from news.ycombinator.com would not exactly interfere with this and in the grand scheme of things isn't a huge burden in terms of network traffic.
- simonebrunozzi 6y agoI bet everyone on HN would have wanted to take a look eventually.
- Donckele 6y ago“ Thank you everyone for the comments and suggestions. I just wanted to share that we have identified the app and will update this task tomorrow. (And yes, it is a mobile app.)”
- kayxspre 6y agoI remember that MediaWiki installation allowed the configuration that essentially permits the use of Commons files, albeit in that case, the file will be downloaded and cached in the Wiki's own server [1]. That being said, though the image wasn't hotlinked directly, they expressed concerns of DDOS and the possible costs the Foundation has to incur from each load (they even pointed out that it's "fair and reasonable" to point donation link to them). I would be interested to see how the licensing issue will be handled, though. The photographer licensed this photo as GFDL/CC BY-SA 3.0 [2], and hotlinking may break the term of these licenses. 1: https://www.mediawiki.org/wiki/InstantCommons https://www.mediawiki.org/wiki/InstantCommons 2: https://commons.wikimedia.org/wiki/File:AsterNovi-belgii-flower-1mb.jpg https://commons.wikimedia.org/wiki/File:AsterNovi-belgii-flo...
- mitchs 6y agoHeh, a popular consumer electronics product a room mate worked on shipped an update that used example.com as a connectivity test. Apparently they were on pace to rack up $20k/month in server costs. At least their user agent made it obvious who to contact.
- V-2 6y ago"Please avoid adding drive-by comments such as "hello from Hacker News" to this task as they are not helpful. Thank you" Why would anyone do such stuff is, as usual, beyond me... PS. "First!"
- calibas 6y ago999 out of a 1,000 people know better, but when there's thousands of people...
- tyingq 6y agoThe funny thing is, the first instance of that in the thread wasn't "hello from hacker news". It was a "hello to hackernews" from an engineer on the WikiMedia team. https://phabricator.wikimedia.org/T273741#6813995 https://phabricator.wikimedia.org/T273741#6813995
- vultour 6y agoEven more curiously, the person that did that registered an account and put up a profile picture (I assume of himself), just for that comment...
- aquadrop 6y agoIt's not 20% of all requests, it's 20% of media requests to one of the clusters (it's said in the issue description) There are 5 clusters.
- rozab 6y agoThis page someone noticed is very interesting. https://newshimalaya.com/2021/02/09/%E2%9A%93-t273741-investigate-unusual-media-traffic-pattern-for-asternovi-belgii-flower-1mb-jpg-on-commons/ https://newshimalaya.com/2021/02/09/%E2%9A%93-t273741-invest... I was sure I'd seen this website before, and sure enough, it's scraping and rehosting almost everything that's posted on HN...
- bottled_poe 6y agoNothing unusual here, just run of the mill online copyright violation.
- BlueTemplar 6y agoYeah, here's the unusual one : http://n-gate.com/ http://n-gate.com/
- fariss 6y agocan this be some sort of botnet checking whether a host is connected to the internet or not?
- navotgil 6y agoWell... posting it here will only increase the number of requests and will make the investigation harder
- ct520 6y agoWell I hope they implement some good caching
- siltpotato 6y agoFirst thing I think of seeing just the headline: something to do with ML datasets? Now that I've read it: Hmm, never heard of phabricator.
- ip26 6y ago145kB for a connectivity check, ouch. This is a poster child for why many apps guzzle so much data. (On a 500MB/mo plan you start noticing)
- blakesterz 6y agoThey did figure it out, a popular chat app in India (they won't name yet) fetches the image but does not display it. https://phabricator.wikimedia.org/T273741#6815828 https://phabricator.wikimedia.org/T273741#6815828
- airhead969 6y agoThose goddamn white Hare Krishnas are in the airport again, handing out flowers no one wants. Also: What's your vector, Victor?
- airhead969 6y agoDoes anyone here have a sense of humor or know the movie Airplane? Good l'erd.
- Razengan 6y ago> Please avoid adding drive-by comments such as "hello from Hacker News" to this task as they are not helpful. Thank you. Ironic. Dang could save himself from spam, but not others.
- kazinator 6y ago> I would suggest that we start banning requests for this image without a UA Why on Earth would you do anything like that, instead of just renaming the image so the URL stops working (and banning the old URL unconditionally?)