10 ms·
Wikimedia narrows down the app sendin 90M requests to a pic of flower
- kzrdude 6y ago> To recap, we were aware of the following at this stage: > it is a popular chat/social media mobile app used in India > it sets the User-Agent and Referer to '-' > it fetches the image from Wikimedia Commons but does not display it And then they identified which app it was, but it is not revealed
- hackonr 6y agoAny idea why they don't name the app?
- segfaultbuserr 6y agoWikimedia is simply trying to be polite and not to publicly shame the company.
- luckystarr 6y agoIt could provoke an angry mob, which doesn't help anybody.
- Blikkentrekker 6y agoGiven that it doesn't display it, and that the image u.r.i. is frequently used in example code rather than something with example.com, it was almost certainly an innocent mistake of copying example code.
- lifthrasiir 6y agoIn addition to that, one of the culprits here is a widespread sample code that was carelessly copied to a popular app. Shaming does penalize the other culprit but not that one.
- Thiez 6y agoSurely the code sample is not to blame here? Or do you truly think the author of the sample is also deserving of being called a "culprit"?
- lifthrasiir 6y agoJust to be clear, I don't think every author using this image for their sample code is to blame. I'm specifically looking for someone using the public Wikimedia CDN for speed tests [1] and I think that someone is probably the sample code author. [1] https://news.ycombinator.com/item?id=26073450 https://news.ycombinator.com/item?id=26073450 has located the actual app and intended purpose, for your information.
- MattGaiser 6y ago> it is a popular chat/social media mobile app used in India Indians of Hacker News, what are the likely candidates?
- revendell_elf 6y agoShareChat may be? Or some video apps like TikTok
- kylehotchkiss 6y agoMoj? That was the seemingly most viable TikTok alternative that cropped up.
- bgdam 6y agoConsidering that this seems to have been code accidentally left in while copy pasting from a tutorial, it's very hard to say, without doing the exact same investigation the Wikimedia team did. There are a lot of apps that have launched in India around that time frame with huge numbers of users thanks to nationalistic rhetoric. They are terrible apps, but they are made in India terrible apps, and that apparently is enough to get a large following in India of late.
- rossdavidh 6y agoTo be fair, a nation has to make lots of terrible apps first, before they can make mediocre apps, and then good apps. The logic of wanting homegrown apps could still be correct, even though there is a painful period of terrible apps. Now, what the U.S. excuse is for its terrible apps, I'm not sure...
- 908B64B197 6y ago> They are terrible apps, but they are made in India terrible apps, and that apparently is enough to get a large following in India of late. That's awesome! Every app has to start somewhere. I mean, even ISRO started by transporting rocket parts on bikes, and now has a satellite around Mars. https://www.indiatimes.com/technology/science-and-future/from-bicycle-to-a-billion-dreams-the-inspiring-history-of-isro-powering-india-s-space-fantasy-371930.html https://www.indiatimes.com/technology/science-and-future/fro...
- sydd 6y agoGuys can you stop the "hello from Hacker news" comments? its spam.
- user-the-name 6y agoNever link to issues on bug tracker. This always, without fail, happens. It makes more work for people who are already stressed about trying to solve a problem. Just don't link to bug trackers, ever.
- joadha 6y agoNah, if it's a historically or culturally significant issue, I'm definitely linking to it so people can read about it. Just add a warning not to post meaningless garbage in the ticket's thread, if you're that concerned about it.
- soneca 6y agoThere was ”Hello to Hacker News” from one of the moderators, to which someone replied ”Hello from Hacker News”. No that much of a spam IMO. And it was from one person, not much all of ”guys” in HN
- kumukomo 6y agoNo, just f** o* with your childish s**. We are a legion. We do not forget. We do not serve you or anybody else. We bring chaos when and where we want. Wikipedia is just another target for us. Down with the flames. Losers like you are nothing but worthless air to us.
- lxgr 6y agoThis seems interesting from a legal point of view as well: Is an app downloading, but never displaying, creative commons content infringing on copyright (by not showing correct attribution and violating the CC terms)? Besides copyright, could this be considered theft of service?
- qeternity 6y ago> This seems interesting from a legal point of view as well Is it? If you make a resource freely available to people online, and people access said resource, what's the legal ramification there? It would appear there is no malicious intent which would be necessary to make the case for abuse, and theft of service would be a stretch given that Wikimedia doesn't charge for their service.
- londons_explore 6y agoI think OP is referring to the fact that your device is internally making a copy of the image during the download process, yet the creative commons license requires that copies of an image have attribution. The terms of the license are therefore likely not being met. Very unlikely anyone will care...
- boomboomsubban 6y agoIt's a violation of the terms of use >Disrupting the services by placing an undue burden on a Project website or the networks or servers connected with a Project website;
- boomboomsubban 6y agoThough it's interesting to think of the possible legal ramifications, I doubt there will be a court case. The "damages" looks like about ten terabytes of bandwidth, and lawyer fees would surpass that in days . NTP domains have had a history of similar problems, and they seem to be resolved by apologizing, fixing the problem, and sometimes a donation.
- WJW 6y agoIt's difficult to steal something offered for free. You could try to charge them with attempted DDOS or something like that but since wikimedia did not suffer any actual degradation of service. I think that at most you can go for something like "causing harmful traffic through negligence" but you'd need to prove the traffic was actually harmful. In any case let's not get carried away. 90 million requests for a 70KB file is only 5.8 TB. Wikimedia mentions in their about pages that they are hosted on bare metal servers in various places around the world. Just going on the bandwidth charges of the first provider in the list, that'd be about $30 USD per month if they have the "bulk" pricing or $300 USD per month if they use the list pricing. I don't think that is worth going to court over for the Wikimedia foundation.
- gillesjacobs 6y agoTL;DR The thread doesn't name the specific app, but it's Indian, likely a social Android app, probably Say Namaste or Mutton TV. The right app was found by installing apps and checking the connection logs.
- tomglynch 6y agoWasn't say namaste. I suggested that within the thread and contacted them, but ssingh has said it's not them.
- 4cao 6y ago> 12. By this time, we had isolated the app and were convinced that this is the one that is fetching the image on startup. We could not find the image anywhere in the app, confirming our theory that it fetches the image but does not display it. The analysis stops right when things start to become interesting. I was hoping there'd be a decompiled code snippet to see what the app in question is actually doing with the image, since it's not displaying it.
- inopinatus 6y agoEvidently the download has been specially crafted and surreptitiously emplaced to globally disseminate a steganographically embedded key that decrypts tailored malware aimed at disrupting the [REDACTED] nuclear weapons programme and for which the app is a weaponised delivery sabot distributed and marketed as part of the same covert operation. What I'm trying to say is, the image is a plant
- 4cao 6y ago> What I'm trying to say is [...] Since you seem to be responding to me, how does anything you wrote relate to anything I wrote?
- rob74 6y agoIt's usually not a good idea to explain a joke, and I'm not the GP so I'm not sure that's what they meant, but "plant" has multiple meanings, and I suspect they are referring to meanings 3 to 5 from this list: https://www.oxfordlearnersdictionaries.com/definition/american_english/plant_2 https://www.oxfordlearnersdictionaries.com/definition/americ... . Basically, you were asking for some kind of elaborate meaning to a random image downloaded by a random app, and they provided an elaborate conspiracy theory to match your question...
- 4cao 6y agoThanks, I appreciate your effort but there really isn't any need to explain this to me. I understood the parent comment this way too (i.e. as a snide remark trying way too hard to be funny in the worst possible, low effort, Reddit kind of way). It's just that since it's rare to see this kind of response here, I was wondering if the author was trying to make any finer point, although admittedly that was unlikely to begin with. > Basically, you were asking for some kind of elaborate meaning to a random image downloaded by a random app [...] Basically, since they already traced the culprit with a lot of effort (as opposed to just blocking the request URL/UA string pair, which was also an option), the logical ultimate step to conclude their investigation should be to see what the code does, especially considering it's trivial to do so. While I would not expect to discover any "elaborate meaning" behind it, and never claimed anything like that, I certainly think it would be prudent to check what was going on if I had to decide what to do about it next. For example, it could have been intended as some proxy/filtering/DPI check. I've also seen similar stuff before incorporated into some custom Android builds to generate fake ad traffic. It really beggars belief that on a website called _Hacker_ News it's necessary to explain why sometimes it's worth it to be curious to people who themselves were curious enough to read this story and the associated comments but then halfway through decided their curiosity was satisfied and thus nobody else should be asking any more questions either (to be clear, I'm referring to the parent commenter here).
- segfaultbuserr 6y ago> [...] We will thus hold back the banning of the url for now, awaiting for confirmation of the desired effect to reduce the potential harmful impact on the application users. Given how much "sample code" we found around the internet using that url, it might still be a good idea to merge the patch later just to prevent this from happening again. Looks like a case that the developers carelessly copied and pasted some "sample code" into the app... > [...] it seems that there is no good way to get in touch with them through email (I sent an email to all publicly available channels, only to get back an autoresponder that assumes I'm an user of the app and asking for my phone number). I eventually resorted to DM their CEO on twitter. Resorting to Twitter for support is increasingly common. The importance of having an "abuse@" email (and possibly some social media bots to DM all sysops when a mail arrives)...
- raverbashing 6y agoOn one side, StackOverflow has been a blessing. On the other, it made "Copy Paste Programming" go to eleven. (There was even a C# example the other day that famously broke in a big project but I can't find it) Maybe it would be a case of Stack Overflow linting examples to remove stuff like builtin urls and such. I've seen "developers" complaining that example code with a very explicit >replace this part for your use case< complaining that the example didn't work. I guess making some things harder would just be an overall gain.
- cypressious 6y agoYou're probably thinking about some app not being able to start when some other specific app is running because both were using a GUID copied from SO for implementing single-instance apps. Possibly the SO question in question: https://stackoverflow.com/a/522874/615306 https://stackoverflow.com/a/522874/615306
- raverbashing 6y agoIt's a good example but I think it's not exactly that. Maybe it was a GUID generation code that would generate the same one for every instance? > I know it has something to do with some mythical thing called a mutex, rarely can I find someone that bothers to stop and explain what one of these are. Oh my
- clawoo 6y ago> it fetches the image from Wikimedia Commons but does not display it If I were to guess, they use the picture as a connectivity/speed test. They probably figured Wikipedia has unlimited free bandwidth, so they didn't care.
- londons_explore 6y agoThe fact the picture has "1mb.jpg" in the file name suggests it is 1 megabyte and therefore a good candidate for use as a speed test. However the image is in fact 160 kilobytes, so I suspect whatever speedtest is being done is getting the wrong results...
- segfaultbuserr 6y agoCould be worse. Imagine if the original "1mb.jpg" was correctly linked, boom, 10x more traffic...
- johnx123-up 6y agoRelated: https://news.ycombinator.com/item?id=26072025 https://news.ycombinator.com/item?id=26072025, 20% of requests for Wikimedia Commons are for one image of a flower, 2 days ago
- abetusk 6y agoThis reminds me of a Stack Overflow answer that became popular but instead of using 'example.com', they used some other random, but valid, URL that suddenly created a huge spike in traffic for the unsuspecting web page.
- segfaultbuserr 6y agoI'm now curious, how much traffic does example.com receive? Does it use Anycast? Does IETF publish statistics? Searched and found the answer here, no known statistics, but it's backed by a CDN. * Ask HN: What does traffic to example.com look like? https://news.ycombinator.com/item?id=8057442 https://news.ycombinator.com/item?id=8057442
- GekkePrutser 6y agoThis is going to be so much fun when the owner of foo.bar brings their domain online :D
- oefrha 6y ago> To narrow down the app, we decided to observe connections to the image from clients (phones) to our servers. We did this by opening the popular apps one-by-one and noting down the time. After doing this for all the apps, we then ran this query in Hive: SELECT * FROM wmf.webrequest WHERE year=2021 AND month=2 AND day=9 AND parse_media_file_url(uri_path).base_name='/wikipedia/commons/1/16/AsterNovi-belgii-flower-1mb.jpg' AND webrequest_source='upload' AND uri_host = 'upload.wikimedia.org' AND user_agent='-' AND ip=<IP>; > We then found the specific app that was making the request by matching the time when it was opened and the time image was requested from our servers, restricting the results to the User-Agent '-' and from the IP we tested. Unless I missed something, running mitmproxy/Charles etc. in front of the phone would have been way easier than querying the entirety of Wikimedia server logs and trying to match IP & timing windows.
- simonh 6y agoIf they have the logs in Elasticsearch or something similar it wouldn't be all that hard.
- Muley 6y agoCertificate pinning has made this a pain in the arse.
- oefrha 6y agoYou can only pin your own certificate, not someone else’s. In this case you probably don’t even need SSL proxying to pin down the culprit, as I dare say not many apps connect to wikimedia on startup. You do need SSL proxying to be sure though.
- jgrahamc 6y agoSome time we'll get round to writing this up but there's a small customer of Cloudflare that gets a very high HTTP requests per second rate. It's a simple service (bit like a "what's my IP address" but not that) and it turns out that a quite popular hardware device hard-coded requests to this service and doesn't appear to cache the results and so it asks over and over and over again for the same information. We've contacted the manufacturer and I think it's been patched but the life time of installed equipment is long... Yesterday: over a billion HTTP requests...
- Thiez 6y agoAre you not tempted to just block the requests from these devices, and let the manufacturer take the loss? I imagine serving all those requests is costing real money.
- jgrahamc 6y agoIt's not a TOS violation. It did cause us some ops pain at one point (they were getting hit with > 50,000rps concentrated in certain locations). But one of the reasons Cloudflare can operate our service is we have 3.2 million customers who are doing all sorts of stuff. We get so much stronger from that great variety of traffic.
- pests 6y agoRight. I'm sure you can explain more but I've read due to peering agreements and the like having a lot of one-directional traffic can be a good thing.
- RyJones 6y agoThis is similar to how Qualcomm's DNS servers got knocked off the air. An OEM shipped an update which would query a development TURN server we were running - once per connection, over millions of devices. It was a crazy day.
- DaniloDias 6y agoCould you be more explicit on the nature of the service? I’d like to explore mechanisms for tests that detect IoT devices that misbehave this way (and other ways as well). Your anecdote sounds interesting. Is it unrelated to time servers? Unrelated to internet connectivity tests?
- _kdave 6y agoInsert a delay before the image is returned by the server, watch who starts complaining.
- mooseburger 6y agoIt's pretty lame they didn't name the app, it's clear some programmer(s) somewhere needs the wake up call.
- yborg 6y agoThe most impressive thing about this is that the offending app doesn't even display the image, it was some copypasta code the app developers apparently didn't even understand.
- thitcanh 6y agoWelcome to the Internet. Here we welcome amateurs and it shows. Or not, like in this case.
- trianglem 6y agoDon’t know why you would go with didn’t understand rather than didn’t care. I thought the prevailing theory was that it was used for measuring internet speeds.
- walrus01 6y agothe hard part about this, if it were not wikimedia but some individual person's server, is that the traditional method of using something like an apache rewrite rule to replace the jpg/png with goatse wouldn't work, because the image isn't even being displayed.
- msla 6y agoEven worse: You might be seen as an easy way to Goatse people. Just give the yahoos the URL of an image you're rewriting to be the infamous gape. The traffic would only grow once you came to be known for Goatse as a Service.
- stickfigure 6y ago> Goatse as a Service (YC S21)
- walrus01 6y agoImplemented via several dozen geographically distributed 1U servers with 10 or 100GbE network interfaces, and anycast DNS. You could do it pretty much the same as anycast DNS recursive resolvers, but all sharing the same hostname, IP and TLS keys.
- deleted 6y ago[deleted]
- MPSimmons 6y agoI wonder if the chat app will stop working. I'm betting it is just using this image retrieval as a test to see if it's online.
- GekkePrutser 6y agoI'm pretty sure that this pic was also linked to in some of the examples in the Coding Together course from Apple and Stanford a few years back. It makes sense that's still included deep down in some copy/paste app stuff.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- mro_name 6y agofrankly, I would have redirected the image to their own homepage. If only for the outrageous user agent.
- ChrisArchitect 6y agoDupe More discussion https://news.ycombinator.com/item?id=26072025 https://news.ycombinator.com/item?id=26072025
- olliej 6y agoit's not a dupe. This is the follow on.