5 ms·

The Wiki is in French (2011)

80 points by luu 7y ago

lol768 7y ago
I don't understand how a crawler could change the language of the wiki without being authenticated.
- trampi 7y ago
  > After determining the source IP of the crawler (one of the admins was experimenting with Nutch as a supplement to the wiki's impoverished search capabilities and had authenticated the crawler using their admin credentials) Quoting the article.
  - lol768 7y ago
    Thanks, I missed this on first read! The lines are quite wide. Now wondering what the wiki software was that had implemented it as a GET (of course, opening up to CSRF too if there's no token param) and implemented poor search functionality.
- masklinn 7y ago
  Internal crawling to build a search index: > one of the admins was experimenting with Nutch as a supplement to the wiki's impoverished search capabilities and had authenticated the crawler using their admin credentials
- deleted 7y ago
  [deleted]
walrus01 7y ago
Company wikis should not be accessible from outside a VPN or physically being on the office lan.
- trampi 7y ago
  While your comment is correct, this would not have prevented the issue stated in the article. > After determining the source IP of the crawler (one of the admins was experimenting with Nutch as a supplement to the wiki's impoverished search capabilities and had authenticated the crawler using their admin credentials) The real problem is that a GET request is meant to be side-effect free. A crawler only issuing GET requests should not be able to modify e.g. global settings. Even when using an admin token.
  - avmich 7y ago
    > GET request is meant to be side-effect free Meant, yes, by somebody, but a web service creator can decide otherwise, for some reasons, like simplicity. If I remember correctly, there was a good story about Viaweb, how they figured that sending requests to follow links can be used as commands - and I wasn't sure they didn't use GET for that... but maybe I'm wrong.
    - TuringTest 7y ago
      > Meant, yes, by somebody, but a web service creator can decide otherwise, for some reasons, like simplicity. As the conclusion of the article shows, if you do that you're breaking the contracts of the web platform. It's a sure way to build a web service that won't play along well with other web services, like a search crawler.
    - mcv 7y ago
      > Meant, yes, by somebody, but a web service creator can decide otherwise, for some reasons, like simplicity. Simply introducing unexpected side effects, you mean? Using methods in ways they're not meant to be used doesn't create simplicity, it creates complexity. Suddenly there are exceptions to the standards that you need to remember and take into account. This story demonstrates that very well. Laziness is not the same thing as simplicity.
    - cesarb 7y ago
      > but a web service creator can decide otherwise This is like "undefined behavior" in C. Sure, you can make a GET have side effects in your application, but everything else will still assume that HEAD and GET is free of side effects, and might repeat requests, omit requests (using cached data), or even do requests speculatively in advance.
- ThePadawan 7y ago
  The article is unclear whether this applies in this scenario. It describes an admin using their own credentials to run a crawler - it seems quite possible that the crawler was also running on a local development machine or local server.
pierlu 7y ago
Modern crawlers can easily follow javascript or some basic POSTs. So, apart from the insane state-changing GET requests in the code, unless you are spidering your own application, I would not give a crawler admin credential. Anyway a very nice anecdote of the two-thousands decade. Never ever forget the basics.
- deleted 7y ago
  [deleted]
skrebbel 7y ago
Pretty lucky that the "delete everything, really, also the backups!" button needed a POST.
ncmncm 7y ago
Funny, the original author appears not aware of the derivation of the expression "all too".
collyw 7y ago
I have seen a lot of web apps doingthe opposite - using POST requests for everything. Has this ever caused anyone problems?
- vollmond 7y ago
  It makes it impossible to bookmark a specific search result, for one example; either the page of results or a specific instance, because the bookmarked URL doesn't have any of the search/ID metadata that a GET-loaded page would.
klodolph 7y ago
It could have been worse, and it has been. "The Spider of Doom" (2006) A crawler deleted all of the content in a CMS, because the delete button used a GET request, without authentication, and with the URLs embedded in the HTML. https://thedailywtf.com/articles/The_Spider_of_Doom https://thedailywtf.com/articles/The_Spider_of_Doom
- Drdrdrq 7y ago
  I know of someone who tried to backup company wiki with wget. There were links that would delete pages, "protected" by some JavaScript which didn’t get triggered. Fortunately they had some real backups, but the error seems to be quite common.
  - mort96 7y ago
    Part of the problem is that we don't really have nice ways to send POST requests from something which looks like links, from what I can tell. An <a> tag can only ever send a GET request. It can't be made to submit a form (without JS) afaik. The only simple way to have something the user can click to send a POST request is to use a <button> or <input type="button"> in a form, and styling that to look like a link is absolute hell. I'm sure that if we had proper HTML tools from the start to say, "I want this thing to look like a link, but I want it to send a POST request", we'd at least have less such issues.
    - klodolph 7y ago
      Sure, but styling is possible (http://jsfiddle.net/adardesign/5vHGc/ http://jsfiddle.net/adardesign/5vHGc/) even though it’s not perfect, and requiring JS is completely acceptable for >99% of people. If you want it both ways, you can always use JS to replace the button with a link, that way you get a functional button without JS that looks like a link (but is slightly off), and a well-styled link with JS, and you get zero GET requests that delete resources. Progressive enhancement! It’s a mess, but as far as I’m considered “not messy” HTML+CSS+JS was never an option to begin with, at least for most people.
1337shadow 7y ago
One day I got a similar call from an old customer I had not heard about for 6 months or so "Please help, my customer website has been hacked ! All content has been deleted !". A simple grep delete on the Apache logs showed that a User-Agent called GoogleBot visited all delete pages. Looking at these pages source codes, the developers that coded their website had forgotten to check authentication on a number of administration pages. And of course, the list page showed links to delete each content item, that was intercepted by a javascript callback doing a confirm(), which didn't matter for GoogleBot.
deleted 7y ago
[deleted]
inflatableDodo 7y ago
Reminds me of something that happened to a mate. He created a database for a project and while it was still in testing he generated a very long random number and made it into a URL to reset everything. It was not linked to from anywhere and he sent it once to the client over a Skype message. He gets a call from the client a few days later to ask if there is a problem as the database had emptied itself. He finds one visit to the page in his logs. From an IP located in Redmond. He reckons they were scanning URLs from Skype and using them to help populate Bing.
- ambicapter 7y ago
  Wow, that's...Redmond's fault in this case? You can't guarantee every URL you scan is going to be idempotent. That's a hell of an oversight on their part.
  - lucideer 7y ago
    I completely agree with your comment but... automatically hitting URLs to create link previews in messenger apps is a common practice, and one that seems to have been (frustratingly) accepted by the mainstream of developers and users alike. This might be MS' fault, but it's also Facebook's, Slack's, pre-FB-WhatsApp's, to name just a small few of many. <aside> On the other hand, if you're putting up a GET endpoint that's not only not idempotent, but unauthenticated and wipes your whole DB. Even on a test site. Blaming MS for your troubles is a bit much.
    - Kalium 7y ago
      This is a practice that exposes the tension between protecting users from malicious use and privacy at scale. A lot of malicious links get sent over messenger systems, and loading them is often the only good way to find out which ones are or aren't. But wait! Surely there are other possible approaches. Can't users just report spam? Surely there are clear usage patterns that can be detected too! These are valuable and useful approaches, but have historically proven unequal to the task. It can be tricky to spot a spam campaign spread over N accounts - only the most basic forms of spam are really obvious. URLs are easily disguised and changed. Loading the URLs and checking the contents that come back is one of the better approaches. Users also don't tolerate a bunch of spam. They will migrate away. So, yes, anyone who makes a GET endpoint into an unauthenticated change-things URL deserves all the misery they get.
robbrit 7y ago
And this is why GET requests should not modify state.
thanatropism 7y ago
Once somewhere in 2016 on Facebook I clicked a slate.fr link. Ever since, my Facebook has been in French.

80 points by luu 7y ago

lol768 7y ago

I don't understand how a crawler could change the language of the wiki without being authenticated.

trampi 7y ago
> After determining the source IP of the crawler (one of the admins was experimenting with Nutch as a supplement to the wiki's impoverished search capabilities and had authenticated the crawler using their admin credentials) Quoting the article.
- lol768 7y ago
  Thanks, I missed this on first read! The lines are quite wide. Now wondering what the wiki software was that had implemented it as a GET (of course, opening up to CSRF too if there's no token param) and implemented poor search functionality.
masklinn 7y ago
Internal crawling to build a search index: > one of the admins was experimenting with Nutch as a supplement to the wiki's impoverished search capabilities and had authenticated the crawler using their admin credentials
deleted 7y ago
[deleted]

walrus01 7y ago

Company wikis should not be accessible from outside a VPN or physically being on the office lan.

trampi 7y ago
While your comment is correct, this would not have prevented the issue stated in the article. > After determining the source IP of the crawler (one of the admins was experimenting with Nutch as a supplement to the wiki's impoverished search capabilities and had authenticated the crawler using their admin credentials) The real problem is that a GET request is meant to be side-effect free. A crawler only issuing GET requests should not be able to modify e.g. global settings. Even when using an admin token.
- avmich 7y ago
  > GET request is meant to be side-effect free Meant, yes, by somebody, but a web service creator can decide otherwise, for some reasons, like simplicity. If I remember correctly, there was a good story about Viaweb, how they figured that sending requests to follow links can be used as commands - and I wasn't sure they didn't use GET for that... but maybe I'm wrong.
  - TuringTest 7y ago
    > Meant, yes, by somebody, but a web service creator can decide otherwise, for some reasons, like simplicity. As the conclusion of the article shows, if you do that you're breaking the contracts of the web platform. It's a sure way to build a web service that won't play along well with other web services, like a search crawler.
  - mcv 7y ago
    > Meant, yes, by somebody, but a web service creator can decide otherwise, for some reasons, like simplicity. Simply introducing unexpected side effects, you mean? Using methods in ways they're not meant to be used doesn't create simplicity, it creates complexity. Suddenly there are exceptions to the standards that you need to remember and take into account. This story demonstrates that very well. Laziness is not the same thing as simplicity.
  - cesarb 7y ago
    > but a web service creator can decide otherwise This is like "undefined behavior" in C. Sure, you can make a GET have side effects in your application, but everything else will still assume that HEAD and GET is free of side effects, and might repeat requests, omit requests (using cached data), or even do requests speculatively in advance.
ThePadawan 7y ago
The article is unclear whether this applies in this scenario. It describes an admin using their own credentials to run a crawler - it seems quite possible that the crawler was also running on a local development machine or local server.

pierlu 7y ago

Modern crawlers can easily follow javascript or some basic POSTs. So, apart from the insane state-changing GET requests in the code, unless you are spidering your own application, I would not give a crawler admin credential. Anyway a very nice anecdote of the two-thousands decade. Never ever forget the basics.

deleted 7y ago
[deleted]

skrebbel 7y ago

Pretty lucky that the "delete everything, really, also the backups!" button needed a POST.

ncmncm 7y ago

Funny, the original author appears not aware of the derivation of the expression "all too".

collyw 7y ago

I have seen a lot of web apps doingthe opposite - using POST requests for everything. Has this ever caused anyone problems?

vollmond 7y ago
It makes it impossible to bookmark a specific search result, for one example; either the page of results or a specific instance, because the bookmarked URL doesn't have any of the search/ID metadata that a GET-loaded page would.

klodolph 7y ago

It could have been worse, and it has been. "The Spider of Doom" (2006) A crawler deleted all of the content in a CMS, because the delete button used a GET request, without authentication, and with the URLs embedded in the HTML. https://thedailywtf.com/articles/The_Spider_of_Doom https://thedailywtf.com/articles/The_Spider_of_Doom

Drdrdrq 7y ago
I know of someone who tried to backup company wiki with wget. There were links that would delete pages, "protected" by some JavaScript which didn’t get triggered. Fortunately they had some real backups, but the error seems to be quite common.
- mort96 7y ago
  Part of the problem is that we don't really have nice ways to send POST requests from something which looks like links, from what I can tell. An <a> tag can only ever send a GET request. It can't be made to submit a form (without JS) afaik. The only simple way to have something the user can click to send a POST request is to use a <button> or <input type="button"> in a form, and styling that to look like a link is absolute hell. I'm sure that if we had proper HTML tools from the start to say, "I want this thing to look like a link, but I want it to send a POST request", we'd at least have less such issues.
  - klodolph 7y ago
    Sure, but styling is possible (http://jsfiddle.net/adardesign/5vHGc/ http://jsfiddle.net/adardesign/5vHGc/) even though it’s not perfect, and requiring JS is completely acceptable for >99% of people. If you want it both ways, you can always use JS to replace the button with a link, that way you get a functional button without JS that looks like a link (but is slightly off), and a well-styled link with JS, and you get zero GET requests that delete resources. Progressive enhancement! It’s a mess, but as far as I’m considered “not messy” HTML+CSS+JS was never an option to begin with, at least for most people.

1337shadow 7y ago

One day I got a similar call from an old customer I had not heard about for 6 months or so "Please help, my customer website has been hacked ! All content has been deleted !". A simple grep delete on the Apache logs showed that a User-Agent called GoogleBot visited all delete pages. Looking at these pages source codes, the developers that coded their website had forgotten to check authentication on a number of administration pages. And of course, the list page showed links to delete each content item, that was intercepted by a javascript callback doing a confirm(), which didn't matter for GoogleBot.

deleted 7y ago

[deleted]

inflatableDodo 7y ago

Reminds me of something that happened to a mate. He created a database for a project and while it was still in testing he generated a very long random number and made it into a URL to reset everything. It was not linked to from anywhere and he sent it once to the client over a Skype message. He gets a call from the client a few days later to ask if there is a problem as the database had emptied itself. He finds one visit to the page in his logs. From an IP located in Redmond. He reckons they were scanning URLs from Skype and using them to help populate Bing.

ambicapter 7y ago
Wow, that's...Redmond's fault in this case? You can't guarantee every URL you scan is going to be idempotent. That's a hell of an oversight on their part.
- lucideer 7y ago
  I completely agree with your comment but... automatically hitting URLs to create link previews in messenger apps is a common practice, and one that seems to have been (frustratingly) accepted by the mainstream of developers and users alike. This might be MS' fault, but it's also Facebook's, Slack's, pre-FB-WhatsApp's, to name just a small few of many. <aside> On the other hand, if you're putting up a GET endpoint that's not only not idempotent, but unauthenticated and wipes your whole DB. Even on a test site. Blaming MS for your troubles is a bit much.
  - Kalium 7y ago
    This is a practice that exposes the tension between protecting users from malicious use and privacy at scale. A lot of malicious links get sent over messenger systems, and loading them is often the only good way to find out which ones are or aren't. But wait! Surely there are other possible approaches. Can't users just report spam? Surely there are clear usage patterns that can be detected too! These are valuable and useful approaches, but have historically proven unequal to the task. It can be tricky to spot a spam campaign spread over N accounts - only the most basic forms of spam are really obvious. URLs are easily disguised and changed. Loading the URLs and checking the contents that come back is one of the better approaches. Users also don't tolerate a bunch of spam. They will migrate away. So, yes, anyone who makes a GET endpoint into an unauthenticated change-things URL deserves all the misery they get.

robbrit 7y ago

And this is why GET requests should not modify state.

thanatropism 7y ago

Once somewhere in 2016 on Facebook I clicked a slate.fr link. Ever since, my Facebook has been in French.