3 ms·

1. We identify ourselves very clearly in our request header through our user-agent, including a link to our website. It's not "savvy" to look for this. It's b

by jdrock 17y ago

1. We identify ourselves very clearly in our request header through our user-agent, including a link to our website. It's not "savvy" to look for this. It's basic webmaster knowledge. 2. Per another commenter, I can see how the language may be confusing; we'll change it. 3. Webmasters tend to rush to grab their pitchforks instead of thinking things through. We follow all the standard, accepted rules. 4. Some people may not want to use your API because they are interested in more than just your site. Crawling is a one-stop solution to get information from a ton of sites, instead of implementing an API for a single site that is just one data point on the web.

OmarIsmail 17y ago
At the end of the day users of your service won't be accessing our data, which means they'll be missing out on some data, and we'll be missing out on being included in potentially some cool applications. This spat isn't going to have a real effect on either of our businesses in the long run. However, something that may have a bigger effect on your business is the seemingly dismissive and patronizing attitude you have towards webmasters. At this stage of the game you need us way more than we need you. I'm sure in this exchange there's defensiveness on your end because I attacked your company, and you've probably dealt with many PO'd guys like me (patience wears thin easily). However, that's the nature of the web. Us webmasters grab our pitchforks and bitch and moan. I imagine if a large enough contingent of webmasters doesn't like you guys it WILL start to have an effect on your overall business. In which case, it's probably smart business to either: A) play nice, even when we're not, and/or B) Demonstrate a clear business case why it's better for us to expose our data to your crawlers.
- jdrock 17y ago
  The thing is I was playing nice - responding immediately to your Twitter message, changing the rate asap. But then I come onto HN and see a complaint from you without mentioning we responded quickly. Not cool in my book. You're saying people should play nice when you're just telling one side of the story. Edit: I should note the response would have been even faster if you had emailed us from the contact page that's on the website we link to in our request header. Tweets go to us marketing/biz-dev guys. The contact form goes to everyone in the company. I'm not sure how much nicer we can be beyond doing everything we can to be reachable and identifiable.
  - OmarIsmail 17y ago
    By the time we identified you as the problem the damage was already done. In fact, when combined with potential performance penalty that Google may apply (could be 5%, 10%, for one week, two weeks... who knows) the actual amount of damage in lost traffic/business to us could easily run into the thousands. Are you going to reimburse us the damages? Obviously not. I want you to realize that the timeliness of your response is completely irrelevant to the matter at hand. Proper net etiquette would be to have a reasonable limit of an ACTUAL 1 req/sec, not an average. Furthermore, allow only the webmaster of the site to increase the rate. Also, while nofollow sculpting is not the original intention of the metatag, MANY people use it as such, so respect that directive. Also, you can have your system be aware when a site is slow to respond and adjust crawl rates/times as necessary. Also, user-agent protections are not enough. Many webservers have built-in automatic IP-based attack detection/prevention, but as far as I know, most don't have user-agent based systems. The new virtual land is distributed and webservers have to react to that new reality, but the onus is on you to be aware that many people are still running legacy systems. If you guys did that from the beginning there wouldn't be an issue. And I'm sure all those complaints you see on other webmaster forums would largely go away as well. There are many things you can/could've done to prevent this problem. There's things we could have done as well, but placing the blame on us is akin to blaming a robbed family for not locking the door. Sure, said family should have locked the door, but that in no way excuses the robber of their actions. I know it's a lot easier to ask for forgiveness than permission, but when you're performing the equivalent of a DDoS attack and causing immediate and direct financial damages to a company... that doesn't fly.
    - jdrock 17y ago
      The reason the request rate was at 4 was that at some point your site responded to 4 req/sec without problems. User-agent is the accepted and standardized way to communicate with bots. Getting angry at us for following robots.txt is like getting angry at someone for showing you her ID when she goes through airport security. From our perspective, your house put up a sign that said "Hey everyone come in 4 at a time".
    - jacquesm 17y ago

by jdrock 17y ago

OmarIsmail 17y ago

At the end of the day users of your service won't be accessing our data, which means they'll be missing out on some data, and we'll be missing out on being included in potentially some cool applications. This spat isn't going to have a real effect on either of our businesses in the long run. However, something that may have a bigger effect on your business is the seemingly dismissive and patronizing attitude you have towards webmasters. At this stage of the game you need us way more than we need you. I'm sure in this exchange there's defensiveness on your end because I attacked your company, and you've probably dealt with many PO'd guys like me (patience wears thin easily). However, that's the nature of the web. Us webmasters grab our pitchforks and bitch and moan. I imagine if a large enough contingent of webmasters doesn't like you guys it WILL start to have an effect on your overall business. In which case, it's probably smart business to either: A) play nice, even when we're not, and/or B) Demonstrate a clear business case why it's better for us to expose our data to your crawlers.

jdrock 17y ago
The thing is I was playing nice - responding immediately to your Twitter message, changing the rate asap. But then I come onto HN and see a complaint from you without mentioning we responded quickly. Not cool in my book. You're saying people should play nice when you're just telling one side of the story. Edit: I should note the response would have been even faster if you had emailed us from the contact page that's on the website we link to in our request header. Tweets go to us marketing/biz-dev guys. The contact form goes to everyone in the company. I'm not sure how much nicer we can be beyond doing everything we can to be reachable and identifiable.
- OmarIsmail 17y ago
  By the time we identified you as the problem the damage was already done. In fact, when combined with potential performance penalty that Google may apply (could be 5%, 10%, for one week, two weeks... who knows) the actual amount of damage in lost traffic/business to us could easily run into the thousands. Are you going to reimburse us the damages? Obviously not. I want you to realize that the timeliness of your response is completely irrelevant to the matter at hand. Proper net etiquette would be to have a reasonable limit of an ACTUAL 1 req/sec, not an average. Furthermore, allow only the webmaster of the site to increase the rate. Also, while nofollow sculpting is not the original intention of the metatag, MANY people use it as such, so respect that directive. Also, you can have your system be aware when a site is slow to respond and adjust crawl rates/times as necessary. Also, user-agent protections are not enough. Many webservers have built-in automatic IP-based attack detection/prevention, but as far as I know, most don't have user-agent based systems. The new virtual land is distributed and webservers have to react to that new reality, but the onus is on you to be aware that many people are still running legacy systems. If you guys did that from the beginning there wouldn't be an issue. And I'm sure all those complaints you see on other webmaster forums would largely go away as well. There are many things you can/could've done to prevent this problem. There's things we could have done as well, but placing the blame on us is akin to blaming a robbed family for not locking the door. Sure, said family should have locked the door, but that in no way excuses the robber of their actions. I know it's a lot easier to ask for forgiveness than permission, but when you're performing the equivalent of a DDoS attack and causing immediate and direct financial damages to a company... that doesn't fly.
  - jdrock 17y ago
    The reason the request rate was at 4 was that at some point your site responded to 4 req/sec without problems. User-agent is the accepted and standardized way to communicate with bots. Getting angry at us for following robots.txt is like getting angry at someone for showing you her ID when she goes through airport security. From our perspective, your house put up a sign that said "Hey everyone come in 4 at a time".
  - jacquesm 17y ago