6 ms·
If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally ques
by marginalia_nu 5y ago
If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general.
Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching some resource that is expensive that bots in general refuse to lay off. Bots with seconds between the requests rarely get blocked even by CDNs.
- hbgl 5y agoTechnically it should be illegal to scrape websites without the consent of the server owner because it would be a violation of their property rights. That being said, I think that in reality it would be pretty difficult to get courts to agree with you and then to enforce it.
- soheil 5y agoGoogle can access any site without being blocked. They dominate the search space and give little incentive for site owners to allow other bots. I'd say bypassing these measures is fair game while there is a monopoly in search space. We don't want a web that only Google can access. By the way great work on Marginalia search engine, I love it.
- marginalia_nu 5y agoI've honestly not had much problem at all crawling the web as an indie search engine operator. If you want to get past CloudFlare you can register your bot fingerprint with them. A small number of sites has blocked my crawler , but that's almost always been my own fault, and happened a few instances when the crawler was misbehaving and actually fetching too aggressively (or repeatedly). In every case just sending an email to the site explaining what happened and humbly asking for a second chance been enough to be allowed back in. Most website owners don't seem to mind small search engines at all, what they don't want is scrapers that aggressively scrape their entire site 10 times a day, ignoring robots.txt, and being a general nuisance.
- gzer0 5y agoHow did you go about registering your bot fingerprint with Cloudflare? I did a quick search but cannot find anything with regards to that.
- marginalia_nu 5y agoGood question, I can't find the link now either. I found it somewhere on Cloudflare's site. --edit-- Nevermind, here it is: https://support.cloudflare.com/hc/en-us/articles/360035387431#h_5itGQRBabQ51RwT5cNJX8u https://support.cloudflare.com/hc/en-us/articles/36003538743... It should skip to "I run a good bot...", but if it doesn't, that's where you want to scroll.
- kristjansson 5y agoThere's also the alignment of interests between you and the site operators, even for a small search engine. Cooperation improves outcomes for both sides. Whereas the more aggressive scrapers almost certainly have interests that conflict with the site operators, regardless of the costs to serve the traffic itself.
- TruthWillHurt 5y agoYou're forgetting a case where a website offers a garbage API that doesn't provide all the data available via the web interface, either due to neglegance (Apple Store developer console), Security (Google Play Store accounting data), Or financial gain (AppsFlyer "premium API").
- Dah00n 5y agoThat doesn't change anything. The site owner decides what you can and cannot do. If a badly made API meant you could do anything you wanted to then everything could be done to those sites running them. That is not how this work. Outside normal usage you need permission.
- boolemancer 5y agoIf I can access the information in Chrome, I should be able to access the information with my own client, and the site operator shouldn't have any say in that.
- temp8964 5y agoI really don't believe "The site owner decides what you can and cannot do." statement. What is the base of this? This does not seem to apply to anything in the real world. In most cases you have very limited ability to decide what other people cannot do. And other people has mostly infinite choices of what they can do. I never heard anything as broad as you said. What you said is like a person standing on the street with a T-shirt says "do not look at me more than twice" and claim it has a legal binding to the whole world.
- arp242 5y ago> I really don't believe "The site owner decides what you can and cannot do." statement. What is the base of this? This does not seem to apply to anything in the real world. It kind of does though; if I own a store and say that only people with hats can enter, then I'm free to do so. Silly? Yes. Legal? Also yes. There are some circumstances where it's not legal, mostly centred around discrimination. Details on this differ per jurisdiction, but generally speaking you have a right to refuse customers.
- jonatron 5y agoLegitimate uses of scraping include price comparison, and finding copyright or trademark infringement.
- Dah00n 5y ago>Legitimate uses of scraping include price comparison "Legitimate uses" is what the site operator says it is, nothing more nothing less. There are no laws that says you can scrape a site and circumvent their protection against doing so.
- EMIRELADERO 5y agoOn the contrary, there are no laws that say you can't scrape a site. If it's available to the public internet, it's legally scrapable.
- dragonwriter 5y ago> On the contrary, there are no laws that say you can't scrape a site. You are both wrong: copyright law both says you can't (in some cases for some uses) and that you can (under implicit license, fair use, and other rules) in others.
- EMIRELADERO 5y agoDepends on what exactly is being scraped. If it's something like price data or exact values then it isn't protected by copyright at all.
- kingcharles 5y agoPrice data can be protected by copyright as a compliation of data. https://www.bitlaw.com/copyright/database.html https://www.bitlaw.com/copyright/database.html
- 5y ago
- hcta 5y agoI'm a bot, you're a bot, we are all bots, so what does this even mean. If someone is trying to discriminate between one set of visitors that they welcome and another that they deny service to, by definition they are the ones creating an adversarial environment. They are going to have to work to get the terms they want. Asking the players on the opposite side of the table to comply with their imagined rules out of altruism is a non-starter.
- gjs278 5y agoit’s not illegal to scrape a website
- oefrha 5y ago> Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching some resource that is expensive that bots in general refuse to lay off. Bots with seconds between the requests rarely get blocked even by CDNs. Tell that to any Cloudflare site on security level High or "I'm Under Attack!" year round.
- marginalia_nu 5y agoUsually means they have large ongoing problems with aggressive bots.
- herbst 5y agoOr they using it as lazy approach to fix long-dos attacks by competitors.
- marginalia_nu 5y agoWhat is the non-lazy alternative that stops a botnet DDOS without affecting regular visitors?
- herbst 5y agoLong-dos is usually not a botnet but a single or a few attackers requesting a resource heavy page on and on or using a technic like slowloris. If you are using WordPress I am sure you find some signs in your log. The solution other than with ddos is to make the app better. Caching, access control and limiting, etc.
- svnpenn 5y ago> Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive Obviously you haven't used Instagram recently.
- matheusmoreira 5y ago> If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Maybe respect user freedom? If I can access the data using my browser, why can't I access it using my script? Why is Google the only one who can do it? Must they have yet another monopoly? > Bots with seconds between the requests rarely get blocked even by CDNs. I've had scripts that made 1 request per day get blocked for no reason. Not to mention the endless cloudflare javascript bullshit they made me support for it to even work.
- Spivak 5y agoBut you have no freedom when contacting my servers. I can send you 403's just because I don't like your face. There is zero entitlement that you have any access to my servers in any way I don't permit. If I say no automated access then on what grounds to you have to do it anyway? > Why is Google the only one who can do it? Because site operators explicitly allow them automated access. If you want the same treatment you have to ask for it.
- matheusmoreira 5y ago> I can send you 403's just because I don't like your face. Sure. At least then you're being honest. If you hate me, it doesn't matter what user agent I use to access your site. Browsers, scripts, they are all me.
- Spivak 5y agoWhy do you think it’s dishonest to send 403s to bots but not browsers? Method of access matters — you the human might have access to your safety deposit box but the bank is still allowed to make rules about your access — like you have to come during business hours and you can’t send someone on your behalf.
- matheusmoreira 5y ago> you can’t send someone on your behalf I totally can though. If sign a document saying another person can do such and such on my behalf, that person can totally do that. Yes, even at the bank. No different from a user agent, really.