5 ms·
> This isn’t an astronomical number, but it’s large enough that we (at least, as a bootstrapped company) have to care about cost efficiency. ... by externalizi
by jsnell 6y ago
> This isn’t an astronomical number, but it’s large enough that we (at least, as a bootstrapped company) have to care about cost efficiency.
... by externalizing the costs to a third party.
In general, I'm really surprised that they published this article. It's like they described exactly the data that somebody working on preventing scraping would need to block this traffic, in totally unnecessary level of detail. (E.g. telling exactly which ASN this traffic would be arriving from, describing the very specific timing of their traffic spikes, the kind of multi-city searches that probably see almost no organic traffic).
I just don't get it. It's like they're intentionally trying to get blocked so that they can write a follow-up "how Google blocked our bootstrapped business" blog post.
- fxtentacle 6y agoOr they just don't understand that what they are doing is illegal. I'm always surprised by the level of ignorance, but I've seen more than one startup burn because the founders didn't understand which taxes were due and, thus, failed to account for them in their pricing.
- aardvark291 6y agoUnethical? Yes. Illegal? How?
- sneak 6y agoIt’s not even unethical. Google publishes the data to the public for all to see.
- digsy 6y agoUnethical to build a business scraping data from a company that makes money scraping data?
- aardvark291 6y agoThe Google Flights data is not "scraped." They interface directly to the airline reservations systems.
- meritt 6y ago> what they are doing is illegal It's not illegal. Google can sue them and bury them in court fees and potentially win a civil suit, but it sure as hell isn't illegal.
- fxtentacle 6y agoI'm pretty sure this is legal wrt CFAA because they are not circumventing any access control mechanism. However, 300k requests per day surely is enough that this could be considered a sort of denial of service attack and violate fair use. If Google wanted to, they would scale down their servers for a day, wait for this traffic peak to hit, document how it made the service unavailable to others, and now you have a valid offense to sue for.
- meritt 6y ago> now you have a valid offense to sue for. Yes. That's also what I said above when I said "Google can sue them". It's not illegal is my entire point. You can sue anyone, for any reason, at any time, and even win. That doesn't make something illegal. If Google sends them a C&D, and expressly forbids them from doing this activity, and implements technical measures to prevent them from doing so, and they continue doing so then they may start approaching the area of illegal (Craigslist v. 3Taps would agree, hiQ v. LinkedIn would disagree).
- bdcravens 6y ago> what they are doing is illegal illegal means violate criminal code. > now you have a valid offense to sue for That's civil, not criminal. It's pedantic, but words matter.
- occamrazor 6y agoThey are using residential proxies to evade rate-limiting by google. I don’t know if it’s enough to trigger CFAA, but it shows at a minimum that they know that what they are doing is not what google considers fair use.
- tmerr 6y agoI am surprised people think this could affect the servers? If they scrape 300k prices from 25k pages a day, and the crawl runs every hour from 15-22 UTC, then that means the 25k is spread across 3.5k pages per crawl. Even if the crawl is aggressive and completes within a minute that is still only 58 QPS.
- temp43t453 6y ago? https://news.ycombinator.com/item?id=22180559 https://news.ycombinator.com/item?id=22180559 i dont think it's unethical in the age of ad driven web
- deleted 6y ago[deleted]
- sdinsn 6y agoScraping public data is not illegal in the US. > I'm always surprised by the level of ignorance Such as the ignorance displayed in your comment?
- sneak 6y ago> E.g. telling exactly which ASN this traffic would be arriving from The article mentions that they are using rotating residential proxies.
- fxtentacle 6y agoWhy would they need those proxies if what they are doing is fully legal?
- asciident 6y agoYou do know that "legal" means "against the law" right? And when you say something is illegal, then you need to produce the law it violates; there's no proof from a company trying to prevent you from doing it.
- jsnell 6y agoAh, thanks! I missed that part in the writeup.
- fsaintjacques 6y agoThere's a very high chance this is monetization of botnets, which adds even more to the overall shadyness.
- dylz 6y agoThis is almost certainly guaranteed to be either monetization of botnets, or inadvertently installed adware.
- splonk 6y agoI agree with you in principle, but having worked on both sides of this, I think there's very little chance they get blocked at their current traffic levels. I do think that if they ever get traction they'll have a lot of problems - there's a reason GDS access to flight availability is slow, expensive, and difficult to implement well. Scraping definitely won't scale.