7 ms·
> But companies that “crawl” Reddit for data It's not possible to hide crawling at a large enough scale, right? At some point, certain IPs/user agents will (sh
by MuffinFlavored 4y ago
> But companies that “crawl” Reddit for data
It's not possible to hide crawling at a large enough scale, right? At some point, certain IPs/user agents will (should be?) hit with CAPTCHAs to be able to have access to content and no amount of user agent/cookie/session/whatever spoofing will get around that, yeah?
- wswope 4y agoCat & mouse game. If you’re defending against a whitehat business scraping with curl from data center IPs, sure. Against a less-savory actor using hundreds of IPs from residential proxies/compromised hosts, you’re gonna have a rough time, especially if you’re unwilling or unable to use aggresive fingerprinting or (vomit) CloudFlare. Not to mention CAPTCHAs are generally already a solved problem for scrapers.
- AlphaSite 4y agoRequire auth and it’ll help a ton.
- Zak 4y agoReddit wants people to visit the site, become interested in the content they see, and start participating regularly. That's not compatible with hiding enough content behind a registration wall to thwart sufficiently sophisticated scraping.
- thomastjeffery 4y agoOne of the core features of Reddit is that any person may create as many accounts as they like for free. Changing that would be incredibly disruptive. You could set a minimum karma threshold, but that would only promote karma farming; which is already widespread.
- MuffinFlavored 4y ago> One of the core features of Reddit is that any person may create as many accounts as they like for free. Changing that would be incredibly disruptive. I wonder what their monthly active users look like if you filter out 1 person switching through 3 usernames/accounts for example.
- deleted 4y ago[deleted]
- asdadsdad 4y agoreddit might be one of the few last places on the internet that hold the old times of pseudo-names and mindless anonymity. I don't see how changing that would benefit the company. see twitter
- kayson 4y agoWhy is cloudflare vomit-inducing?
- wswope 4y agoWhile I appreciate it as an irreplaceable tool for countering DDoS, its premise is antithetical to a reliable and open web IMO, and it suffers from the same lack of accessible, customer-facing support as other big tech players. Lazy examples from HN algolia search: https://news.ycombinator.com/item?id=32912075 https://news.ycombinator.com/item?id=32912075 https://news.ycombinator.com/item?id=17750801 https://news.ycombinator.com/item?id=17750801 https://news.ycombinator.com/item?id=22109969 https://news.ycombinator.com/item?id=22109969 https://news.ycombinator.com/item?id=30764757 https://news.ycombinator.com/item?id=30764757 https://news.ycombinator.com/item?id=29839960 https://news.ycombinator.com/item?id=29839960 https://news.ycombinator.com/item?id=22406277 https://news.ycombinator.com/item?id=22406277 https://news.ycombinator.com/item?id=23897705 https://news.ycombinator.com/item?id=23897705 https://news.ycombinator.com/item?id=34639212 https://news.ycombinator.com/item?id=34639212 (Hypocrisy disclaimer: I have sites behind CloudFlare.)
- gnicholas 4y agoUpvoted for “hypocrisy disclaimer”
- tomwheeler 4y agoI intensely dislike them taking over as gatekeepers of the web. Perhaps because my browser is configured to resist fingerprinting and to avoid running arbitrary scripts from random websites, it is very frequently blocked by Cloudflare. As one example, I can no longer browse the site for Lowe's (big box home improvement chain). Consequently, I now buy everything from Home Depot (their competitor). It's astonishing how Cloudflare can do such a poor job of determining the difference between a potential customer and an attacker. Life's too short to solve captchas for an intermediary, so I don't bother, I just find a competitor who wants my business.
- travisjungroth 4y ago
- SheinhardtWigCo 4y agoResidential proxies are a completely solved problem, for companies that actually lose money to them (e.g. Ticketmaster, whose profit is maximized by blocking third-party scalpers so they can do the scalping themselves) For companies that make money by having more MAUs, well, yeah, they're going to have a real "rough time" detecting inauthentic traffic
- zer0tonin 4y agoIt's possible but might end up more expensive (and definitely less reliable) than just paying whatever reddit asks for.
- stingraycharles 4y agoThere are residential proxies that allow you bypass most of these things. I’ve been using them to crawl e.g. Amazon or Instagram without any issues, but they’re expensive. IIRC something like $10/GB
- sonofhans 4y agoSerious question — is “residential proxies” a euphemism for “botnet?”
- lobsterthief 4y agoI’m curious about this too.
- jamesfinlayson 4y agoLast week someone here said that some of the big VPN players use botnets to residential IP addresses. I assumed they got residential IP addresses from ISPs but maybe not all ISPs in all parts of the world offer that.
- bikingbismuth 4y agoEssentially yes. Sometimes it also includes people who have installed “free” VPNs.
- aeyes 4y agoNo because they are only HTTP proxies. But you don’t actually know how these companies get them, rumor is that they are part of browser extensions or free VPNs which users might install on their devices. The most “reputable” company in this space is Bright Data (formerly Luminati).
- sleepybrett 4y agoYes and also people who install certain shady "VPN" software.
- celestialcheese 4y ago
- henryfjordan 4y agoThere are services out there that have a large pool of consumer IPs that are marketed at crawlers for exactly this reason. A lot of them are either using hacked hardware or one of those free VPN browser plugins so it would be very hard to distinguish the traffic from a legit user.
- pocket_cheese 4y agoIP restrictions are easy to overcome using mobile networks. Basically, mobile networks assigns your device an internal ip and NATs out to a very small pool of ip public addresses. If they block you, they also block a very large chunk of legitimate mobile users. I'm a big ol' dummy when it comes to networking, so I imagine I explained something poorly... so any mobile network nerds feel free to pile on! Captchas are super easy! There's a gagillion captcha bypass services for every type of captcha. Just snag the captcha token, send it in an API call, and then you get a verified captcha token. See CGNAT for more details about mobile networks. https://en.wikipedia.org/wiki/Carrier-grade_NAT#cite_note-ofcom-cgnat-7 https://en.wikipedia.org/wiki/Carrier-grade_NAT#cite_note-of... It's pretty much impossible to stop the top 1% of the most dedicated scrapers without affecting end user experience.
- tjohns 4y ago> IP restrictions are easy to overcome using mobile networks. Only if the connection is over IPv4. The mobile networks were among the first major adopters of IPv6, and most now give each device a unique IPv6 address.
- abhibeckert 4y agoMy mobile device (iPhone) relays most traffic through the nearest Akamai datacenter. So they don't get my IP address. And that datacenter has a massive number of IP addresses, which are rotated.
- denkmoon 4y agoOut of interest how do you know it's being relayed through an Akamai DC? I assume you're talking about private relay which I also use, but I thought cloudflare was the 2nd hop for that?
- lathiat 4y agoThey're using multiple providers including Fastly, Akamai and Cloudflare: https://www.streamingmediablog.com/2021/06/apple-private-relay.html https://www.streamingmediablog.com/2021/06/apple-private-rel...
- oceanplexian 4y agoCAPTCHAs have been broken by primitive AI for a long time (Long before GPT4-like tools). Their only purpose is to deter the lazy bots. User agents, and any other arbitrary HTTP headers, cookies, etc. have been easy to circumvent as long as the internet has existed. The only thing that sort of works is IP reputation but with IPV6 you can have as many legitimate IPs as you want. tl;dr Dedicated crawlers built by sophisticated actors are more or less impossible to defeat.
- wlesieutre 4y agoIt's hard to imagine captchas being a workable solution as better AI models get cheaper. At this point computers are probably already better at solving them than humans are.
- gloosx 4y agoIt is very easy and cheap to scale. 1$ for 1000 captchas solved, 10$ for 1000 proxies. Then you have 1000 users, and these are kinda impossible to distinguish from your typical common users if you cared to randomize the digital fingerprints for each client to some extent. Paid APIs for publicly accessible data are not something that makes sense or works well in this world.
- asdadsdad 4y agowhy would paid scraping services work then?
- gloosx 4y agoYou're right... I went off-lane there. It makes total sense of course since there is a demand for data, and clearly, just a minority of the people can just scrape everything at will, even if it sounds like kids play to me. And actually, it all makes sense now, since pay walling your own API is just throwing some competition to scrapers, which is totally legit. Sometimes a simple question can do a good deed, thank you:)
- asdadsdad 4y agoHa, no worries. I ask because I've also contemplated that kind of thing and realized that I was chasing my own tail perhaps.