6 ms·
Nice effort, but after one successful NYT session, it fails and treats the access as though it were an end user. But don't take my word for it : try it. One acc
by lutusp 2y ago
Nice effort, but after one successful NYT session, it fails and treats the access as though it were an end user. But don't take my word for it : try it. One access, succeeds. Two or more ... fails.
The reason is the staff at the NYT appear to be very well versed in the technical tricks people use to gain access.
- echelon 2y agoWe need a P2P swarm for content. Just like Bittorrent back in the day. Pop in your favorite news article (or paraphrase it yourself), and everyone gets it. With recommender systems, attention graph modeling, etc. it'd probably be a perfect information ingestion and curation engine. And nobody else could modify the algorithm on my behalf.
- Swarmeggcute 2y agoI get the feeling there is some sort of vector of attack behind this idea, but I'm not well versed enough to figure it out.
- deleted 2y ago[deleted]
- bredren 2y agoI think you have something here. I don’t know about paraphrased versions but it would need to handle content revisions by the publisher somehow.
- WatchDog 2y agoThey probably asynchronously verify that the IP address actually belongs to googlebot, then ban the IP when it fails. Synchronously verifying it, would probably be too slow. You can verify googlebot authenticity by doing a reverse dns lookup, then checking that reverse dns name resolves correctly to the expected IP address[0]. [0]: https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/...
- katzgrau 2y agoThere are easily installable databases of IP block info, super easy to do it synchronously, especially if it’s stored in memory. I run a small group of servers that each have to do it thousands of times per second.
- selcuka 2y ago> Synchronously verifying it, would probably be too slow. Why would it be slow? There is a JSON documenbt that lists all IP ranges on the same page you linked to: https://developers.google.com/static/search/apis/ipranges/googlebot.json https://developers.google.com/static/search/apis/ipranges/go...
- WatchDog 2y agoSure, that's one option, and I don't have any insight into what nyt actually does with regards to it's handling of googlebot traffic. But if I were implementing filtering, I might prefer a solution that doesn't require keeping a whitelist up to date.
- Gooblebrai 2y agoThe whitelist can be updated asynchronously
- bomewish 2y agoMaybe we could use GCP infra and the trick will work better ?
- immibis 2y agoWhich leads to the possibility of triggering a self-inflicted DoS. I am behind a CGNAT right now. You reckon that if I set myself to Googlebot and loaded NYT, they'd ban the entire o2 mobile network in Germany? (or possibly shared infrastructure with Deutsche Telekom - not sure) Not to mention the possibility of just filling up the banned IP table.
- 2y ago
- noodlesUK 2y agoFWIW, if you happen to be based in the U.S., you might find that your local public library provides 3-day NYT subscriptions free of charge, which whilst annoying is probably easier than fighting the paywall. Of course this only applies to the NYT.
- akvadrako 2y agoIn the Netherlands the library provides free access to thousands of newspapers for 5 days after visiting, including The Economist and WSJ, which actually have paywalls that aren't trivial to bypass. https://www.pressreader.com/ https://www.pressreader.com/
- letier 2y agoI just checked and the Berlin public library offers press reader as well. Will need to check that out. Thanks for the tip!
- letier 2y agoHow do you bypass the paywall with it? I can only read the “newspaper version” on their site it seems. Edit: Just read your comment again. I assume that’s exactly what you meant.
- noodlesUK 2y agoPressReader allows reading various newspapers (in their newspaper form), the short sub to NYT I mentioned is a bit different and gives you access to the online version. Example for LA libraries: https://www.lapl.org/new-york-times-digital https://www.lapl.org/new-york-times-digital Your local library might have a similar offering specifically for the NYT.
- 1vuio0pswjnm7 2y ago"The reason is the staff at the NYT appear to be very well versed in the technical tricks people use to gain access." It appears anyone can read any new NYT article in the Internet Archive. I use a text-only browser. I am not using Javascript. I do not send a User-Agent header. Don't take my word for it. Here is an example: https://web.archive.org/web/20240603124838if_/https://www.nytimes.com/2024/06/03/magazine/benjamin-bolger-college-harvard-yale.html https://web.archive.org/web/20240603124838if_/https://www.ny... If I am not mistaken the NYT recently had their entire private Github repository, a very large one, made public without their consent. This despite the staff at the NYT being "well-versed" in whatever it is the HN commenter thinks they are well versed in.
- markerz 2y agohttps://securityboulevard.com/2024/08/the-secrets-of-the-new-york-times-source-code-breach/ https://securityboulevard.com/2024/08/the-secrets-of-the-new... Because I had to learn more, sounds like a pretty bad breach. But I’m still pretty impressed by NYTs technical staff for the most part for the things they do accomplish, like the interactive web design of some very complicated data visualizations.
- redxtech 2y agoWasn't the creator/maintainer of svelte on the NYT data visualisation/web technology team?
- stareatgoats 2y agoRich Harris, creator of Svelte, worked at the Guardian. Svelte has been adopted at NYT however [0]. You might be thinking of Mike Bostock, the creator of D3.js and ObservableHQ.com, who led data visualization work at NYT for several years [1]. I'm not sure if they have people of that magnitude working for them now. [0] https://en.wikipedia.org/wiki/Svelte https://en.wikipedia.org/wiki/Svelte [1] https://en.wikipedia.org/wiki/Mike_Bostock https://en.wikipedia.org/wiki/Mike_Bostock
- roshankhan28 2y agoi tried to access my own website and it says internal server error. i also tried to access Youtube and it said the same.
- lodovic 2y agoSome sites do work, but others such as WSJ just give a blank page. Worse, Economist actively blocks this through Cloudflare.