4 ms·

One article mentioned in the OP was discussed here: Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 https://ne

by dang 3mo ago

One article mentioned in the OP was discussed here: Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)

fragmede 3mo ago
How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests?
- dredmorbius 3mo ago
  For one datapoint ... I have a custom HN CSS which includes some formatting of different sets of user accounts. Admins, for example, get orange highlighting and a dragon emoji (for one does not meddle in the affairs of ...). Also included are leaders, which is the one part of my CSS build script which is, or at least was until a few minutes ago, dynamic. Presently HN is returning "sorry" to my curl request. Given that I run that build manually a few times a month, it's not a matter of hitting HN with frequent scrapes. But HN has become increasingly scrape-hostile over time. Back in 2023 I did a crawl of all of HN's front-page daily history (365.25 days/year * 17 years, so about 6,200 requests), to answer a question which had come up about what was/wasn't mentioned in submission titles. That scrape included a delay (probably either 1 or 10 seconds, possibly more, I don't recall which and may have run the fetch directly from the command line), and ran (initially) without issues. I don't think it would fly today. I reported on findings at the time and several times since: <https://news.ycombinator.com/item?id=36078578 https://news.ycombinator.com/item?id=36078578> <https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=by%3Adredmorbius%20front%20page%20analysis&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...>
  - fragmede 3mo ago
    HN is exported to firebase, which you can hit directly, for that sort of purpose https://github.com/HackerNews/API https://github.com/HackerNews/API
    - dredmorbius 3mo ago
      I know that. I've not worked with the API, and there's the blessing/curse (blurse‽) that HTML is a known, if poor, standard. API always translates to "one more thing to learn, that's applicable to a single-use case". HTML scraping / sorting I can apply across multiple sites. That said, a standard, say, JSON packaging of website contents available on request might be fun to have.
      - fragmede 3mo ago
        I feel less bad hammering firebase in a "while True:" loop vs hitting HN's servers.
        dredmorbius 3mo ago
        12,000 times less bad? <https://news.ycombinator.com/item?id=48868910 https://news.ycombinator.com/item?id=48868910>
        fragmede 3mo ago
        Between me and you, I don't think Google would register that amongst the noise of just running the service anyway on their monitoring systems. You're right to point out that if you're trying to get the contents of dead objects the API is of no use though.
        dredmorbius 3mo ago
        Google might not register it, but in some related testing, even a few hundred requests through the API (serial, not concurrent) takes much longer than a single "Past" page request would, even if the latter were substantially rate-limited. Of course, if those requests (to HN itself rather than the API) are blocked entirely, that's a moot issue. On dead/flagged items, there's some value. Whilst the title/URL context aren't available, just knowing what fraction of submissions and comments are moderated is interesting data, and it is possible to construct patterns against specific accounts. I'm frequently encountering what appear to be banned accounts. Being able to trace those through the API to see where and when they were banned, or now much moderated activity they're generating, can be useful. I'm relying heavily on the "/replies?id=<UID>&by=<moderator>" search endpoint (generally dang, tomhow, sctb, or pg as mod) currently to find out if there was a specific ban admonishment from a moderator. That's often but not always the case. But it's not possible, say, to tell through the API what sites are banned. Looking at site history with "showdead" enabled can tell you that though, e.g.: <https://news.ycombinator.com/from?site=synthetica.cloud https://news.ycombinator.com/from?site=synthetica.cloud> (From the New queue, one of several "dead" submissions not flagged, suggesting a site ban.) Hypothesizing an undocumented "site" API endpoint ... doesn't seem to check out: https://hacker-news.firebaseio.com/v0/site/synthetica.cloud?print=pretty Returns: not found (Similarly for "domain", "url", and "URL".) And there's the "day" endpoint doesn't seem to work either, though it conspicuously does not report "not found", e.g.: https://hacker-news.firebaseio.com/v0/day/2025-07-10?print=pretty (I've tried a few other date format variants, including Unix time (seconds) without success.)
- Bender 3mo ago
  Not dang and not the person you are asking but there is no CDN. HN is just two servers running BSD one active and one standby. HN is all text so there is not much bandwidth usage. I did an experiment and linked from HN to my lame blog site and disabled all my anti-scraping measures. Even with all the bots I did not see that much traffic. I suspect some people are specifically being targeted by very poorly configured or very poorly written archiving scripts. Just one example thread discussing this with someone on HN [1]. Each case of being targeted will require looking at generalized characteristics but most are easy to stop in my opinion and experience. [1] - https://news.ycombinator.com/item?id=48416693 https://news.ycombinator.com/item?id=48416693
  - dredmorbius 3mo ago
    And the backup is field-tested to fall over within a few hours of the primary ;-) <https://news.ycombinator.com/item?id=32048148 https://news.ycombinator.com/item?id=32048148> <https://news.ycombinator.com/item?id=32031243 https://news.ycombinator.com/item?id=32031243> (AFAIK that specific failure mode has in fact been addressed.)
- dang 3mo ago
  It's really bad. I found myself identifying with everything Jonathan wrote in the OP - so much so that I thought of asking to compare notes on mitigation measures.
  - khurs 3mo ago
    Ive been seeing a 'sorry' message occasionally when accessing older pages. Is that a side effect of whatever you are doing?
    - dang 3mo ago
      It was, but there were too many legit users getting affected, so we turned most of that off a few days ago. Are you still seeing it?
      - khurs 3mo ago
        Not in last day no.
        dang 3mo ago
        Ok that's good - if you get it again can you let us know at hn@ycombinator.com? We definitely don't want to exclude legit users.
        khurs 3mo ago
        Yes will do
        Aachen 3mo ago
        LI've emailed there in May and June about different topics and gotten no reply to a request to confirm receipt. Is there a backup method for when your algorithm throws people's email away without even informing the sender with a delivery failure notification?
        3mo ago

by dang 3mo ago

fragmede 3mo ago

How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests?

dredmorbius 3mo ago
For one datapoint ... I have a custom HN CSS which includes some formatting of different sets of user accounts. Admins, for example, get orange highlighting and a dragon emoji (for one does not meddle in the affairs of ...). Also included are leaders, which is the one part of my CSS build script which is, or at least was until a few minutes ago, dynamic. Presently HN is returning "sorry" to my curl request. Given that I run that build manually a few times a month, it's not a matter of hitting HN with frequent scrapes. But HN has become increasingly scrape-hostile over time. Back in 2023 I did a crawl of all of HN's front-page daily history (365.25 days/year * 17 years, so about 6,200 requests), to answer a question which had come up about what was/wasn't mentioned in submission titles. That scrape included a delay (probably either 1 or 10 seconds, possibly more, I don't recall which and may have run the fetch directly from the command line), and ran (initially) without issues. I don't think it would fly today. I reported on findings at the time and several times since: <https://news.ycombinator.com/item?id=36078578 https://news.ycombinator.com/item?id=36078578> <https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=by%3Adredmorbius%20front%20page%20analysis&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...>
- fragmede 3mo ago
  HN is exported to firebase, which you can hit directly, for that sort of purpose https://github.com/HackerNews/API https://github.com/HackerNews/API
  - dredmorbius 3mo ago
    I know that. I've not worked with the API, and there's the blessing/curse (blurse‽) that HTML is a known, if poor, standard. API always translates to "one more thing to learn, that's applicable to a single-use case". HTML scraping / sorting I can apply across multiple sites. That said, a standard, say, JSON packaging of website contents available on request might be fun to have.
    - fragmede 3mo ago
      I feel less bad hammering firebase in a "while True:" loop vs hitting HN's servers.
      - dredmorbius 3mo ago
        12,000 times less bad? <https://news.ycombinator.com/item?id=48868910 https://news.ycombinator.com/item?id=48868910>
        fragmede 3mo ago
        Between me and you, I don't think Google would register that amongst the noise of just running the service anyway on their monitoring systems. You're right to point out that if you're trying to get the contents of dead objects the API is of no use though.
        dredmorbius 3mo ago
        Google might not register it, but in some related testing, even a few hundred requests through the API (serial, not concurrent) takes much longer than a single "Past" page request would, even if the latter were substantially rate-limited. Of course, if those requests (to HN itself rather than the API) are blocked entirely, that's a moot issue. On dead/flagged items, there's some value. Whilst the title/URL context aren't available, just knowing what fraction of submissions and comments are moderated is interesting data, and it is possible to construct patterns against specific accounts. I'm frequently encountering what appear to be banned accounts. Being able to trace those through the API to see where and when they were banned, or now much moderated activity they're generating, can be useful. I'm relying heavily on the "/replies?id=<UID>&by=<moderator>" search endpoint (generally dang, tomhow, sctb, or pg as mod) currently to find out if there was a specific ban admonishment from a moderator. That's often but not always the case. But it's not possible, say, to tell through the API what sites are banned. Looking at site history with "showdead" enabled can tell you that though, e.g.: <https://news.ycombinator.com/from?site=synthetica.cloud https://news.ycombinator.com/from?site=synthetica.cloud> (From the New queue, one of several "dead" submissions not flagged, suggesting a site ban.) Hypothesizing an undocumented "site" API endpoint ... doesn't seem to check out: https://hacker-news.firebaseio.com/v0/site/synthetica.cloud?print=pretty Returns: not found (Similarly for "domain", "url", and "URL".) And there's the "day" endpoint doesn't seem to work either, though it conspicuously does not report "not found", e.g.: https://hacker-news.firebaseio.com/v0/day/2025-07-10?print=pretty (I've tried a few other date format variants, including Unix time (seconds) without success.)
Bender 3mo ago
Not dang and not the person you are asking but there is no CDN. HN is just two servers running BSD one active and one standby. HN is all text so there is not much bandwidth usage. I did an experiment and linked from HN to my lame blog site and disabled all my anti-scraping measures. Even with all the bots I did not see that much traffic. I suspect some people are specifically being targeted by very poorly configured or very poorly written archiving scripts. Just one example thread discussing this with someone on HN [1]. Each case of being targeted will require looking at generalized characteristics but most are easy to stop in my opinion and experience. [1] - https://news.ycombinator.com/item?id=48416693 https://news.ycombinator.com/item?id=48416693
- dredmorbius 3mo ago
  And the backup is field-tested to fall over within a few hours of the primary ;-) <https://news.ycombinator.com/item?id=32048148 https://news.ycombinator.com/item?id=32048148> <https://news.ycombinator.com/item?id=32031243 https://news.ycombinator.com/item?id=32031243> (AFAIK that specific failure mode has in fact been addressed.)
dang 3mo ago
It's really bad. I found myself identifying with everything Jonathan wrote in the OP - so much so that I thought of asking to compare notes on mitigation measures.
- khurs 3mo ago
  Ive been seeing a 'sorry' message occasionally when accessing older pages. Is that a side effect of whatever you are doing?
  - dang 3mo ago
    It was, but there were too many legit users getting affected, so we turned most of that off a few days ago. Are you still seeing it?
    - khurs 3mo ago
      Not in last day no.
      - dang 3mo ago
        Ok that's good - if you get it again can you let us know at hn@ycombinator.com? We definitely don't want to exclude legit users.
        khurs 3mo ago
        Yes will do
        Aachen 3mo ago
        LI've emailed there in May and June about different topics and gotten no reply to a request to confirm receipt. Is there a backup method for when your algorithm throws people's email away without even informing the sender with a delivery failure notification?
        3mo ago