5 ms·

It's worth a read, even if it's not obvious what it's about from that title. "To get the actual data, you need to go through a website maintained by the US T

It's worth a read, even if it's not obvious what it's about from that title. "To get the actual data, you need to go through a website maintained by the US Trade Commission. This website has good and bad aspects. On the one hand, it’s slow and clunky and confusing and often randomly fails to deliver any results. On the other hand, when you re-submit, it clears your query and then blocks you for submitting too many requests, which is nice."

andai 1y ago
On a vaguely related note, I hacked together a WayBack machine type thing on top of Common Crawl in like 50 lines of Python. So it transparently lets you load any page directly from the dumps (it loads only those few bytes from the crawl dumps, and gunzips them). It was really easy, and I was surprised it apparently hadn't been done before. The only problem is the index IP blocks you if you query it too often, and it gets queried once for every resource on the page. I've been informed that there are ways around this (download an index?), does anyone know more about that?
- speerer 1y ago
  This sounds a fantasic idea. Will you be releasing it?
- RestartKernel 1y ago
  Couldn't this still work as a service if you run all queries through the front-end?
  - skylerwiernik 1y ago
    I imagine that they'd get blocked by CORS
    - RestartKernel 1y ago
      Good point. CORS somehow didn't cross my mind.
  - andai 1y ago
    Thanks. I meant that light usage from a single user was already enough to get banned. (Though if you rewrite the HTML so there's no resources to fetch, it should be OK...) Also, it's just a proxy, there's no front-end. (Being a 50 line Python script with one dependency (Flask), the idea is that the end user would run it.) It's just http://localhost/URL http://localhost/URL, and it loads it from the hardcoded snapshot. I guess https://localhost/snapshot/url https://localhost/snapshot/url might work better. I'd also like to see how many lines it ends up being if you remove the one dependency. I've done this for other projects and the answer is "not much more" and "also runs way faster for some reason."
    - RestartKernel 1y ago
      I see, thanks for the extra details. I'm pretty sure you can forgo Flask in favour of directly using the underlying WSGI protocol, while keeping the the Web server you are already using (Gunicorn, probably). The Web server would still be a dependency though.
ytpete 1y ago
> even if it's not obvious what it's about from that title Tangent, but: I wish Hacker News was a bit less dogmatic about preferring the original site's title over any better semantic summary the submitter might offer. Blog posts and news headlines have a strong incentive to be clickbait-ey or just plain catchy, which often makes them less informative and in turn makes the HN homepage harder to scan for interesting posts. It also feels like an arbitrary dividing line where if the original title is too long to fit HN's max limit, the submitter edits it and their take on what's a good summary stands. But if the original title could fit, the submitter's headline is often overwritten with the article title by mods even if it's less useful.
- zahlman 1y ago
  Agreed. I feel awkward sharing my blog here, because I title my posts to appeal to a hypothetical regular audience, but I'm trying to build a regular audience by showing it to non-regulars who lack the context to interpret the title. (Or rather: I want the title to be sensible and entertaining in hindsight.)
  - taneq 1y ago
    Could you title your posts in a HN-friendly style and subtitle them with your actual preferred title? And then (if this wouldn’t be considered Poor Form) swap back to the original title once it’s dropped off the front page?
    - zahlman 1y ago
      Possibly, but I already use title and subtitle in other ways. I suppose I could create aliases and redirects instead.
- smcin 1y ago
  Agreed. That's the second in the last few days: https://news.ycombinator.com/item?id=43880951 https://news.ycombinator.com/item?id=43880951
- qwertox 1y ago

andai 1y ago

On a vaguely related note, I hacked together a WayBack machine type thing on top of Common Crawl in like 50 lines of Python. So it transparently lets you load any page directly from the dumps (it loads only those few bytes from the crawl dumps, and gunzips them). It was really easy, and I was surprised it apparently hadn't been done before. The only problem is the index IP blocks you if you query it too often, and it gets queried once for every resource on the page. I've been informed that there are ways around this (download an index?), does anyone know more about that?

speerer 1y ago
This sounds a fantasic idea. Will you be releasing it?
RestartKernel 1y ago
Couldn't this still work as a service if you run all queries through the front-end?
- skylerwiernik 1y ago
  I imagine that they'd get blocked by CORS
  - RestartKernel 1y ago
    Good point. CORS somehow didn't cross my mind.
- andai 1y ago
  Thanks. I meant that light usage from a single user was already enough to get banned. (Though if you rewrite the HTML so there's no resources to fetch, it should be OK...) Also, it's just a proxy, there's no front-end. (Being a 50 line Python script with one dependency (Flask), the idea is that the end user would run it.) It's just http://localhost/URL http://localhost/URL, and it loads it from the hardcoded snapshot. I guess https://localhost/snapshot/url https://localhost/snapshot/url might work better. I'd also like to see how many lines it ends up being if you remove the one dependency. I've done this for other projects and the answer is "not much more" and "also runs way faster for some reason."
  - RestartKernel 1y ago
    I see, thanks for the extra details. I'm pretty sure you can forgo Flask in favour of directly using the underlying WSGI protocol, while keeping the the Web server you are already using (Gunicorn, probably). The Web server would still be a dependency though.

ytpete 1y ago

> even if it's not obvious what it's about from that title Tangent, but: I wish Hacker News was a bit less dogmatic about preferring the original site's title over any better semantic summary the submitter might offer. Blog posts and news headlines have a strong incentive to be clickbait-ey or just plain catchy, which often makes them less informative and in turn makes the HN homepage harder to scan for interesting posts. It also feels like an arbitrary dividing line where if the original title is too long to fit HN's max limit, the submitter edits it and their take on what's a good summary stands. But if the original title could fit, the submitter's headline is often overwritten with the article title by mods even if it's less useful.

zahlman 1y ago
Agreed. I feel awkward sharing my blog here, because I title my posts to appeal to a hypothetical regular audience, but I'm trying to build a regular audience by showing it to non-regulars who lack the context to interpret the title. (Or rather: I want the title to be sensible and entertaining in hindsight.)
- taneq 1y ago
  Could you title your posts in a HN-friendly style and subtitle them with your actual preferred title? And then (if this wouldn’t be considered Poor Form) swap back to the original title once it’s dropped off the front page?
  - zahlman 1y ago
    Possibly, but I already use title and subtitle in other ways. I suppose I could create aliases and redirects instead.
smcin 1y ago
Agreed. That's the second in the last few days: https://news.ycombinator.com/item?id=43880951 https://news.ycombinator.com/item?id=43880951
qwertox 1y ago