4 ms·

Is there a dataset of HN titles? This made me want to fiddle with this, but step one is to get the data, and I don't want to crawl HN if the data has already be

by remram 4y ago

Is there a dataset of HN titles? This made me want to fiddle with this, but step one is to get the data, and I don't want to crawl HN if the data has already been collected.

krapp 4y ago
There's an API[0] but it's frustratingly limited in capabilities (albeit not rate-limited.) You'll have to iterate all post IDs, download each post as JSON and get the titles that way. There's also a Google dataset but I don't know the URL for it or if it's up to date. [0]https://github.com/HackerNews/API https://github.com/HackerNews/API
- capableweb 4y ago
  > frustratingly limited in capabilities What's missing precisely? Seems to be good enough for every use I could think of. One time I even downloaded every single item from it, with a threaded fetcher of I think 16 threads, iterating from 1 up to latest ID and it was done in some like 2 hours I think.
  - krapp 4y ago
    No ability to directly download threads with a single request, for one, or query it like a database to sort or filter results, exclude unwanted fields, etc.
    - capableweb 4y ago
      Those things should be trivial to achieve with most general purpose languages, as the API is so simple. No need for pagination or other things, just request things by ID recursively and you get the full thing, then after than filter/select whatever you want. Pseudo-code to show how simple it would be: function get_thread(id) { let item = http.get(`{api}/?id={id}`).body if item.childs { item.fetched_children = item.childs.map((id) => { return get_thread(id) }) } return item } (untested, but you don't really need more than that, besides checking if the item was deleted)
      - krapp 4y ago
        Yes, and i've done it. But you still wind up having to make a separate request for each item, which makes building threads incredibly slow. It's also a waste of time if you're filtering out anything, because you still have to make the request and download the item to filter it out. Which is why it would be preferable for the API itself to support these features.
FiloSottile 4y ago
There are a few sources. There's the official API [0], the Algolia search API [1], and the BigQuery dataset which is pretty up to date [2]. I used the Algolia search API, it has extremely generous rate limits and page limits. [0]: https://github.com/HackerNews/API https://github.com/HackerNews/API [1]: https://hn.algolia.com/api https://hn.algolia.com/api [2]: https://console.cloud.google.com/bigquery?p=bigquery-public-data&d=hacker_news&page=dataset https://console.cloud.google.com/bigquery?p=bigquery-public-...
lumenwrites 4y ago
Can you somehow fine-tune GPT3 on a dataset? I just assumed the OP generated them using a prompt like "top hacker news threads" or somehing like that.

by remram 4y ago

Is there a dataset of HN titles? This made me want to fiddle with this, but step one is to get the data, and I don't want to crawl HN if the data has already been collected.

krapp 4y ago

There's an API[0] but it's frustratingly limited in capabilities (albeit not rate-limited.) You'll have to iterate all post IDs, download each post as JSON and get the titles that way. There's also a Google dataset but I don't know the URL for it or if it's up to date. [0]https://github.com/HackerNews/API https://github.com/HackerNews/API

capableweb 4y ago
> frustratingly limited in capabilities What's missing precisely? Seems to be good enough for every use I could think of. One time I even downloaded every single item from it, with a threaded fetcher of I think 16 threads, iterating from 1 up to latest ID and it was done in some like 2 hours I think.
- krapp 4y ago
  No ability to directly download threads with a single request, for one, or query it like a database to sort or filter results, exclude unwanted fields, etc.
  - capableweb 4y ago
    Those things should be trivial to achieve with most general purpose languages, as the API is so simple. No need for pagination or other things, just request things by ID recursively and you get the full thing, then after than filter/select whatever you want. Pseudo-code to show how simple it would be: function get_thread(id) { let item = http.get(`{api}/?id={id}`).body if item.childs { item.fetched_children = item.childs.map((id) => { return get_thread(id) }) } return item } (untested, but you don't really need more than that, besides checking if the item was deleted)
    - krapp 4y ago
      Yes, and i've done it. But you still wind up having to make a separate request for each item, which makes building threads incredibly slow. It's also a waste of time if you're filtering out anything, because you still have to make the request and download the item to filter it out. Which is why it would be preferable for the API itself to support these features.

FiloSottile 4y ago

There are a few sources. There's the official API [0], the Algolia search API [1], and the BigQuery dataset which is pretty up to date [2]. I used the Algolia search API, it has extremely generous rate limits and page limits. [0]: https://github.com/HackerNews/API https://github.com/HackerNews/API [1]: https://hn.algolia.com/api https://hn.algolia.com/api [2]: https://console.cloud.google.com/bigquery?p=bigquery-public-data&d=hacker_news&page=dataset https://console.cloud.google.com/bigquery?p=bigquery-public-...

lumenwrites 4y ago

Can you somehow fine-tune GPT3 on a dataset? I just assumed the OP generated them using a prompt like "top hacker news threads" or somehing like that.