2 ms·

I had a 20 GiB JSON file of everything that has ever happened on Hacker News I'm actually surprised at that volume, given this is a text-only site. Humans have

I had a 20 GiB JSON file of everything that has ever happened on Hacker News I'm actually surprised at that volume, given this is a text-only site. Humans have managed to post over 20 billion bytes of text to it over the 18 years that HN existed? That averages to over 2MB per day, or around 7.5KB/s.

deleted 2y ago
[deleted]
sph 2y ago
2 MB per day doesn't sound like a lot. The amount of posts probably has increased exponentially over the years, especially after the Reddit fiasco when we had our latest, and biggest neverending September. Also, I bet a decent amount of that is not from humans. /newest is full of bot spam.
- samplatt 2y ago
  Plus the JSON structure metadata, which for the average comment is going to add, what, 10%?
  - deleted 2y ago
    [deleted]
  - kevincox 2y ago
    I suspect it is closer to 100% increase for the average comment. If the average comment is a few senteces and the metadata has id, parent id, author, timestamp and a vote count that can add up pretty fast.
- FabHK 2y ago
  Around one book every 12 hours.
xnx 2y ago
20 GB JSON is surprising to me. I have an sqlite file of all HN data that is 20 GB, it would be much larger as JSON.
- wolfgang42 2y ago
  20 GB of JSON is correct; here’s the entire dump straight from the API up to last Monday: $ du -c ~/feepsearch-prod/datasource/hacker-news/data/dump/*.jsonl | tail -n1 19428360 total Not sure how your sqlite file is structured but my intuition is that the sizes being roughly the same sounds plausible: JSON has a lot of overhead from redundant structure and ASCII-formatted values; but sqlite has indexes, btrees, ptrmaps, overflow pages, freelists, and so on.
  - elcritch 2y ago
    Sqlite also doesn’t have fixed types, but uses a tagged value system to store data. Well according to what I’ve read on the topic.
- kortilla 2y ago
  SQLite files are optimized for fast querying, not size.
olalonde 2y ago
7.5KB/s (aka 7500 characters per second) didn't sound realistic... So I did the math[0] and it turns out it's closer to 34 bytes/s (0.03 KB/s). And it's really lower than that because of all the metadata and syntax in the JSON. You were right about the "over 2MB per day" though. [0] Well, ChatGPT did the math but it seems to check out: https://chatgpt.com/share/68124afc-c914-800b-8647-74e7dc4f21b4 https://chatgpt.com/share/68124afc-c914-800b-8647-74e7dc4f21...
NitpickLawyer 2y ago
The entire reddit archive was ~4TB sometime close to them removing the API. That's fully compressed, it used to be hosted on the-eye. There are stil arrrr places where you can torrent the files if you're inclined to do so. A lot of that is garbage, but the early years are probably worth a look at, especially before 2018-2019 when smarter bots came to be.
dredmorbius 2y ago
The total strikes me as small. That's nearly two decades of contributions from several 100k active members, and a few million total. HN is what would have been a substantial social network prior to Facebook, and (largely on account of its modest size and active moderation) a high-value one. I did some modelling of how much contributed text data there was on Google+ as that site was shutting down in 2019. By "text data", I'm excluding both media (images, audio, video), and all the extraneous page throw-weight (HTML scaffolding, CSS, JS). Given the very low participation rates, and finding that posts on average ran about 120 characters (I strongly suspect that much activity was part of a Twitter-oriented social strategy, though it's possible that SocMed posts just trend short), seven years' of history from a few tens of millions of active accounts (out of > 4 billion registered profiles) only amounted to a few GiB. This has a bearing on a few other aspects: - The Archive Team (AT, working with, but unaffiliated with, the Internet Archive, IA) was engaged in an archival effort aimed at G+. That had ... mixed success (much content was archived, one heck of a lot wasn't, very few comments survive (threads were curtailed to the most recent ten or so, absent search it remains fairly useless, those with "vanity accounts" (based on a selected account name rather than a random hash) prove to be even less accessible). In addition to all of that, by scraping full pages and attempting to present the site as it presented online, AT/IA are committing to a tremendous increase in the stored data requirements whilst missing much of what actually made the site actually of interest. - Those interested in storing text contributions of even large populations face very modest storage requirements. If, say, average online time is 45 minutes daily, typing speed is 45 wpm, and only half of online time is spent writing vs. reading, that's roughly 1,000 words/(person*day), or about 6 KiB/(person*day). That's 6 MiB per 1,000 people, 6 GiB per 1 million, 6 PiB per billion. And ... the true values are almost certainly far lower: I'm pretty certain I've overstated writing time (it's likely closer to 10%), and typing speed (typing on mobile is likely closer to 20--30 wpm, if that). E.g., Facebook sees about 2.45 billion "pieces of content" posted per day, of which half is video. If we assume 120 characters (bytes) per post, that's a surprisingly modest amount, substantially less than 300 GiB/day of text data. (Images, audio, and video will of course inflate that markedly). - The amount of non-entered data (e.g., location, video, online interactions, commerce) is the bulk of current data collection / surveillance state & capitalism systems.

deleted 2y ago

[deleted]

sph 2y ago

2 MB per day doesn't sound like a lot. The amount of posts probably has increased exponentially over the years, especially after the Reddit fiasco when we had our latest, and biggest neverending September. Also, I bet a decent amount of that is not from humans. /newest is full of bot spam.

samplatt 2y ago
Plus the JSON structure metadata, which for the average comment is going to add, what, 10%?
- deleted 2y ago
  [deleted]
- kevincox 2y ago
  I suspect it is closer to 100% increase for the average comment. If the average comment is a few senteces and the metadata has id, parent id, author, timestamp and a vote count that can add up pretty fast.
FabHK 2y ago
Around one book every 12 hours.

xnx 2y ago

20 GB JSON is surprising to me. I have an sqlite file of all HN data that is 20 GB, it would be much larger as JSON.

wolfgang42 2y ago
20 GB of JSON is correct; here’s the entire dump straight from the API up to last Monday: $ du -c ~/feepsearch-prod/datasource/hacker-news/data/dump/*.jsonl | tail -n1 19428360 total Not sure how your sqlite file is structured but my intuition is that the sizes being roughly the same sounds plausible: JSON has a lot of overhead from redundant structure and ASCII-formatted values; but sqlite has indexes, btrees, ptrmaps, overflow pages, freelists, and so on.
- elcritch 2y ago
  Sqlite also doesn’t have fixed types, but uses a tagged value system to store data. Well according to what I’ve read on the topic.
kortilla 2y ago
SQLite files are optimized for fast querying, not size.

olalonde 2y ago

7.5KB/s (aka 7500 characters per second) didn't sound realistic... So I did the math[0] and it turns out it's closer to 34 bytes/s (0.03 KB/s). And it's really lower than that because of all the metadata and syntax in the JSON. You were right about the "over 2MB per day" though. [0] Well, ChatGPT did the math but it seems to check out: https://chatgpt.com/share/68124afc-c914-800b-8647-74e7dc4f21b4 https://chatgpt.com/share/68124afc-c914-800b-8647-74e7dc4f21...

NitpickLawyer 2y ago

The entire reddit archive was ~4TB sometime close to them removing the API. That's fully compressed, it used to be hosted on the-eye. There are stil arrrr places where you can torrent the files if you're inclined to do so. A lot of that is garbage, but the early years are probably worth a look at, especially before 2018-2019 when smarter bots came to be.

dredmorbius 2y ago

The total strikes me as small. That's nearly two decades of contributions from several 100k active members, and a few million total. HN is what would have been a substantial social network prior to Facebook, and (largely on account of its modest size and active moderation) a high-value one. I did some modelling of how much contributed text data there was on Google+ as that site was shutting down in 2019. By "text data", I'm excluding both media (images, audio, video), and all the extraneous page throw-weight (HTML scaffolding, CSS, JS). Given the very low participation rates, and finding that posts on average ran about 120 characters (I strongly suspect that much activity was part of a Twitter-oriented social strategy, though it's possible that SocMed posts just trend short), seven years' of history from a few tens of millions of active accounts (out of > 4 billion registered profiles) only amounted to a few GiB. This has a bearing on a few other aspects: - The Archive Team (AT, working with, but unaffiliated with, the Internet Archive, IA) was engaged in an archival effort aimed at G+. That had ... mixed success (much content was archived, one heck of a lot wasn't, very few comments survive (threads were curtailed to the most recent ten or so, absent search it remains fairly useless, those with "vanity accounts" (based on a selected account name rather than a random hash) prove to be even less accessible). In addition to all of that, by scraping full pages and attempting to present the site as it presented online, AT/IA are committing to a tremendous increase in the stored data requirements whilst missing much of what actually made the site actually of interest. - Those interested in storing text contributions of even large populations face very modest storage requirements. If, say, average online time is 45 minutes daily, typing speed is 45 wpm, and only half of online time is spent writing vs. reading, that's roughly 1,000 words/(person*day), or about 6 KiB/(person*day). That's 6 MiB per 1,000 people, 6 GiB per 1 million, 6 PiB per billion. And ... the true values are almost certainly far lower: I'm pretty certain I've overstated writing time (it's likely closer to 10%), and typing speed (typing on mobile is likely closer to 20--30 wpm, if that). E.g., Facebook sees about 2.45 billion "pieces of content" posted per day, of which half is video. If we assume 120 characters (bytes) per post, that's a surprisingly modest amount, substantially less than 300 GiB/day of text data. (Images, audio, and video will of course inflate that markedly). - The amount of non-entered data (e.g., location, video, online interactions, commerce) is the bulk of current data collection / surveillance state & capitalism systems.