11 ms·

Show HN: 22 GB of Hacker News in SQLite

730 points by keepamovin 10mo ago

Community, All the HN belong to you. This is an archive of hacker news that fits in your browser. When I made HN Made of Primes I realized I could probably do this offline sqlite/wasm thing with the whole GBs of archive. The whole dataset. So I tried it, and this is it. Have Hacker News on your device. Go to this repo (https://github.com/DOSAYGO-STUDIO/HackerBook https://github.com/DOSAYGO-STUDIO/HackerBook): you can download it. Big Query -> ETL -> npx serve docs - that's it. 20 years of HN arguments and beauty, can be yours forever. So they'll never die. Ever. It's the unkillable static archive of HN and it's your hands. That's my Year End gift to you all. Thank you for a wonderful year, have happy and wonderful 2026. make something of it.

yupyupyups 10mo ago
1 hour passed and it's already nuked? Thank you btw
joshcsimmons 10mo ago
Link appears broken
- ra 10mo ago
  confirmed - I wonder what happened?
deleted 10mo ago
[deleted]
asdefghyk 10mo ago
How much space is needed? ...for the data .... Im wondering if it would work on a tablet? ....
- keepamovin 10mo ago
  ~9GB gzipped.
- asdefghyk 10mo ago
  FYI I did NOT see the size info in the title. Impossible to edit / delete my comment now ........
carbocation 10mo ago
That repo is throwing up a 404 for me. Question - did you consider tradeoffs between duckdb (or other columnar stores) and SQLite?
- keepamovin 10mo ago
  No, I just went straight to sqlite. What is duckdb?
  - cess11 10mo ago
    It is very similar to SQLite in that it can run in-process and store its data as a file. It's different in that it is tailored to analytics, among other things storage is columnar, and it can run off some common data analytics file formats.
  - fsiefken 10mo ago
    DuckDB is an open-source column-oriented Relational Database Management System (RDBMS). It's designed to provide high performance on complex queries against large databases in embedded configuration. It has transparent compression built-in and has support for natural language queries. https://buckenhofer.com/2025/11/agentic-ai-with-duckdb-and-smolagents-natural-language-queries-for-analytics/ https://buckenhofer.com/2025/11/agentic-ai-with-duckdb-and-s... "DICT FSST (Dictionary FSST) represents a hybrid compression technique that combines the benefits of Dictionary Encoding with the string-level compression capabilities of FSST. This approach was implemented and integrated into DuckDB as part of ongoing efforts to optimize string storage and processing performance." https://homepages.cwi.nl/~boncz/msc/2025-YanLannaAlexandre.pdf https://homepages.cwi.nl/~boncz/msc/2025-YanLannaAlexandre.p...
  - simonw 10mo ago
    One interesting feature of DuckDB is that it can run queries against HTTP ranges of a static file hosted via HTTPS, and there's an official WebAssembly build of it that can do that same trick. So you can dump e.g. all of Hacker News in a single multi-GB Parquet file somewhere and build a client-side JavaScript application that can run queries against that without having to fetch the whole thing. You can run searches on https://lil.law.harvard.edu/data-gov-archive/ https://lil.law.harvard.edu/data-gov-archive/ and watch the network panel to see DuckDB in action.
wslh 10mo ago
Is this updated regularly? 404 on GitHub as the other comment. With all due respect it would be great if there is an official HN public dump available (and not requiring stuff such as BigQuery which is expensive).
- scsh 10mo ago
  The BQ dataset is only ~17GB and the free tier of BQ lets you query 1TB per month. If you're not doing select * on every query you should be able to do a lot with that.
zX41ZdbW 10mo ago
The query tab looks quite complex with all these content shards: https://hackerbook.dosaygo.com/?view=query https://hackerbook.dosaygo.com/?view=query I have a much simpler database: https://play.clickhouse.com/play?user=play#U0VMRUNUIHRpbWUsIHR5cGUsIHNjb3JlLCBpZCwgdGl0bGUsIHRleHQgRlJPTSBoYWNrZXJuZXdzX2hpc3RvcnkgV0hFUkUgYnkgPSAna2VlcGFtb3ZpbicgT1JERVIgQlkgdGltZSBERVND https://play.clickhouse.com/play?user=play#U0VMRUNUIHRpbWUsI...
- embedding-shape 10mo ago
  Does your database also runs offline/locally in the browser? Seems to be the reason for the large number of shards.
  - zX41ZdbW 10mo ago
    You can run it locally, but it is a client-server architecture, which means that something has to run behind the browser.
abixb 10mo ago
Wonder if you could turn this into a .zim file for offline browsing with an offline browser like Kiwix, etc. [0] I've been taking frequent "offline-only-day" breaks to consolidate whatever I've been learning, and Kiwix has been a great tool for reference (offline Wikipedia, StackOverflow and whatnot). [0] https://kiwix.org/en/the-new-kiwix-library-is-available/ https://kiwix.org/en/the-new-kiwix-library-is-available/
- Barbing 10mo ago
  Oh this should TOTALLY be available to those who are scrolling through sources on the Kiwix app!
- keepamovin 10mo ago
  Oh that's a cool idea. If you want to take a crack at writing the script, the repo is open!
Paul-E 10mo ago
That's pretty neat! I did something similar. I build a tool[1] to import the Project Arctic Shift dumps[2] of reddit into sqlite. It was mostly an exercise to experiment with Rust and SQLite (HN's two favorite topics). If you don't build a FTS5 index and import without WAL (--unsafe-mode), import of every reddit comment and submission takes a bit over 24 hours and produces a ~10TB DB. SQLite offers a lot of cool json features that would let you store the raw json and operate on that, but I eschewed them in favor of parsing only once at load time. THat also lets me normalize the data a bit. I find that building the DB is pretty "fast", but queries run much faster if I immediately vacuum the DB after building it. The vacuum operation is actually slower than the original import, taking a few days to finish. [1] https://github.com/Paul-E/Pushshift-Importer https://github.com/Paul-E/Pushshift-Importer [2] https://github.com/ArthurHeitmann/arctic_shift/blob/master/download_links.md https://github.com/ArthurHeitmann/arctic_shift/blob/master/d...
- s_ting765 10mo ago
  You could check out SQLite's auto_vacuum which reclaims space without rebuilding the entire db https://sqlite.org/pragma.html#pragma_auto_vacuum https://sqlite.org/pragma.html#pragma_auto_vacuum
  - Paul-E 10mo ago
    I haven't tested that, so I'm not sure if it would work. The import only inserts rows, it doesn't delete, so I don't think that is the cause of fragmentation. I suspect this line in the vacuum docs: > The VACUUM command may change the ROWIDs of entries in any tables that do not have an explicit INTEGER PRIMARY KEY. means SQLite does something to organize by rowid and that this is doing most of the work. Reddit post/comment IDs are 1:1 with integers, though expressed in a different base that is more friendly to URLs. I map decoded post/comment IDs to INTEGER PRIMARY KEYs on their respective tables. I suspect the vacuum operation sorts the tables by their reddit post ID and something about this sorting improves tables scans, which in turn helps building indices quickly after standing up the DB.
- Xyra 10mo ago
  Holy cow, I didn't know getting reddit was that straightforward. I am building public readonly-SQL+vector databases optimized for exploring high-quality public commons with Claude Code (https://exopriors.com/scry https://exopriors.com/scry), I so cannot wait until some funding source comes in and I can upgrade to a $1500/month Hetzner server and pay the ~$1k to embed all that.
fao_ 10mo ago
[flagged]
- rantingdemon 10mo ago
  Why do you say that?
  - sundarurfriend 10mo ago
    Because anything that even slightly differs from the standard American phrasing of something must be "LLM generated" these days.
    - JavGull 10mo ago
      With the em dashes I see you. But at this point idrc so long as it reads well. Everyone uses spell check…
      - naikrovek 10mo ago
        I add em dashes to everything I write now, solely to throw people who look for them off. Lots of editors add them automatically when you have two sequential dashes between words — a common occurrence, like that one. And this is is Chrome on iOS doing it automatically. Ooh, I used “sequential”, ooh, I used an em dash. ZOMG AI IS COMING FOR US ALL
        Barbing 10mo ago
        Ya—in fact, globally replaced on iOS (sent from Safari) Also for reference: “this shortcut can be toggled using the switch labeled 'Smart Punctuation' in General > Keyboard settings.”
        3eb7988a1663 10mo ago
        Anyone demonstrating above a high-school vocabulary/reading level is obviously a machine.
simonw 10mo ago
Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqlite.gz https://hackerbook.dosaygo.com/static-shards/shard_1635.sqlite.gz https://hackerbook.dosaygo.com/static-shards/shard_1634.sqlite.gz As I paginated to previous days. It's reminiscent of that brilliant SQLite.js VFS trick from a few years ago: https://github.com/phiresky/sql.js-httpvfs https://github.com/phiresky/sql.js-httpvfs - only that one used HTTP range headers, this one uses sharded files instead. The interactive SQL query interface at https://hackerbook.dosaygo.com/?view=query https://hackerbook.dosaygo.com/?view=query asks you to select which shards to run the query against, there are 1636 total.
- maxloh 10mo ago
  I am curios why they don't use a single file and HTTP Range Requests instead. PMTiles (a distribution of OpenStreetMap) uses that.
  - keepamovin 10mo ago
    This would be a neat idea to try. Want to add a PR? Bench different "hackends" to see how DuckDB, SQLite shards, or range queries perform?
- tehlike 10mo ago
  Vfs support is amazing.
- nextaccountic 10mo ago
  Is there anything more production grade built around the same idea of HTTP range requests like that sqlite thing? This has so much potential
  - simonw 10mo ago
sieep 10mo ago
What a reminder on how text is so much more efficient than video, its crazy! Could you imagine the same amount of knowledge (or dribble) but in video form? I wonder how large that would be.
- ivanjermakov 10mo ago
  Average high quality 1080p60 video has bitrate of 5Mbps, which is equivalent to 120k English words per second. With average English speech being 150wpm, we end up with text being 50 thousand times more space efficient. Converting 22GB of uncompressed text into video essay lands us at ~1PB or 1000TB.
  - sieep 10mo ago
    Cheers to you for doing the math, I hope 2026 is excellent for you!
- fsiefken 10mo ago
  one could use a video llm to generate the video, diagrams or the stills automatically based on the text. except when it's boardgames playthroughs or programming i just transcribe to text, summarise and read youtube video's.
  - Barbing 10mo ago
    Can be nice to pull a raw transcript and have it formatted as HTML (formatting/punctuation fixes applied). Best locally of course to avoid “I burned a lake for this?” guilt.
    - fsiefken 10mo ago
      yes, yt-dlp can download the transcript, and if it's not available i can get the audio file and run it through parakeet locally.
  - deskamess 10mo ago
    How do you read youtube videos? Very curious as I have been wanting to watch PDF's scroll by slowly on a large TV. I am interested in the workflow of getting a pdf/document into a scrolling video format. These days NotebookLM may be an option but I am curious if there is something custom. If I can get it into video form (mp4) then I can even deliver it via plex.
sirjaz 10mo ago
This would be awesome as a cross platform app.
- keepamovin 10mo ago
  Good idea. HN.exe
tevon 10mo ago
The link seems to be down, was it taken down?
- scsh 10mo ago
  Probably just forgot to make it public.
zkmon 10mo ago
Similar to Single-page applications (SPA), single-table application (STA) might become a thing. Just a shard a table on multiple keys and serve the shards as static files, provided that the data is Ok to share, similar to sharing static html content.
- jesprenj 10mo ago
  do you mean single database? it'd be quite hard if not impossible to make applications using a single table (no relations). reddit did it though, they have a huge table of "things" iirc.
  - mburns 10mo ago
    That is a common misconception. > Next, we've got more than just two tables. The quote/paraphrase doesn't make it clear, but we've got two tables per thing. That means Accounts have an "account_thing" and an "account_data" table, Subreddits have a "subreddit_thing" and "subreddit_data" table, etc. https://www.reddit.com/r/programming/comments/z9sm8/comment/c62uz89/ https://www.reddit.com/r/programming/comments/z9sm8/comment/...
    - rplnt 10mo ago
      And the important lesson from that the k/v-like aspect of it. That the "schema" is horizontal (is that a thing?) and not column-based. But I actually only read it on their blog IIRC and never even got the full details - that there's still a third ID column. Thanks for the link.
- jhd3 10mo ago
  [The Baked Data architectural pattern](https://simonwillison.net/2021/Jul/28/baked-data/ https://simonwillison.net/2021/Jul/28/baked-data/)
yread 10mo ago
I wonder how much smaller it could get with some compression. You could probably encode "This website hijacks the scrollbar and I don't like it" comments into just a few bits.
- hamburglar 10mo ago
  It might be a neat experiment to use ai to produce canonicalized paraphrasings of HN arguments so they could be compared directly and compress well.
- jacquesm 10mo ago
  That's at least 45%, then you can leave out all of my comments and you're left with only 5!
- Rendello 10mo ago
  The hard-coded dictionary wouldn't be much stranger than Brotli's: https://news.ycombinator.com/item?id=27160590 https://news.ycombinator.com/item?id=27160590
  - maxbond 10mo ago
    You can use a BPE variant like SentencePiece to identify these patterns rather than hard coding them.
- rossant 10mo ago
  Guilty.
- deleted 10mo ago
  [deleted]
- keepamovin 10mo ago
  Dear it is already compressed using G zip – nine for every SQLlight shard and manifest 22 GB is uncompressed and compressed the entire things about 9 GB
dmarwicke 10mo ago
22gb for mostly text? tried loading the site, it's pretty slow. curious how the query performance is with this much data in sqlite
spit2wind 10mo ago
This is pretty neat! The calendar didn't work well for me. I could only seem to navigate by month. And when I selected the earliest day (after much tapping), nothing seemed to be updated. Nonetheless, random access history is cool.
- keepamovin 10mo ago
  Cna you let me know? I'm sure there's some weirdness lurking there and I want to smooth it out. Calendar is essential.
Sn0wCoder 10mo ago
Site does not load on Firefox console error says 'Uncaught (in promise) TypeError: can't access property "wasm", sqlite3 is null' Guess its common knowledge that SharedArrayBuffer (SQLite wasm) does not work with FF due to Cross-Origin Attacks (i just found out ;). Once the initial chunk of data loads the rest load almost instantly on Chrome. Can you please fix the GitHub link (current 404) would like to peak at the code. Thank you!
- deleted 10mo ago
  [deleted]
- keepamovin 10mo ago
  Damn. Will try to fix for FF. edit: I just tested with FF latest, seems to be working.
  - Sn0wCoder 10mo ago
    Strange now the first few days load (getting a new error) 'Ignoring inability to install OPFS sqlite3_vfs: Cannot install OPFS: Missing SharedArrayBuffer and/or Atomics. The server must emit the COOP/COEP response headers to enable those. See https://sqlite.org/wasm/doc/trunk/persistence.md#coop-coep https://sqlite.org/wasm/doc/trunk/persistence.md#coop-coep' But when go back to the 26th none of the shards will load, error out. Using Windows 11, FF 146.0.1 Since you tested it seems its just a me problem and thanks for fixing the GitHub link
    - keepamovin 10mo ago
      No I've seen that error too, on Safari. I think it's related to the wasm being sent with wrong headers. CF pages _headers file should be ensuring correctness. Can you try busting your cache (or wait for a new Dec 29 Data dump version coming in a couple minutes), or from incognito to see if that fixes the issue? It's possible an earlier version had stale headers or sth. Idk.
layer8 10mo ago
Apparently the comment counts are only the top-level comments? It would be nice for the thread pages to show a comment count.
- keepamovin 10mo ago
  Yes, because comments in a thread can span shards. It’s just a bit too heavy to add comment counts of an entire thread. So I give a low bound ha ha
dspillett 10mo ago
Is there a public dump of the data anywhere that this is based upon, or have they scraped it themselves? Such as DB might be entertaining to play with, and the threadedness of comments would be useful for beginners to practise efficient recursive queries (more so than the StackExchange dumps, for instance).
- thomasmarton 10mo ago
  While not a dump per se, there is an API where you can get HN data programmatically, no scraping needed. https://github.com/HackerNews/API https://github.com/HackerNews/API
- keepamovin 10mo ago
  Yes, you can see the download HN bash script in the repository now that simply extract the data to your local machine from BigQuery and saves it as a series of gzip JSON files
  - dspillett 10mo ago
    Ah, the repo was 404ing for me last time I checked (seems fine now) so I couldn't inspect that. I'll have a play later.
abetusk 10mo ago
Alas, HN does not belong to us, and the existence of projects like this are subject to the whims of the legal owners of HN. From the terms of use [0]: """ Commercial Use: Unless otherwise expressly authorized herein or in the Site, you agree not to display, distribute, license, perform, publish, reproduce, duplicate, copy, create derivative works from, modify, sell, resell, exploit, transfer or upload for any commercial purposes, any portion of the Site, use of the Site, or access to the Site. The buying, exchanging, selling and/or promotion (commercial or otherwise) of upvotes, comments, submissions, accounts (or any aspect of your account or any other account), karma, and/or content is strictly prohibited, constitutes a material breach of these Terms of Use, and could result in legal liability. """ [0] https://www.ycombinator.com/legal/#tou https://www.ycombinator.com/legal/#tou
- tom1337 10mo ago
  But is this really a commercial use? There doesn’t seem to be any intention of monetising this so I guess it doesn’t as specify commercial?
m-p-3 10mo ago
Looks like the repo was taken down (404). That's too bad, I'd like to see the inner-working with a subset of data, even with placeholders for the posts and comments.
- 3abiton 10mo ago
  That was fast. I was looking into recent HN datasets, and they are impossible find.
  - xnx 10mo ago
    Complete and continuously updated: https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0aW1lKSBGUk9NIGhhY2tlcm5ld3NfaGlzdG9yeQ== https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a...
    - gettingoverit 10mo ago
      If the last story on HN was at December 26, that is.
      - rolymath 10mo ago
        Continuously updated != instantly updated
        dspillett 10mo ago
        Continuously would suggest to me that the data is never far out of date, and a few days might be considered far in this case. Perhaps “regularly updated” would be less contentious wordage?
  - scsh 10mo ago
    It's available on BigQuery and is updated frequently enough(daily I think).
- octoberfranklin 10mo ago
KomoD 10mo ago
How do I download it? That repo is a 404.
solarized 10mo ago
Beautiful ! 2026 prayer: for all you AI junkies—please don’t pollute H/N with your dirty AI gaming. Don’t bot posts, comments, or upvote/downvote just to maximize karma. Please. We can’t identify anymore who’s a bot and who’s human. I just want to hang out with real humans here.
kristianp 10mo ago
I tried "select * from items limit 10" and it is slowly iterating through the shards without returning. I got up to 60 shards before I stopped. Selecting just one shard makes that query return instantly. As mentioned elsewhere I think duckdb can work faster by only reading the part of a parquet file it needs over http. I was getting an error that the users and user_domains tables aren't available, but you just need to change the shard filter to the user stats shard.
- ncruces 10mo ago
  That's odd. If it was a VFS, that's not what I'd expect would happen. Maybe it's not a VFS?
  - keepamovin 10mo ago
    What is a VFS?
    - ncruces 10mo ago
      https://sqlite.org/vfs.html https://sqlite.org/vfs.html While designed for OS portability, you can use it to convince SQLite to read from something other than a file on disk.
- piperswe 10mo ago
  Doesn't `LIMIT` just limit the amount of rows returned, rather than the amount read & processed?
  - lucb1e 10mo ago
    That's what it does, but if I'm not mistaken (at least in my experience with MariaDB) it'll also return immediately once it ran up to the limit and not try to process further rows. If you have an expensive subquery in the SELECT (...) AS `column_name`, it won't run that for every row before returning the first 10 (when using LIMIT 10) unless you ORDERed BY that column_name. Other components like the WHERE clause might also require that it reads every row before finding the ten matches. So mostly yes but not necessarily
diyseguy 10mo ago
link no workie: https://github.com/DOSAYGO-STUDIO/HackerBook https://github.com/DOSAYGO-STUDIO/HackerBook
- keepamovin 10mo ago
  Fixed now. Forgot to make public. I also added a script: ./toool/download-site.mjs --help To let you download the entire site over HTTPS so you don't need to "build it" by running the pipeline. That way it's truly offline.
kamranjon 10mo ago
It'd be great if you could add it to Kiwix[1] somehow (not sure what the process is for that but 100rabbits figured it out for their site) - I use it all the time now that I have a dumb phone - I have the entirety of wikipedia, wiktionary and 100rabbits all offline. https://kiwix.org/en/ https://kiwix.org/en/
- codazoda 10mo ago
  I love that you have 100r.ca on that short list.
- endofreach 10mo ago
  what dumb phone do you use? and why do you want wikipedia in your pocket, but not a smartphone? where do you draw the line? (doing a lot of work in that area, so i am asking to learn from someone who might think alike)
  - kamranjon 10mo ago
    I use the Mudita Kompakt specifically cause it allows sideloading so I can still have a few extras. Right now I have Kiwix and Libby. It works really well. I have a $10 a month plan from US cellular with only 2gigs so I try to keep everything offline that I can. Honestly it's mostly the news... so I draw the line at browser, I'll never install a browser, that's basically something I can do when I sit down at a PC. I read quite a bit and I like to have the ability to look up a word or a historical event or some reference from something I read using Kiwix and it's been great for that, just needed to add a 512gb micro sd card. And Libby I just use at the gym when I'm on the treadmill.
    - endofreach 10mo ago
      interesting. thank you. any way i can reach out to you regardibg a project i am working on? your input would be very valuable.
- keepamovin 10mo ago
  This is a good idea — we should do it. I also want to make sure we can build this in CI. My goal is to have this updated every day using the BigQuery update process, so it becomes a 1–2 day delayed static archive of the current state of Hacker News, which is honestly very cool. I can probably run the build for free on GitHub Actions runners, as long as the runner has about 40 GB of disk space available. If needed, I can free up space on the runner before the build starts. I’ll also write to GitHub support and ask if they can sponsor the cost of a larger runner, mainly because I need the extra disk space to run the build reliably.
3eb7988a1663 10mo ago
Did anyone get a copy of this before it was pulled? If GitHub is not keen, could it be uploaded to HuggingFace or some other service which hosts large assets? I have always known I could scrape HN, but I would much rather take a neat little package.
modeless 10mo ago
It's really a shame that comment scores are hidden forever. Would the admins consider publishing them after stories are old enough that voting is closed? It would be great to have them for archives and search indices and projects like this.
- pilingual 10mo ago
  I wrote to hn@ and asked for this as a feature request: "1. Delayed Karma Display. I understand why comment karma was hidden. I don't see the harm in un-hiding karma after some time. If not 24 hours, then 72-168 hours. This would help me read through threads with 1300 comments." This was last January. While I asked for a few more features, it is the only one that seems essential as HN grows with massive threads.
- keepamovin 10mo ago
  Fear not. I have a collaborative project designed to address this.
  - vunderba 10mo ago
    They're referring to scores on individual COMMENTS - this information isn't available via the HN Firebase API. The only way you could theoretically extract everyone's comment scores (at least the top level ones) would be like this if you're a complete madman: 1. Wait 48 hours so the article is effectively dead 2. Post a new comment using an account called ThePresident 3. Create a swarm of a thousand shill user accounts called Voter1, Voter2, etc. 4. Use a single account at a time and upvote ThePresident 5. Recheck the page to see if ThePresident has moved above a user(s) post 6. Record the score for that user and assign it to the tracked story's history 7. Repeat from (4)
    - keepamovin 10mo ago
      I know that! I have a collaborative project to make it sort of available. But the idea I have is not like that at all - it's much nicer on everyone's ethics. Stay tuned! :)
fouc 10mo ago
Suddenly occurs to me that it would be neat to pair a small LLM (3-7B) with an HN dataset
- codazoda 10mo ago
  Does the SQLite version of this already exist somewhere? The github link on the footer of the page fails for me.
Xyra 10mo ago
Similar in spirit to a recent tool I recently posted Show HN on, https://exopriors.com/scry https://exopriors.com/scry. You can use Claude Code to SQL+vector query HackerNews and many other high quality public commons sites, exceptionally well-indexed and usually 5+ minute query timeout limits, so you can run seriously large research queries, to rapidly refine your worldview (particular because you can do easily to EXHAUSTIVE exploration).
- visarga 10mo ago
  I like your concept of indexing high quality sources for RAG. For many queries we might not need the usual search engines.
- keepamovin 10mo ago
  This looks cool but can you make a "Google Search Box" page where I don't have to sign in but can use it? It's just a bit of friction and I feel unbothered to overcome it. It's not personal to you - it's just how I feel about anything that looks unknown and interesting I just want to try, not have to sign up. For now. You know?
DenisDolya 10mo ago
Hahaha, now you can be prepared for the apocalypse when the internet disappears. ;)
RyJones 10mo ago
Neat. I keep wanting to build something like this for GitHub audit logs, but at ~5 tb, probably a little much
WadeGrimridge 10mo ago
threw some heatmaps together of post volume and average score by day and time (15min intervals) story volume (all time): https://ibb.co/pBTTRznP https://ibb.co/pBTTRznP average score (all time): https://ibb.co/KcvVjx8p https://ibb.co/KcvVjx8p story volume (since 2020): https://ibb.co/cKC5d7Pp https://ibb.co/cKC5d7Pp average score (since 2020): https://ibb.co/WpN20kfh https://ibb.co/WpN20kfh
- WadeGrimridge 10mo ago
  added median too. median score (all time): https://ibb.co/gZV5QVMG https://ibb.co/gZV5QVMG median score (since 2020): https://ibb.co/Gfv8T7k8 https://ibb.co/Gfv8T7k8
  - keepamovin 10mo ago
    This is fascinating, I love this! What do you create these heat maps with? Did you use the BigQuery data or download from the site?
    - WadeGrimridge 10mo ago
      thanks! i downloaded them and used python to make these
      - keepamovin 10mo ago
        So cool! Would it be impossible fro me to use them on the Archive stats page (https://hackerbook.dosaygo.com/?view=archive https://hackerbook.dosaygo.com/?view=archive) ? If you're okay with that any links/credit line details? Totally cool if not, just super interesting!
        WadeGrimridge 10mo ago
        yes, please, that'd be cool! you can link my site grimridge.net if you'd like. here are the numbers so you can plot graphs that fit the site's style: mean (all time): https://katb.in/yutupojerux https://katb.in/yutupojerux mean (since 2020): https://katb.in/omoyibisava https://katb.in/omoyibisava median (all time): https://katb.in/kilopofivet https://katb.in/kilopofivet median (since 2020): https://katb.in/ukefetuyuhi https://katb.in/ukefetuyuhi
7777777phil 10mo ago
Absolutely love this!! I have been analyzing a lot of HN data lately [1] so I backtested my hypothesis on your dataset and ran some stats: https://philippdubach.com/standalone/hackerbook-stats/ https://philippdubach.com/standalone/hackerbook-stats/ [1]https://news.ycombinator.com/item?id=46434575 https://news.ycombinator.com/item?id=46434575
- keepamovin 10mo ago
  That is so cool! I love that this passion fun project inspired and helped your paper. So cool!
adamszakal 10mo ago
Is it a thing that the design is almost unusable on a mobile phone? The tech making this possible is beyond cool, but it's just presented in such a brutal way for phone users, even though fixing it would be super simple.
- keepamovin 10mo ago
  Really? Let me know how I can help. What would you like to see fixed?
  - adamszakal 10mo ago
    Just following the ordinary guidelines when doing responsive designs, like increasing the text size and sizes of buttons and inputs, so my fat fingers don't missklick every other try. HN has gotten better, but is still below average, hence why I thought it was some kind of aesthetic choice.
    - keepamovin 10mo ago
      I tried to keep to HN look. But I’m not a mobile design expert. I will give it a once over see if i can make it better for you
      - adamszakal 10mo ago
        You rock dude, hope it didn't come out as condescending! It's an awesome project.
rcarmo 10mo ago
Nice. I wonder if there’s any way to quickly get a view for a whole year.
- keepamovin 10mo ago
  You mean like stuff ranked per year? Edit: Good idea! I implemented a "year" selector so all main views (front/show/ask/jobs) will be from that entire year rather than just a single day.
ComputerGuru 10mo ago
Awesome work. Minor bug/suggestion: right-aligned text inputs (eg the username input on the “me” page) aren’t ideal since they are often obscured by input helpers (autocomplete or form fill helper icons).
keepamovin 10mo ago
I added a weekly update, so it should remain current with the actual site BigQuery does so. It's funny that it now archives itself: https://hackerbook.dosaygo.com/?view=item&id=46435308 https://hackerbook.dosaygo.com/?view=item&id=46435308

730 points by keepamovin 10mo ago

yupyupyups 10mo ago

1 hour passed and it's already nuked? Thank you btw

joshcsimmons 10mo ago

Link appears broken

ra 10mo ago
confirmed - I wonder what happened?

deleted 10mo ago

[deleted]

asdefghyk 10mo ago

How much space is needed? ...for the data .... Im wondering if it would work on a tablet? ....

keepamovin 10mo ago
~9GB gzipped.
asdefghyk 10mo ago
FYI I did NOT see the size info in the title. Impossible to edit / delete my comment now ........

carbocation 10mo ago

That repo is throwing up a 404 for me. Question - did you consider tradeoffs between duckdb (or other columnar stores) and SQLite?

keepamovin 10mo ago
No, I just went straight to sqlite. What is duckdb?
- cess11 10mo ago
  It is very similar to SQLite in that it can run in-process and store its data as a file. It's different in that it is tailored to analytics, among other things storage is columnar, and it can run off some common data analytics file formats.
- fsiefken 10mo ago
  DuckDB is an open-source column-oriented Relational Database Management System (RDBMS). It's designed to provide high performance on complex queries against large databases in embedded configuration. It has transparent compression built-in and has support for natural language queries. https://buckenhofer.com/2025/11/agentic-ai-with-duckdb-and-smolagents-natural-language-queries-for-analytics/ https://buckenhofer.com/2025/11/agentic-ai-with-duckdb-and-s... "DICT FSST (Dictionary FSST) represents a hybrid compression technique that combines the benefits of Dictionary Encoding with the string-level compression capabilities of FSST. This approach was implemented and integrated into DuckDB as part of ongoing efforts to optimize string storage and processing performance." https://homepages.cwi.nl/~boncz/msc/2025-YanLannaAlexandre.pdf https://homepages.cwi.nl/~boncz/msc/2025-YanLannaAlexandre.p...
- simonw 10mo ago
  One interesting feature of DuckDB is that it can run queries against HTTP ranges of a static file hosted via HTTPS, and there's an official WebAssembly build of it that can do that same trick. So you can dump e.g. all of Hacker News in a single multi-GB Parquet file somewhere and build a client-side JavaScript application that can run queries against that without having to fetch the whole thing. You can run searches on https://lil.law.harvard.edu/data-gov-archive/ https://lil.law.harvard.edu/data-gov-archive/ and watch the network panel to see DuckDB in action.

wslh 10mo ago

Is this updated regularly? 404 on GitHub as the other comment. With all due respect it would be great if there is an official HN public dump available (and not requiring stuff such as BigQuery which is expensive).

scsh 10mo ago
The BQ dataset is only ~17GB and the free tier of BQ lets you query 1TB per month. If you're not doing select * on every query you should be able to do a lot with that.

zX41ZdbW 10mo ago

The query tab looks quite complex with all these content shards: https://hackerbook.dosaygo.com/?view=query https://hackerbook.dosaygo.com/?view=query I have a much simpler database: https://play.clickhouse.com/play?user=play#U0VMRUNUIHRpbWUsIHR5cGUsIHNjb3JlLCBpZCwgdGl0bGUsIHRleHQgRlJPTSBoYWNrZXJuZXdzX2hpc3RvcnkgV0hFUkUgYnkgPSAna2VlcGFtb3ZpbicgT1JERVIgQlkgdGltZSBERVND https://play.clickhouse.com/play?user=play#U0VMRUNUIHRpbWUsI...

embedding-shape 10mo ago
Does your database also runs offline/locally in the browser? Seems to be the reason for the large number of shards.
- zX41ZdbW 10mo ago
  You can run it locally, but it is a client-server architecture, which means that something has to run behind the browser.

abixb 10mo ago

Wonder if you could turn this into a .zim file for offline browsing with an offline browser like Kiwix, etc. [0] I've been taking frequent "offline-only-day" breaks to consolidate whatever I've been learning, and Kiwix has been a great tool for reference (offline Wikipedia, StackOverflow and whatnot). [0] https://kiwix.org/en/the-new-kiwix-library-is-available/ https://kiwix.org/en/the-new-kiwix-library-is-available/

Barbing 10mo ago
Oh this should TOTALLY be available to those who are scrolling through sources on the Kiwix app!
keepamovin 10mo ago
Oh that's a cool idea. If you want to take a crack at writing the script, the repo is open!

Paul-E 10mo ago

That's pretty neat! I did something similar. I build a tool[1] to import the Project Arctic Shift dumps[2] of reddit into sqlite. It was mostly an exercise to experiment with Rust and SQLite (HN's two favorite topics). If you don't build a FTS5 index and import without WAL (--unsafe-mode), import of every reddit comment and submission takes a bit over 24 hours and produces a ~10TB DB. SQLite offers a lot of cool json features that would let you store the raw json and operate on that, but I eschewed them in favor of parsing only once at load time. THat also lets me normalize the data a bit. I find that building the DB is pretty "fast", but queries run much faster if I immediately vacuum the DB after building it. The vacuum operation is actually slower than the original import, taking a few days to finish. [1] https://github.com/Paul-E/Pushshift-Importer https://github.com/Paul-E/Pushshift-Importer [2] https://github.com/ArthurHeitmann/arctic_shift/blob/master/download_links.md https://github.com/ArthurHeitmann/arctic_shift/blob/master/d...

s_ting765 10mo ago
You could check out SQLite's auto_vacuum which reclaims space without rebuilding the entire db https://sqlite.org/pragma.html#pragma_auto_vacuum https://sqlite.org/pragma.html#pragma_auto_vacuum
- Paul-E 10mo ago
  I haven't tested that, so I'm not sure if it would work. The import only inserts rows, it doesn't delete, so I don't think that is the cause of fragmentation. I suspect this line in the vacuum docs: > The VACUUM command may change the ROWIDs of entries in any tables that do not have an explicit INTEGER PRIMARY KEY. means SQLite does something to organize by rowid and that this is doing most of the work. Reddit post/comment IDs are 1:1 with integers, though expressed in a different base that is more friendly to URLs. I map decoded post/comment IDs to INTEGER PRIMARY KEYs on their respective tables. I suspect the vacuum operation sorts the tables by their reddit post ID and something about this sorting improves tables scans, which in turn helps building indices quickly after standing up the DB.
Xyra 10mo ago
Holy cow, I didn't know getting reddit was that straightforward. I am building public readonly-SQL+vector databases optimized for exploring high-quality public commons with Claude Code (https://exopriors.com/scry https://exopriors.com/scry), I so cannot wait until some funding source comes in and I can upgrade to a $1500/month Hetzner server and pay the ~$1k to embed all that.

fao_ 10mo ago

[flagged]

rantingdemon 10mo ago
Why do you say that?
- sundarurfriend 10mo ago
  Because anything that even slightly differs from the standard American phrasing of something must be "LLM generated" these days.
  - JavGull 10mo ago
    With the em dashes I see you. But at this point idrc so long as it reads well. Everyone uses spell check…
    - naikrovek 10mo ago
      I add em dashes to everything I write now, solely to throw people who look for them off. Lots of editors add them automatically when you have two sequential dashes between words — a common occurrence, like that one. And this is is Chrome on iOS doing it automatically. Ooh, I used “sequential”, ooh, I used an em dash. ZOMG AI IS COMING FOR US ALL
      - Barbing 10mo ago
        Ya—in fact, globally replaced on iOS (sent from Safari) Also for reference: “this shortcut can be toggled using the switch labeled 'Smart Punctuation' in General > Keyboard settings.”
      - 3eb7988a1663 10mo ago
        Anyone demonstrating above a high-school vocabulary/reading level is obviously a machine.

simonw 10mo ago

Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqlite.gz https://hackerbook.dosaygo.com/static-shards/shard_1635.sqlite.gz https://hackerbook.dosaygo.com/static-shards/shard_1634.sqlite.gz As I paginated to previous days. It's reminiscent of that brilliant SQLite.js VFS trick from a few years ago: https://github.com/phiresky/sql.js-httpvfs https://github.com/phiresky/sql.js-httpvfs - only that one used HTTP range headers, this one uses sharded files instead. The interactive SQL query interface at https://hackerbook.dosaygo.com/?view=query https://hackerbook.dosaygo.com/?view=query asks you to select which shards to run the query against, there are 1636 total.

maxloh 10mo ago
I am curios why they don't use a single file and HTTP Range Requests instead. PMTiles (a distribution of OpenStreetMap) uses that.
- keepamovin 10mo ago
  This would be a neat idea to try. Want to add a PR? Bench different "hackends" to see how DuckDB, SQLite shards, or range queries perform?
tehlike 10mo ago
Vfs support is amazing.
nextaccountic 10mo ago
Is there anything more production grade built around the same idea of HTTP range requests like that sqlite thing? This has so much potential
- simonw 10mo ago

sieep 10mo ago

What a reminder on how text is so much more efficient than video, its crazy! Could you imagine the same amount of knowledge (or dribble) but in video form? I wonder how large that would be.

ivanjermakov 10mo ago
Average high quality 1080p60 video has bitrate of 5Mbps, which is equivalent to 120k English words per second. With average English speech being 150wpm, we end up with text being 50 thousand times more space efficient. Converting 22GB of uncompressed text into video essay lands us at ~1PB or 1000TB.
- sieep 10mo ago
  Cheers to you for doing the math, I hope 2026 is excellent for you!
fsiefken 10mo ago
one could use a video llm to generate the video, diagrams or the stills automatically based on the text. except when it's boardgames playthroughs or programming i just transcribe to text, summarise and read youtube video's.
- Barbing 10mo ago
  Can be nice to pull a raw transcript and have it formatted as HTML (formatting/punctuation fixes applied). Best locally of course to avoid “I burned a lake for this?” guilt.
  - fsiefken 10mo ago
    yes, yt-dlp can download the transcript, and if it's not available i can get the audio file and run it through parakeet locally.
- deskamess 10mo ago
  How do you read youtube videos? Very curious as I have been wanting to watch PDF's scroll by slowly on a large TV. I am interested in the workflow of getting a pdf/document into a scrolling video format. These days NotebookLM may be an option but I am curious if there is something custom. If I can get it into video form (mp4) then I can even deliver it via plex.

sirjaz 10mo ago

This would be awesome as a cross platform app.

keepamovin 10mo ago
Good idea. HN.exe

tevon 10mo ago

The link seems to be down, was it taken down?

scsh 10mo ago
Probably just forgot to make it public.

zkmon 10mo ago

Similar to Single-page applications (SPA), single-table application (STA) might become a thing. Just a shard a table on multiple keys and serve the shards as static files, provided that the data is Ok to share, similar to sharing static html content.

jesprenj 10mo ago
do you mean single database? it'd be quite hard if not impossible to make applications using a single table (no relations). reddit did it though, they have a huge table of "things" iirc.
- mburns 10mo ago
  That is a common misconception. > Next, we've got more than just two tables. The quote/paraphrase doesn't make it clear, but we've got two tables per thing. That means Accounts have an "account_thing" and an "account_data" table, Subreddits have a "subreddit_thing" and "subreddit_data" table, etc. https://www.reddit.com/r/programming/comments/z9sm8/comment/c62uz89/ https://www.reddit.com/r/programming/comments/z9sm8/comment/...
  - rplnt 10mo ago
    And the important lesson from that the k/v-like aspect of it. That the "schema" is horizontal (is that a thing?) and not column-based. But I actually only read it on their blog IIRC and never even got the full details - that there's still a third ID column. Thanks for the link.
jhd3 10mo ago
[The Baked Data architectural pattern](https://simonwillison.net/2021/Jul/28/baked-data/ https://simonwillison.net/2021/Jul/28/baked-data/)

yread 10mo ago

I wonder how much smaller it could get with some compression. You could probably encode "This website hijacks the scrollbar and I don't like it" comments into just a few bits.

hamburglar 10mo ago
It might be a neat experiment to use ai to produce canonicalized paraphrasings of HN arguments so they could be compared directly and compress well.
jacquesm 10mo ago
That's at least 45%, then you can leave out all of my comments and you're left with only 5!
Rendello 10mo ago
The hard-coded dictionary wouldn't be much stranger than Brotli's: https://news.ycombinator.com/item?id=27160590 https://news.ycombinator.com/item?id=27160590
- maxbond 10mo ago
  You can use a BPE variant like SentencePiece to identify these patterns rather than hard coding them.
rossant 10mo ago
Guilty.
deleted 10mo ago
[deleted]
keepamovin 10mo ago
Dear it is already compressed using G zip – nine for every SQLlight shard and manifest 22 GB is uncompressed and compressed the entire things about 9 GB

dmarwicke 10mo ago

22gb for mostly text? tried loading the site, it's pretty slow. curious how the query performance is with this much data in sqlite

spit2wind 10mo ago

This is pretty neat! The calendar didn't work well for me. I could only seem to navigate by month. And when I selected the earliest day (after much tapping), nothing seemed to be updated. Nonetheless, random access history is cool.

keepamovin 10mo ago
Cna you let me know? I'm sure there's some weirdness lurking there and I want to smooth it out. Calendar is essential.

Sn0wCoder 10mo ago

Site does not load on Firefox console error says 'Uncaught (in promise) TypeError: can't access property "wasm", sqlite3 is null' Guess its common knowledge that SharedArrayBuffer (SQLite wasm) does not work with FF due to Cross-Origin Attacks (i just found out ;). Once the initial chunk of data loads the rest load almost instantly on Chrome. Can you please fix the GitHub link (current 404) would like to peak at the code. Thank you!

deleted 10mo ago
[deleted]
keepamovin 10mo ago
Damn. Will try to fix for FF. edit: I just tested with FF latest, seems to be working.
- Sn0wCoder 10mo ago
  Strange now the first few days load (getting a new error) 'Ignoring inability to install OPFS sqlite3_vfs: Cannot install OPFS: Missing SharedArrayBuffer and/or Atomics. The server must emit the COOP/COEP response headers to enable those. See https://sqlite.org/wasm/doc/trunk/persistence.md#coop-coep https://sqlite.org/wasm/doc/trunk/persistence.md#coop-coep' But when go back to the 26th none of the shards will load, error out. Using Windows 11, FF 146.0.1 Since you tested it seems its just a me problem and thanks for fixing the GitHub link
  - keepamovin 10mo ago
    No I've seen that error too, on Safari. I think it's related to the wasm being sent with wrong headers. CF pages _headers file should be ensuring correctness. Can you try busting your cache (or wait for a new Dec 29 Data dump version coming in a couple minutes), or from incognito to see if that fixes the issue? It's possible an earlier version had stale headers or sth. Idk.

layer8 10mo ago

Apparently the comment counts are only the top-level comments? It would be nice for the thread pages to show a comment count.

keepamovin 10mo ago
Yes, because comments in a thread can span shards. It’s just a bit too heavy to add comment counts of an entire thread. So I give a low bound ha ha

dspillett 10mo ago

Is there a public dump of the data anywhere that this is based upon, or have they scraped it themselves? Such as DB might be entertaining to play with, and the threadedness of comments would be useful for beginners to practise efficient recursive queries (more so than the StackExchange dumps, for instance).

thomasmarton 10mo ago
While not a dump per se, there is an API where you can get HN data programmatically, no scraping needed. https://github.com/HackerNews/API https://github.com/HackerNews/API
keepamovin 10mo ago
Yes, you can see the download HN bash script in the repository now that simply extract the data to your local machine from BigQuery and saves it as a series of gzip JSON files
- dspillett 10mo ago
  Ah, the repo was 404ing for me last time I checked (seems fine now) so I couldn't inspect that. I'll have a play later.

abetusk 10mo ago

Alas, HN does not belong to us, and the existence of projects like this are subject to the whims of the legal owners of HN. From the terms of use [0]: """ Commercial Use: Unless otherwise expressly authorized herein or in the Site, you agree not to display, distribute, license, perform, publish, reproduce, duplicate, copy, create derivative works from, modify, sell, resell, exploit, transfer or upload for any commercial purposes, any portion of the Site, use of the Site, or access to the Site. The buying, exchanging, selling and/or promotion (commercial or otherwise) of upvotes, comments, submissions, accounts (or any aspect of your account or any other account), karma, and/or content is strictly prohibited, constitutes a material breach of these Terms of Use, and could result in legal liability. """ [0] https://www.ycombinator.com/legal/#tou https://www.ycombinator.com/legal/#tou

tom1337 10mo ago
But is this really a commercial use? There doesn’t seem to be any intention of monetising this so I guess it doesn’t as specify commercial?

m-p-3 10mo ago

Looks like the repo was taken down (404). That's too bad, I'd like to see the inner-working with a subset of data, even with placeholders for the posts and comments.

3abiton 10mo ago
That was fast. I was looking into recent HN datasets, and they are impossible find.
- xnx 10mo ago
  Complete and continuously updated: https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0aW1lKSBGUk9NIGhhY2tlcm5ld3NfaGlzdG9yeQ== https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a...
  - gettingoverit 10mo ago
    If the last story on HN was at December 26, that is.
    - rolymath 10mo ago
      Continuously updated != instantly updated
      - dspillett 10mo ago
        Continuously would suggest to me that the data is never far out of date, and a few days might be considered far in this case. Perhaps “regularly updated” would be less contentious wordage?
- scsh 10mo ago
  It's available on BigQuery and is updated frequently enough(daily I think).
octoberfranklin 10mo ago

KomoD 10mo ago

How do I download it? That repo is a 404.

solarized 10mo ago

Beautiful ! 2026 prayer: for all you AI junkies—please don’t pollute H/N with your dirty AI gaming. Don’t bot posts, comments, or upvote/downvote just to maximize karma. Please. We can’t identify anymore who’s a bot and who’s human. I just want to hang out with real humans here.

kristianp 10mo ago

I tried "select * from items limit 10" and it is slowly iterating through the shards without returning. I got up to 60 shards before I stopped. Selecting just one shard makes that query return instantly. As mentioned elsewhere I think duckdb can work faster by only reading the part of a parquet file it needs over http. I was getting an error that the users and user_domains tables aren't available, but you just need to change the shard filter to the user stats shard.

ncruces 10mo ago
That's odd. If it was a VFS, that's not what I'd expect would happen. Maybe it's not a VFS?
- keepamovin 10mo ago
  What is a VFS?
  - ncruces 10mo ago
    https://sqlite.org/vfs.html https://sqlite.org/vfs.html While designed for OS portability, you can use it to convince SQLite to read from something other than a file on disk.
piperswe 10mo ago
Doesn't `LIMIT` just limit the amount of rows returned, rather than the amount read & processed?
- lucb1e 10mo ago
  That's what it does, but if I'm not mistaken (at least in my experience with MariaDB) it'll also return immediately once it ran up to the limit and not try to process further rows. If you have an expensive subquery in the SELECT (...) AS `column_name`, it won't run that for every row before returning the first 10 (when using LIMIT 10) unless you ORDERed BY that column_name. Other components like the WHERE clause might also require that it reads every row before finding the ten matches. So mostly yes but not necessarily

diyseguy 10mo ago

link no workie: https://github.com/DOSAYGO-STUDIO/HackerBook https://github.com/DOSAYGO-STUDIO/HackerBook

keepamovin 10mo ago
Fixed now. Forgot to make public. I also added a script: ./toool/download-site.mjs --help To let you download the entire site over HTTPS so you don't need to "build it" by running the pipeline. That way it's truly offline.

kamranjon 10mo ago

It'd be great if you could add it to Kiwix[1] somehow (not sure what the process is for that but 100rabbits figured it out for their site) - I use it all the time now that I have a dumb phone - I have the entirety of wikipedia, wiktionary and 100rabbits all offline. https://kiwix.org/en/ https://kiwix.org/en/

codazoda 10mo ago
I love that you have 100r.ca on that short list.
endofreach 10mo ago
what dumb phone do you use? and why do you want wikipedia in your pocket, but not a smartphone? where do you draw the line? (doing a lot of work in that area, so i am asking to learn from someone who might think alike)
- kamranjon 10mo ago
  I use the Mudita Kompakt specifically cause it allows sideloading so I can still have a few extras. Right now I have Kiwix and Libby. It works really well. I have a $10 a month plan from US cellular with only 2gigs so I try to keep everything offline that I can. Honestly it's mostly the news... so I draw the line at browser, I'll never install a browser, that's basically something I can do when I sit down at a PC. I read quite a bit and I like to have the ability to look up a word or a historical event or some reference from something I read using Kiwix and it's been great for that, just needed to add a 512gb micro sd card. And Libby I just use at the gym when I'm on the treadmill.
  - endofreach 10mo ago
    interesting. thank you. any way i can reach out to you regardibg a project i am working on? your input would be very valuable.
keepamovin 10mo ago
This is a good idea — we should do it. I also want to make sure we can build this in CI. My goal is to have this updated every day using the BigQuery update process, so it becomes a 1–2 day delayed static archive of the current state of Hacker News, which is honestly very cool. I can probably run the build for free on GitHub Actions runners, as long as the runner has about 40 GB of disk space available. If needed, I can free up space on the runner before the build starts. I’ll also write to GitHub support and ask if they can sponsor the cost of a larger runner, mainly because I need the extra disk space to run the build reliably.

3eb7988a1663 10mo ago

Did anyone get a copy of this before it was pulled? If GitHub is not keen, could it be uploaded to HuggingFace or some other service which hosts large assets? I have always known I could scrape HN, but I would much rather take a neat little package.

modeless 10mo ago

It's really a shame that comment scores are hidden forever. Would the admins consider publishing them after stories are old enough that voting is closed? It would be great to have them for archives and search indices and projects like this.

pilingual 10mo ago
I wrote to hn@ and asked for this as a feature request: "1. Delayed Karma Display. I understand why comment karma was hidden. I don't see the harm in un-hiding karma after some time. If not 24 hours, then 72-168 hours. This would help me read through threads with 1300 comments." This was last January. While I asked for a few more features, it is the only one that seems essential as HN grows with massive threads.
keepamovin 10mo ago
Fear not. I have a collaborative project designed to address this.
- vunderba 10mo ago
  They're referring to scores on individual COMMENTS - this information isn't available via the HN Firebase API. The only way you could theoretically extract everyone's comment scores (at least the top level ones) would be like this if you're a complete madman: 1. Wait 48 hours so the article is effectively dead 2. Post a new comment using an account called ThePresident 3. Create a swarm of a thousand shill user accounts called Voter1, Voter2, etc. 4. Use a single account at a time and upvote ThePresident 5. Recheck the page to see if ThePresident has moved above a user(s) post 6. Record the score for that user and assign it to the tracked story's history 7. Repeat from (4)
  - keepamovin 10mo ago
    I know that! I have a collaborative project to make it sort of available. But the idea I have is not like that at all - it's much nicer on everyone's ethics. Stay tuned! :)

fouc 10mo ago

Suddenly occurs to me that it would be neat to pair a small LLM (3-7B) with an HN dataset

codazoda 10mo ago
Does the SQLite version of this already exist somewhere? The github link on the footer of the page fails for me.

Xyra 10mo ago

Similar in spirit to a recent tool I recently posted Show HN on, https://exopriors.com/scry https://exopriors.com/scry. You can use Claude Code to SQL+vector query HackerNews and many other high quality public commons sites, exceptionally well-indexed and usually 5+ minute query timeout limits, so you can run seriously large research queries, to rapidly refine your worldview (particular because you can do easily to EXHAUSTIVE exploration).

visarga 10mo ago
I like your concept of indexing high quality sources for RAG. For many queries we might not need the usual search engines.
keepamovin 10mo ago
This looks cool but can you make a "Google Search Box" page where I don't have to sign in but can use it? It's just a bit of friction and I feel unbothered to overcome it. It's not personal to you - it's just how I feel about anything that looks unknown and interesting I just want to try, not have to sign up. For now. You know?

DenisDolya 10mo ago

Hahaha, now you can be prepared for the apocalypse when the internet disappears. ;)

RyJones 10mo ago

Neat. I keep wanting to build something like this for GitHub audit logs, but at ~5 tb, probably a little much

WadeGrimridge 10mo ago

threw some heatmaps together of post volume and average score by day and time (15min intervals) story volume (all time): https://ibb.co/pBTTRznP https://ibb.co/pBTTRznP average score (all time): https://ibb.co/KcvVjx8p https://ibb.co/KcvVjx8p story volume (since 2020): https://ibb.co/cKC5d7Pp https://ibb.co/cKC5d7Pp average score (since 2020): https://ibb.co/WpN20kfh https://ibb.co/WpN20kfh

WadeGrimridge 10mo ago
added median too. median score (all time): https://ibb.co/gZV5QVMG https://ibb.co/gZV5QVMG median score (since 2020): https://ibb.co/Gfv8T7k8 https://ibb.co/Gfv8T7k8
- keepamovin 10mo ago
  This is fascinating, I love this! What do you create these heat maps with? Did you use the BigQuery data or download from the site?
  - WadeGrimridge 10mo ago
    thanks! i downloaded them and used python to make these
    - keepamovin 10mo ago
      So cool! Would it be impossible fro me to use them on the Archive stats page (https://hackerbook.dosaygo.com/?view=archive https://hackerbook.dosaygo.com/?view=archive) ? If you're okay with that any links/credit line details? Totally cool if not, just super interesting!
      - WadeGrimridge 10mo ago
        yes, please, that'd be cool! you can link my site grimridge.net if you'd like. here are the numbers so you can plot graphs that fit the site's style: mean (all time): https://katb.in/yutupojerux https://katb.in/yutupojerux mean (since 2020): https://katb.in/omoyibisava https://katb.in/omoyibisava median (all time): https://katb.in/kilopofivet https://katb.in/kilopofivet median (since 2020): https://katb.in/ukefetuyuhi https://katb.in/ukefetuyuhi

7777777phil 10mo ago

Absolutely love this!! I have been analyzing a lot of HN data lately [1] so I backtested my hypothesis on your dataset and ran some stats: https://philippdubach.com/standalone/hackerbook-stats/ https://philippdubach.com/standalone/hackerbook-stats/ [1]https://news.ycombinator.com/item?id=46434575 https://news.ycombinator.com/item?id=46434575

keepamovin 10mo ago
That is so cool! I love that this passion fun project inspired and helped your paper. So cool!

adamszakal 10mo ago

Is it a thing that the design is almost unusable on a mobile phone? The tech making this possible is beyond cool, but it's just presented in such a brutal way for phone users, even though fixing it would be super simple.

keepamovin 10mo ago
Really? Let me know how I can help. What would you like to see fixed?
- adamszakal 10mo ago
  Just following the ordinary guidelines when doing responsive designs, like increasing the text size and sizes of buttons and inputs, so my fat fingers don't missklick every other try. HN has gotten better, but is still below average, hence why I thought it was some kind of aesthetic choice.
  - keepamovin 10mo ago
    I tried to keep to HN look. But I’m not a mobile design expert. I will give it a once over see if i can make it better for you
    - adamszakal 10mo ago
      You rock dude, hope it didn't come out as condescending! It's an awesome project.

rcarmo 10mo ago

Nice. I wonder if there’s any way to quickly get a view for a whole year.

keepamovin 10mo ago
You mean like stuff ranked per year? Edit: Good idea! I implemented a "year" selector so all main views (front/show/ask/jobs) will be from that entire year rather than just a single day.

ComputerGuru 10mo ago

Awesome work. Minor bug/suggestion: right-aligned text inputs (eg the username input on the “me” page) aren’t ideal since they are often obscured by input helpers (autocomplete or form fill helper icons).

keepamovin 10mo ago

I added a weekly update, so it should remain current with the actual site BigQuery does so. It's funny that it now archives itself: https://hackerbook.dosaygo.com/?view=item&id=46435308 https://hackerbook.dosaygo.com/?view=item&id=46435308