7 ms·

Show HN: Self-host Reddit – 2.38B posts, works offline, yours forever

286 points by 19-84 9mo ago

Reddit's API is effectively dead for archival. Third-party apps are gone. Reddit has threatened to cut off access to the Pushshift dataset multiple times. But 3.28TB of Reddit history exists as a torrent right now, and I built a tool to turn it into something you can browse on your own hardware. The key point: This doesn't touch Reddit's servers. Ever. Download the Pushshift dataset, run my tool locally, get a fully browsable archive. Works on an air-gapped machine. Works on a Raspberry Pi serving your LAN. Works on a USB drive you hand to someone. What it does: Takes compressed data dumps from Reddit (.zst), Voat (SQL), and Ruqqus (.7z) and generates static HTML. No JavaScript, no external requests, no tracking. Open index.html and browse. Want search? Run the optional Docker stack with PostgreSQL – still entirely on your machine. API & AI Integration: Full REST API with 30+ endpoints – posts, comments, users, subreddits, full-text search, aggregations. Also ships with an MCP server (29 tools) so you can query your archive directly from AI tools. Self-hosting options: - USB drive / local folder (just open the HTML files) - Home server on your LAN - Tor hidden service (2 commands, no port forwarding needed) - VPS with HTTPS - GitHub Pages for small archives Why this matters: Once you have the data, you own it. No API keys, no rate limits, no ToS changes can take it away. Scale: Tens of millions of posts per instance. PostgreSQL backend keeps memory constant regardless of dataset size. For the full 2.38B post dataset, run multiple instances by topic. How I built it: Python, PostgreSQL, Jinja2 templates, Docker. Used Claude Code throughout as an experiment in AI-assisted development. Learned that the workflow is "trust but verify" – it accelerates the boring parts but you still own the architecture. Live demo: https://online-archives.github.io/redd-archiver-example/ https://online-archives.github.io/redd-archiver-example/ GitHub: https://github.com/19-84/redd-archiver https://github.com/19-84/redd-archiver (Public Domain) Pushshift torrent: https://academictorrents.com/details/1614740ac8c94505e4ecb9d88be8bed7b6afddd4 https://academictorrents.com/details/1614740ac8c94505e4ecb9d...

elSidCampeador 9mo ago
I wonder if this can be hooked up with the now-dead Apollo app in some way, to get back a slice of time that is forever lost now?
- 19-84 9mo ago
  the API should allow for a lot of different integrations
kylehotchkiss 9mo ago
_Hacker News collectively grabs the dataset to train their models on how to become effective reddit trolls_
- 19-84 9mo ago
  the API and MCP server is very powerful ;)
- layer8 9mo ago
  Don’t we have enough of those already? ;)
NickNaraghi 9mo ago
Data is available via torrent in this section: https://github.com/19-84/redd-archiver?tab=readme-ov-file#-get-involved-help-preserve-internet-history https://github.com/19-84/redd-archiver?tab=readme-ov-file#-g...
- 19-84 9mo ago
  I have also published sub statistics and profiling for each platform. these can be used to help identify which subs to prioritize for archiving. reddit: https://github.com/19-84/redd-archiver/blob/main/tools/subreddits_complete.json https://github.com/19-84/redd-archiver/blob/main/tools/subre... voat: https://github.com/19-84/redd-archiver/blob/main/tools/subverses.json https://github.com/19-84/redd-archiver/blob/main/tools/subve... ruqqus: https://github.com/19-84/redd-archiver/blob/main/tools/guilds.json https://github.com/19-84/redd-archiver/blob/main/tools/guild...
Aurornis 9mo ago
Cool way to self-host archives. What I'd really like is a plugin that automatically pulls from archives somewhere and replaces deleted comments and those bot-overwritten comments with the original context. Reddit is becoming maddening to use because half the old links I click have comments overwritten with garbage out of protest for something. Ironically the original content is available in these archives (which are used for AI training) but now missing for actual users like me just trying to figure out how someone fixed their printer driver 2 years ago.
- anonymous908213 9mo ago
  That would only really be ironic if the reason for people overwriting their comments was out of protest for LLM training, but the main reason that resulted in by far the biggest wave of deletions was Reddit locking down their API. If the result of their protest is that the site is less useful for you, the user, then in fact it served its purpose, as the entire point was an attempt to boycott Reddit, ie. get people to stop using it by removing the user contributions that give the site its only value in the first place.
  - Aurornis 9mo ago
    > If the result of their protest is that the site is less useful for you, the user, then in fact it served its purpose, as the entire point was an attempt to boycott Reddit, ie. get people to stop using it by removing the user contributions that give the site its only value in the first place. In practice I just give them more page views because I have to view more threads before I find the answer. Reddit's DAU numbers have only gone up since the protest.
    - anonymous908213 9mo ago
      I did phrase it as "an attempt". In the end the protest probably wasn't as effective as protestors might have hoped, and it didn't get Reddit to change course on their enshittification decisions. I do think it was good that there was an attempt at pushback, at least, when most software users just accept enshittification as normal and continue tolerating whatever abuse their masters throw at them.
Jordan-117 9mo ago
[flagged]
- 19-84 9mo ago
  thank you for your comment, I will support any platform that has complete dataset available. I will take submissions for any complete datasets through github issues. https://github.com/19-84/redd-archiver/blob/main/.github/ISSUE_TEMPLATE/submit-data-source.yml https://github.com/19-84/redd-archiver/blob/main/.github/ISS...
- diggyhole 9mo ago
  Wat?
  - Jordan-117 9mo ago
    It sold itself as a healthier alternative to Reddit, but by the end of its run virtually every post sitewide was some flavor of virulently racist, misogynistic, anti-semitic, fringe conspiratorial, etc.
    - tempsaasexample 9mo ago
      [flagged]
      - Jordan-117 9mo ago
        Funny defense to use for a crowd that spent all their time regularly, hatefully dehumanizing people. The front page was routinely plastered with shit like "Why interracial children are an abomination," Hitler-did-nothing-wrong propaganda, usernames that echoed Nazi slogans and fantasized about mass-murdering non-white people, etc. It was an utter cesspool, and preserving and perpetuating that is a really weird use of dev time and effort.
        lazyasciiart 9mo ago
        It’ll be useful to have when the posters pop up as presidential advisors.
dvngnt_ 9mo ago
I want to do the same thing for tiktok. I have 5k videos starting from the pandemic downloaded. want to find a way to use AI to tag and categorize the videos to scroll locally.
syngrog66 9mo ago
Did you pay all the people who created its content?
- devilsdata 9mo ago
  I have no problem with this being downloaded for personal use, in fact that's a good thing. But of course we both know it'll be used to train AI.
- nullandvoid 9mo ago
  Did anyone ever comment on reddit with an expectation of pay? It's an open forum - similar to here, whatever I post I it's in the public forum and therefore I expect it to be used / remixed however anyone wants.
  - nozzlegear 9mo ago
    > Did anyone ever comment on reddit with an expectation of pay? Maybe Gallowboob
    - Sohcahtoa82 9mo ago
      That's a name I haven't seen in a LONG time.
- antisthenes 9mo ago
  Reddit didn't pay me for posting either. Not that I posted in the last decade.
deleted 9mo ago
[deleted]
alcroito 9mo ago
I tried spinning up the local approach with docker compose, but it fails. There's no `.env.example` file to copy from. And even if the env vars are set manually, there are issues with the mentioned volumes not existing locally. Seems like this needs more polish.
- 19-84 9mo ago
  thank you for your comment, some example dot files were not copied in my original repo, they have now been added. https://github.com/19-84/redd-archiver/commit/0bb103952195ae5897d7c7d32f65aeb0471cb664 https://github.com/19-84/redd-archiver/commit/0bb103952195ae... the docs have been updated with mkdir steps https://github.com/19-84/redd-archiver/commit/c3754ea3a0238f9ec573604558e0594480142e6e https://github.com/19-84/redd-archiver/commit/c3754ea3a0238f...
  - alcroito 9mo ago
    Cheers. I checked the updated steps. This is still missing creating the `output/.postgres-data` dir, without which docker compose refuses to start. After creating that manually, going to http://localhost/ shows a 403 Forbidden page, which makes you believe that something might have gone wrong. This is before running `reddarchiver-builder python reddarc.py` to generate the necessary DB from the input data.
    - 19-84 9mo ago
      I've updated the workflow and added a placeholder page that will serve before archives are created. thanks again! https://github.com/19-84/redd-archiver/commit/0dfd505ca81cb20dade5f2047631ddbb60926c7a https://github.com/19-84/redd-archiver/commit/0dfd505ca81cb2...
diggings 9mo ago
This is a neat project, nice work. You've probably come across this already but there are alternative archives to PushShift that may have differing sets of posts and comments (perhaps depending on removal request coverage?) One is Arctic Shift: https://github.com/ArthurHeitmann/arctic_shift/releases https://github.com/ArthurHeitmann/arctic_shift/releases Another is PullPush: https://pullpush.io/ https://pullpush.io/
bkovacev 9mo ago
Is there any way to check if a subreddit that was made private (2-3 years ago) is in the data dump?
- tourist2d 9mo ago
  [dead]
- 19-84 9mo ago
  I included a metadata dump of every subreddit found in the torrent. it includes a status field which will show of a subreddit is private along with a much more details data catalog readme: https://github.com/19-84/redd-archiver/blob/main/tools/README.md https://github.com/19-84/redd-archiver/blob/main/tools/READM... reddit data: https://github.com/19-84/redd-archiver/blob/main/tools/subreddits_complete.json https://github.com/19-84/redd-archiver/blob/main/tools/subre...
m463 9mo ago
I wonder if you could use this to "Seed" a new distributed social media thing and just take over from there. sort of like forking a project.
- 19-84 9mo ago
  ive created tooling for an instance registry and team based leaderboard. the API has function to support this as well, so that we can collectively host archives in a decentralized and distributed manner registry readme: https://github.com/19-84/redd-archiver/blob/main/docs/REGISTRY_SETUP.md https://github.com/19-84/redd-archiver/blob/main/docs/REGIST... register instances: https://github.com/19-84/redd-archiver/blob/main/.github/ISSUE_TEMPLATE/register-instance.yml https://github.com/19-84/redd-archiver/blob/main/.github/ISS...
drob518 9mo ago
This is a great way to participate in arguments you missed three years ago.
justsomehnguy 9mo ago
Appreciated. EDIT: Is there any cheap way to search? I have MS TechNet archive which is useless without search, so I realky want to know a way to have a cheap local search w/o grepping everyting.
- 19-84 9mo ago
  redd-archiver uses postgres full text search. for static search you could use lunr.js
twobitshifter 9mo ago
If reddit was a squeaky clean place, or if I could pick certain subs, maybe I would be interested, but I really wouldn't want ALL of reddit on my machine even temporarily.
- 19-84 9mo ago
  the torrent has data for the top 40,000 subs on reddit. thanks to watchful1 splitting the data by subreddit, you can download only the subreddit you want from the torrent
  - Imustaskforhelp 9mo ago
    I am going to be honest and this looks really cool. 40,000 subs are good numbers and I hope that the number can be spread to even more subreddits Perhaps we can finally migrate all or much of the data to lemmy instances as well to finally get the lemmy instance up and running as well. Thank you for creating this. It opens up a lots of interesting opportunities.
feconroses 9mo ago
Very cool project! Quick question: is the underlying Pushshift dataset updated with new Reddit data on any regular cadence (daily/weekly/monthly), or is this essentially a fixed historical snapshot up to a certain date? Just want to understand if self-hosters would need to periodically re-download for fresh content or if it's archival-only.
- 19-84 9mo ago
  the data from 2025-12 has been released already, it is usually released every month, it just needs to be split and reprocessed for 2025 by watchful1. i will probably eventually add support for importing data from the monthly arctic shift dumps so that archives can be updated monthly. https://github.com/ArthurHeitmann/arctic_shift/releases https://github.com/ArthurHeitmann/arctic_shift/releases Arctic Shift https://academictorrents.com/browse.php?search=RaiderBDev https://academictorrents.com/browse.php?search=RaiderBDev Watchful1 https://academictorrents.com/browse.php?search=Watchful1 https://academictorrents.com/browse.php?search=Watchful1
  - riku_iki 9mo ago
    Is data web scrapped? Is reddit ok with that?..
    - inquirerGeneral 9mo ago
      [dead]
blks 9mo ago
Does it also contains countless NSFW content?
blks 9mo ago
Opened the live demo, went into programming subreddit, felt like I was showered with liquid shit. I tend to forget what kind of edgelord hellhole Reddit was (and stil is sometimes).
vivzkestrel 9mo ago
- slightly offtopic here but does anyone have a similar data set of all youtube channels out there? - details probably include the 400 million youtube accounts, channel id, name, creator url, etc
- 19-84 9mo ago
  there is nearly 10TB of youtube metadata available on archive.org https://archive.org/details/youtube-metadata https://archive.org/details/youtube-metadata
  - chicagojoe 9mo ago
    These show as unavailable/with lock icons for me. Is there some process to download locked content from IA?
inquirerGeneral 9mo ago
[dead]
tetrisgm 9mo ago
Is there a docker compose?
nick007x 9mo ago
Hey, I’m working on a similar project and have uploaded Pushshift Reddit data to Hugging Face Datasets. If anyone wants to download specific files when torrents aren’t seeding well, you can use: https://huggingface.co/datasets/nick007x/pushshift-reddit https://huggingface.co/datasets/nick007x/pushshift-reddit It’s handy for grabbing individual months or subreddit slices without needing to pull the full torrent. Might be useful for smaller-scale archiving or testing.
- nick007x 9mo ago
  [dead]

286 points by 19-84 9mo ago

elSidCampeador 9mo ago

I wonder if this can be hooked up with the now-dead Apollo app in some way, to get back a slice of time that is forever lost now?

19-84 9mo ago
the API should allow for a lot of different integrations

kylehotchkiss 9mo ago

_Hacker News collectively grabs the dataset to train their models on how to become effective reddit trolls_

19-84 9mo ago
the API and MCP server is very powerful ;)
layer8 9mo ago
Don’t we have enough of those already? ;)

NickNaraghi 9mo ago

Data is available via torrent in this section: https://github.com/19-84/redd-archiver?tab=readme-ov-file#-get-involved-help-preserve-internet-history https://github.com/19-84/redd-archiver?tab=readme-ov-file#-g...

19-84 9mo ago
I have also published sub statistics and profiling for each platform. these can be used to help identify which subs to prioritize for archiving. reddit: https://github.com/19-84/redd-archiver/blob/main/tools/subreddits_complete.json https://github.com/19-84/redd-archiver/blob/main/tools/subre... voat: https://github.com/19-84/redd-archiver/blob/main/tools/subverses.json https://github.com/19-84/redd-archiver/blob/main/tools/subve... ruqqus: https://github.com/19-84/redd-archiver/blob/main/tools/guilds.json https://github.com/19-84/redd-archiver/blob/main/tools/guild...

Aurornis 9mo ago

Cool way to self-host archives. What I'd really like is a plugin that automatically pulls from archives somewhere and replaces deleted comments and those bot-overwritten comments with the original context. Reddit is becoming maddening to use because half the old links I click have comments overwritten with garbage out of protest for something. Ironically the original content is available in these archives (which are used for AI training) but now missing for actual users like me just trying to figure out how someone fixed their printer driver 2 years ago.

anonymous908213 9mo ago
That would only really be ironic if the reason for people overwriting their comments was out of protest for LLM training, but the main reason that resulted in by far the biggest wave of deletions was Reddit locking down their API. If the result of their protest is that the site is less useful for you, the user, then in fact it served its purpose, as the entire point was an attempt to boycott Reddit, ie. get people to stop using it by removing the user contributions that give the site its only value in the first place.
- Aurornis 9mo ago
  > If the result of their protest is that the site is less useful for you, the user, then in fact it served its purpose, as the entire point was an attempt to boycott Reddit, ie. get people to stop using it by removing the user contributions that give the site its only value in the first place. In practice I just give them more page views because I have to view more threads before I find the answer. Reddit's DAU numbers have only gone up since the protest.
  - anonymous908213 9mo ago
    I did phrase it as "an attempt". In the end the protest probably wasn't as effective as protestors might have hoped, and it didn't get Reddit to change course on their enshittification decisions. I do think it was good that there was an attempt at pushback, at least, when most software users just accept enshittification as normal and continue tolerating whatever abuse their masters throw at them.

Jordan-117 9mo ago

[flagged]

19-84 9mo ago
thank you for your comment, I will support any platform that has complete dataset available. I will take submissions for any complete datasets through github issues. https://github.com/19-84/redd-archiver/blob/main/.github/ISSUE_TEMPLATE/submit-data-source.yml https://github.com/19-84/redd-archiver/blob/main/.github/ISS...
diggyhole 9mo ago
Wat?
- Jordan-117 9mo ago
  It sold itself as a healthier alternative to Reddit, but by the end of its run virtually every post sitewide was some flavor of virulently racist, misogynistic, anti-semitic, fringe conspiratorial, etc.
  - tempsaasexample 9mo ago
    [flagged]
    - Jordan-117 9mo ago
      Funny defense to use for a crowd that spent all their time regularly, hatefully dehumanizing people. The front page was routinely plastered with shit like "Why interracial children are an abomination," Hitler-did-nothing-wrong propaganda, usernames that echoed Nazi slogans and fantasized about mass-murdering non-white people, etc. It was an utter cesspool, and preserving and perpetuating that is a really weird use of dev time and effort.
      - lazyasciiart 9mo ago
        It’ll be useful to have when the posters pop up as presidential advisors.

dvngnt_ 9mo ago

I want to do the same thing for tiktok. I have 5k videos starting from the pandemic downloaded. want to find a way to use AI to tag and categorize the videos to scroll locally.

syngrog66 9mo ago

Did you pay all the people who created its content?

devilsdata 9mo ago
I have no problem with this being downloaded for personal use, in fact that's a good thing. But of course we both know it'll be used to train AI.
nullandvoid 9mo ago
Did anyone ever comment on reddit with an expectation of pay? It's an open forum - similar to here, whatever I post I it's in the public forum and therefore I expect it to be used / remixed however anyone wants.
- nozzlegear 9mo ago
  > Did anyone ever comment on reddit with an expectation of pay? Maybe Gallowboob
  - Sohcahtoa82 9mo ago
    That's a name I haven't seen in a LONG time.
antisthenes 9mo ago
Reddit didn't pay me for posting either. Not that I posted in the last decade.

deleted 9mo ago

[deleted]

alcroito 9mo ago

I tried spinning up the local approach with docker compose, but it fails. There's no `.env.example` file to copy from. And even if the env vars are set manually, there are issues with the mentioned volumes not existing locally. Seems like this needs more polish.

19-84 9mo ago
thank you for your comment, some example dot files were not copied in my original repo, they have now been added. https://github.com/19-84/redd-archiver/commit/0bb103952195ae5897d7c7d32f65aeb0471cb664 https://github.com/19-84/redd-archiver/commit/0bb103952195ae... the docs have been updated with mkdir steps https://github.com/19-84/redd-archiver/commit/c3754ea3a0238f9ec573604558e0594480142e6e https://github.com/19-84/redd-archiver/commit/c3754ea3a0238f...
- alcroito 9mo ago
  Cheers. I checked the updated steps. This is still missing creating the `output/.postgres-data` dir, without which docker compose refuses to start. After creating that manually, going to http://localhost/ shows a 403 Forbidden page, which makes you believe that something might have gone wrong. This is before running `reddarchiver-builder python reddarc.py` to generate the necessary DB from the input data.
  - 19-84 9mo ago
    I've updated the workflow and added a placeholder page that will serve before archives are created. thanks again! https://github.com/19-84/redd-archiver/commit/0dfd505ca81cb20dade5f2047631ddbb60926c7a https://github.com/19-84/redd-archiver/commit/0dfd505ca81cb2...

diggings 9mo ago

This is a neat project, nice work. You've probably come across this already but there are alternative archives to PushShift that may have differing sets of posts and comments (perhaps depending on removal request coverage?) One is Arctic Shift: https://github.com/ArthurHeitmann/arctic_shift/releases https://github.com/ArthurHeitmann/arctic_shift/releases Another is PullPush: https://pullpush.io/ https://pullpush.io/

bkovacev 9mo ago

Is there any way to check if a subreddit that was made private (2-3 years ago) is in the data dump?

tourist2d 9mo ago
[dead]
19-84 9mo ago
I included a metadata dump of every subreddit found in the torrent. it includes a status field which will show of a subreddit is private along with a much more details data catalog readme: https://github.com/19-84/redd-archiver/blob/main/tools/README.md https://github.com/19-84/redd-archiver/blob/main/tools/READM... reddit data: https://github.com/19-84/redd-archiver/blob/main/tools/subreddits_complete.json https://github.com/19-84/redd-archiver/blob/main/tools/subre...

m463 9mo ago

I wonder if you could use this to "Seed" a new distributed social media thing and just take over from there. sort of like forking a project.

19-84 9mo ago
ive created tooling for an instance registry and team based leaderboard. the API has function to support this as well, so that we can collectively host archives in a decentralized and distributed manner registry readme: https://github.com/19-84/redd-archiver/blob/main/docs/REGISTRY_SETUP.md https://github.com/19-84/redd-archiver/blob/main/docs/REGIST... register instances: https://github.com/19-84/redd-archiver/blob/main/.github/ISSUE_TEMPLATE/register-instance.yml https://github.com/19-84/redd-archiver/blob/main/.github/ISS...

drob518 9mo ago

This is a great way to participate in arguments you missed three years ago.

justsomehnguy 9mo ago

Appreciated. EDIT: Is there any cheap way to search? I have MS TechNet archive which is useless without search, so I realky want to know a way to have a cheap local search w/o grepping everyting.

19-84 9mo ago
redd-archiver uses postgres full text search. for static search you could use lunr.js

twobitshifter 9mo ago

If reddit was a squeaky clean place, or if I could pick certain subs, maybe I would be interested, but I really wouldn't want ALL of reddit on my machine even temporarily.

19-84 9mo ago
the torrent has data for the top 40,000 subs on reddit. thanks to watchful1 splitting the data by subreddit, you can download only the subreddit you want from the torrent
- Imustaskforhelp 9mo ago
  I am going to be honest and this looks really cool. 40,000 subs are good numbers and I hope that the number can be spread to even more subreddits Perhaps we can finally migrate all or much of the data to lemmy instances as well to finally get the lemmy instance up and running as well. Thank you for creating this. It opens up a lots of interesting opportunities.

feconroses 9mo ago

Very cool project! Quick question: is the underlying Pushshift dataset updated with new Reddit data on any regular cadence (daily/weekly/monthly), or is this essentially a fixed historical snapshot up to a certain date? Just want to understand if self-hosters would need to periodically re-download for fresh content or if it's archival-only.

19-84 9mo ago
the data from 2025-12 has been released already, it is usually released every month, it just needs to be split and reprocessed for 2025 by watchful1. i will probably eventually add support for importing data from the monthly arctic shift dumps so that archives can be updated monthly. https://github.com/ArthurHeitmann/arctic_shift/releases https://github.com/ArthurHeitmann/arctic_shift/releases Arctic Shift https://academictorrents.com/browse.php?search=RaiderBDev https://academictorrents.com/browse.php?search=RaiderBDev Watchful1 https://academictorrents.com/browse.php?search=Watchful1 https://academictorrents.com/browse.php?search=Watchful1
- riku_iki 9mo ago
  Is data web scrapped? Is reddit ok with that?..
  - inquirerGeneral 9mo ago
    [dead]

blks 9mo ago

Does it also contains countless NSFW content?

blks 9mo ago

Opened the live demo, went into programming subreddit, felt like I was showered with liquid shit. I tend to forget what kind of edgelord hellhole Reddit was (and stil is sometimes).

vivzkestrel 9mo ago

- slightly offtopic here but does anyone have a similar data set of all youtube channels out there? - details probably include the 400 million youtube accounts, channel id, name, creator url, etc

19-84 9mo ago
there is nearly 10TB of youtube metadata available on archive.org https://archive.org/details/youtube-metadata https://archive.org/details/youtube-metadata
- chicagojoe 9mo ago
  These show as unavailable/with lock icons for me. Is there some process to download locked content from IA?

inquirerGeneral 9mo ago

[dead]

tetrisgm 9mo ago

Is there a docker compose?

nick007x 9mo ago

Hey, I’m working on a similar project and have uploaded Pushshift Reddit data to Hugging Face Datasets. If anyone wants to download specific files when torrents aren’t seeding well, you can use: https://huggingface.co/datasets/nick007x/pushshift-reddit https://huggingface.co/datasets/nick007x/pushshift-reddit It’s handy for grabbing individual months or subreddit slices without needing to pull the full torrent. Might be useful for smaller-scale archiving or testing.

nick007x 9mo ago
[dead]