4 ms·

Show HN: Restoredrill – proves your Postgres backups restore

50 points by ahmadpiran 1mo ago

ahmadpiran 1mo ago
Nobody disputes that backups need testing. Almost nobody actually does it, because there's no safe place to restore to and no time to build one. The people who do automate it usually end up with a cron job and a shell script that fails in its own quiet way. The real risk was never a bad backup. It's that quietly not running, or quietly restoring the same stale file, month after month, with nobody noticing. restoredrill turns that drill into a single command. It grabs your latest backup, restores it into a disposable Docker container, runs whatever checks you configure, and writes out a JSON report with the restore time. Every check is fail-closed: if it can't run, that's a failure, not something quietly skipped. Before writing a line of code I spent a couple weeks asking around r/devops and a few compliance subreddits how people actually handle this. Nobody had a clean answer, it was all bespoke scripts or nothing at all. A handful had already built close to this exact tool for themselves and hit the same wall every time: automating the restore was the easy part, getting a report format an auditor would sign off on took months. That's what shaped most of the report schema here. The quickstart is meant to take about 10 minutes start to finish: pg_dump a Supabase or RDS database, point restoredrill at the file, run it against a local container. No production access, no S3, no CI required just to try it. Worth naming the alternatives up front. Databasus is a solid self-hosted backup platform, several database engines, full dashboard, restore verification built in. BackupDrill does something close to this specifically for Supabase. restoredrill is narrower on purpose: no dashboard, a report built for an auditor (fail-closed checks, RPO/RTO evidence, your own SQL assertions), and it runs as a CI job instead of infrastructure you have to host. Postgres only for the moment, pgBackRest and WAL support are on the roadmap. MIT licensed. Repo: https://github.com/ahmadpiran/restoredrill https://github.com/ahmadpiran/restoredrill Would like to hear what people think, especially if you've built something like this yourself and ran into the same report-format problem.
davedx 1mo ago
This is a great idea. Anything that makes this easier is a win for ops teams.
- ahmadpiran 1mo ago
  [flagged]
sebmellen 1mo ago
> There are other tools in this space. Worth naming plainly instead of pretending they don't exist. Why do LLMs like this kind of writing? Why did you need to “plainly” name the competition as opposed to “flamboyantly” naming them? This is a cool idea but the README is absurdly not-to-the-point.
- petercooper 1mo ago
  CLaude, in particular, quietly, but genuinely and deliberately, overuses adverbs.
- ahmadpiran 1mo ago
  Fair, you're right. Thanks for the point. I used ai for a chunk of the build and writing. However, the actual work behind it wasn't AI-generated. I spent a few weeks interviewing people in r/devops and some compliance subs about how they actually test backup restores, and the report schema especially came out directly out of those conversations. You are also right about the REAME. I'll tighten it up right now, and enhance it. Thank you for the feedback.
  - progbits 1mo ago
    What bothers me personally the most is trying to hide it. Why did you remove Claude's (or whatever you used, but looks like opus to me from the style) co-author info? I'm willing to use (semi-)vibecoded projects under certain circumstances, but I like to know up front what I'm getting. The github contributors list makes that easy to see as long as people don't try to hide it.
    - ahmadpiran 1mo ago
      I don't have a good defense here, it's fair. But part of that was because of messy commit history and I wanted to have a clean commit history. Maybe it's not a good reason as well, but it was my real intention and my desire of perfectionism :) reason is that I wanted to take the full control, when ai do the commits you normally don't check the code and you loose the control, well I think it's the most important thing. But Going forward, for building trust, definitely take your recommendation. The actual design, the report schema especially, came out of around 30 interviews with people who handle backup testing as part of their job. Thank you for the feedback.
deleted 1mo ago
[deleted]
lbrito 1mo ago
This is a great idea and something I've faced and had to implement manually in the past. The most common scenario for me was: I implement backup, done by a daily cron calling pg_dump. It worked and I moved on, until one day I needed to restore and the backups did not work. Typically at some point the VM failed to run the commands to clear temporary backup files, the disk would fill up and the new backups would be truncated mid way. psql is pretty awful at identifying this which must be a very common scenario. Instead of a header check leading to "Corrupted backup, aborting" or something, it simply executes the SQL commands until it reaches the very last line, which will be truncated midway and thus invalid SQL. Anyway, it took some time to figure out, but later I did basically what your tool does, probably simpler: another daily cron job that downloads the file and checks if it ends with the correct pg_dump success signature. It has worked so far.
- ahmadpiran 1mo ago
  [dead]
- ahmadpiran 1mo ago
  [dead]
deleted 1mo ago
[deleted]
csydas 1mo ago
i saw your now dead comment about how no one you interviewed had an answer… just it’s surprising to me that devops folk aren’t in an org that has backup software, cause this kind of testing is standard load out for most backup softwares, and they even solve the space and isolation issue with various instant restore / publishing capabilities where data from the backup is mounted directly to some staging server with postgres or whatever installed in an isolated area not trying to knock the idea you are correct most ppl don’t test their backups, but it’s a bit strange no one you talked to used any of the most common backup softwares definitely in favor of free solutions as backup software field is awful like many ent tech fields, but like the workflows do exist
- ahmadpiran 1mo ago
  One person I talked to had Veeam SureBackup running and still built a separate check for the database. As I know SureBackup proved the VM boots, but it doesn't know or care if your orders table has the right row count, or if a sequence is broken. That needs a tool that actually understand the specific platform or data, like Postgres, not only the underlying VM. Also a lot of people I talked to were never use Veeam or Commvault, since they run Postgres on RDS, Supabase, or a simple VM. And even where the tooling exists, it's usually a separate infra tam running it, not the people who actually own this one database, so it never gets wired up for this specific check anyway. This tool is more for people whose stack never had such a thing, or it's there and nobody's actually using it for this.
  - csydas 1mo ago
    aha that’s sort of what i mean though — your concept is two parts: isolated restore and bespoke tests the isolation part is something that is more difficult due to space and resource constraints, and a lot of backup apps handle that but expect the user to handle the bespoke testing part your tool covers both but expects the user to provide both which isn’t always feasible for the application teams (not impossible but lot of red tape to request resources and a lot of resistance if the company already has recovery verification tooling) if i were you i would refocus your pitch on making it work as part of those backup software workflows, “tell your backup team about this test suite and include it in their recovery verification workflows” and i’m sure you’ll get a lot of attention most of the dba’s probably have a backup and recovery verification solution at their company with the isolation and resources but they don’t have the bespoke application testing
vladigtr 1mo ago
[flagged]
sebastiangrill 1mo ago
The problem i see with this approach is that nearly nobody uses dumps as backups. I consulted Postgres for a few hundred customers and the amount of times that I saw dumps as Backup can probably be counted on two hands. Does not help that the description reads like AI slop.
- ahmadpiran 1mo ago
  v1 is pg_dump only because it's the thing that every Postgres install can do natively, and the fastest way to get something really working and tested. Physical backups, pgBackRest specifically are already on the roadmap. One person on interviews doing this kind of testing was on plain dumps and moving off them, about 18 hours to restore versus 2 with a physical backup tool. I don't want to disagree here, I know dump-based isn't a great long term strategy. I will enhance the REAME, thank you for the feedback.
mannyv 1mo ago
This tool's existence is vaguely horrifying. Why wouldn't a database dump restore? Why wouldn't your dump command tell you it failed?
- JodieBenitez 1mo ago
  You'd be surprised... we do have horror stories.
- ahmadpiran 1mo ago
  [dead]
shidesheng 1mo ago
[flagged]
brianpg 1mo ago
[flagged]
martinlucas1994 1mo ago
[dead]

50 points by ahmadpiran 1mo ago

Nobody disputes that backups need testing. Almost nobody actually does it, because there's no safe place to restore to and no time to build one. The people who do automate it usually end up with a cron job and a shell script that fails in its own quiet way. The real risk was never a bad backup. It's that quietly not running, or quietly restoring the same stale file, month after month, with nobody noticing. restoredrill turns that drill into a single command. It grabs your latest backup, restores it into a disposable Docker container, runs whatever checks you configure, and writes out a JSON report with the restore time. Every check is fail-closed: if it can't run, that's a failure, not something quietly skipped. Before writing a line of code I spent a couple weeks asking around r/devops and a few compliance subreddits how people actually handle this. Nobody had a clean answer, it was all bespoke scripts or nothing at all. A handful had already built close to this exact tool for themselves and hit the same wall every time: automating the restore was the easy part, getting a report format an auditor would sign off on took months. That's what shaped most of the report schema here. The quickstart is meant to take about 10 minutes start to finish: pg_dump a Supabase or RDS database, point restoredrill at the file, run it against a local container. No production access, no S3, no CI required just to try it. Worth naming the alternatives up front. Databasus is a solid self-hosted backup platform, several database engines, full dashboard, restore verification built in. BackupDrill does something close to this specifically for Supabase. restoredrill is narrower on purpose: no dashboard, a report built for an auditor (fail-closed checks, RPO/RTO evidence, your own SQL assertions), and it runs as a CI job instead of infrastructure you have to host. Postgres only for the moment, pgBackRest and WAL support are on the roadmap. MIT licensed. Repo: https://github.com/ahmadpiran/restoredrill https://github.com/ahmadpiran/restoredrill Would like to hear what people think, especially if you've built something like this yourself and ran into the same report-format problem.

davedx 1mo ago

This is a great idea. Anything that makes this easier is a win for ops teams.

ahmadpiran 1mo ago
[flagged]

sebmellen 1mo ago

> There are other tools in this space. Worth naming plainly instead of pretending they don't exist. Why do LLMs like this kind of writing? Why did you need to “plainly” name the competition as opposed to “flamboyantly” naming them? This is a cool idea but the README is absurdly not-to-the-point.

petercooper 1mo ago
CLaude, in particular, quietly, but genuinely and deliberately, overuses adverbs.
ahmadpiran 1mo ago
Fair, you're right. Thanks for the point. I used ai for a chunk of the build and writing. However, the actual work behind it wasn't AI-generated. I spent a few weeks interviewing people in r/devops and some compliance subs about how they actually test backup restores, and the report schema especially came out directly out of those conversations. You are also right about the REAME. I'll tighten it up right now, and enhance it. Thank you for the feedback.
- progbits 1mo ago
  What bothers me personally the most is trying to hide it. Why did you remove Claude's (or whatever you used, but looks like opus to me from the style) co-author info? I'm willing to use (semi-)vibecoded projects under certain circumstances, but I like to know up front what I'm getting. The github contributors list makes that easy to see as long as people don't try to hide it.
  - ahmadpiran 1mo ago
    I don't have a good defense here, it's fair. But part of that was because of messy commit history and I wanted to have a clean commit history. Maybe it's not a good reason as well, but it was my real intention and my desire of perfectionism :) reason is that I wanted to take the full control, when ai do the commits you normally don't check the code and you loose the control, well I think it's the most important thing. But Going forward, for building trust, definitely take your recommendation. The actual design, the report schema especially, came out of around 30 interviews with people who handle backup testing as part of their job. Thank you for the feedback.

deleted 1mo ago

[deleted]

lbrito 1mo ago

This is a great idea and something I've faced and had to implement manually in the past. The most common scenario for me was: I implement backup, done by a daily cron calling pg_dump. It worked and I moved on, until one day I needed to restore and the backups did not work. Typically at some point the VM failed to run the commands to clear temporary backup files, the disk would fill up and the new backups would be truncated mid way. psql is pretty awful at identifying this which must be a very common scenario. Instead of a header check leading to "Corrupted backup, aborting" or something, it simply executes the SQL commands until it reaches the very last line, which will be truncated midway and thus invalid SQL. Anyway, it took some time to figure out, but later I did basically what your tool does, probably simpler: another daily cron job that downloads the file and checks if it ends with the correct pg_dump success signature. It has worked so far.

ahmadpiran 1mo ago
[dead]
ahmadpiran 1mo ago
[dead]

deleted 1mo ago

[deleted]

csydas 1mo ago

i saw your now dead comment about how no one you interviewed had an answer… just it’s surprising to me that devops folk aren’t in an org that has backup software, cause this kind of testing is standard load out for most backup softwares, and they even solve the space and isolation issue with various instant restore / publishing capabilities where data from the backup is mounted directly to some staging server with postgres or whatever installed in an isolated area not trying to knock the idea you are correct most ppl don’t test their backups, but it’s a bit strange no one you talked to used any of the most common backup softwares definitely in favor of free solutions as backup software field is awful like many ent tech fields, but like the workflows do exist

ahmadpiran 1mo ago
One person I talked to had Veeam SureBackup running and still built a separate check for the database. As I know SureBackup proved the VM boots, but it doesn't know or care if your orders table has the right row count, or if a sequence is broken. That needs a tool that actually understand the specific platform or data, like Postgres, not only the underlying VM. Also a lot of people I talked to were never use Veeam or Commvault, since they run Postgres on RDS, Supabase, or a simple VM. And even where the tooling exists, it's usually a separate infra tam running it, not the people who actually own this one database, so it never gets wired up for this specific check anyway. This tool is more for people whose stack never had such a thing, or it's there and nobody's actually using it for this.
- csydas 1mo ago
  aha that’s sort of what i mean though — your concept is two parts: isolated restore and bespoke tests the isolation part is something that is more difficult due to space and resource constraints, and a lot of backup apps handle that but expect the user to handle the bespoke testing part your tool covers both but expects the user to provide both which isn’t always feasible for the application teams (not impossible but lot of red tape to request resources and a lot of resistance if the company already has recovery verification tooling) if i were you i would refocus your pitch on making it work as part of those backup software workflows, “tell your backup team about this test suite and include it in their recovery verification workflows” and i’m sure you’ll get a lot of attention most of the dba’s probably have a backup and recovery verification solution at their company with the isolation and resources but they don’t have the bespoke application testing

vladigtr 1mo ago

[flagged]

sebastiangrill 1mo ago

The problem i see with this approach is that nearly nobody uses dumps as backups. I consulted Postgres for a few hundred customers and the amount of times that I saw dumps as Backup can probably be counted on two hands. Does not help that the description reads like AI slop.

ahmadpiran 1mo ago
v1 is pg_dump only because it's the thing that every Postgres install can do natively, and the fastest way to get something really working and tested. Physical backups, pgBackRest specifically are already on the roadmap. One person on interviews doing this kind of testing was on plain dumps and moving off them, about 18 hours to restore versus 2 with a physical backup tool. I don't want to disagree here, I know dump-based isn't a great long term strategy. I will enhance the REAME, thank you for the feedback.

mannyv 1mo ago

This tool's existence is vaguely horrifying. Why wouldn't a database dump restore? Why wouldn't your dump command tell you it failed?

JodieBenitez 1mo ago
You'd be surprised... we do have horror stories.
ahmadpiran 1mo ago
[dead]

shidesheng 1mo ago

[flagged]

brianpg 1mo ago

[flagged]

martinlucas1994 1mo ago

[dead]