6 ms·

Is spending time to make it reliable worth it vs working on your actual product? Databases are THE most critical things your company has.

davedx 3y ago
All that infra doesn’t integrate itself. Everywhere I’ve worked that had this kind of stack employed at least one if not a team of DevOps people to maintain it all, full time, the year round. Automating a database backup and testing it works takes half a day unless you’re doing something weird
- isbvhodnvemrwvn 3y ago
  Setting up a multi-az db with automatic failover, incremental backups and PiTR, automated runbooks and monitoring all that doesn't take half a day, not even with RDS.
  - davedx 3y ago
    No, but again, that sounds like a lot of complexity your average startup does not need. Multi-az? Why?
    - marcinzm 3y ago
      Because their Enterprise client requires it on their due diligence paperwork.
      - dvfjsdhgfv 3y ago
        Which makes little sense anyway as in practice the real problems you have are from region/connectivity issues, not AZ failures.
- fullstackchris 3y ago
  A startup sized company using this many tools? They're for sure doing something weird (and that's not a compliment :) ) Totally on your side with this one - but alas, people associate value with complexity.
- ffsm8 3y ago
  > Automating a database backup and testing it works takes half a day unless you’re doing something weird True story bro I'm sure that's possible if you're storing the backup on the same server you're restoring on and everything is on top of the line nvme storage. Otherwise your backup just started to run and will need another few days to finish. And that's only if you're running single master. You're massively underestimating the challenge to get that kind of automation done in a stable manner - and the maintenance required to keep it working over the years.
  - davedx 3y ago
    I’ve implemented such a process for companies multiple times, bro. I know what I’m talking about.
    - marcinzm 3y ago
      And that's the problem. "It's easy for me because I've done it a dozen times so it's easy for everyone" is a very common fallacy.
      - layer8 3y ago
        What happened to having people trained by external trainers for what you need? That’s much cheaper than having everything externally “managed” and still having to integrate all of it. The number of services listed in TFA is just ridiculous.
      - davedx 3y ago
        This is an oversimplification, but! Dumping postgres to a file is one command. scp the file to a different server is two commands. (Granted you need to setup ssh keys there too). I have implemented backups this way. With sqlite you only need the scp part. You can even push your backup file to an S3 bucket... with one command! Honestly, this argument mystifies me. Of course you can make it as complicated as you want to, too. I've also worked on replicating anonymized data from a production OLTP database to a data warehouse. That's a lot more work.
Draiken 3y ago
I see this argument a lot. Then most startups use that time to create rushed half-assed features instead of spending a week on their db that'll end up saving hundreds of thousands of dollars. Forever. For me that's short-sighted.
- orthoxerox 3y ago
  Startups are in the job of earning millions. If they can spend $100k on a managed DB now and just spend every braincell on getting their product right, it's a win for their investors.
  - Draiken 3y ago
    That mentality is working wonders right now, isn't it? Hundreds of dead startups because after all that unnecessary spending, they still have unnecessary buggy software that got sold to other startups that, when push comes to shove, will cut spending in those same startups that offer half-baked buggy products. What you say is definitely what they preach. But I don't agree or see that as a good logic. Way too many startup founders decide to build shitty products with short-sighted solutions like these, following whatever is trendy (crypto, AI, etc) because investors advise them to. Guess what: the investor doesn't care about creating a good business. He wants a unicorn. So they advise them to make all-or-nothing moves knowing it will most likely kill the startup. It's definitely "a strategy". But I think it's short-sighted as hell.
    - orthoxerox 3y ago
      That's the whole point of startups and VC: not spending money on safe investments that provide a 10% return, but spending large amounts of money on risky investments that provide, when averaged together (rare unicorns on top of a mound of dead startups), a 20% return. Both numbers completely arbitrary, of course.
      - Draiken 3y ago
        Isn't this only the latest definition of "startup"? What exactly defines a startup? Is it growth at all costs? Is it reckless spending? Complete focus on short-term gains? Something else? These investment rounds are only there to provide money to start. Unless the founders sign away the control of their company to investors, they are still at the helm of the ship. They can choose how they will approach growth. We can see how the entire scene is gearing towards profitability now that the money dried up, so this growth focus is no longer the only game in town. You can still take investments and accelerate growth without having to recklessly go all-in. But I've never taken VC money. Maybe that's baked into the contracts?
eptcyka 3y ago
So investing in a critical part of my business is the bad thing to do?

davedx 3y ago

All that infra doesn’t integrate itself. Everywhere I’ve worked that had this kind of stack employed at least one if not a team of DevOps people to maintain it all, full time, the year round. Automating a database backup and testing it works takes half a day unless you’re doing something weird

isbvhodnvemrwvn 3y ago
Setting up a multi-az db with automatic failover, incremental backups and PiTR, automated runbooks and monitoring all that doesn't take half a day, not even with RDS.
- davedx 3y ago
  No, but again, that sounds like a lot of complexity your average startup does not need. Multi-az? Why?
  - marcinzm 3y ago
    Because their Enterprise client requires it on their due diligence paperwork.
    - dvfjsdhgfv 3y ago
      Which makes little sense anyway as in practice the real problems you have are from region/connectivity issues, not AZ failures.
fullstackchris 3y ago
A startup sized company using this many tools? They're for sure doing something weird (and that's not a compliment :) ) Totally on your side with this one - but alas, people associate value with complexity.
ffsm8 3y ago
> Automating a database backup and testing it works takes half a day unless you’re doing something weird True story bro I'm sure that's possible if you're storing the backup on the same server you're restoring on and everything is on top of the line nvme storage. Otherwise your backup just started to run and will need another few days to finish. And that's only if you're running single master. You're massively underestimating the challenge to get that kind of automation done in a stable manner - and the maintenance required to keep it working over the years.
- davedx 3y ago
  I’ve implemented such a process for companies multiple times, bro. I know what I’m talking about.
  - marcinzm 3y ago
    And that's the problem. "It's easy for me because I've done it a dozen times so it's easy for everyone" is a very common fallacy.
    - layer8 3y ago
      What happened to having people trained by external trainers for what you need? That’s much cheaper than having everything externally “managed” and still having to integrate all of it. The number of services listed in TFA is just ridiculous.
    - davedx 3y ago
      This is an oversimplification, but! Dumping postgres to a file is one command. scp the file to a different server is two commands. (Granted you need to setup ssh keys there too). I have implemented backups this way. With sqlite you only need the scp part. You can even push your backup file to an S3 bucket... with one command! Honestly, this argument mystifies me. Of course you can make it as complicated as you want to, too. I've also worked on replicating anonymized data from a production OLTP database to a data warehouse. That's a lot more work.

Draiken 3y ago

I see this argument a lot. Then most startups use that time to create rushed half-assed features instead of spending a week on their db that'll end up saving hundreds of thousands of dollars. Forever. For me that's short-sighted.

orthoxerox 3y ago
Startups are in the job of earning millions. If they can spend $100k on a managed DB now and just spend every braincell on getting their product right, it's a win for their investors.
- Draiken 3y ago
  That mentality is working wonders right now, isn't it? Hundreds of dead startups because after all that unnecessary spending, they still have unnecessary buggy software that got sold to other startups that, when push comes to shove, will cut spending in those same startups that offer half-baked buggy products. What you say is definitely what they preach. But I don't agree or see that as a good logic. Way too many startup founders decide to build shitty products with short-sighted solutions like these, following whatever is trendy (crypto, AI, etc) because investors advise them to. Guess what: the investor doesn't care about creating a good business. He wants a unicorn. So they advise them to make all-or-nothing moves knowing it will most likely kill the startup. It's definitely "a strategy". But I think it's short-sighted as hell.
  - orthoxerox 3y ago
    That's the whole point of startups and VC: not spending money on safe investments that provide a 10% return, but spending large amounts of money on risky investments that provide, when averaged together (rare unicorns on top of a mound of dead startups), a 20% return. Both numbers completely arbitrary, of course.
    - Draiken 3y ago
      Isn't this only the latest definition of "startup"? What exactly defines a startup? Is it growth at all costs? Is it reckless spending? Complete focus on short-term gains? Something else? These investment rounds are only there to provide money to start. Unless the founders sign away the control of their company to investors, they are still at the helm of the ship. They can choose how they will approach growth. We can see how the entire scene is gearing towards profitability now that the money dried up, so this growth focus is no longer the only game in town. You can still take investments and accelerate growth without having to recklessly go all-in. But I've never taken VC money. Maybe that's baked into the contracts?

eptcyka 3y ago

So investing in a critical part of my business is the bad thing to do?