8 ms·

Multiple load-balanced “webscale” servers This is the biggest one. I have seen many, many projects held back by the idea that they need eight nines of reliabil

Multiple load-balanced “webscale” servers This is the biggest one. I have seen many, many projects held back by the idea that they need eight nines of reliability and a geodiverse HA cluster before they can even start, where a single server with a half-decent backup plan would be more than enough. Good security from the start You don't need that either. You absolutely should think about security but it's by no means necessary to succeed. And really, backups and letting someone else handle payments will usually make it impossible to fail catastrophically†. † Some exceptions apply.

hanief 13y ago
Do you have a recommendation on decent backup practice? Thanks.
- jvehent 13y ago
  On small websites, I use git to track /etc and /var/www (where the webapp code is installed). 5 years ago, I would have recommended to rsync the entire server daily. But nowadays it's a lot easier to snapshot an EC2 instance daily. For the database, just use RDS which handles point-in-time snapshot for you. If you're a developer with little linux experience, don't try to play the sunday morning sysadmin with fancy bash scripts on your startup. It takes a lot more time and resources to keep a system in good state than you can foresee.
  - ixmatus 13y ago
    This is the reason for idempotent server configuration via Ansible, Chef, Salt, Puppet, or even better NixOS. Backing up a deployed application is kind of silly when the tagged releases should be sitting in your source control and that the press of a button have an entire server built and configured from scratch that is identical to the one you need. Then storing that configuration in source control. Backing up configuration files and what-not is also pretty silly and indicative of bad devop practices. The database should be versioned and maintained with a migration tool. So the only feature of the database that needs backing up should be the data it holds - live replication servers that can act as background backup slaves are perfect for this. The only stuff you should be backing up on your servers are the latest log file rotations, user-generated data in the database, and any critical files that were also generated / uploaded by your users (but technically, you should be storing those in S3 with duplicates in another availability zone). One other thought: people usually don't think to encrypt their backups. Duplicity + GPG + S3 is what I use right now and is super easy to setup with some fiddling. Once Tarsnap becomes a bit more professional I'll be moving all of my server backups to Tarsnap.
    - joevandyk 13y ago
      > So the only feature of the database that needs backing up should be the data it holds - live replication servers that can act as background backup slaves are perfect for this. Replication won't help if a "drop table users;" happens. You need point in time recovery. You may want to backup log files and uploaded images/files. The backups should be stored at your main service provider and also at a different unrelated place, so if AWS or whoever deletes your account, your backups aren't toast. You also want an automated system for monitoring that your backups are happening and that you can restore from them. If encrypting backups, don't forget to backup your encryption keys.
      - ixmatus 13y ago
        That's what I meant by "background backup slave". The slave can handle point-in-time backups which is better than doing dumps or hot backups on the primary DB server. Good point re: multiple providers.
- adwf 13y ago
  The most important - and most forgotten/ignored - part of a good backup practice, is actually checking that you can restore the backup successfully. Ok, it's slightly less important than actually backing up at all in the first place, but only slightly. You won't believe just how many organisations will have an automated script for a daily/weekly/monthly rolling backup of their database, but won't have any procedure to restore the previous days database to check the backup procedure went ok! At the most basic, it's just another automated script on a spare box to ensure the restore returns ok. But people don't even bother with that...
  - sjclemmy 13y ago
    Word to that. I was contracting once and I asked about the backups. Someone was diligently changing the tapes on a daily basis but the backup had actually been failing for months. How we laughed...
- deleted 13y ago
  [deleted]
- deleted 13y ago
  [deleted]
antidaily 13y ago
I consulted on a project pulling in 6 figures that stored bank account routing info in plain text and had zero source code control. Extreme example but proves your point, I'd say.
- deleted 13y ago
  [deleted]
- andruby 13y ago
  Let me guess, something with crypto currencies?
  - dllthomas 13y ago
    Probably not. Cryptocurrency disasters are largely a result of the same poor practices we see everywhere, and there are more non-cryptocurrency projects than cryptocurrency ones.
jimmaswell 13y ago
>Multiple load-balanced “webscale” servers >This is the biggest one. I have seen many, many projects held back by the idea that they need eight nines of reliability and a geodiverse HA cluster before they can even start, where a single server with a half-decent backup plan would be more than enough. This seems weird to me because I'd gotten the impression, from reading around, that it was more often a problem that someone's servers weren't ready to handle the throughput due to not enough servers, not being written scalably from the start, etc. And I've seen this happen a ton of times with people's new web apps that they post to HN or reddit. It even has a name over there, the "reddit hug of death" when it's overloaded by the new demand.
- dangrossman 13y ago
  In 99.9% of cases, a $5 VPS would have been able to handle the Reddit/HN traffic, if only it were configured better. If you "yum/apt-get install httpd" and call it a day, your default configuration will cause Apache to spawn more processes than there is available memory as soon as you get more than a few visits at once, and the disk swapping will kill the box. If you set reasonable limits for the number of processes/threads/connections to keep them within the available RAM, then most sites will have no issue with thousands of visitors per hour. Here's a screenshot of the real-time analytics for my blog on a day it was front page of HN. That was a WordPress site, with no caching, on Linode's 512MB VPS (the cheapest plan). The server barely registered any load at all. http://www.dangrossman.info/wp-content/themes/2012/hnpulse.png http://www.dangrossman.info/wp-content/themes/2012/hnpulse.p... I also launched my web analytics startup on a VPS. It was handling millions of visits per day before I rented the first real server. You can absolutely have thousands of users, thousands of dollars in revenue, and a $20 hosting bill. Sometimes that's all that's required. Plan for what you're going to do when you outgrow the hosting, but there's no need to buy hardware you don't need up front.
  - extesy 13y ago
    Would you mind sharing the specific parameters of your apache configuration to get these results on 512mb VPS?
  - 13y ago

hanief 13y ago

Do you have a recommendation on decent backup practice? Thanks.

jvehent 13y ago
On small websites, I use git to track /etc and /var/www (where the webapp code is installed). 5 years ago, I would have recommended to rsync the entire server daily. But nowadays it's a lot easier to snapshot an EC2 instance daily. For the database, just use RDS which handles point-in-time snapshot for you. If you're a developer with little linux experience, don't try to play the sunday morning sysadmin with fancy bash scripts on your startup. It takes a lot more time and resources to keep a system in good state than you can foresee.
- ixmatus 13y ago
  This is the reason for idempotent server configuration via Ansible, Chef, Salt, Puppet, or even better NixOS. Backing up a deployed application is kind of silly when the tagged releases should be sitting in your source control and that the press of a button have an entire server built and configured from scratch that is identical to the one you need. Then storing that configuration in source control. Backing up configuration files and what-not is also pretty silly and indicative of bad devop practices. The database should be versioned and maintained with a migration tool. So the only feature of the database that needs backing up should be the data it holds - live replication servers that can act as background backup slaves are perfect for this. The only stuff you should be backing up on your servers are the latest log file rotations, user-generated data in the database, and any critical files that were also generated / uploaded by your users (but technically, you should be storing those in S3 with duplicates in another availability zone). One other thought: people usually don't think to encrypt their backups. Duplicity + GPG + S3 is what I use right now and is super easy to setup with some fiddling. Once Tarsnap becomes a bit more professional I'll be moving all of my server backups to Tarsnap.
  - joevandyk 13y ago
    > So the only feature of the database that needs backing up should be the data it holds - live replication servers that can act as background backup slaves are perfect for this. Replication won't help if a "drop table users;" happens. You need point in time recovery. You may want to backup log files and uploaded images/files. The backups should be stored at your main service provider and also at a different unrelated place, so if AWS or whoever deletes your account, your backups aren't toast. You also want an automated system for monitoring that your backups are happening and that you can restore from them. If encrypting backups, don't forget to backup your encryption keys.
    - ixmatus 13y ago
      That's what I meant by "background backup slave". The slave can handle point-in-time backups which is better than doing dumps or hot backups on the primary DB server. Good point re: multiple providers.
adwf 13y ago
The most important - and most forgotten/ignored - part of a good backup practice, is actually checking that you can restore the backup successfully. Ok, it's slightly less important than actually backing up at all in the first place, but only slightly. You won't believe just how many organisations will have an automated script for a daily/weekly/monthly rolling backup of their database, but won't have any procedure to restore the previous days database to check the backup procedure went ok! At the most basic, it's just another automated script on a spare box to ensure the restore returns ok. But people don't even bother with that...
- sjclemmy 13y ago
  Word to that. I was contracting once and I asked about the backups. Someone was diligently changing the tapes on a daily basis but the backup had actually been failing for months. How we laughed...
deleted 13y ago
[deleted]
deleted 13y ago
[deleted]

antidaily 13y ago

I consulted on a project pulling in 6 figures that stored bank account routing info in plain text and had zero source code control. Extreme example but proves your point, I'd say.

deleted 13y ago
[deleted]
andruby 13y ago
Let me guess, something with crypto currencies?
- dllthomas 13y ago
  Probably not. Cryptocurrency disasters are largely a result of the same poor practices we see everywhere, and there are more non-cryptocurrency projects than cryptocurrency ones.

jimmaswell 13y ago

>Multiple load-balanced “webscale” servers >This is the biggest one. I have seen many, many projects held back by the idea that they need eight nines of reliability and a geodiverse HA cluster before they can even start, where a single server with a half-decent backup plan would be more than enough. This seems weird to me because I'd gotten the impression, from reading around, that it was more often a problem that someone's servers weren't ready to handle the throughput due to not enough servers, not being written scalably from the start, etc. And I've seen this happen a ton of times with people's new web apps that they post to HN or reddit. It even has a name over there, the "reddit hug of death" when it's overloaded by the new demand.

dangrossman 13y ago
In 99.9% of cases, a $5 VPS would have been able to handle the Reddit/HN traffic, if only it were configured better. If you "yum/apt-get install httpd" and call it a day, your default configuration will cause Apache to spawn more processes than there is available memory as soon as you get more than a few visits at once, and the disk swapping will kill the box. If you set reasonable limits for the number of processes/threads/connections to keep them within the available RAM, then most sites will have no issue with thousands of visitors per hour. Here's a screenshot of the real-time analytics for my blog on a day it was front page of HN. That was a WordPress site, with no caching, on Linode's 512MB VPS (the cheapest plan). The server barely registered any load at all. http://www.dangrossman.info/wp-content/themes/2012/hnpulse.png http://www.dangrossman.info/wp-content/themes/2012/hnpulse.p... I also launched my web analytics startup on a VPS. It was handling millions of visits per day before I rented the first real server. You can absolutely have thousands of users, thousands of dollars in revenue, and a $20 hosting bill. Sometimes that's all that's required. Plan for what you're going to do when you outgrow the hosting, but there's no need to buy hardware you don't need up front.
- extesy 13y ago
  Would you mind sharing the specific parameters of your apache configuration to get these results on 512mb VPS?
- 13y ago