3 ms·
I run Reverb (https://reverb.com https://reverb.com) and there's a lot of good information in this thread. Backups, Security, Alerts, Monitoring, all good thing
by atom_enger 12y ago
I run Reverb (https://reverb.com https://reverb.com) and there's a lot of good information in this thread. Backups, Security, Alerts, Monitoring, all good things.
However, the biggest piece to scaling your application is the automation of everything you possibly can so that you can scale when you need to. You're going to be in a bit of pain if you need to scale everything manually.
Here's a few things I automate using Jenkins:
* Creation of web application servers(whether it be Puppet, Chef or Ansible, etc) make sure you can bring up a new node quickly and scale your app layer horizontally. Ideally automate the addition of this node to the LB.
* Data store backup/restores to all staging environments on a schedule(tests backups/restores) this is done using some custom code and the Backup gem. This way your dev team has access to an env that closely resembles prod and can resolve current prod bugs.
* External security scans using NMap (again using custom scripts). The jenkins job will fail if output is not as it expects. This way if we change a layer of our infrastructure we can know if something is exposed and shouldn't be.
* Static code analysis using Brakeman
Information you're going to need to scale your infra:
* Metrics on each one of your hosts. Use DataDog if you can afford it, integrates with all major systems and technologies. Great tool.
* Log collection via something like Logstash or Loggly and being able to visualize your application and web logs.
* Application response time measurements using something like NewRelic or building your own using StatsD and tracking the heck out of your application actions
Last but not least, have a plan for failure. While you're laying in bed at night, ask yourself these question:
* What would happen if the DB went poof? Can I restore it? How much data will I lose from the last backup? Will I know when this happens?
* What would happen if you're now being scanned by 30 ips from the netherlands, all of which are submitting garbage data into your forms. Are you protected against this? How will the added load effect your app layer? Do you have a way to automate the responses to those requests so as to deny them? This is a case of when, not if. Be ready.
* What would happen if my site gets put on Digg(lol)?
There's no magic bullet here. It's just practice, failure and learning from yours and others mistakes.
Good luck!