4 ms·

You're right, if we manage our own db, then we need everybody to understand how to troubleshoot across both software and hardware issues, rather than just file

by ashrust 14y ago

You're right, if we manage our own db, then we need everybody to understand how to troubleshoot across both software and hardware issues, rather than just file a ticket and follow instructions/suggestions. Let's say we have 5 engineers and that kind of training costs us 1 hour per eng/per week. That's 20 hours a month, which is easily more than $1k/month and the cost grows as the team expands. This is before you account for the losses of having your own engineers on call for issues and setting up early warning systems specific to the database etc.

josegonzalez 14y ago
Good point. Hardware aside - thats largely moot on both Heroku and AWS - the 5x cost of the server will very quickly outstrip the cost of a good developer/operations guy. It seems like a good idea - I initially just throw money at a problem too! - but at some point it does not make financial sense, which is what I was getting at. Note that we don't have every engineer on staff capable of bringing up a new server/rebuild a damaged mysql replication setup, but we do have engineers that can: - ssh (or attempt to ssh onto) a dead instance - tail a log - check disk space, memory usage and cpu load - check if something is running - restart something that just randomly stopped All that stuff is pretty basic, and will get you 80% of the way there - the other 20% being experience. I guess at some point you are paying for the experience of working with a datastore, so there it makes sense. As far as early warning etc., that does take time, but it's not the big deal it appears to be. EDIT: I can't math, and 80 + 10 = 90, not 100. Good thing I'm not a data scientist :)
- jeltz 14y ago
  For getting early warnings and helping your employees debug what went wrong there are several good monitoring tools. They wont magically solve your problems but it will make it much easier for non-database experts to run a database server. For monitoring PostgreSQL I have used the excellent plugins that are shipped with munin. http://munin-monitoring.org/ http://munin-monitoring.org/ EDIT: munin does also ship monitoring plugins for MySQL but since I have never used them I cannot vouch for them. The general health monitoring (disk usage, CPU usage, SMART status, inode usage, ...) of the machines provided by munin is also probably more valuable than the database specific monitoring.

by ashrust 14y ago

josegonzalez 14y ago

Good point. Hardware aside - thats largely moot on both Heroku and AWS - the 5x cost of the server will very quickly outstrip the cost of a good developer/operations guy. It seems like a good idea - I initially just throw money at a problem too! - but at some point it does not make financial sense, which is what I was getting at. Note that we don't have every engineer on staff capable of bringing up a new server/rebuild a damaged mysql replication setup, but we do have engineers that can: - ssh (or attempt to ssh onto) a dead instance - tail a log - check disk space, memory usage and cpu load - check if something is running - restart something that just randomly stopped All that stuff is pretty basic, and will get you 80% of the way there - the other 20% being experience. I guess at some point you are paying for the experience of working with a datastore, so there it makes sense. As far as early warning etc., that does take time, but it's not the big deal it appears to be. EDIT: I can't math, and 80 + 10 = 90, not 100. Good thing I'm not a data scientist :)

jeltz 14y ago
For getting early warnings and helping your employees debug what went wrong there are several good monitoring tools. They wont magically solve your problems but it will make it much easier for non-database experts to run a database server. For monitoring PostgreSQL I have used the excellent plugins that are shipped with munin. http://munin-monitoring.org/ http://munin-monitoring.org/ EDIT: munin does also ship monitoring plugins for MySQL but since I have never used them I cannot vouch for them. The general health monitoring (disk usage, CPU usage, SMART status, inode usage, ...) of the machines provided by munin is also probably more valuable than the database specific monitoring.