5 ms·

If you have a HA DB cluster with predictable usage pattern (most fall into that category), it should be cheaper to run it yourself (with the employees needed fo

by dx034 9y ago

If you have a HA DB cluster with predictable usage pattern (most fall into that category), it should be cheaper to run it yourself (with the employees needed for it) than using RDS for most sizes. That doesn't mean having your own data centre but renting some dedicated servers somewhere. The additional work after setting it up isn't that much more than for the cloud where you still have to monitor uptime and load.

zbentley 9y ago
Strongly disagree. Hand-waving the initial setup period away hides a substantial footer cost (it is quite hard to learn how to safely, properly install a large database stack, especially if you're unfamiliar with it), and a substantial ongoing cost (updates etc.) that is often eliminated or ameliorated with cloud-hosted options. And the additional work after setting it up is much greater. You have to monitor uptime/load/etc. regardless, sure, but when running your own hardware you also need to monitor uptime and load of other things as well: internet connectivity to your servers, electrical power, human access to servers/physical security and so on. Especially for a small site, those are quite expensive; their cost decreases a bit when you're running or leasing a lot of hardware, but it's still quite high.
- quicksilver03 9y ago
  If you have to learn how to perform a proper setup, you're a good candidate for the kind of managed services provided by a cloud-based solution. As long as you're aware that you're paying a sort of "ignorance tax" for not knowing how to do things, that's perfectly fine. While researching for https://www.thecloudhostinghandbook.com/ https://www.thecloudhostinghandbook.com/ I discovered that there are as many possible setups as there are companies: the best solution depends on what kind of assumptions the developers have materialized in the code, how much the company wants to spend, and ultimately on the world view of the people running the system.
- stephenr 9y ago
  Why do "cloud" advocates always jump to the extreme opposite: you manage everything yourself down to mowing the lawn in front of your own custom built DC. You can buy and colo or even rent dedicated hardware. I've never heard of a DC that didn't offer a remote hands service for hardware issues that requires a physical presence.
  - toomuchtodo 9y ago
    If cloud advocates don’t jump to the extreme, the benefits of cloud computing quickly fall apart (there’s a reason AWS’ margins are ridiculously high).
  - zbentley 9y ago
    I have managed my own DC and colo'd. Some examples of issues that occurred: - Colo: unauthorized personnel tripped over a cable, killing network to a lot of our servers. This was in a huge colo hosting facility that was home to some fortune 500 companies' servers. - Own datacenter: local internet providers both had a planned maintenance/outage on the same day. This isn't "mowing the lawn in front of your DC", this is the most basic utility after power that needs to be available for your DC to work. - Colo: hard drive failure after hours. Colo security staff wouldn't let us disassemble hardware that wasn't ours. Had to wait 6hrs for a technician to show up to pop and swap a drive (this with a 1hr incident response in our contract and the same "highly reliable" colo; we got money back per our contract, but not the customers we lost, even after apologetic/refunding communications to them). - Own datacenter, third-party NAS appliance professionally installed by a vendor: persistent performance issues and service dropouts, eventually traced to the "professional installation" being at the bottom of a rack with a large, detachable rear panel mostly covering the fan intakes for the appliance, leading to persistent throttling due to heat. - Colo (rented hardware, managed OS install): When the business started, our sysadmins/DBAs (me and one other guy) were primarily experienced with RHEL and old school init. The colo only provisioned Debian/systemd servers. We learned, but it slowed us down for a week or so. Sure, it was only an extra few minutes per task, but it added up. - Own datacenter: management needed us to move into a new server room because of an ending lease, with a hard deadline. The air conditioning installation vendor showed up two weeks late; we had no cooling at all when we needed to cut over, causing days of service interruptions and downtime. I have also managed cloud services. Some examples of issues that occurred: - Amazon's S3-gate. That sucked. When it happened, we were able to email our customers a copy of Amazon's status, and links to reporting of exactly how wide-spread the issue was. We had impact nearly identical to the colo hard-drive-failure incident above (it was the same service, migrated to AWS/S3), but we didn't lose nearly as many customers. - The DNS DDoS that affected the East Coast of the US earlier this year. Same kind of service interruption, same communication. We even got replies back from customers saying "I couldn't get to Reddit either; I figured you guys might be in the same boat". I'm not a cloud evangelist. I think that there are very good reasons to host entirely locally, or colocate/rent, or anything in between. I do, however, think that businesses, especially small ones that are not typical software-centric startups, massively and regularly underestimate both the initial and ongoing costs of running their own infrastructure. There are significant technical benefits to being cloud-hosted (these benefits also apply, only a little less potently, to fully managed hosting operations a la Rackspace), but people often miss the political and financial benefits. The political benefits are things like "our customers are less pissed because widespread issues with AWS probably affected them in other ways that day as well, so not as many people will knee-jerk blame us". The financial benefits are primarily that there are fewer moments of the "oh shit, setting these servers up ended up costing way more than we thought it would", and "we thought this would take a day to turn up; it actually took two weeks" varieties. Those things still happen with cloud services, but much less often.
- dx034 9y ago
  I was talking about dedicated servers, not running your data centre. Data centres are a scale business but cheap to run at reasonable scale. Dedicated servers are significantly cheaper than cloud instances and there's not much more to monitor than for the cloud. Compared to hosted DB clusters you need someone to set it up and run it but you'd also need an AWS expert for a production cluster in the cloud.
throwaway2016a 9y ago
I also disagree on this one. We manages our own cluster for years. Even had an expert in that database technology help us optimize the config. Then we switched to RDS... Sure it was a few hundred dollars more per month in AWS fees but we saved thousands a month in consulting fees.
- dx034 9y ago
  Why do you need consultants to run a database cluster? And do you run AWS database clusters in production without expert knowledge? If you need consultants for self-hosted databases you'd probably also need consultants to properly set up AWS services.
  - throwaway2016a 9y ago
    Performance tuning, scaling, troubleshooting, etc. We had terabytes of active data (as opposed to warehoused data). For example, if a disk fills or is fails, adding a disk to a mySQL setup without causing downtime is a pretty involved process but in RDS it is one config change in the admin. It does the same process just AWS put a huge amount of work into automating it. We only brought in the consultant for an average of a day every other week but that adds up and we had to bring in 0 people after switching to RDS.
- KaiserPro 9y ago
  but you are already there. At present company if I want a new database, I have to make it from scratch, and I'm not a DBA. One size really doesn't fit all.
KaiserPro 9y ago
it _is_ cheaper to run. But thats only part of the lifecycle. Where is the tooling to make the HA DB cluster? where is the tooling that securely and reliably does point in time recovery with a minute resolution for the last _n_ months? Where is the tooling to measure the health of the underlying FS, OS and hardware. What about scaling, what happens when you change the schema and shit gets slow, how long does it take to re-tool the scripts to make it work on new hardware? All of these will have to be built by your company, or bought in. Who tests them? how do you get support when shit goes wrong? For sure, it can be cheaper for the whole lifecycle, but often its not. Look I love real steel, I looked after a large cluster, but the prospect of making changes to the three-node tungsten, or creating a new one, gave me the fear.

by dx034 9y ago

zbentley 9y ago

Strongly disagree. Hand-waving the initial setup period away hides a substantial footer cost (it is quite hard to learn how to safely, properly install a large database stack, especially if you're unfamiliar with it), and a substantial ongoing cost (updates etc.) that is often eliminated or ameliorated with cloud-hosted options. And the additional work after setting it up is much greater. You have to monitor uptime/load/etc. regardless, sure, but when running your own hardware you also need to monitor uptime and load of other things as well: internet connectivity to your servers, electrical power, human access to servers/physical security and so on. Especially for a small site, those are quite expensive; their cost decreases a bit when you're running or leasing a lot of hardware, but it's still quite high.

quicksilver03 9y ago
If you have to learn how to perform a proper setup, you're a good candidate for the kind of managed services provided by a cloud-based solution. As long as you're aware that you're paying a sort of "ignorance tax" for not knowing how to do things, that's perfectly fine. While researching for https://www.thecloudhostinghandbook.com/ https://www.thecloudhostinghandbook.com/ I discovered that there are as many possible setups as there are companies: the best solution depends on what kind of assumptions the developers have materialized in the code, how much the company wants to spend, and ultimately on the world view of the people running the system.
stephenr 9y ago
Why do "cloud" advocates always jump to the extreme opposite: you manage everything yourself down to mowing the lawn in front of your own custom built DC. You can buy and colo or even rent dedicated hardware. I've never heard of a DC that didn't offer a remote hands service for hardware issues that requires a physical presence.
- toomuchtodo 9y ago
  If cloud advocates don’t jump to the extreme, the benefits of cloud computing quickly fall apart (there’s a reason AWS’ margins are ridiculously high).
- zbentley 9y ago
  I have managed my own DC and colo'd. Some examples of issues that occurred: - Colo: unauthorized personnel tripped over a cable, killing network to a lot of our servers. This was in a huge colo hosting facility that was home to some fortune 500 companies' servers. - Own datacenter: local internet providers both had a planned maintenance/outage on the same day. This isn't "mowing the lawn in front of your DC", this is the most basic utility after power that needs to be available for your DC to work. - Colo: hard drive failure after hours. Colo security staff wouldn't let us disassemble hardware that wasn't ours. Had to wait 6hrs for a technician to show up to pop and swap a drive (this with a 1hr incident response in our contract and the same "highly reliable" colo; we got money back per our contract, but not the customers we lost, even after apologetic/refunding communications to them). - Own datacenter, third-party NAS appliance professionally installed by a vendor: persistent performance issues and service dropouts, eventually traced to the "professional installation" being at the bottom of a rack with a large, detachable rear panel mostly covering the fan intakes for the appliance, leading to persistent throttling due to heat. - Colo (rented hardware, managed OS install): When the business started, our sysadmins/DBAs (me and one other guy) were primarily experienced with RHEL and old school init. The colo only provisioned Debian/systemd servers. We learned, but it slowed us down for a week or so. Sure, it was only an extra few minutes per task, but it added up. - Own datacenter: management needed us to move into a new server room because of an ending lease, with a hard deadline. The air conditioning installation vendor showed up two weeks late; we had no cooling at all when we needed to cut over, causing days of service interruptions and downtime. I have also managed cloud services. Some examples of issues that occurred: - Amazon's S3-gate. That sucked. When it happened, we were able to email our customers a copy of Amazon's status, and links to reporting of exactly how wide-spread the issue was. We had impact nearly identical to the colo hard-drive-failure incident above (it was the same service, migrated to AWS/S3), but we didn't lose nearly as many customers. - The DNS DDoS that affected the East Coast of the US earlier this year. Same kind of service interruption, same communication. We even got replies back from customers saying "I couldn't get to Reddit either; I figured you guys might be in the same boat". I'm not a cloud evangelist. I think that there are very good reasons to host entirely locally, or colocate/rent, or anything in between. I do, however, think that businesses, especially small ones that are not typical software-centric startups, massively and regularly underestimate both the initial and ongoing costs of running their own infrastructure. There are significant technical benefits to being cloud-hosted (these benefits also apply, only a little less potently, to fully managed hosting operations a la Rackspace), but people often miss the political and financial benefits. The political benefits are things like "our customers are less pissed because widespread issues with AWS probably affected them in other ways that day as well, so not as many people will knee-jerk blame us". The financial benefits are primarily that there are fewer moments of the "oh shit, setting these servers up ended up costing way more than we thought it would", and "we thought this would take a day to turn up; it actually took two weeks" varieties. Those things still happen with cloud services, but much less often.
dx034 9y ago
I was talking about dedicated servers, not running your data centre. Data centres are a scale business but cheap to run at reasonable scale. Dedicated servers are significantly cheaper than cloud instances and there's not much more to monitor than for the cloud. Compared to hosted DB clusters you need someone to set it up and run it but you'd also need an AWS expert for a production cluster in the cloud.

throwaway2016a 9y ago

I also disagree on this one. We manages our own cluster for years. Even had an expert in that database technology help us optimize the config. Then we switched to RDS... Sure it was a few hundred dollars more per month in AWS fees but we saved thousands a month in consulting fees.

dx034 9y ago
Why do you need consultants to run a database cluster? And do you run AWS database clusters in production without expert knowledge? If you need consultants for self-hosted databases you'd probably also need consultants to properly set up AWS services.
- throwaway2016a 9y ago
  Performance tuning, scaling, troubleshooting, etc. We had terabytes of active data (as opposed to warehoused data). For example, if a disk fills or is fails, adding a disk to a mySQL setup without causing downtime is a pretty involved process but in RDS it is one config change in the admin. It does the same process just AWS put a huge amount of work into automating it. We only brought in the consultant for an average of a day every other week but that adds up and we had to bring in 0 people after switching to RDS.
KaiserPro 9y ago
but you are already there. At present company if I want a new database, I have to make it from scratch, and I'm not a DBA. One size really doesn't fit all.

KaiserPro 9y ago

it _is_ cheaper to run. But thats only part of the lifecycle. Where is the tooling to make the HA DB cluster? where is the tooling that securely and reliably does point in time recovery with a minute resolution for the last _n_ months? Where is the tooling to measure the health of the underlying FS, OS and hardware. What about scaling, what happens when you change the schema and shit gets slow, how long does it take to re-tool the scripts to make it work on new hardware? All of these will have to be built by your company, or bought in. Who tests them? how do you get support when shit goes wrong? For sure, it can be cheaper for the whole lifecycle, but often its not. Look I love real steel, I looked after a large cluster, but the prospect of making changes to the three-node tungsten, or creating a new one, gave me the fear.