5 ms·

A few people in the comments are saying 'isn't better to just setup your own SQL server instead of RDS?' and similar. I don't want to post a reply to each, so I

by gedrap 11y ago

A few people in the comments are saying 'isn't better to just setup your own SQL server instead of RDS?' and similar. I don't want to post a reply to each, so I will say it here. While I can totally sympathize from a programmer point of view (setting up, tweaking stuff and all is a great fun), but you need to ask yourself whether it is in the interests of business to do so. Especially if you're working in a small team with no dedicated infrastructure staff or a startup with a short runway and a lot of urgent user facing changes. Doing something on your own (e.g. setting up your own alternative to S3, or configuring your own SQL servers), comes with a cost and it's not only the programming/initial setup time. It's also opportunity cost (instead of setting up a server, I could, for example, analyze some user data); maintenance (more things to worry about which you can outsource); skills set required to run the infrastructure (running your own SQL cluster requires more knowledge, more training than running one on RDS), etc. So is it in the interest of the business to run your own infrastructure? If you have thousands of servers and spending millions on it - probably, but then probably you can make an attractive deal with GCE or AWS :) If your application needs some complex performance related stuff which is harder to do in the cloud (e.g. some custom hardware or whatever), then again, running your own infrastructure might be better. But if you are like the majority of the companies/products (you just need infrastructure to run reliably and performance should be just good enough), using AWS and friends might make a big difference.

latch 11y ago
If you come to the conclusion that the opportunity cost is too high for your team/company, fine...I can believe it....as long as you're also weighing the benefit of learning. All learning has an opportunity cost. I do believe that someone who can setup and configure nginx to do load balancing, caching, rate limiting and build middlewares in LUA is potentially going to be a more productive full stack developer than someone who can't. As a programmer, having managed postgresql, has made me a more effective programmer. I have a good understanding of how it vacuum and collects statistics, the relationship between connections and sorts and work_mem, so I'm better equipped at writing queries and troubleshooting issues. The gain to me personally and to my employer (and future employers) is not trivial.
- gedrap 11y ago
  Great point and I totally agree with you. Treating everything like a black box that just shove money into isn't going to take you far either :)
- rashkov 11y ago
  This reminds me of recent discussions about how countries that outsource their manufacturing quickly lose knowledge of manufacturing technology and fall behind in innovation and self-reliance. I don't think we are there yet, but it is conceivable that in the future system administration could become a lost art to many. Something to consider as more of our infrastructure needs are met by the cloud. I'm sure I'm overstating this, but I figured I'd share anyway
- deleted 11y ago
  [deleted]
- pjlegato 11y ago
  It's true that you gain some technical knowledge by learning all that. The question is whether that is worth the opportunity cost of not learning other things or doing something else during that time. Every minute you spend studying nginx config files or adjusting work_mem is a minute you are not spending working on your actual product or app. To some extent it is indeed beneficial to know how all that stuff works. But at some point your time is better spent working on things that differentiate your specific company from others, rather than spending it on twiddling the same set of knobs that everyone else has.
tn13 11y ago
I agree to that with respect to many AWS services more especially s3. However I find it hard to believe about RDS. I have spent thousands for dollars on AWS bills and never ever used RDS. Always setup my own db servers.
cookiecaper 11y ago
You still have to tune the configuration of RDS instances, and they can make some administrative tasks more difficult because they don't grant SUPER and/or don't allow system-level access. For example, you can't use innobackupex/xtrabackup on an RDS instance. I appreciate that it is often worthwhile to just pay someone else to handle things for your company, but I think people are too quick to flip the switch one way or the other; they either run everything "in the cloud" or nothing there. The truth is that as with most things, there is a happy medium tailored to each company's specific circumstances. AWS is very expensive. People underestimate how expensive it is. When we switched to AWS, our monthly bill was about 70% the total cost to buy (not rent, buy) all the machines in our old bare-metal datacenter, which were still perfectly performant (we switched because the execs wanted to be super-duper cool cloud users, not because of any real technical limitation, though there are pros and cons to it). I have to think this is fairly common. I hear a popular trick to net a big savings in your first year as CFO is to just force the tech guys to cut out Amazon and replace it with a cheaper cloud provider or colocated bare-metal.
- sokoloff 11y ago
  The financial picture of AWS only makes sense if you are using it to reduce (either in absolute terms or in growth rate) your spend on IT staffing costs. If you're paying 100% of your old IT staffing costs and paying the AWS bill, you're double paying part of it, IMO.
  - cookiecaper 11y ago
    Sure, that's an argument that I'm sure crosses the threshold sometimes. However, with the amount of money we're paying AWS, we could've easily brought on several new full-time, dedicated guys to handle datacenter, hardware, colo, and sysadmin stuff and still come out ahead. AWS is very expensive. People underestimate how expensive it is. I don't preclude the possibility that it's cost-effective for someone out there, but I seriously doubt that's the case for most of its users.
    - pjlegato 11y ago
nunez 11y ago
The biggest advantage that makes RDS and friends (DynamoDB, Redshift) so extremely attractive is that it (almost) completely offloads operating system and database server responsibilities onto Amazon. While there are some workloads that legitimately require physical hardware to execute on, there are a LOT of costs associated with that decision (networking, storage, power, hosting, licensing, etc.) that people tend to forget in cost comparisons. I've worked with hardware for several years and love the challenges that come with it, but in most cases, the cost of debugging those problems and working with vendors to get fixes/replacements down isn't worth it. It's a HUGE time suck. I've been working with AWS for the last few months, and being able to safely dispose of a bad instance and spin up whatever underlying services that ran on it on something else is amazing. If I were in charge of building infrastructure at an early-stage startup (or even one that's at a later-stage), I would absolutely 100% start with AWS and really squeeze every bit of performance out of the code for their products before thinking about going physical.
aftbit 11y ago
This can definitely happen in some cases, but in plenty of other cases, moving off the cloud can give you an enormous cost savings _and_ dramatically improve your performance at the same time. A coworker of mine was employee ~10 at a startup with some basic photogrammetry workloads. They ran jobs over ~0.1 to 1GB of images (lasting anywhere from minutes to hours) and operated primarily on Digital Ocean. They also had VPSes at AWS and Azure. My coworker helped them move this to LXC running on top of cheap last-gen dedicated servers. He built the entire (pretty simple) job management stack in about two weeks. As a result, the company was able to scale to ~100x the previous number of jobs at roughly the same cost. The improvement was so dramatic that they were able to offer a free tier for their services - a business-model change enabled by cheap physical hardware. In my own experience, I recently moved a production Postgres database off of Compose, who was charging us something like $400 a month for 2GB of RAM and 20GB of storage. I moved it to a hot-swap pair running in Rackspace on boxes that cost $2000 /month for the pair. That's 5x the cost. However, they each have 64GB of RAM and an 800GB SSD. The migration took me about two days total, including learning how to set up the hot-swap, and we have had very few operational issues with the database (knock on wood). This decreased our mean page load times by an order of magnitude. In some cases, dedicated hardware can really be worth the time cost.

by gedrap 11y ago

latch 11y ago

If you come to the conclusion that the opportunity cost is too high for your team/company, fine...I can believe it....as long as you're also weighing the benefit of learning. All learning has an opportunity cost. I do believe that someone who can setup and configure nginx to do load balancing, caching, rate limiting and build middlewares in LUA is potentially going to be a more productive full stack developer than someone who can't. As a programmer, having managed postgresql, has made me a more effective programmer. I have a good understanding of how it vacuum and collects statistics, the relationship between connections and sorts and work_mem, so I'm better equipped at writing queries and troubleshooting issues. The gain to me personally and to my employer (and future employers) is not trivial.

gedrap 11y ago
Great point and I totally agree with you. Treating everything like a black box that just shove money into isn't going to take you far either :)
rashkov 11y ago
This reminds me of recent discussions about how countries that outsource their manufacturing quickly lose knowledge of manufacturing technology and fall behind in innovation and self-reliance. I don't think we are there yet, but it is conceivable that in the future system administration could become a lost art to many. Something to consider as more of our infrastructure needs are met by the cloud. I'm sure I'm overstating this, but I figured I'd share anyway
deleted 11y ago
[deleted]
pjlegato 11y ago
It's true that you gain some technical knowledge by learning all that. The question is whether that is worth the opportunity cost of not learning other things or doing something else during that time. Every minute you spend studying nginx config files or adjusting work_mem is a minute you are not spending working on your actual product or app. To some extent it is indeed beneficial to know how all that stuff works. But at some point your time is better spent working on things that differentiate your specific company from others, rather than spending it on twiddling the same set of knobs that everyone else has.

tn13 11y ago

I agree to that with respect to many AWS services more especially s3. However I find it hard to believe about RDS. I have spent thousands for dollars on AWS bills and never ever used RDS. Always setup my own db servers.

cookiecaper 11y ago

You still have to tune the configuration of RDS instances, and they can make some administrative tasks more difficult because they don't grant SUPER and/or don't allow system-level access. For example, you can't use innobackupex/xtrabackup on an RDS instance. I appreciate that it is often worthwhile to just pay someone else to handle things for your company, but I think people are too quick to flip the switch one way or the other; they either run everything "in the cloud" or nothing there. The truth is that as with most things, there is a happy medium tailored to each company's specific circumstances. AWS is very expensive. People underestimate how expensive it is. When we switched to AWS, our monthly bill was about 70% the total cost to buy (not rent, buy) all the machines in our old bare-metal datacenter, which were still perfectly performant (we switched because the execs wanted to be super-duper cool cloud users, not because of any real technical limitation, though there are pros and cons to it). I have to think this is fairly common. I hear a popular trick to net a big savings in your first year as CFO is to just force the tech guys to cut out Amazon and replace it with a cheaper cloud provider or colocated bare-metal.

sokoloff 11y ago
The financial picture of AWS only makes sense if you are using it to reduce (either in absolute terms or in growth rate) your spend on IT staffing costs. If you're paying 100% of your old IT staffing costs and paying the AWS bill, you're double paying part of it, IMO.
- cookiecaper 11y ago
  Sure, that's an argument that I'm sure crosses the threshold sometimes. However, with the amount of money we're paying AWS, we could've easily brought on several new full-time, dedicated guys to handle datacenter, hardware, colo, and sysadmin stuff and still come out ahead. AWS is very expensive. People underestimate how expensive it is. I don't preclude the possibility that it's cost-effective for someone out there, but I seriously doubt that's the case for most of its users.
  - pjlegato 11y ago

nunez 11y ago

The biggest advantage that makes RDS and friends (DynamoDB, Redshift) so extremely attractive is that it (almost) completely offloads operating system and database server responsibilities onto Amazon. While there are some workloads that legitimately require physical hardware to execute on, there are a LOT of costs associated with that decision (networking, storage, power, hosting, licensing, etc.) that people tend to forget in cost comparisons. I've worked with hardware for several years and love the challenges that come with it, but in most cases, the cost of debugging those problems and working with vendors to get fixes/replacements down isn't worth it. It's a HUGE time suck. I've been working with AWS for the last few months, and being able to safely dispose of a bad instance and spin up whatever underlying services that ran on it on something else is amazing. If I were in charge of building infrastructure at an early-stage startup (or even one that's at a later-stage), I would absolutely 100% start with AWS and really squeeze every bit of performance out of the code for their products before thinking about going physical.

aftbit 11y ago

This can definitely happen in some cases, but in plenty of other cases, moving off the cloud can give you an enormous cost savings _and_ dramatically improve your performance at the same time. A coworker of mine was employee ~10 at a startup with some basic photogrammetry workloads. They ran jobs over ~0.1 to 1GB of images (lasting anywhere from minutes to hours) and operated primarily on Digital Ocean. They also had VPSes at AWS and Azure. My coworker helped them move this to LXC running on top of cheap last-gen dedicated servers. He built the entire (pretty simple) job management stack in about two weeks. As a result, the company was able to scale to ~100x the previous number of jobs at roughly the same cost. The improvement was so dramatic that they were able to offer a free tier for their services - a business-model change enabled by cheap physical hardware. In my own experience, I recently moved a production Postgres database off of Compose, who was charging us something like $400 a month for 2GB of RAM and 20GB of storage. I moved it to a hot-swap pair running in Rackspace on boxes that cost $2000 /month for the pair. That's 5x the cost. However, they each have 64GB of RAM and an 800GB SSD. The migration took me about two days total, including learning how to set up the hot-swap, and we have had very few operational issues with the database (knock on wood). This decreased our mean page load times by an order of magnitude. In some cases, dedicated hardware can really be worth the time cost.