9 ms·

Amazon are pushing very hard to convince the next generation of devs who grew up writing front-end JS that databases, servers etc. are some kind of technical wi

by Jedi72 8y ago

Amazon are pushing very hard to convince the next generation of devs who grew up writing front-end JS that databases, servers etc. are some kind of technical wizardy best outsourced, when a few weeks of reading and playing around would be enough to get them up to speed.

scarface74 8y ago
It’s not about “getting up to speed”. It’s about not having to manage it on and ongoing basis. I wouldn’t work for a company that expects devs to manage resources that can be managed by a cloud provider and develop. How well can you “manage” a Mysql database with storage redundancy across three availability zones and synchronous autoscaling read replicas? How well can you manage an autoscaling database that costs basically nothing when you’re not using but scales to handle spikey traffic when you do?
- Jedi72 8y ago
  Did you read the article? You are not Google. If you ever do really need that kind of redundancy and scale you will have the team to support it, with all the benefits of doing it in-house. No uptime guarantee or web dashboard will ever substitute for simply having people who know what they're doing on staff. How a company which seems entirely driven by vertical integration is able to convince other companies that outsourcing is the way to go is an absolute mystery to me.
  - scarface74 8y ago
    No, we are not Google. We do need to be able to handle spiky loads - see the other reply. No we don’t “need a team” to support. Yes “the people in the know” are at AWS. They handle failover, autoscaling, etc. We also use Serverless Aurora/MySQL for non production environments with production like size of data. When we don’t need to access the database, we only pay for storage. When we do need it, it’s there.
- jacoblambda 8y ago
  The question is how often is that necessary? Once again the point goes back to the article title. You are not Google. Unless your product is actually large, you probably don't need all of that and even if you do, you can probably just do part of it in the cloud for significantly cheaper and get close to the same result. This obsession with making something completely bulletproof and scalable is the exact problem they are discussing. You probably don't need it in most cases but just want it. I am guilty of this as well and it is very difficult to avoid doing.
  - scarface74 8y ago
    You think only Google needs to protect against data loss? We have a process that reads a lot of data from the database on a periodic basis and sends it to ElasticSearch. We would either have to spend more and overprovision it to handle peak load or we can just turn on autoscaling for read replicas. Since the read replicas use the same storage as the reader/writer it’s much faster. Yes we need “bulletproof” and scalability or our clients we have six and seven figure contracts with won’t be happy and will be up in arms.
    - aprdm 8y ago
      > You think only Google needs to protect against data loss? Just have a cron job taking backups in a server and sending somewhere else? It has been working for the last 40 years...
      - scarface74 8y ago
        So a cron job can give me point in time recovery? Can a cron job give me automatic failover to another server that is in sync with the master? Can it give me autoscaling read replicas? Who is going to get up in the middle of the night when the cron job fails? Yes I am sure my company that has six and seven figure contracts would be just as well served self hosting MySQL on Linode.
        candiodari 8y ago
- mcny 8y ago
  I agree for production because you want to be able to blame someone when autoscaling fails but we trust developers to run applications locally, right? Then why can't we trust them with dev and staging? By the way what is autoscaling and why are we autoscaling databases? Im guessing the only resource thst autoscales is the bandwidth? Why can't we all get shared access to a fat pipe in production? I was under the impression that products like Google Cloud Spanner have this figured out. What exactly needs to auto scale? Isn't there just one database server in production? In dev (which is the use case I'm talking about) you should be able to just reset the vm whenever you want, no?
  - scarface74 8y ago
    No, it’s not just the bandwidth. It’s also the CPU for read and writes in the case of Serverless Aurora. For regular autoscaling of read replicas. It brings additional servers on line using the same storage array. It isn’t just “one database”. That’s just it. There isn’t “just one database”.
  - davidjnelson 8y ago
    Cloud spanner seems awesome but has a base price of $740/mo, doesn’t seem to autoscale, and doesn’t support a mysql or postgres interface.
  - davidjnelson 8y ago
    The big innovation of aurora is they decoupled the compute from the storage [1]. Both compute and storage need to auto scale, but compute is the really important one that is hardest. Aurora serverless scales the compute automatically by keeping warm pools of db capacity around [2]. This is great for spiky traffic without degraded performance. 1. https://www.allthingsdistributed.com/files/p1041-verbitski.pdf https://www.allthingsdistributed.com/files/p1041-verbitski.p... 2. https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.how-it-works.html https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...
- stephenr 8y ago
  Right.. so now your developers don't need to understand how to configure and tune open source directory services and RDBMS's and in-memory caches... they just need to understand how to configure and tune a cloud-provider's implementation of a directory service and RDBMS and in-memory cache..... ? If you think using a cloud "service" out-of-the-box will "just work" your scale is probably small enough that a single server with the equivalent package installed with the default settings is going to "just work" too.
  - scarface74 8y ago
    You did just read our use case didn’t you? Yes we could overprovision a single server with 5x the resources for the once a week indexing. We could also have 4 other servers running all of the time even when we weren’t demoing anything in our UAT environment. We could also not have any redundancy and separate out the reads and writes. No one said the developers didn’t need to understand how to do it. I said we didn’t have to worry about maintaining infrastructure and overprovisioning. We also have bulk processors that run messages at a trickle based on incoming instances during the day but at night and especially st the end of the week, we need 8 times the resources to meet our SLAs. Should we also overprovision that and run 8 servers all of the time?
    - nilkn 8y ago
      > Yes we could overprovision a single server with 5x the resources for the once a week indexing. You assume this is bad, but why? Your alternative seems to be locking yourself into expensive and highly proprietary vendor solutions that have their own learning curve and a wide variety of tradeoffs and complications. The point being made is that you are still worrying about maintaining infrastructure and overprovisioning, because you're now spending time specializing your system and perhaps entire company to a specific vendor's serverless solutions. To be clear, I don't have anything against cloud infrastructure really, but I do think some folks really don't seem to understand how powerful simple, battle-tested tools like PostgreSQL are. "Overprovisioning" may be way less of an issue than you imply (especially if you seriously only need 8 machines), and replication for PostgreSQL is a long solved problem.
felbane 8y ago
Very much this. For most use cases, the out-of-the-box configuration is fine until you hit ridiculous scale, and it's not really all that complicated to keep a service running if you take time to read the docs.
- rodolphoarruda 8y ago
  It just came to mind Jason Fried's Getting Real chapter titled "Scale Later". Page 44. "For example, we ran Basecamp on a single server for the first year. Because we went with such a simple setup, it only took a week to implement. We didn’t start with a cluster of 15 boxes or spend months worrying about scaling. Did we experience any problems? A few. But we also realized that most of the problems we feared, like a brief slowdown, really weren’t that big of a deal to customers. As long as you keep people in the loop, and are honest about the situation, they’ll understand."
noobermin 8y ago
Hate to be super paranoid, but isn't it rather convenient the top comment on this section expresses exactly this sentiment? If anything, it proves that this perspective is working, or at least, a huge host of devs online really are front-end JS'ers who have this opinion already.
rswail 8y ago
From an engineering/development sense, this is a good thing, because it means that devs are cheaper. Most devs can't even handle being a good client of things like databases. They barely comprehend what the underlying theories are behind SQL (eg sets etc). Just like early electricity, people ran their own generators. That got outsourced, so the standard "sparky" wouldn't have the faintest idea of the requirements of the generation side, only the demand side. The same is happening to programming.
- Jedi72 8y ago
  The difference is that in the case of programming, for the cost of some side-study you can get yourself a competitive advantage.
  - VeronikaMaier 8y ago
    I am thinking the same!
- mcguire 8y ago
  1. Programmers are supposed to be professionals. They certainly want to be paid like professionals. 2. The effects of the back end have a nasty habit of poking through in a way that the differences between windmill generators and pumped hydraulic storage don't.
ElFitz 8y ago
I've had servers with identical configurations, identical setup, identical programs installed the exact same way (via docker images), setup the same day One worked, one didn't ¯\_(ツ)_/¯ That's when I gave up on servers, virtual or not
- fredsir 8y ago
  It stands to reason that they weren't identical because if they were they would both have worked. The thing separating you (in this case) and Amazon/Google/* is that they make sure their processes produce predictable outcomes that if they were to fail can be shutdown and the process can be redone until it yields the correct outcome.
  - user5994461 8y ago
    Unless they were identical running in the same directory and listening to the same port. First one to start won.
    - fredsir 8y ago
      That’s a whole different problem in a whole other place. It’s not either of them not working.
mcguire 8y ago
I spent this last week finally being forced to deal with Hibernate. Devs are more than capable of pushing each other to the same conclusions.

by Jedi72 8y ago

scarface74 8y ago

It’s not about “getting up to speed”. It’s about not having to manage it on and ongoing basis. I wouldn’t work for a company that expects devs to manage resources that can be managed by a cloud provider and develop. How well can you “manage” a Mysql database with storage redundancy across three availability zones and synchronous autoscaling read replicas? How well can you manage an autoscaling database that costs basically nothing when you’re not using but scales to handle spikey traffic when you do?

Jedi72 8y ago
Did you read the article? You are not Google. If you ever do really need that kind of redundancy and scale you will have the team to support it, with all the benefits of doing it in-house. No uptime guarantee or web dashboard will ever substitute for simply having people who know what they're doing on staff. How a company which seems entirely driven by vertical integration is able to convince other companies that outsourcing is the way to go is an absolute mystery to me.
- scarface74 8y ago
  No, we are not Google. We do need to be able to handle spiky loads - see the other reply. No we don’t “need a team” to support. Yes “the people in the know” are at AWS. They handle failover, autoscaling, etc. We also use Serverless Aurora/MySQL for non production environments with production like size of data. When we don’t need to access the database, we only pay for storage. When we do need it, it’s there.
jacoblambda 8y ago
The question is how often is that necessary? Once again the point goes back to the article title. You are not Google. Unless your product is actually large, you probably don't need all of that and even if you do, you can probably just do part of it in the cloud for significantly cheaper and get close to the same result. This obsession with making something completely bulletproof and scalable is the exact problem they are discussing. You probably don't need it in most cases but just want it. I am guilty of this as well and it is very difficult to avoid doing.
- scarface74 8y ago
  You think only Google needs to protect against data loss? We have a process that reads a lot of data from the database on a periodic basis and sends it to ElasticSearch. We would either have to spend more and overprovision it to handle peak load or we can just turn on autoscaling for read replicas. Since the read replicas use the same storage as the reader/writer it’s much faster. Yes we need “bulletproof” and scalability or our clients we have six and seven figure contracts with won’t be happy and will be up in arms.
  - aprdm 8y ago
    > You think only Google needs to protect against data loss? Just have a cron job taking backups in a server and sending somewhere else? It has been working for the last 40 years...
    - scarface74 8y ago
      So a cron job can give me point in time recovery? Can a cron job give me automatic failover to another server that is in sync with the master? Can it give me autoscaling read replicas? Who is going to get up in the middle of the night when the cron job fails? Yes I am sure my company that has six and seven figure contracts would be just as well served self hosting MySQL on Linode.
      - candiodari 8y ago
mcny 8y ago
I agree for production because you want to be able to blame someone when autoscaling fails but we trust developers to run applications locally, right? Then why can't we trust them with dev and staging? By the way what is autoscaling and why are we autoscaling databases? Im guessing the only resource thst autoscales is the bandwidth? Why can't we all get shared access to a fat pipe in production? I was under the impression that products like Google Cloud Spanner have this figured out. What exactly needs to auto scale? Isn't there just one database server in production? In dev (which is the use case I'm talking about) you should be able to just reset the vm whenever you want, no?
- scarface74 8y ago
  No, it’s not just the bandwidth. It’s also the CPU for read and writes in the case of Serverless Aurora. For regular autoscaling of read replicas. It brings additional servers on line using the same storage array. It isn’t just “one database”. That’s just it. There isn’t “just one database”.
- davidjnelson 8y ago
  Cloud spanner seems awesome but has a base price of $740/mo, doesn’t seem to autoscale, and doesn’t support a mysql or postgres interface.
- davidjnelson 8y ago
  The big innovation of aurora is they decoupled the compute from the storage [1]. Both compute and storage need to auto scale, but compute is the really important one that is hardest. Aurora serverless scales the compute automatically by keeping warm pools of db capacity around [2]. This is great for spiky traffic without degraded performance. 1. https://www.allthingsdistributed.com/files/p1041-verbitski.pdf https://www.allthingsdistributed.com/files/p1041-verbitski.p... 2. https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.how-it-works.html https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...
stephenr 8y ago
Right.. so now your developers don't need to understand how to configure and tune open source directory services and RDBMS's and in-memory caches... they just need to understand how to configure and tune a cloud-provider's implementation of a directory service and RDBMS and in-memory cache..... ? If you think using a cloud "service" out-of-the-box will "just work" your scale is probably small enough that a single server with the equivalent package installed with the default settings is going to "just work" too.
- scarface74 8y ago
  You did just read our use case didn’t you? Yes we could overprovision a single server with 5x the resources for the once a week indexing. We could also have 4 other servers running all of the time even when we weren’t demoing anything in our UAT environment. We could also not have any redundancy and separate out the reads and writes. No one said the developers didn’t need to understand how to do it. I said we didn’t have to worry about maintaining infrastructure and overprovisioning. We also have bulk processors that run messages at a trickle based on incoming instances during the day but at night and especially st the end of the week, we need 8 times the resources to meet our SLAs. Should we also overprovision that and run 8 servers all of the time?
  - nilkn 8y ago
    > Yes we could overprovision a single server with 5x the resources for the once a week indexing. You assume this is bad, but why? Your alternative seems to be locking yourself into expensive and highly proprietary vendor solutions that have their own learning curve and a wide variety of tradeoffs and complications. The point being made is that you are still worrying about maintaining infrastructure and overprovisioning, because you're now spending time specializing your system and perhaps entire company to a specific vendor's serverless solutions. To be clear, I don't have anything against cloud infrastructure really, but I do think some folks really don't seem to understand how powerful simple, battle-tested tools like PostgreSQL are. "Overprovisioning" may be way less of an issue than you imply (especially if you seriously only need 8 machines), and replication for PostgreSQL is a long solved problem.

felbane 8y ago

Very much this. For most use cases, the out-of-the-box configuration is fine until you hit ridiculous scale, and it's not really all that complicated to keep a service running if you take time to read the docs.

rodolphoarruda 8y ago
It just came to mind Jason Fried's Getting Real chapter titled "Scale Later". Page 44. "For example, we ran Basecamp on a single server for the first year. Because we went with such a simple setup, it only took a week to implement. We didn’t start with a cluster of 15 boxes or spend months worrying about scaling. Did we experience any problems? A few. But we also realized that most of the problems we feared, like a brief slowdown, really weren’t that big of a deal to customers. As long as you keep people in the loop, and are honest about the situation, they’ll understand."

noobermin 8y ago

Hate to be super paranoid, but isn't it rather convenient the top comment on this section expresses exactly this sentiment? If anything, it proves that this perspective is working, or at least, a huge host of devs online really are front-end JS'ers who have this opinion already.

rswail 8y ago

From an engineering/development sense, this is a good thing, because it means that devs are cheaper. Most devs can't even handle being a good client of things like databases. They barely comprehend what the underlying theories are behind SQL (eg sets etc). Just like early electricity, people ran their own generators. That got outsourced, so the standard "sparky" wouldn't have the faintest idea of the requirements of the generation side, only the demand side. The same is happening to programming.

Jedi72 8y ago
The difference is that in the case of programming, for the cost of some side-study you can get yourself a competitive advantage.
- VeronikaMaier 8y ago
  I am thinking the same!
mcguire 8y ago
1. Programmers are supposed to be professionals. They certainly want to be paid like professionals. 2. The effects of the back end have a nasty habit of poking through in a way that the differences between windmill generators and pumped hydraulic storage don't.

ElFitz 8y ago

I've had servers with identical configurations, identical setup, identical programs installed the exact same way (via docker images), setup the same day One worked, one didn't ¯\_(ツ)_/¯ That's when I gave up on servers, virtual or not

fredsir 8y ago
It stands to reason that they weren't identical because if they were they would both have worked. The thing separating you (in this case) and Amazon/Google/* is that they make sure their processes produce predictable outcomes that if they were to fail can be shutdown and the process can be redone until it yields the correct outcome.
- user5994461 8y ago
  Unless they were identical running in the same directory and listening to the same port. First one to start won.
  - fredsir 8y ago
    That’s a whole different problem in a whole other place. It’s not either of them not working.

mcguire 8y ago

I spent this last week finally being forced to deal with Hibernate. Devs are more than capable of pushing each other to the same conclusions.