12 ms·

Because that "cheap dedicated server" isn't all that cheap. Just the documentation for backups on your Hetzner server is ridiculous. Why, in the 21st century, s

Because that "cheap dedicated server" isn't all that cheap. Just the documentation for backups on your Hetzner server is ridiculous. Why, in the 21st century, should I as an application developer ever have to worry about that? Let's say that Hetzner server goes completely toast at 2am and you get a pingdom message. You log in, realize you have to restore from backup. You fire up a new instance, and go through your restore procedure, and everything works great. You're lucky to be offline for an hour. I cannot even tell you what would happen if an underlying instance running a Lambda goes offline, because it never happens. But let's say that an underlying machine dies in us-east-1. Lambda just starts running more instances of your code on another underlying group of cpus, and you aren't offline.

crehn 6y ago
And I’d argue the mere possibility of the server breaking at any time contributes to non-negligible amounts of sustained psychological stress.
- mark242 6y ago
  Yes, this exactly. The fact that I can guarantee that my code is running, and have entire massive teams at Amazon whose literal job title is to keep my code running, it is such a freeing feeling. I might be able to delete rows from Dynamo from my code messing up, but Dynamo will never go offline.
  - crehn 6y ago
    The psychological effects of technology seem often overlooked—and they’re hard to quantify. I wish there was some easier way to quantify the “soft” aspects of everything that surrounds us, e.g. the long-term impact of beautiful and usable UI design, the reduction in “ambient” psychological stress, the impacts of chaos/consistency on how we feel, any robust measures of happiness, excitement, relaxation, etc. They seem to be such important attributes, but not as easily measured as a latency. Which in turn makes reaching consensus harder—and probably why design by committee doesn't work?
    - spyspy 6y ago
      My joke around the office is some people optimize for latency, some for throughput, some for memory usage. I, however, optimize for sleep. My entire goal is for my services to never wake me up. And if they do it had better because I clearly messed up and am the sole human alive able to fix it.
      - ethanwillis 6y ago
        I optimize for escape hatches. Because if everything is on fire I want the ability to be able to bail(fix it). There's no point in waking up when everything is blowing up if you don't have the access to fix the underlying issue. AWS doesn't actually guarantee 99.9% uptime. What they guarantee is they'll give you some company currency[1] if they can't meet or exceed 99.9% uptime. So the next time the ship is on fire, don't worry, stay asleep and your Amazon account manager will be by shortly with a thimble of water to throw on you :) [1] - https://aws.amazon.com/about-aws/whats-new/2019/03/aws-systems-manager-announces-service-level-agreement/ https://aws.amazon.com/about-aws/whats-new/2019/03/aws-syste...
      - erikerikson 6y ago
        We optimize for good engineers
      - pm90 6y ago
        > I, however, optimize for sleep. Bravo. Having been on call for some truly monstrous amount of things and seen things break in all kinds of ways... I can't agree more with this principle. While the cloud isn't cheap, if your org can afford it, there is no question about it: go with managed services. Focus on building things, leave the OPS to AWS/GCP/Azure.
  - joana035 6y ago
    Are you really sure?
    - Supermancho 6y ago
      Tracking exactly what individual db entries ran and didn't run when there's one of these "serverless" "painless" interruptions to an ETL, ensures that you have an endless number of ever-changing status columns and re-run processes. Are you really sure that it all re-ran? No. You have to query your latest batch - which means you have to have arbitrary boundaries for when failures might have occurred like say, last 24 hours. If was a weekend, just query the whole database looking for any entry that's missing the "final" status success. Try not to run anything while you're doing this, unless your system is tolerant to the DB being locked for a full-table scan. This is all very fragile and change-averse. Who would choose to run an ETL like this? I have worked for large companies who all end up with this rickety system. Every one, every time. God forbid you want to run tests with a mock set of serverless AWS services.
      - justinram11 6y ago
        I generally agree that if you're setting up multi-step or "enterprise grade" ETL processes that it's a job better handled by something like Airflow (we've done a few smaller ones with Step Functions which hasn't been terrible, but Airflow is still better). Our use of AWS Lambda for ETL is usually just one-off small processes that don't have dependencies. As soon as you start getting into "This ETL depends on X, Y, and Z being run first" I think you're out of "small utility" territory.
      - mark242 6y ago
        Here's the beautiful thing about thinking serverless. "Are you really sure that it all re-ran?" Yes I am, because Dynamo streams or Kinesis streams guarantee message delivery. If my Lambda code has a bug in it, which yes sure that happens, I can as part of my exception catching, put that message into a dead letter queue and process it later or report on it. No polling, no querying, I know exactly what went wrong and when. Meanwhile, I don't have to do things like /make sure the ETL process is running/ or /clear out a bunch of logs when a disk inadvertently fills up/ or any of those other stupid last-century compute problems. It's just a far, far easier way of thinking about application development.
  - mmaunder 6y ago
    This is so naive. Unless you’re just creating utilities all day long, at some point you’ll actually have a production service that people are using that needs to be monitored, even if it is build out of lambda functions. And unless you’ve run AWS vs colo at scale, you have no concept of the orders of magnitude cost savings on bandwidth (at 95th percentile billing vs per gig) and compute.
    - crehn 6y ago
      Nobody’s saying serverless makes monitoring unnecessary. But a serverless architecture gets rid of an entire category of bureaucracy and issues that can arise from dedicated servers.
    - mark242 6y ago
      I have run AWS vs colo at scale. For the cost of precisely 1 engineer fully loaded, I can have 500TB of bandwidth per month, every month. Guess what, it isn't going to take just 1 engineer to maintain your homegrown worldwide CDN. Does it make sense for FAANG to run their own datacenters? Yes of course. Does it make sense for you? No. This is what I don't get about arguments for running a bunch of VPSs-- for literally any environment that isn't thousands of engineers, it is more cost effective to let someone manage your underlying infrastructure for you, and any workload at that scale, you aren't contracting with some random bare metal hosting shop.
      - ethanwillis 6y ago
        Maybe it's hubris, but I'm 99% sure I could run a homegrown worldwide CDN for the cost of one engineer salary. I'd be willing to take that bet from anyone who thinks it can't be done.
        calt 6y ago
        But are you including your own salary into that cost? I believe he is comparing aws bills vs. engineer time+other hosting bills.
  - mrmonkeyman 6y ago
    Yet outages are everywhere and data leaks all over the place..
  - bufferoverflow 6y ago
    > Dynamo will never go offline. That's some ridiculous propaganda. AWS has whole regions going down for hours.
- kranner 6y ago
  What about the psychological stress of a massive AWS bill because of a misconfiguration by you or someone on your team?
  - crehn 6y ago
    It’s acute.
  - SiVal 6y ago
    A very good point. Exchange the risk of a problem taking you offline, losing sales, for the risk of a problem not taking you offline, costing you a fortune. That's a big selling point for DigitalOcean: you sleep easy knowing what your bill will be. If you have an unexpected spike in traffic--whether a great opportunity, a mistaken test run wild, or a DDoS--it doesn't increase your bill. AWS offers the opposite: no matter what happens, we'll keep you online and just send you the bill.
    - t0astbread 6y ago
      A middle ground between both would be ideal. Does AWS offer settings to cap your spendings? (I googled and found AWS Budgets but it seems to be only an alerting system, not a cap.)
      - nurettin 6y ago
        AFAIK at the moment there is no cap. It's pretty much like going long on a crashing market. However, you can later have a bunch of emails and tweets basically begging to be reimbursed and it works most of the time.
        vanviegen 6y ago
        Going long will limit your losses to the amount invested. The point is that your AWS bill has no such limit whatsoever.
        6y ago
- brylie 6y ago
  My psychological stress is knowing our company is hemorrhaging 50k €/month on AWS fees (mainly Lambda, step functions, and CloudWatch) to run a production app that could easily run on a couple of VMs (for failover) and a managed DB instance. Edit: add RDS to the parenthetical list of AWS services above.
tomc1985 6y ago
> Just the documentation for backups on your Hetzner server is ridiculous. Nothing a server template, code repo, and daily db dump cronjob can't solve
- crehn 6y ago
  I’m assuming this is a joke, but this is exactly what most people who just want to run some code don’t want to deal with.
  - tomc1985 6y ago
    Then maybe they should learn how to deal with it
    - nostrebored 6y ago
      Learning how to deal with it is great, until you find out what you don't know. Dealing with server infrastructure isn't business differentiating. Doing it poorly can certainly be business terminating, though.
      - jlokier 6y ago
        > Dealing with server infrastructure isn't business differentiating I agree, for most businesses no it isn't. However, saving on cloud hosting costs to replace them with a handful of dedicated servers in different data centres can make a large difference to costs, depending on what the cloud usage is. For a funded rocketship startup with lots of free AWS credits it makes sense to use AWS; for a steady state or long-term low income business, not so much. Those costs can make all the difference to a business if the cloud rental is creeping up to look comparable with salaries. I've used a combination of dedicated and cloud hosting for a long time. In general the dedicated systems are so much cheaper for the same amount of bandwidth and compute, and you can run VMs on them with great performance, so you can run everything interesting inside VMs and a lot of the arguments around upgrade downtime have completely gone away. With some cloud to assist, you can even use cloud temporarily to cover the downtime for major upgrades such as replacing hosts or reconfiguring the cluster network. 99.9% uptime with excellent bandwidth is easily achievable this way, and you can completely avoid user-facing downtime due to planned upgrades, with a bit of planning around DNS and IP transitions, VM migration, overlay networks, things like that. Sometimes at much lower opex cost (100x) than the modern cloud-recommended equivalents, although it takes knowledge and time to ensure it. If you have enough redundancy and occasional infrastructure stress testing, a similar level of "optimise for sleep" is possible as with cloud deployments. If you care about this (not every business does), it does take some work and knowledge to produce and verify hot redundancy, and of course there are costs. Things will go wrong, but they'll tend to be at the level of things running inside VMs and containers, rather than hosts failing. The same things which go wrong with cloud deployments anyway. The cloud can be used alongside dedicated to provide some of that redundancy and extra scaling. Cloud is rented in small units of time, so you can use cloud as "free most of the time" cold-backup service, and if there's a burst in traffic beyond what the dedicated systems can provide, or if you want to temporarily improve geographic locality for a service. It might seem wasteful to rent larger dedicated services rather than cloud-as-needed, but the cost differential has changed in the last 10 years to favour the former more than it used to.
deleted 6y ago
[deleted]
ashtonkem 6y ago
So get a cheap EC2 machine instead. There are reasonable steps between “buy your own hardware” and “everything is a Lambda”.
zaarn 6y ago
That's why you have three hetzner servers, so if one goes toast, you still have a redundant operation as you can easily failover while you recover the other host.
finnthehuman 6y ago
>Just the documentation for backups on your Hetzner server is ridiculous. Why, in the 21st century, should I as an application developer ever have to worry about that? I dunno, that's a problem for the ops team. It was in the 20th century too. The only reason you would have to worry about it is if you're pulling double duty as under-trained ops. If that were the case the right answer to every "build or buy" question is "buy" because you don't have anyone with the domain knowledge to bring more architectural options to the table than 'half-ass server management with internet tutorials.'
1337shadow 6y ago
> Why, in the 21st century, should I as an application developer ever have to worry about that? Then don't have backups at all, let's see how that goes.

crehn 6y ago

And I’d argue the mere possibility of the server breaking at any time contributes to non-negligible amounts of sustained psychological stress.

mark242 6y ago
Yes, this exactly. The fact that I can guarantee that my code is running, and have entire massive teams at Amazon whose literal job title is to keep my code running, it is such a freeing feeling. I might be able to delete rows from Dynamo from my code messing up, but Dynamo will never go offline.
- crehn 6y ago
  The psychological effects of technology seem often overlooked—and they’re hard to quantify. I wish there was some easier way to quantify the “soft” aspects of everything that surrounds us, e.g. the long-term impact of beautiful and usable UI design, the reduction in “ambient” psychological stress, the impacts of chaos/consistency on how we feel, any robust measures of happiness, excitement, relaxation, etc. They seem to be such important attributes, but not as easily measured as a latency. Which in turn makes reaching consensus harder—and probably why design by committee doesn't work?
  - spyspy 6y ago
    My joke around the office is some people optimize for latency, some for throughput, some for memory usage. I, however, optimize for sleep. My entire goal is for my services to never wake me up. And if they do it had better because I clearly messed up and am the sole human alive able to fix it.
    - ethanwillis 6y ago
      I optimize for escape hatches. Because if everything is on fire I want the ability to be able to bail(fix it). There's no point in waking up when everything is blowing up if you don't have the access to fix the underlying issue. AWS doesn't actually guarantee 99.9% uptime. What they guarantee is they'll give you some company currency[1] if they can't meet or exceed 99.9% uptime. So the next time the ship is on fire, don't worry, stay asleep and your Amazon account manager will be by shortly with a thimble of water to throw on you :) [1] - https://aws.amazon.com/about-aws/whats-new/2019/03/aws-systems-manager-announces-service-level-agreement/ https://aws.amazon.com/about-aws/whats-new/2019/03/aws-syste...
    - erikerikson 6y ago
      We optimize for good engineers
    - pm90 6y ago
      > I, however, optimize for sleep. Bravo. Having been on call for some truly monstrous amount of things and seen things break in all kinds of ways... I can't agree more with this principle. While the cloud isn't cheap, if your org can afford it, there is no question about it: go with managed services. Focus on building things, leave the OPS to AWS/GCP/Azure.
- joana035 6y ago
  Are you really sure?
  - Supermancho 6y ago
    Tracking exactly what individual db entries ran and didn't run when there's one of these "serverless" "painless" interruptions to an ETL, ensures that you have an endless number of ever-changing status columns and re-run processes. Are you really sure that it all re-ran? No. You have to query your latest batch - which means you have to have arbitrary boundaries for when failures might have occurred like say, last 24 hours. If was a weekend, just query the whole database looking for any entry that's missing the "final" status success. Try not to run anything while you're doing this, unless your system is tolerant to the DB being locked for a full-table scan. This is all very fragile and change-averse. Who would choose to run an ETL like this? I have worked for large companies who all end up with this rickety system. Every one, every time. God forbid you want to run tests with a mock set of serverless AWS services.
    - justinram11 6y ago
      I generally agree that if you're setting up multi-step or "enterprise grade" ETL processes that it's a job better handled by something like Airflow (we've done a few smaller ones with Step Functions which hasn't been terrible, but Airflow is still better). Our use of AWS Lambda for ETL is usually just one-off small processes that don't have dependencies. As soon as you start getting into "This ETL depends on X, Y, and Z being run first" I think you're out of "small utility" territory.
    - mark242 6y ago
      Here's the beautiful thing about thinking serverless. "Are you really sure that it all re-ran?" Yes I am, because Dynamo streams or Kinesis streams guarantee message delivery. If my Lambda code has a bug in it, which yes sure that happens, I can as part of my exception catching, put that message into a dead letter queue and process it later or report on it. No polling, no querying, I know exactly what went wrong and when. Meanwhile, I don't have to do things like /make sure the ETL process is running/ or /clear out a bunch of logs when a disk inadvertently fills up/ or any of those other stupid last-century compute problems. It's just a far, far easier way of thinking about application development.
- mmaunder 6y ago
  This is so naive. Unless you’re just creating utilities all day long, at some point you’ll actually have a production service that people are using that needs to be monitored, even if it is build out of lambda functions. And unless you’ve run AWS vs colo at scale, you have no concept of the orders of magnitude cost savings on bandwidth (at 95th percentile billing vs per gig) and compute.
  - crehn 6y ago
    Nobody’s saying serverless makes monitoring unnecessary. But a serverless architecture gets rid of an entire category of bureaucracy and issues that can arise from dedicated servers.
  - mark242 6y ago
    I have run AWS vs colo at scale. For the cost of precisely 1 engineer fully loaded, I can have 500TB of bandwidth per month, every month. Guess what, it isn't going to take just 1 engineer to maintain your homegrown worldwide CDN. Does it make sense for FAANG to run their own datacenters? Yes of course. Does it make sense for you? No. This is what I don't get about arguments for running a bunch of VPSs-- for literally any environment that isn't thousands of engineers, it is more cost effective to let someone manage your underlying infrastructure for you, and any workload at that scale, you aren't contracting with some random bare metal hosting shop.
    - ethanwillis 6y ago
      Maybe it's hubris, but I'm 99% sure I could run a homegrown worldwide CDN for the cost of one engineer salary. I'd be willing to take that bet from anyone who thinks it can't be done.
      - calt 6y ago
        But are you including your own salary into that cost? I believe he is comparing aws bills vs. engineer time+other hosting bills.
- mrmonkeyman 6y ago
  Yet outages are everywhere and data leaks all over the place..
- bufferoverflow 6y ago
  > Dynamo will never go offline. That's some ridiculous propaganda. AWS has whole regions going down for hours.
kranner 6y ago
What about the psychological stress of a massive AWS bill because of a misconfiguration by you or someone on your team?
- crehn 6y ago
  It’s acute.
- SiVal 6y ago
  A very good point. Exchange the risk of a problem taking you offline, losing sales, for the risk of a problem not taking you offline, costing you a fortune. That's a big selling point for DigitalOcean: you sleep easy knowing what your bill will be. If you have an unexpected spike in traffic--whether a great opportunity, a mistaken test run wild, or a DDoS--it doesn't increase your bill. AWS offers the opposite: no matter what happens, we'll keep you online and just send you the bill.
  - t0astbread 6y ago
    A middle ground between both would be ideal. Does AWS offer settings to cap your spendings? (I googled and found AWS Budgets but it seems to be only an alerting system, not a cap.)
    - nurettin 6y ago
      AFAIK at the moment there is no cap. It's pretty much like going long on a crashing market. However, you can later have a bunch of emails and tweets basically begging to be reimbursed and it works most of the time.
      - vanviegen 6y ago
        Going long will limit your losses to the amount invested. The point is that your AWS bill has no such limit whatsoever.
        6y ago
brylie 6y ago
My psychological stress is knowing our company is hemorrhaging 50k €/month on AWS fees (mainly Lambda, step functions, and CloudWatch) to run a production app that could easily run on a couple of VMs (for failover) and a managed DB instance. Edit: add RDS to the parenthetical list of AWS services above.

tomc1985 6y ago

> Just the documentation for backups on your Hetzner server is ridiculous. Nothing a server template, code repo, and daily db dump cronjob can't solve

crehn 6y ago
I’m assuming this is a joke, but this is exactly what most people who just want to run some code don’t want to deal with.
- tomc1985 6y ago
  Then maybe they should learn how to deal with it
  - nostrebored 6y ago
    Learning how to deal with it is great, until you find out what you don't know. Dealing with server infrastructure isn't business differentiating. Doing it poorly can certainly be business terminating, though.
    - jlokier 6y ago
      > Dealing with server infrastructure isn't business differentiating I agree, for most businesses no it isn't. However, saving on cloud hosting costs to replace them with a handful of dedicated servers in different data centres can make a large difference to costs, depending on what the cloud usage is. For a funded rocketship startup with lots of free AWS credits it makes sense to use AWS; for a steady state or long-term low income business, not so much. Those costs can make all the difference to a business if the cloud rental is creeping up to look comparable with salaries. I've used a combination of dedicated and cloud hosting for a long time. In general the dedicated systems are so much cheaper for the same amount of bandwidth and compute, and you can run VMs on them with great performance, so you can run everything interesting inside VMs and a lot of the arguments around upgrade downtime have completely gone away. With some cloud to assist, you can even use cloud temporarily to cover the downtime for major upgrades such as replacing hosts or reconfiguring the cluster network. 99.9% uptime with excellent bandwidth is easily achievable this way, and you can completely avoid user-facing downtime due to planned upgrades, with a bit of planning around DNS and IP transitions, VM migration, overlay networks, things like that. Sometimes at much lower opex cost (100x) than the modern cloud-recommended equivalents, although it takes knowledge and time to ensure it. If you have enough redundancy and occasional infrastructure stress testing, a similar level of "optimise for sleep" is possible as with cloud deployments. If you care about this (not every business does), it does take some work and knowledge to produce and verify hot redundancy, and of course there are costs. Things will go wrong, but they'll tend to be at the level of things running inside VMs and containers, rather than hosts failing. The same things which go wrong with cloud deployments anyway. The cloud can be used alongside dedicated to provide some of that redundancy and extra scaling. Cloud is rented in small units of time, so you can use cloud as "free most of the time" cold-backup service, and if there's a burst in traffic beyond what the dedicated systems can provide, or if you want to temporarily improve geographic locality for a service. It might seem wasteful to rent larger dedicated services rather than cloud-as-needed, but the cost differential has changed in the last 10 years to favour the former more than it used to.

deleted 6y ago

[deleted]

ashtonkem 6y ago

So get a cheap EC2 machine instead. There are reasonable steps between “buy your own hardware” and “everything is a Lambda”.

zaarn 6y ago

That's why you have three hetzner servers, so if one goes toast, you still have a redundant operation as you can easily failover while you recover the other host.

finnthehuman 6y ago

>Just the documentation for backups on your Hetzner server is ridiculous. Why, in the 21st century, should I as an application developer ever have to worry about that? I dunno, that's a problem for the ops team. It was in the 20th century too. The only reason you would have to worry about it is if you're pulling double duty as under-trained ops. If that were the case the right answer to every "build or buy" question is "buy" because you don't have anyone with the domain knowledge to bring more architectural options to the table than 'half-ass server management with internet tutorials.'

1337shadow 6y ago

> Why, in the 21st century, should I as an application developer ever have to worry about that? Then don't have backups at all, let's see how that goes.