4 ms·

Yes, this exactly. The fact that I can guarantee that my code is running, and have entire massive teams at Amazon whose literal job title is to keep my code run

Yes, this exactly. The fact that I can guarantee that my code is running, and have entire massive teams at Amazon whose literal job title is to keep my code running, it is such a freeing feeling. I might be able to delete rows from Dynamo from my code messing up, but Dynamo will never go offline.

crehn 6y ago
The psychological effects of technology seem often overlooked—and they’re hard to quantify. I wish there was some easier way to quantify the “soft” aspects of everything that surrounds us, e.g. the long-term impact of beautiful and usable UI design, the reduction in “ambient” psychological stress, the impacts of chaos/consistency on how we feel, any robust measures of happiness, excitement, relaxation, etc. They seem to be such important attributes, but not as easily measured as a latency. Which in turn makes reaching consensus harder—and probably why design by committee doesn't work?
- spyspy 6y ago
  My joke around the office is some people optimize for latency, some for throughput, some for memory usage. I, however, optimize for sleep. My entire goal is for my services to never wake me up. And if they do it had better because I clearly messed up and am the sole human alive able to fix it.
  - ethanwillis 6y ago
    I optimize for escape hatches. Because if everything is on fire I want the ability to be able to bail(fix it). There's no point in waking up when everything is blowing up if you don't have the access to fix the underlying issue. AWS doesn't actually guarantee 99.9% uptime. What they guarantee is they'll give you some company currency[1] if they can't meet or exceed 99.9% uptime. So the next time the ship is on fire, don't worry, stay asleep and your Amazon account manager will be by shortly with a thimble of water to throw on you :) [1] - https://aws.amazon.com/about-aws/whats-new/2019/03/aws-systems-manager-announces-service-level-agreement/ https://aws.amazon.com/about-aws/whats-new/2019/03/aws-syste...
  - erikerikson 6y ago
    We optimize for good engineers
  - pm90 6y ago
    > I, however, optimize for sleep. Bravo. Having been on call for some truly monstrous amount of things and seen things break in all kinds of ways... I can't agree more with this principle. While the cloud isn't cheap, if your org can afford it, there is no question about it: go with managed services. Focus on building things, leave the OPS to AWS/GCP/Azure.
joana035 6y ago
Are you really sure?
- Supermancho 6y ago
  Tracking exactly what individual db entries ran and didn't run when there's one of these "serverless" "painless" interruptions to an ETL, ensures that you have an endless number of ever-changing status columns and re-run processes. Are you really sure that it all re-ran? No. You have to query your latest batch - which means you have to have arbitrary boundaries for when failures might have occurred like say, last 24 hours. If was a weekend, just query the whole database looking for any entry that's missing the "final" status success. Try not to run anything while you're doing this, unless your system is tolerant to the DB being locked for a full-table scan. This is all very fragile and change-averse. Who would choose to run an ETL like this? I have worked for large companies who all end up with this rickety system. Every one, every time. God forbid you want to run tests with a mock set of serverless AWS services.
  - justinram11 6y ago
    I generally agree that if you're setting up multi-step or "enterprise grade" ETL processes that it's a job better handled by something like Airflow (we've done a few smaller ones with Step Functions which hasn't been terrible, but Airflow is still better). Our use of AWS Lambda for ETL is usually just one-off small processes that don't have dependencies. As soon as you start getting into "This ETL depends on X, Y, and Z being run first" I think you're out of "small utility" territory.
  - mark242 6y ago
    Here's the beautiful thing about thinking serverless. "Are you really sure that it all re-ran?" Yes I am, because Dynamo streams or Kinesis streams guarantee message delivery. If my Lambda code has a bug in it, which yes sure that happens, I can as part of my exception catching, put that message into a dead letter queue and process it later or report on it. No polling, no querying, I know exactly what went wrong and when. Meanwhile, I don't have to do things like /make sure the ETL process is running/ or /clear out a bunch of logs when a disk inadvertently fills up/ or any of those other stupid last-century compute problems. It's just a far, far easier way of thinking about application development.
mmaunder 6y ago
This is so naive. Unless you’re just creating utilities all day long, at some point you’ll actually have a production service that people are using that needs to be monitored, even if it is build out of lambda functions. And unless you’ve run AWS vs colo at scale, you have no concept of the orders of magnitude cost savings on bandwidth (at 95th percentile billing vs per gig) and compute.
- crehn 6y ago
  Nobody’s saying serverless makes monitoring unnecessary. But a serverless architecture gets rid of an entire category of bureaucracy and issues that can arise from dedicated servers.
- mark242 6y ago
  I have run AWS vs colo at scale. For the cost of precisely 1 engineer fully loaded, I can have 500TB of bandwidth per month, every month. Guess what, it isn't going to take just 1 engineer to maintain your homegrown worldwide CDN. Does it make sense for FAANG to run their own datacenters? Yes of course. Does it make sense for you? No. This is what I don't get about arguments for running a bunch of VPSs-- for literally any environment that isn't thousands of engineers, it is more cost effective to let someone manage your underlying infrastructure for you, and any workload at that scale, you aren't contracting with some random bare metal hosting shop.
  - ethanwillis 6y ago
    Maybe it's hubris, but I'm 99% sure I could run a homegrown worldwide CDN for the cost of one engineer salary. I'd be willing to take that bet from anyone who thinks it can't be done.
    - calt 6y ago
      But are you including your own salary into that cost? I believe he is comparing aws bills vs. engineer time+other hosting bills.
mrmonkeyman 6y ago
Yet outages are everywhere and data leaks all over the place..
bufferoverflow 6y ago
> Dynamo will never go offline. That's some ridiculous propaganda. AWS has whole regions going down for hours.

crehn 6y ago

The psychological effects of technology seem often overlooked—and they’re hard to quantify. I wish there was some easier way to quantify the “soft” aspects of everything that surrounds us, e.g. the long-term impact of beautiful and usable UI design, the reduction in “ambient” psychological stress, the impacts of chaos/consistency on how we feel, any robust measures of happiness, excitement, relaxation, etc. They seem to be such important attributes, but not as easily measured as a latency. Which in turn makes reaching consensus harder—and probably why design by committee doesn't work?

spyspy 6y ago
My joke around the office is some people optimize for latency, some for throughput, some for memory usage. I, however, optimize for sleep. My entire goal is for my services to never wake me up. And if they do it had better because I clearly messed up and am the sole human alive able to fix it.
- ethanwillis 6y ago
  I optimize for escape hatches. Because if everything is on fire I want the ability to be able to bail(fix it). There's no point in waking up when everything is blowing up if you don't have the access to fix the underlying issue. AWS doesn't actually guarantee 99.9% uptime. What they guarantee is they'll give you some company currency[1] if they can't meet or exceed 99.9% uptime. So the next time the ship is on fire, don't worry, stay asleep and your Amazon account manager will be by shortly with a thimble of water to throw on you :) [1] - https://aws.amazon.com/about-aws/whats-new/2019/03/aws-systems-manager-announces-service-level-agreement/ https://aws.amazon.com/about-aws/whats-new/2019/03/aws-syste...
- erikerikson 6y ago
  We optimize for good engineers
- pm90 6y ago
  > I, however, optimize for sleep. Bravo. Having been on call for some truly monstrous amount of things and seen things break in all kinds of ways... I can't agree more with this principle. While the cloud isn't cheap, if your org can afford it, there is no question about it: go with managed services. Focus on building things, leave the OPS to AWS/GCP/Azure.

joana035 6y ago

Are you really sure?

Supermancho 6y ago
Tracking exactly what individual db entries ran and didn't run when there's one of these "serverless" "painless" interruptions to an ETL, ensures that you have an endless number of ever-changing status columns and re-run processes. Are you really sure that it all re-ran? No. You have to query your latest batch - which means you have to have arbitrary boundaries for when failures might have occurred like say, last 24 hours. If was a weekend, just query the whole database looking for any entry that's missing the "final" status success. Try not to run anything while you're doing this, unless your system is tolerant to the DB being locked for a full-table scan. This is all very fragile and change-averse. Who would choose to run an ETL like this? I have worked for large companies who all end up with this rickety system. Every one, every time. God forbid you want to run tests with a mock set of serverless AWS services.
- justinram11 6y ago
  I generally agree that if you're setting up multi-step or "enterprise grade" ETL processes that it's a job better handled by something like Airflow (we've done a few smaller ones with Step Functions which hasn't been terrible, but Airflow is still better). Our use of AWS Lambda for ETL is usually just one-off small processes that don't have dependencies. As soon as you start getting into "This ETL depends on X, Y, and Z being run first" I think you're out of "small utility" territory.
- mark242 6y ago
  Here's the beautiful thing about thinking serverless. "Are you really sure that it all re-ran?" Yes I am, because Dynamo streams or Kinesis streams guarantee message delivery. If my Lambda code has a bug in it, which yes sure that happens, I can as part of my exception catching, put that message into a dead letter queue and process it later or report on it. No polling, no querying, I know exactly what went wrong and when. Meanwhile, I don't have to do things like /make sure the ETL process is running/ or /clear out a bunch of logs when a disk inadvertently fills up/ or any of those other stupid last-century compute problems. It's just a far, far easier way of thinking about application development.

mmaunder 6y ago

This is so naive. Unless you’re just creating utilities all day long, at some point you’ll actually have a production service that people are using that needs to be monitored, even if it is build out of lambda functions. And unless you’ve run AWS vs colo at scale, you have no concept of the orders of magnitude cost savings on bandwidth (at 95th percentile billing vs per gig) and compute.

crehn 6y ago
Nobody’s saying serverless makes monitoring unnecessary. But a serverless architecture gets rid of an entire category of bureaucracy and issues that can arise from dedicated servers.
mark242 6y ago
I have run AWS vs colo at scale. For the cost of precisely 1 engineer fully loaded, I can have 500TB of bandwidth per month, every month. Guess what, it isn't going to take just 1 engineer to maintain your homegrown worldwide CDN. Does it make sense for FAANG to run their own datacenters? Yes of course. Does it make sense for you? No. This is what I don't get about arguments for running a bunch of VPSs-- for literally any environment that isn't thousands of engineers, it is more cost effective to let someone manage your underlying infrastructure for you, and any workload at that scale, you aren't contracting with some random bare metal hosting shop.
- ethanwillis 6y ago
  Maybe it's hubris, but I'm 99% sure I could run a homegrown worldwide CDN for the cost of one engineer salary. I'd be willing to take that bet from anyone who thinks it can't be done.
  - calt 6y ago
    But are you including your own salary into that cost? I believe he is comparing aws bills vs. engineer time+other hosting bills.

mrmonkeyman 6y ago

Yet outages are everywhere and data leaks all over the place..

bufferoverflow 6y ago

> Dynamo will never go offline. That's some ridiculous propaganda. AWS has whole regions going down for hours.