8 ms·

Actually your comment is a great demonstration of poor engineering leadership. You've already decided on an outcome and are shifting the narrative to adhere to

Actually your comment is a great demonstration of poor engineering leadership. You've already decided on an outcome and are shifting the narrative to adhere to this outcome. The labelling of JVM + K8 as boring, to posture why it's automatically superior to "serverless integration crap" is exactly the sort of thing poor engineering leadership does to avoid having the sort of difficult conversations necessary. There's no metrics involved, no studies, no prototypes, no financial assessments, none of that it's purely based on emotion. You can envision the same exact thing happening (but to the preference of serverless) on the Prime Video team.

jpgvm 3y ago
It's not emotion. I make money from porting people off of poor technical choices onto solid ones. It comes from experience working with all these various systems. I have used (at large scale) GAE, Google Cloud Functions and Lambda, before that I also used all the previous "serverless-like" things including Beanstalk (anyone else even remember that?), Heroku and bunch of other PaaS things for various stuff. It's all come and gone. To me serverless is just the latest in a long line of things that I get paid to replace with things that actually solve the problem and doesn't turn into a ball of spaghetti that no one is able to modify anymore.
- Spivak 3y ago
  Seconded, just got done with one of these. Literally 1200 separate lambdas, step functions, hundreds of kinesis streams, and sqs topics plus a smattering of ecs fargate tasks for when invariably something takes longer than the max lambda time. And oh god all the stupid supporting infrastructure to build layers. There primary app was Rails, you know what replaced all of it? Running the synchronous tasks on the app servers themselves, the async ones on Sidekiq and changing the max instances on the two ASGs. The only downside has been not being able to meme about how we're web scale anymore.
  - jpgvm 3y ago
    Not being able to make jokes about how fast it is to serve requests from /dev/zero must be absolutely killing your team. However, great job. Once they get to that scale of objects it becomes really hard to clean it up - especially if most the people that wrote them have already left the company for example. Curious if you ended up also saving money in the process?
- babyshake 3y ago
  One potential reason to advocate for serverless architecture is it is relatively stateless and event-driven and therefore modular and easy to reason about. Can you be more specific about how serverless tends to turn into a ball of spaghetti? It is simply because the lack of state becomes more of a bug than a feature?
  - jpgvm 3y ago
    State can be part of it. For example it can be way simpler to use a in-memory cache than a network attached cache and by doing so you save yourself a dependency, a potential network request that can fail, etc. Also for event processing consider the case you need to process events in order for instance. That is a ton easier with stateful consumers, batching etc which while possible in serverless is usually more difficult/off the beaten path. However by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system. It's pretty much accepted fact now that distributed systems are by far the most challenging area of both computer science and software engineering. Serverless doesn't technically -mandate- this, you could put all your code in a giant function. However it strongly incentivises you not to. a) generally it's deployment times and cold start issues get worse with function size and b) it pushes you to adopt other "building blocks" which are themselves network attached. Essentially it leads people into a massively distributed micro-service architecture right out of the gate. For most applications this is definitely the wrong choice even if you have the folks on staff that specialise in distributed systems, usually those folk are the last to advocate for anything distributed ironically. Getting into the weeds as for why this is hard and all the problems you can run into is probably a bit much for a single HN comment but if you really are curious I suggest reading Leslie Lamport and John Ousterhout. Both have covered the problem space in depth with foundational papers, Leslies "Time, Clocks, and the Ordering of Events in a Distributed System" in particular is foundational.
    - eternalban 3y ago
      (quite impressed by the sheer number of comments you have made in this thread, btw) > For example it can be way simpler to use a in-memory cache than a network attached cache and by doing so you save yourself a dependency, a potential network request that can fail, etc. Elsewhere you decided against using cloudflare's kv because it was a poor fit for "cache invalidation" -- "way simpler"? -- and then of course turns out you really don't know if it's a bad fit. > by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system. That is a novel definition of "spagetti". It's a pejorative term in software sense. Why? > It's pretty much accepted fact now that distributed systems are by far the most challenging area of both computer science and software engineering. Ahem. So it caching and cache invalidation. But that didn't stop you, so exactly how is this line you seem to draw with such vocal authority regarding "bad hard", "good hard" informed technically?
- plugin-baby 3y ago
  Has GAE come and gone? Surely it’s still there, and capable of running the JVM? K8s seems like overkill for small teams which don’t already have experience
  - jpgvm 3y ago
    GAE is actually still pretty good tbh. However it has it's limitations and even though there are things like Cloud Run et al I still find GKE is the best option.
- vlovich123 3y ago
  Have you tried Cloudflare Workers? I work at Cloudflare but I’m genuinely curious what your feedback is on it as it does work quite differently from Lambda/Cloudflare functions.
  - jpgvm 3y ago
    Yes actually, currently using in production. I actually advocated for the move to Cloudflare and we ended up adopting workers because Cloudflare unfortunately has broken stale-while-revalidate support. So I built an implementation in Workers that abuses the Cache API to store our own TTLs and also our revalidating state. It works pretty well as we don't end up with too many isolates per POP but it's not perfect. I didn't end up using KV or other features yet, they weren't a good fit for the cache revalidation implementation. I think at-least for now I see it differently to FaaS services like Lambda/CloudFunctions because I don't intend to use it to implement application logic, rather just augment (or in this case shore up deficiencies) in the edge. We may consider doing more stuff at the edge later for latency advantages. This to me changes the calculus because Lamba (well vanilla, not Lamda@Edge) and CF solve my usual problems just with higher costs. CloudFlare workers actually let me do things I couldn't do. So the cost is still just as high as FaaS platforms BUT importantly I actually get something for it, something I can't otherwise have and important enough I am willing to pay the costs and overheads for. Please advocate for real state-while-revalidate support internally! It would make tons of folks super happy.
    - vlovich123 3y ago
      Interesting. Since I TL KV currently I’m particularly interested in the ways it’s not working for you. We’re in the process of rolling out major changes here that might help you here so hit me up on the KV Discord or my username at cloudflare.com. I actually do have a meeting with the Cache team on Monday about something related and I’ll ask about it if you could message me what specifically is broke / what the semantics you want are (I’m not intimately familiar with all the HTTP semantics).
      - jpgvm 3y ago
- grogenaut 3y ago
  I remember beanstalk. We used it for years at twitch. We got a shit ton of mileage out of it. It's easy to talk shit about the problems of the old tech, and beanstalk has a lot of issues. But it also let us build a ton of stuff much easier than the alternatives. This cycle will forever be this way. You only think that the new stuff is the best if this is your first rodeo. If it's not, it's just the right or wrong tradeoff for the point in time you are at.
- noduerme 3y ago
  As a one man shop, I made a decision years ago to try Beanstalk for a particular app backend / API. It's the only "serverless" deployment I've done. It still works fine for that purpose, and it's less trouble than manually deploying to multiple servers. Forced updates are an occasional headache but at least they're all in one place. The code does a lot of serverside computation for a lot of client connections, so that part needs to scale, and since there's relatively little database stuff it just connects to a single RDS instance. I can't think of any other code of mine for which I'd use Beanstalk or something similar now, but it doesn't hurt to evaluate each project independently and pick the right tools for the job.
  - jpgvm 3y ago
    Yeah Beanstalk was pretty simple and decent for it's time. I told anyone that wasn't comfortable with Cloudformation + Packer and wanted to be on AWS to use it during that era. At that time things were pretty binary, you used PaaS like Beanstalk or Heroku or you were off in the wild west where there was no standards at all. People were doing immutable AMIs on ASGs at the high end and then everything down to pet AWS instances w/Capistrano or similar tools at the low end. These days things are different, containers were a big deal. If you target k8s your application is pretty portable, can be run locally in something like kind to integration test the infra components, etc. You can take it to the 3 major clouds and bring most of your ingress, secrets, volume, etc configuration with you. For me k8s has been a godsend because for once I haven't had to learn a whole new set of idioms each new place that are all equivalent but different in their own special snowflake ways.
    - emodendroket 3y ago
      > You can take it to the 3 major clouds and bring most of your ingress, secrets, volume, etc configuration with you. How often have you seen someone actually do this, and was it ever actually as "easy" as they imagined? I feel like the advantages of "going with the flow" and using the stuff specific to the cloud environment you're in are substantial and I feel like the efforts to avoid vendor lock-in start to look insufficient once you try to actually execute most of the time.
- emodendroket 3y ago
  > I make money from porting people off of poor technical choices onto solid ones So it was successful enough for long enough to get them to the point where someone cared about optimizing and paid you to come in, right? I don't see how you are disproving the article's argument that this is a great way to prototype ideas or get a first version of the product out there.
  - rat9988 3y ago
    "So it was successful enough for long enough " Which doesn't necessarily make it a good choice at any point. It just says it wasn't enough of a bad technical choice to make the product sink.
    - emodendroket 3y ago
      OK, sure, it's not ironclad proof that it was the right decision. But it's certainly not proof it was the wrong decision when it was made.
      - namaria 3y ago
        "It works and makes money" is at the root of so much shoddy workmanship in software I think you're making a different point than what you're trying to.
        emodendroket 3y ago
        The argument of TFA is that starting with serverless and evolving away from it bit by bit as it is revealed to be necessary is a sound strategy most of the time. Whatever you may think of that argument, “I have been hired to optimize projects built using serverless and get them off of it” is not a rebuttal.
        namaria 3y ago
        Thanks for rehashing how this thread got started for me?
- trollied 3y ago
  Amen, brother. Love you.
- golemiprague 3y ago
  [dead]
- unusualmonkey 3y ago
  Bit of survivorship bias there! No-one is going to hire you to port off of serverless, if they're happy with it. Ergo you only see cases where it isn't working well.
  - jpgvm 3y ago
    Yeah that is fair. There is probably serverless chugging away fine in lots of places. Doesn't make it better than alternatives but still a valid point.
    - unusualmonkey 3y ago
      I use serverless successfully - handing tens of millions of annual revenue. We do this in a handful of small services, none of which gets much traffic. The alternatives (e.g. app-engine, container in vm, or k8s) all require more infrastructure, would cost more (no scaling to zero), and are overall less suited for the kinds of problems we expect to see.
    - slrainka 3y ago
      I too have a small team that built a greenfield application that generates about $100M annually for a large hotel chain. Serverless allowed us to get to market fast and everyone in the team is a feature developer — 0 infra folks. I think that’s the biggest value we have been able to derive from this compared all the other internal App teams who struggle with release cadences, experimentation and TTV. But I fully acknowledge this is not the ideal and optimal setup and we are paying more to AWS on the OpEx. However, we have gone 3 years without any downtime at full speed, and the fact that we were able to unlock net new revenue very quickly has made all project costs and OpEx very minuscule. But I think once the application boundaries stabilize and we are able to take a breather from full on feature development, hope to migrate some of this to simpler containerized infrastructure. To make this transition easier, we use Serverless-framework and decouple the app component architecture from deployment architecture, so developers don’t worry whether this is getting deployed to a lambda or container.
- stickfigure 3y ago
  > I make money from porting people off of poor technical choices I make money by delivering features my customers care about. We tried k8s, it was a ton of mental effort and time wasted when we could have been building more features. That particular bit of infrastructure is currently running in elastic beanstalk, where it's cheap and I don't have to think about it.
  - jpgvm 3y ago
    I actually think Beanstalk/Cloud Run/friends are fine. I should have specified that Functions as a Service is the main mess I think people should be avoiding. If Beanstalk + ELB + RDS or whatever is getting you by just fine then by all means. It's super easy to port that to k8s whenever you end up needing what it does, which might be never if your money/business scales well vs infra complexity.
- deleted 3y ago
  [deleted]
- alex_lav 3y ago
  > I make money from porting people off of poor technical choices onto solid ones. Is this not just openly broadcasting your financial incentive to say “Whatever is popular is bad, pay me and I’ll make it good” ?
mk89 3y ago
> Actually your comment is a great demonstration of poor engineering leadership. You've already decided on an outcome and are shifting the narrative to adhere to this outcome. For me it's actually the decision of a CTO of one of top 10 companies in the world in terms of software size/value/etc. to push it down to very smart people - that's bad leadership. Why? Because I guess this thing didn't go as planned - a self fulfilling prophecy that shows how awesome serverless is. It backfired in a period where people are questioning whether all these megahypes are not actually just some marketing. Yes, he is responsible for that, for pushing it down to his people that "everything new must be started with serverless" first. When actually in some situations, as the article showed, a monolith or a (big) microservice would do the job better. Why the hell hire super smart people when you have to tell them how to do things?
- hyperhopper 3y ago
  > Why the hell hire super smart people when you have to tell them how to do things? A very hard question. One side is that every smart engineer will do what's best for their use case. The other is now you might have 100 smart engineers with 100 smart different solutions which can cause repeated work, integration issues, missed common solutions, etc. Both approaches have lots of pros cons and tradeoffs
  - jpgvm 3y ago
    Generally the best approach there is to elevate a bunch of your smartest engineers into an architecture team that a) set good standards but also b) appreciates the nuance of problems sufficiently to know when to deviate. A small team can probably only handle one tech stack, an org with 10 teams can probably handle 10 stacks if they needed to but can get efficiency by standardising on some number less than 10. However making those calls shouldn't be coming down from the CTO, it should be made among the architects that actually design and build the systems.
    - phil-m 3y ago
      Where I'd consider a (good) CTO should be somehow responsible for the high-level architectural decisions. Accumulate enough expertise and know-how (via "architects" or rather very senior engineers) and then discuss the pros/cons and at the end weigh each argument and decide what may work best.
AndrewKemendo 3y ago
Fully agree with this - it completely ignores the context and domain for a one size fits all solution, which is precisely what is wrong with engineering today - lack of holism. The OP's comment "Also generally the conception that building things with containers + k8s actually takes more time than serverless is a horseshit take IMO. It doesn't and if it's taking you longer you are probably doing something wrong." I couldn't disagree with more from a hands on practical perspective. That will 100% take anyone longer to spin up for an MVP (for more cost) than spinning up a FreeBSD server and I'm unaware of anything that needs more than that for an MVP. This isn't even mentioning the fact that every second your data sits in an AWS/CGP/Azure datacenter - you are not only telling them what kind of data you have (don't pretend like the hosts don't know and can't tell) but you're increasing your switching/egress costs
- maccard 3y ago
  > I couldn't disagree with more from a hands on practical perspective. That will 100% take anyone longer to spin up for an MVP (for more cost) than spinning up a FreeBSD server and I'm unaware of anything that needs more than that for an MVP. I agree with OP. If you hand me a binary and instructions on how to run it, I can have it running on either AWS or DO in a container, connected to a managed database with snapshots enabled in about 30 minutes. If the toolchain is easy to set up, I can have CI/CD with 0-downtime deployments with auto-rollback in another 30 minutes, and you won't need to touch either of those until you're hitting thousands of concurrent users for a typical web app. > you are not only telling them what kind of data you have (don't pretend like the hosts don't know and can't tell) this is FUD of the n'th degree - if you suspect AWS are spying on your managed RDS data, I don't know what to tell you.
  - AndrewKemendo 3y ago
    Great…So you quickly put yourself in a position where the longer you’re on the service the more expensive it will be to migrate And yes Amazon uses AWS utilization to keep track of promising startups or places that they want signal about market changes.
echelon 3y ago
> poor engineering leadership. Or fantastic engineering leadership. OP has constrained the technology choices of the org into a very capable set of primitives. The entire org can move faster by sitting atop the same rails. K8s + tried and true tech lets the org spin up additional compute anywhere and easily migrate off a particular vendor. Lift and shift is work, but not nearly as bad when the rails are simple.
- jpgvm 3y ago
  Having done the dreaded AWS + kops -> GKE migration I must agree, it was hard but nowhere near as hard as it could have been.
- nostrebored 3y ago
  > simple > k8s Pick one
  - jpgvm 3y ago
    Eh. There are different types of complexity. k8s has a nice contained local complexity that I prefer. It's not simple internally but it's simple to use if you know how it works. So really it's not "k8s isn't simple" it's more "nothing is simple" so pick your preferred type of complexity.
    - scarface74 3y ago
      > It's not simple internally but it's simple to use if you know how it works. Isn’t that literally true about anything?
stcroixx 3y ago
I agree with him that serverless should not be the default and there should be a clear justification for needing those features. He then fails to apply the same rule to k8. Almost nobody needs that either and treating it as the default not a good strategy.
- Aperocky 3y ago
  Exactly, not everything needs scale. For a low traffic, low use system, lambda + dynamo might just be the cheapest and easiest to implement and maintain. And there are generally more of those type of systems.
  - Aeolun 3y ago
    Depends? Is your load consistent and latency sensitive? Then having a small ECS instance may be a lot better.
- jasonm23 3y ago
  Yes, to call Serverless hype train and then talk of K8s as the "boring solid choice" is pretty funny.
Aeolun 3y ago
> it's purely based on emotion Maybe charitably assume it’s based on years of experience doing this exact thing? It’s certainly my experience.

jpgvm 3y ago

It's not emotion. I make money from porting people off of poor technical choices onto solid ones. It comes from experience working with all these various systems. I have used (at large scale) GAE, Google Cloud Functions and Lambda, before that I also used all the previous "serverless-like" things including Beanstalk (anyone else even remember that?), Heroku and bunch of other PaaS things for various stuff. It's all come and gone. To me serverless is just the latest in a long line of things that I get paid to replace with things that actually solve the problem and doesn't turn into a ball of spaghetti that no one is able to modify anymore.

Spivak 3y ago
Seconded, just got done with one of these. Literally 1200 separate lambdas, step functions, hundreds of kinesis streams, and sqs topics plus a smattering of ecs fargate tasks for when invariably something takes longer than the max lambda time. And oh god all the stupid supporting infrastructure to build layers. There primary app was Rails, you know what replaced all of it? Running the synchronous tasks on the app servers themselves, the async ones on Sidekiq and changing the max instances on the two ASGs. The only downside has been not being able to meme about how we're web scale anymore.
- jpgvm 3y ago
  Not being able to make jokes about how fast it is to serve requests from /dev/zero must be absolutely killing your team. However, great job. Once they get to that scale of objects it becomes really hard to clean it up - especially if most the people that wrote them have already left the company for example. Curious if you ended up also saving money in the process?
babyshake 3y ago
One potential reason to advocate for serverless architecture is it is relatively stateless and event-driven and therefore modular and easy to reason about. Can you be more specific about how serverless tends to turn into a ball of spaghetti? It is simply because the lack of state becomes more of a bug than a feature?
- jpgvm 3y ago
  State can be part of it. For example it can be way simpler to use a in-memory cache than a network attached cache and by doing so you save yourself a dependency, a potential network request that can fail, etc. Also for event processing consider the case you need to process events in order for instance. That is a ton easier with stateful consumers, batching etc which while possible in serverless is usually more difficult/off the beaten path. However by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system. It's pretty much accepted fact now that distributed systems are by far the most challenging area of both computer science and software engineering. Serverless doesn't technically -mandate- this, you could put all your code in a giant function. However it strongly incentivises you not to. a) generally it's deployment times and cold start issues get worse with function size and b) it pushes you to adopt other "building blocks" which are themselves network attached. Essentially it leads people into a massively distributed micro-service architecture right out of the gate. For most applications this is definitely the wrong choice even if you have the folks on staff that specialise in distributed systems, usually those folk are the last to advocate for anything distributed ironically. Getting into the weeds as for why this is hard and all the problems you can run into is probably a bit much for a single HN comment but if you really are curious I suggest reading Leslie Lamport and John Ousterhout. Both have covered the problem space in depth with foundational papers, Leslies "Time, Clocks, and the Ordering of Events in a Distributed System" in particular is foundational.
  - eternalban 3y ago
    (quite impressed by the sheer number of comments you have made in this thread, btw) > For example it can be way simpler to use a in-memory cache than a network attached cache and by doing so you save yourself a dependency, a potential network request that can fail, etc. Elsewhere you decided against using cloudflare's kv because it was a poor fit for "cache invalidation" -- "way simpler"? -- and then of course turns out you really don't know if it's a bad fit. > by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system. That is a novel definition of "spagetti". It's a pejorative term in software sense. Why? > It's pretty much accepted fact now that distributed systems are by far the most challenging area of both computer science and software engineering. Ahem. So it caching and cache invalidation. But that didn't stop you, so exactly how is this line you seem to draw with such vocal authority regarding "bad hard", "good hard" informed technically?
plugin-baby 3y ago
Has GAE come and gone? Surely it’s still there, and capable of running the JVM? K8s seems like overkill for small teams which don’t already have experience
- jpgvm 3y ago
  GAE is actually still pretty good tbh. However it has it's limitations and even though there are things like Cloud Run et al I still find GKE is the best option.
vlovich123 3y ago
Have you tried Cloudflare Workers? I work at Cloudflare but I’m genuinely curious what your feedback is on it as it does work quite differently from Lambda/Cloudflare functions.
- jpgvm 3y ago
  Yes actually, currently using in production. I actually advocated for the move to Cloudflare and we ended up adopting workers because Cloudflare unfortunately has broken stale-while-revalidate support. So I built an implementation in Workers that abuses the Cache API to store our own TTLs and also our revalidating state. It works pretty well as we don't end up with too many isolates per POP but it's not perfect. I didn't end up using KV or other features yet, they weren't a good fit for the cache revalidation implementation. I think at-least for now I see it differently to FaaS services like Lambda/CloudFunctions because I don't intend to use it to implement application logic, rather just augment (or in this case shore up deficiencies) in the edge. We may consider doing more stuff at the edge later for latency advantages. This to me changes the calculus because Lamba (well vanilla, not Lamda@Edge) and CF solve my usual problems just with higher costs. CloudFlare workers actually let me do things I couldn't do. So the cost is still just as high as FaaS platforms BUT importantly I actually get something for it, something I can't otherwise have and important enough I am willing to pay the costs and overheads for. Please advocate for real state-while-revalidate support internally! It would make tons of folks super happy.
  - vlovich123 3y ago
    Interesting. Since I TL KV currently I’m particularly interested in the ways it’s not working for you. We’re in the process of rolling out major changes here that might help you here so hit me up on the KV Discord or my username at cloudflare.com. I actually do have a meeting with the Cache team on Monday about something related and I’ll ask about it if you could message me what specifically is broke / what the semantics you want are (I’m not intimately familiar with all the HTTP semantics).
    - jpgvm 3y ago
grogenaut 3y ago
I remember beanstalk. We used it for years at twitch. We got a shit ton of mileage out of it. It's easy to talk shit about the problems of the old tech, and beanstalk has a lot of issues. But it also let us build a ton of stuff much easier than the alternatives. This cycle will forever be this way. You only think that the new stuff is the best if this is your first rodeo. If it's not, it's just the right or wrong tradeoff for the point in time you are at.
noduerme 3y ago
As a one man shop, I made a decision years ago to try Beanstalk for a particular app backend / API. It's the only "serverless" deployment I've done. It still works fine for that purpose, and it's less trouble than manually deploying to multiple servers. Forced updates are an occasional headache but at least they're all in one place. The code does a lot of serverside computation for a lot of client connections, so that part needs to scale, and since there's relatively little database stuff it just connects to a single RDS instance. I can't think of any other code of mine for which I'd use Beanstalk or something similar now, but it doesn't hurt to evaluate each project independently and pick the right tools for the job.
- jpgvm 3y ago
  Yeah Beanstalk was pretty simple and decent for it's time. I told anyone that wasn't comfortable with Cloudformation + Packer and wanted to be on AWS to use it during that era. At that time things were pretty binary, you used PaaS like Beanstalk or Heroku or you were off in the wild west where there was no standards at all. People were doing immutable AMIs on ASGs at the high end and then everything down to pet AWS instances w/Capistrano or similar tools at the low end. These days things are different, containers were a big deal. If you target k8s your application is pretty portable, can be run locally in something like kind to integration test the infra components, etc. You can take it to the 3 major clouds and bring most of your ingress, secrets, volume, etc configuration with you. For me k8s has been a godsend because for once I haven't had to learn a whole new set of idioms each new place that are all equivalent but different in their own special snowflake ways.
  - emodendroket 3y ago
    > You can take it to the 3 major clouds and bring most of your ingress, secrets, volume, etc configuration with you. How often have you seen someone actually do this, and was it ever actually as "easy" as they imagined? I feel like the advantages of "going with the flow" and using the stuff specific to the cloud environment you're in are substantial and I feel like the efforts to avoid vendor lock-in start to look insufficient once you try to actually execute most of the time.
emodendroket 3y ago
> I make money from porting people off of poor technical choices onto solid ones So it was successful enough for long enough to get them to the point where someone cared about optimizing and paid you to come in, right? I don't see how you are disproving the article's argument that this is a great way to prototype ideas or get a first version of the product out there.
- rat9988 3y ago
  "So it was successful enough for long enough " Which doesn't necessarily make it a good choice at any point. It just says it wasn't enough of a bad technical choice to make the product sink.
  - emodendroket 3y ago
    OK, sure, it's not ironclad proof that it was the right decision. But it's certainly not proof it was the wrong decision when it was made.
    - namaria 3y ago
      "It works and makes money" is at the root of so much shoddy workmanship in software I think you're making a different point than what you're trying to.
      - emodendroket 3y ago
        The argument of TFA is that starting with serverless and evolving away from it bit by bit as it is revealed to be necessary is a sound strategy most of the time. Whatever you may think of that argument, “I have been hired to optimize projects built using serverless and get them off of it” is not a rebuttal.
        namaria 3y ago
        Thanks for rehashing how this thread got started for me?
trollied 3y ago
Amen, brother. Love you.
golemiprague 3y ago
[dead]
unusualmonkey 3y ago
Bit of survivorship bias there! No-one is going to hire you to port off of serverless, if they're happy with it. Ergo you only see cases where it isn't working well.
- jpgvm 3y ago
  Yeah that is fair. There is probably serverless chugging away fine in lots of places. Doesn't make it better than alternatives but still a valid point.
  - unusualmonkey 3y ago
    I use serverless successfully - handing tens of millions of annual revenue. We do this in a handful of small services, none of which gets much traffic. The alternatives (e.g. app-engine, container in vm, or k8s) all require more infrastructure, would cost more (no scaling to zero), and are overall less suited for the kinds of problems we expect to see.
  - slrainka 3y ago
    I too have a small team that built a greenfield application that generates about $100M annually for a large hotel chain. Serverless allowed us to get to market fast and everyone in the team is a feature developer — 0 infra folks. I think that’s the biggest value we have been able to derive from this compared all the other internal App teams who struggle with release cadences, experimentation and TTV. But I fully acknowledge this is not the ideal and optimal setup and we are paying more to AWS on the OpEx. However, we have gone 3 years without any downtime at full speed, and the fact that we were able to unlock net new revenue very quickly has made all project costs and OpEx very minuscule. But I think once the application boundaries stabilize and we are able to take a breather from full on feature development, hope to migrate some of this to simpler containerized infrastructure. To make this transition easier, we use Serverless-framework and decouple the app component architecture from deployment architecture, so developers don’t worry whether this is getting deployed to a lambda or container.
stickfigure 3y ago
> I make money from porting people off of poor technical choices I make money by delivering features my customers care about. We tried k8s, it was a ton of mental effort and time wasted when we could have been building more features. That particular bit of infrastructure is currently running in elastic beanstalk, where it's cheap and I don't have to think about it.
- jpgvm 3y ago
  I actually think Beanstalk/Cloud Run/friends are fine. I should have specified that Functions as a Service is the main mess I think people should be avoiding. If Beanstalk + ELB + RDS or whatever is getting you by just fine then by all means. It's super easy to port that to k8s whenever you end up needing what it does, which might be never if your money/business scales well vs infra complexity.
deleted 3y ago
[deleted]
alex_lav 3y ago
> I make money from porting people off of poor technical choices onto solid ones. Is this not just openly broadcasting your financial incentive to say “Whatever is popular is bad, pay me and I’ll make it good” ?

mk89 3y ago

> Actually your comment is a great demonstration of poor engineering leadership. You've already decided on an outcome and are shifting the narrative to adhere to this outcome. For me it's actually the decision of a CTO of one of top 10 companies in the world in terms of software size/value/etc. to push it down to very smart people - that's bad leadership. Why? Because I guess this thing didn't go as planned - a self fulfilling prophecy that shows how awesome serverless is. It backfired in a period where people are questioning whether all these megahypes are not actually just some marketing. Yes, he is responsible for that, for pushing it down to his people that "everything new must be started with serverless" first. When actually in some situations, as the article showed, a monolith or a (big) microservice would do the job better. Why the hell hire super smart people when you have to tell them how to do things?

hyperhopper 3y ago
> Why the hell hire super smart people when you have to tell them how to do things? A very hard question. One side is that every smart engineer will do what's best for their use case. The other is now you might have 100 smart engineers with 100 smart different solutions which can cause repeated work, integration issues, missed common solutions, etc. Both approaches have lots of pros cons and tradeoffs
- jpgvm 3y ago
  Generally the best approach there is to elevate a bunch of your smartest engineers into an architecture team that a) set good standards but also b) appreciates the nuance of problems sufficiently to know when to deviate. A small team can probably only handle one tech stack, an org with 10 teams can probably handle 10 stacks if they needed to but can get efficiency by standardising on some number less than 10. However making those calls shouldn't be coming down from the CTO, it should be made among the architects that actually design and build the systems.
  - phil-m 3y ago
    Where I'd consider a (good) CTO should be somehow responsible for the high-level architectural decisions. Accumulate enough expertise and know-how (via "architects" or rather very senior engineers) and then discuss the pros/cons and at the end weigh each argument and decide what may work best.

AndrewKemendo 3y ago

Fully agree with this - it completely ignores the context and domain for a one size fits all solution, which is precisely what is wrong with engineering today - lack of holism. The OP's comment "Also generally the conception that building things with containers + k8s actually takes more time than serverless is a horseshit take IMO. It doesn't and if it's taking you longer you are probably doing something wrong." I couldn't disagree with more from a hands on practical perspective. That will 100% take anyone longer to spin up for an MVP (for more cost) than spinning up a FreeBSD server and I'm unaware of anything that needs more than that for an MVP. This isn't even mentioning the fact that every second your data sits in an AWS/CGP/Azure datacenter - you are not only telling them what kind of data you have (don't pretend like the hosts don't know and can't tell) but you're increasing your switching/egress costs

maccard 3y ago
> I couldn't disagree with more from a hands on practical perspective. That will 100% take anyone longer to spin up for an MVP (for more cost) than spinning up a FreeBSD server and I'm unaware of anything that needs more than that for an MVP. I agree with OP. If you hand me a binary and instructions on how to run it, I can have it running on either AWS or DO in a container, connected to a managed database with snapshots enabled in about 30 minutes. If the toolchain is easy to set up, I can have CI/CD with 0-downtime deployments with auto-rollback in another 30 minutes, and you won't need to touch either of those until you're hitting thousands of concurrent users for a typical web app. > you are not only telling them what kind of data you have (don't pretend like the hosts don't know and can't tell) this is FUD of the n'th degree - if you suspect AWS are spying on your managed RDS data, I don't know what to tell you.
- AndrewKemendo 3y ago
  Great…So you quickly put yourself in a position where the longer you’re on the service the more expensive it will be to migrate And yes Amazon uses AWS utilization to keep track of promising startups or places that they want signal about market changes.

echelon 3y ago

> poor engineering leadership. Or fantastic engineering leadership. OP has constrained the technology choices of the org into a very capable set of primitives. The entire org can move faster by sitting atop the same rails. K8s + tried and true tech lets the org spin up additional compute anywhere and easily migrate off a particular vendor. Lift and shift is work, but not nearly as bad when the rails are simple.

jpgvm 3y ago
Having done the dreaded AWS + kops -> GKE migration I must agree, it was hard but nowhere near as hard as it could have been.
nostrebored 3y ago
> simple > k8s Pick one
- jpgvm 3y ago
  Eh. There are different types of complexity. k8s has a nice contained local complexity that I prefer. It's not simple internally but it's simple to use if you know how it works. So really it's not "k8s isn't simple" it's more "nothing is simple" so pick your preferred type of complexity.
  - scarface74 3y ago
    > It's not simple internally but it's simple to use if you know how it works. Isn’t that literally true about anything?

stcroixx 3y ago

I agree with him that serverless should not be the default and there should be a clear justification for needing those features. He then fails to apply the same rule to k8. Almost nobody needs that either and treating it as the default not a good strategy.

Aperocky 3y ago
Exactly, not everything needs scale. For a low traffic, low use system, lambda + dynamo might just be the cheapest and easiest to implement and maintain. And there are generally more of those type of systems.
- Aeolun 3y ago
  Depends? Is your load consistent and latency sensitive? Then having a small ECS instance may be a lot better.
jasonm23 3y ago
Yes, to call Serverless hype train and then talk of K8s as the "boring solid choice" is pretty funny.

Aeolun 3y ago

> it's purely based on emotion Maybe charitably assume it’s based on years of experience doing this exact thing? It’s certainly my experience.