34 ms·

The Prime Video microservices to monolith story

512 points by mparnisari 3y ago

goodpoint 3y ago
The points in the Prime Video article still stand. People overuse microservices and serverless and stuff gets expensive quickly.
- j45 3y ago
  It’s interesting that there are folks who think there must be only one architecture to be the best in all cases. It reeks of inexperience and being highly opinionated followers of popular opinion instead of playing with both personally. So few projects actually scale to benefit from microservices where its a benefit. One’s interpretation and benefit for microservices is fine. It’s just not guaranteed to be factual in many used cases. First, the line of where scaling benefits from microservices is also moving higher each year due to the same underlying reason: Whether Monolith or Microservices, the increased performance of Linux networking, availability of inexpensive ram and cpu along with much more optimized databases like Postgres or mysql from the past 10-15 years ago seems to have been missed by many people who seem to have set an interpretation in place and never looked back. When horsepower gets faster and cheaper each year the architectural best practices of microservices adoption was likely relative to its year of adoption. What to do? Keep your stack unapologetically simple. Don’t handcuff your development speed when the complexity arrives to rearchitect things anyways. Because the first version will have the wrong assumptions about what customers want. How? Consider monorepos to let you start as monolith for velocity and move to microservices as needed Monolithic apps can have a service based architecture internally (monorepos can be pretty handy), which in turn lets startups ship much more regularly than they ought think for monoliths. You can design folders in a monorepo at the start to fuel a monolith and keep a path to go to micro services, where and if needed. To know this enough experience (both good and bad) is needed with both microservices and monoliths.
  - groestl 3y ago
    > Monolithic apps can have a service based architecture internally Not only that, but if you absolutely must (for example different scaling requirements), forking out a service from a monolith and deploying it as microservice is relatively straightforward. Provided source code interdependencies have been maintained cleanly before, which can be done with support from tools.
    - tonyarkles 3y ago
      I definitely agree. More than once I've taken a monolith and sliced out a small piece of it to run as a separate service. That's been a significantly less painful task than trying to aggregate multiple microservices into a less expensive monolith, especially when multiple microservices use the same underlying packages but different conflicting versions or different global configuration.
      - j45 3y ago
        The interesting thing about doing this is how things kind of end back at n-tier architecture, where one can have abstractions between each layers of the cake to more easily swap in/out technologies as well.
        tonyarkles 3y ago
        Yeah, definitely. In my experience, the trick is that without doing analysis up front, you don't know exactly what the n-tiers are going to look like. Jumping right to microservices right off the bat is putting a lot of extra overhead at the start of the project (defining HTTP or gRPC or whatever APIs, working out the boilerplate for each project, etc.) without knowing whether the boundaries are appropriate or whether a given component is likely to end up being a hotspot that will need independent scaling. When you start with a monolith that isn't just a ball of spaghetti (e.g. clear module boundaries), you don't have to put as much work into the whole big picture and it's easier to evolve the internal architecture of one larger service than N smaller services. And if you do discover that one piece of it is appropriate to split off (hotspot, reasonably transactional instead of interactive calling, etc), then you just replace the guts of that module with whatever RPC mechanism you're using and the rest of the application doesn't really need to care about it.
zwischenzug 3y ago
I still don't really get it. There are so many simple, easy to build, cheap to run three tier frameworks out there so why bother with all the hassle of porting at all? If your system gets super busy then you can repurpose the code with higher performance and well tuned DBs later. I spent 15 years doing that for some of the busiest systems in the world, but those started on a single cpu.
- gregoriol 3y ago
  The point is: managers and keywords
- gadflyinyoureye 3y ago
  At my client, we go Serverless first in a bastardized state, NestJS. The benefit is quick development at a low cost. We have a series of calculations. Each read from a common S3 and write to it. This allows the teams to do their work with independent deployments. Each one costs about $5 a month. That’s rather hard to get in the AWS EC2 world. They are also easier to manage. We don’t have to manage keys or certain. There is no redeploy after 90s that comes with EC2. However these are low access, slowish apps. Maybe a 1,000 calls a day. They can take a minute to do their thing. Seems like a nice match for lambdas. Benefit of NestJS is if we ever needed to move out of lambda we have a tradition all Express app.
  - zwischenzug 3y ago
    Useful insight, but whether it's 5$ or 50$ a month, compared to dev costs that's a rounding error pretty much everywhere.
    - gadflyinyoureye 3y ago
      That's why we went with NestJS. It's like Spring but for TypeScript. We can run everything locally. Boot up the NestJS service at localhost:8080 with LocalStack running on the side for access to S3 and DynamoDB compatible, working services.
  - 3y ago
jpgvm 3y ago
Serverless first is a mistake. It should be "serverless where it's distinct advantages are actually useful" which is way less common than people think. Also generally the conception that building things with containers + k8s actually takes more time than serverless is a horseshit take IMO. It doesn't and if it's taking you longer you are probably doing something wrong. All the points noted in the slides etc just come from poor engineering leadership. i.e decisions/choices, etc. Constaints do speed up development but they can be enforced by leadership, not by putting yourself in an endless pit of technical debt. The only world where I can make a good case for serverless for small teams is if you have said poor leadership and serverless helps you escape their poor decision making. Otherwise assuming you have the mandate necessary to get stuff built the right way just stick with the Boring Tech stack of JVM + PG + k8s. It solves all problems regardless of size, there are minimal decisions to be made (really just Spring or Quarkus atm) and you won't spend time dicking around with half-baked serverless integration crap. Instead you can just use battle tested high performance libraries for literally everything. Don't get sucked into this serverless nonsense. It's always been bad and it probably always will be. When it's out of fashion Boring Tech will still be here.
- foffoofof 3y ago
  Is jvm pg and k8s boring enough? K8s is overkill. Jvm is heavy. Pg is fine but relational dbs generally suck.
  - beckingz 3y ago
    elaborate on how relational databases suck?
    - vbezhenar 3y ago
      They're not web scale. https://youtu.be/b2F-DItXtZs https://youtu.be/b2F-DItXtZs
    - foffoofof 3y ago
      Try streaming updates to a query without polling…which is essentially all web apps. It’s fundamentally the wrong approach and people don’t realize. The relational model is good for analytics. Think about how often you need to know the internals or you have to debug perf by trying to get a black box to do what you want. Devs should be a layer deeper.
jack_squat 3y ago
Am I alone in feeling there is nothing left to be said on this topic? Correct application design, and balancing that against the constraints of getting a product out the door, the requirements of a particular application, and the resources on hand (skill-sets, developers, infrastructure) does not in my opinion boil down to "microservices vs monolith". Both strategies have tradeoffs, both have produced successes and failures, and choice of approach seems (obviously) too specific to a given context and application for general advice to apply.
- deleted 3y ago
  [deleted]
- TeeWEE 3y ago
  Exactly, tool vs hammer. Sometimes you made the wrong choice in tool, and then you switch tools.. Nothing wrong with that. A craftman just knows its tools better. There is no magic bullet here.
  - matisseverduyn 3y ago
    > Exactly, tool vs hammer. Sometimes you made the wrong choice in tool, and then you switch tools.. Nothing wrong with that. A craftsman just knows its tools better. There is no magic bullet here. Rational take, but I see the debate similar to Roman vs Arabic numerals. Keeping a tally? Roman. Need to use operators? Arabic. Sometimes you can keep a tally in Arabic (not ideal), and sometimes you can do basic operations on Roman numerals (not ideal). However, when you want to start using variables, only one tool enables this easily. I can't architect the types of redundant and properly separated interoperable systems with a monolith that microservices otherwise enable. So the desire to move forward isn't the need to find a magic bullet, but the next evolution of an existing ability that unlocks new capabilities... (I don't think calculus would have been discovered using Roman numerals)
    - TeeWEE 3y ago
      No, but you can start a business with a singlesystem, keep things well decoupled (hard / software engineer is hard).. Then you gradually run some parts of the systems in different docker containers, then at some point you completely decoupled systems and have different teams handle different service boundaries. You could still have the core system be a big bigger, and sometimes referred to as monolith. But monolith vs microservices is a polarises the discussion, its a gradual scale. Its the engineering, and keeping things separated that matters.
balls187 3y ago
I don’t quite understand the point in all this. Design tradeoffs are still a thing; did we all just forget? “Abortions for some... miniature American flags for others!”
- mattgreenrocks 3y ago
  Software architecture is (incorrectly) seen as NP-hard by practitioners, so they avoid it, and instead focus on implementing the current "best practices." However, "best practices" can't be best if they're situational, so we arrive at very local maxima, such as microservices for all.
foffoofof 3y ago
The whole industry is stupid at this point. I don’t think anyone knows what they are doing. Everyone just wants to play with the new shiny thing. Half the engineers probably have ADHD which leads to this behavior of seeking novelty. Everyone is obsessed with their shitty stack they used at their last company, and this shit will keep going on and on. The only things that matters is how easy is something to debug and to change. Think about how often you have heard that as any kind of priority. So many shitty decisions would be avoided if people started thinking about that.
- zwischenzug 3y ago
  I've often mused about creating a language that put readability, maintainability, logging etc over, say, performance.
  - Arbortheus 3y ago
    You might enjoy Go. Lots of emphasis has been put into the development tooling and ecosystem. There is an impressive commitment to avoiding breaking changes. We have a legacy Go microservices codebase that’s over a decade old and because the Go maintainers have paid so much attention to avoiding creating breaking changes and a guarantee to never create a “Go 2”, the codebase has aged much better than stuff written at the same time in Python (some of which is still in Python 2 because it is too risky to upgrade, and that functionality will be killed instead because we don’t know how to safely update it). Most of the time we don’t need to make any changes when bumping Go versions, just fixing up things our linter warns is deprecated.
  - Nextgrid 3y ago
    Performance is absolutely a non-issue when modern "engineering" insists on running everything on underpowered VMs with high-latency network-based storage. Even if your language is truly, actually slow, you can get an instant performance boost just moving that workload to a modern bare-metal server (cloud providers basically ate all the performance improvements of modern hardware), or scale that (most likely stateless) workload horizontally if a single bare-metal machine isn't enough.
nforgerit 3y ago
I'm starting to be convinced that the EU should regulate acceptable software design patterns. This will prevent us all from heaving to implement techbro cargo cult pseudo-innovations like "Serverless for everything" /s
- ipaddr 3y ago
  EU should stop regulating what is acceptable and let society naturally come to conclusions. Limiting software designs seems like an area the government shouldn't be involved with. Regulate cloud providers if necessary not abstract patterns
lopkeny12ko 3y ago
The original, unedited title is "So many bad takes — What is there to learn from the Prime Video microservices to monolith story"
mellosouls 3y ago
The nth posting of this story. https://hn.algolia.com/?dateRange=pastWeek&page=0&prefix=false&query=Primevideo&sort=byDate&type=story https://hn.algolia.com/?dateRange=pastWeek&page=0&prefix=fal...
qwertywert_ 3y ago
This has been posted like 6 times already man.
iamleppert 3y ago
If you have a predictable workload (i.e. we ingest 100,000 videos a month with n size) you should be looking at it from that perspective -- how much compute you are going to need, and when. Serverless works well for predictably unpredictable, infrequent workloads that don't need to have perfect output ("good enough"). The big mistake I see people make is trying to be overly clever and predict the future workload needs. It never works out like that. Design for your current workload now, if your architecture can handle 10x of that great! Each time your scale from 10x workload you will likely need to redesign the system anyway, so you shouldn't pay that tax unless you absolutely have to. There are a lot of limitations of serverless, the big one I experienced was inability to control the host environment and limitations on the slice of memory/CPU you get, such that you must take that into consideration when designing your atomic work units. Also paying the distributed computing tax is real and occurs at development and runtime -- things like job tracking and monitoring are important when you have 10,000 processes running -- you start to get into the territory of things that basically never happen on a single machine or small cluster, but become problems with thousands of disparate machines / processes.
- goostavos 3y ago
  >Design for your current workload Please be my friend. The bulk of my job these days is sitting in design reviews trying to convince people that just making up scenarios where you need to scale to 100x isn't actually engineering. It's exhausting self-indulgence. Nothing is easier than inventing scenarios where you'll need to "scale". It's borderline impossible to get software folks to just live in the world that exists in front of their eyes. "Scale" should go back to its original meaning: change relative to some starting point. Slap a trend line on what you know. Do some estimation for what you don't. Design accordingly.
  - jpgvm 3y ago
    Yeah unless you are designing the replacement for a system that has already reached it's scalability limits you shouldn't be worrying too much other than not doing very silly things architecturally. When you are designing replacements though you need to have an idea of what your scalability runway is before the next re-design will be possible. Sometimes that is just 2x, often times it's 10x, occasionally it's 100x but it's all situational.
mr_tristan 3y ago
What's funny to me, is that I focused on something completely different; how independent these teams at Amazon are. The move from pretty different infrastructure services, all at the same company. I can't think of a single place I've worked at in 20 years where these infra teams wouldn't be either: a.) mandated, i.e., "you will stick with lambda/kubernetes/whatever", or b.) review and be involved in the architectural decisionmaking of the video service approach, i.e., have to politick your way around any deviation. I guess the shift from "mono to micro" just isn't very interesting to me. You can usually change your definition of either concept to fit your architecture. This just seemed like the team did the math and revised their approach. Good for them!
- iLoveOncall 3y ago
  This is in part where the "It's always Day 1" mentality manifests at Amazon. There's only one programming language that is forbidden (PHP - for wrong reasons), and only one cloud vendor that is mandated (obvious). But beside that, teams are able to use literally whatever technology they want. There are some technologies that impose themselves because of the ecosystem, but you can always do your own sauce if needed. I've been in two teams that had basically 90% different tech stacks, but it's never a problem, and it doesn't really ever come up in design reviews (unless there's a real reason, not for personal preferences).
  - jpgvm 3y ago
    Curious though, what is the reason for banning PHP? It must be pretty specific if they singled out just PHP.
    - iLoveOncall 3y ago
      I don't know the full history because it's from way before I joined, but it is honestly a terrible take (by a guy who's now Distinguished Engineer at Apple, no less!) explaining that it is impossible to write secure applications in PHP. Basically, they used "the historical number of vulnerabilities identified in applications developed using a technology" as metric to determine how insecure a technology is. For the argument here, they looked at the CVE list, where at the time (in 2012, with the list existing since 1999) 40% of all software vulnerabilities recorded were in PHP applications. This led to the conclusion that PHP is insecure by nature. Of course, he didn't mention that at the time, PHP was also used by 80% of all websites, because that would have made his argument worthless. That wiki page explaining that is still up. It's so baffling to me when the argument violates so many of Amazon's leadership principles.
ThalesX 3y ago
I completed my role as a CTO for a company several years ago. I chose a boring stack with plenty of developers available: MySQL, PHP with Laravel, and Angular 2. Since then, I've been a founding engineer at some startups using the latest, cutting-edge tech stacks. I'm proud to announce that the stack I managed as CTO is still running stable and smooth with minimal intervention, and it remains easy to modify. Meanwhile, the startups with flashy stacks and nearly impossible-to-debug architectures have gone out of business. It amazes me how some companies still adopt the "build it and they will come" approach, and when confronted with harsh reality, they double down on building (and especially increasing architectural complexity). CEOs, if you can't attract customers with an idea or don't have a vertical that people are eager to pay for, cramming 1000 half-baked features into a microservice architecture and using the latest programming language / paradigm won't save your company!
- goodrubyist 3y ago
  Were those startups that went out of business did so because of their stacks/architecture, or are you confusing correlation with causation? And, there is a good reason people shy away from PHP, and it has nothing to do with trying to be "flashy." There should be a name for this kind of fallacy.
  - iLoveOncall 3y ago
    > there is a good reason people shy away from PHP Please share it. And don't rely on articles that are more than a decade and 3 or 4 versions of PHP old.
    - 5Qn8mNbc2FNCiVV 3y ago
      Afaik you cannot discern null and undefined. So, not there and "null"? Either that or I'm going to have to have a good talk with my colleague
      - iLoveOncall 3y ago
        Go have your talk [1]. But that's a pretty weak argument anyway given that: 1. Other languages also do not have this distinction (Python for example AFAIK), 2. It's relatively useless. I get the value of it in JavaScript which has a lot of asynchronous code and frontend where the behavior may indeed be very different between null and undefined, but in a server-side application I don't think the nuance is necessary at runtime, and potential programming mistakes involving that can be caught by a linter. [1] https://stackoverflow.com/questions/6905491/how-to-tell-whether-a-variable-is-null-or-undefined-in-php https://stackoverflow.com/questions/6905491/how-to-tell-whet...
enterthestartup 3y ago
Very interesting that they were able to reduce costs by 90%, though also agree with the opinions mentioned here.
amluto 3y ago
> they had some overly chatty calls between AWS lambda functions and S3. They were able to re-use most of their working code by combining it into a single long running microservice that is horizontally scaled using ECS, and which is invoked via a lambda function. This is only one of many microservices that make up the Prime Video application. The problem is that they called this refactoring a microservice to monolith transition, when it’s clearly a microservice refactoring step If I understood the post at all, I disagree. One can spend hundreds of HN comments discussing technology stacks, monoliths, etc, and this is important: it affects maintainability, developer hours, and money spent on orchestration. For some applications, almost the entire workload is orchestration, and this discussion makes sense. But for this workload, actual work is being done, and it can be quantified and priced. So let’s get to the basics: an uncompressed 1080p video stream is something like 3 Gbps. It costs a certain amount of CPU to decompress. It costs a certain amount of CPU to analyze. These have price tags. You can pay for that CPU at EC2 rates or on-prem rates or Lambda rates or whatever. You can calculate this! And you pay for that bandwidth. You can pay for 3Gbps per stream using S3: you pay S3 and you pay EC2 (or whatever) because that uses 1/10th of the practical maximum EC2 <-> S3 bandwidth per instance. Or you pay for the fraction of main memory bandwidth used if you decode and analyze on the same EC2 instance (or Lambda function or whatever). Or you pay for the fraction of cache bandwidth used if you decide partial frames and analyze without ever sending to memory. And you’ll find that S3 is not “chatty”. It’s catastrophic. In fact, any use of microservices here is likely catastrophic if not carefully bound to the same machine.
- jayd16 3y ago
  So it didn't make sense to separate some functionality. So what? No one who advocates microservices would say you can cut a service along any arbitrary line and expect good results.
- jiggawatts 3y ago
  This is called mechanical sympathy and most architects and developers do not have it. Computer architectures encompass 13 orders of magnitude of performance! That's roughly the difference between something like a trivial function call processing data in L1 cache to a remote call out to something in another region. People often make relatively "small" mistakes of 3 or 4 orders of magnitude, which is still crazy if you think about it, but that's considered to be a minor sin relative to making the architecture diagram look pretty.
joanne123 3y ago
[dead]
jbverschoor 3y ago
Cant' we just ask the "Application Deployer" to upload an EAR, published by the "Application Assembler", and run it on any of the spec-compatible servers? These problems have been solved decades ago.. 2.5 to be exact
th3h4mm3r 3y ago
What if Amazon take bad choices in "something" and my company is tied up with its services? I know that Prime is always "Amazon" and that's okay. But for a startup (but not only) imho it's better starts with something more "standard" than something as serverless that is obviously an abstraction above other tecnologies.
AndrewKemendo 3y ago
Look, the point is simple, AWS marketing says you can do everything all the time with microservices and their 'DevOps' infrastructure to "abstract away" all the complex you know - engineering - that you cannot automate effectively and have a long running robust system. So it's humorous to see an Amazon team that is doing things correct and holistically in contrast to the sales and marketing bullshit that AWS and Cloud evangelists spew. Us nerds have a great way of missing the point and bikeshedding on the technicals (which is also great sometimes!)
- namaria 3y ago
  It's amazing how cloud companies managed to convince so many people in software engineering that software engineering wasn't a core competence in producing software.
  - AndrewKemendo 3y ago
    Yeah cause engineers aren’t ones typically doing the buying
    - intelVISA 3y ago
      Same reason MSFT SWEs use Mac over Windows -- they've seen too much.
  - atulatul 3y ago
    I think the difference was simple vs easy. They made a few of the things very easy.
- scarface74 3y ago
  > Look, the point is simple, AWS marketing says you can do everything all the time with microservices and their 'DevOps' infrastructure to "abstract away" all the complex you know - engineering - that you cannot automate effectively and have a long running robust system. As someone who knows a little bit about what AWS recommends to customers when first hand, I can absolutely guarantee you that AWS does not recommend serverless to everyone. The vast majority of projects that consultants do by revenue at any cloud consulting company (including AWS’s own internal company) is regular old VMs.
bhauer 3y ago
Obviously, the article is microservice apologia, but... > They were able to re-use most of their working code by combining it into a single long running microservice that is horizontally scaled using ECS... No, it was no longer a microservice; it became a plain service, as in SOA. It was no longer micro. That's the whole point. They could have saved time and money had they just made the thing a plain service to begin with. This thing was not sufficiently complicated to warrant the complexity of starting with a bunch of microservices. The article says many hot takes have missed the point, but I think what we're seeing here is an example of the two sides talking past one another since the author hasn't appreciated the opposition's arguments at all.
- m_mueller 3y ago
  One thing to note is that the transition from smaller to larger services tends to be straight forward, while cutting into smaller can be tricky. Thus IMO there is some merit to keeping them small in the beginning until you can analyse them under production workloads. That being said, in this case here I think some very simple performance / cost modelling would have shown the issues with serverless already in the beginning. I do find serverless architecture useful, but not for such a case with heavy base load. Furthermore, data locality is also an important aspect to consider in anything with strong latency or throughput requirements - serverless or not.
  - mirekrusin 3y ago
    It's the other way around - start with monolith (because it's easier to change things, it's just single pr addressing all places at once) and then, possibly few years later, whatever has crystalized and has clear boundaries with little to no changes coming in or changes contained within this boundary - can be potentially extracted. Just look how over-represented RoR is/was as bootstrap tech in known, successful companies. Microservices is not yes/no - it's a slider. You may find sweet spot at ie. 50/50 split like ie. github does, have more or less microservices while keeping core in monolith or services under single versioned monorepo. Microservices are good for satellite services like system integration, pre/post-processing, gateways etc. As a rule of thumb whatever can fit single (tech lead + team)'s "head" - can be monolithic (single monolith or set of services under single versioned monorepo managed by that team). Their job is to provide stable apis/uis - otherwise it can be seen as black box by other teams. This is natural way things evolve (teams are created around naturally bounded concepts) and the suggestion is - don't break it by creating mismatches, keep it in harmony. If something creates measurable problems, slowly form team around it with new tech lead from existing team and let it grow on it's own - it'll extract itself to separate subsystem by itself. It doesn't have to be done overnight. It's astounding how many people use "scale" as chupacabra to scare everybody in the meeting room without clearly defining what they mean by "scale". To make decision around changing architecture to scale you need to precisely define what it means, have metrics, have benchmarks showing current limits, proof it's a problem now or near future and focus just on those actual issues, if they even exist.
EMM_386 3y ago
> I do think microservices were over sold as the answer to everything, and I think this may have arisen from vendors who wanted to sell Kubernetes with a simple marketing message that enterprises needed to modernize by using Kubernetes to do cloud native microservices for everything Yes, they absolutely were oversold, Kubernetes marketing was part of it. There was also a lot of resume-driven development, and simply "look, new shiny object!". A lot of young engineers entered the workforce over the last couple of decades. That often comes along with "forget this old stuff, what's the new stuff?". I use KISS as much as possible with every project I'm involved in desinging. The microservices push got the point where a dev shops with 10 people had it all broken into microservices and running Kubernetes to manage low-traffic front ends, an API and a database. I would see some of these systems and the only thing I could think was "... but why?". Do microservices have a place? Yes. Have I seen them used where they shouldn't be? Yes. Have I seen more poorly implemented microservices architecture than good ones. Yes. Over-engineering the tech stack, or selecting one part of it vs. another based on which is trendier, only leads to more frusturation, difficulty reasoning about it, added development time ... all of which means it will cost more.
- t0b1 3y ago
  That reminds me of a somewhat funny story. I was listening to a presentation about a startup with about 10 people as you mentioned doing hospital digitalization (gist of it seemed to be sending patient data from one doctor to another). They explained that they wanted to choose a “robust” architecture so they chose microservices (kubernetes and all). I thought that was a bit odd for something hosted at a hospital locally especially because they had like 7 or so services (and maybe even two databases but not sure on that one). Well, we later asked them how much data they were even handling and (after a bit of side-stepping) they said it was around 200mb excluding images. That was my “but why?” moment. Apparently you need 7 services to have a web frontend for 200mb of data with most of it probably never accessed and hospitals can suddenly get three new floors overnight.
bravoteam5 3y ago
[dead]
lr1970 3y ago
Dup. Already discussed on HN less than a day ago [0]. [0] https://news.ycombinator.com/item?id=35847988 https://news.ycombinator.com/item?id=35847988
- jlund-molfese 3y ago
  No, it's a different link. The one in this topic is addressing the link you posted.
kennu 3y ago
Many comments seem to miss the point that nowadays native AWS cloud applications are meant to be built by combining various AWS services using Lambda as the glue between them. It's not about serverless vs non-serverless: it's about using cloud services for the purpose they are each good at. In many cases you might actually have more IaC code than Lambda handler code, especially if you can leverage services like AppSync and Step Functions. The application is no longer a set of microservices; it's a set of CDK / CloudFormation stacks and the resources that they define, some of which might be microservices.
bdcravens 3y ago
Serverless isn't "cheap". It's rightsized. If you aren't serving 24/7, it makes sense to move to an on-demand model. However if you compare the unit of compute pricing, Lambda is comparable to other services, and likely more expensive than buying compute wholesale. You can just buy it in smaller portions.
getcrunk 3y ago
Seeing as how a lot of players are going in on serverless (vercel, netlify, deno, cloudflare) the cynic in me wonders if this is a calculated move on part of Amazon to bring people back to using ec2 .. etc
distantsounds 3y ago
microservices? to sling video files over the internet? tell me you're over-engineering without telling me you're over-engineering
mlhpdx 3y ago
I am so baffled by this whole discussion. Isn’t serverless (aka event-driven distributed systems made up of small components, with no reserved capacity) just one of many options? I write code for 8-bit controllers that I’d never run on Lambda (even though I’m a Lambda/Step Functions freak). I need 90s server software that can only run in a VM. It’s a complicated world and your favorite tool/model just isn’t going to cut it everywhere. Move on maybe?
andy_ppp 3y ago
Nothing is ever truly a mistake, it's much more about understanding what you are trading off between real hardware, cloud servers, something like fly or heroku or vercel and purer serverless functions on AWS or whatever.
GabeIsko 3y ago
Yeah, I don't really understand why people are making a big deal about this. Two lambda functions passing data to step functions? Not exactly the best serverless architecture. And if you think about the problem of checking video streams, there is really no reason to scale different parts of it independently because you just need one process to watch the video and check for errors. Plus they still deployed the new solution on ECS, so it isn't really a monolith application that runs on a fixed number of servers. It's a real nothing story - the Prime Video team was just doing some engineering maintenance. Honestly the bigger story is that it was mischaracterized from the beginning, but I would imagine that is more a result of things coming together from multiple different systems and teams more than the way someone designed this stuff.
chrismsimpson 3y ago
Seeing all the acolytes of serverless get triggered by this has been enjoyable to watch
carabiner 3y ago
I think Agile workflow has a part to play in this. It embraces complexity and moving fast to build out buggy features, where every new bug becomes a new user story. It feels productive but it just creates more work than you need.
hnaccountme 3y ago
I'm surprised Amazon published what they did. AWS make a lot of money selling hyped up buzz word tech to incompetent companies.
- aka878 3y ago
  Prime Video, Lambda, EC2, everything in discussions still revolves around Amazon.
vaidhy 3y ago
I see a lot of about architecture, service design and complexity. One of the key missing piece is around Amazon culture. Promo-driven design is a real deal at Amazon. I am sure this design got someone promoted to principal engineer or higher. If you cannot build complexity into the design, how can you prove your caliber and get promoted.
- theK 3y ago
  The article is more about bundling together lambda functions into horizontally scalable containers. Paraphrasing the author: > I think the prime video team's presentation should have been called "Moving from lambda to container microservices" There is nothing particularly bad about this move. Prototype something in a FaaS, move it to a container based microservices when you know which features you want and how they perform. Not doing much FaaS prototyping myself but I agree with the author that FaaS beats containers in time to production (even if the difference might be negligible in some cases)
  - disruptiveink 3y ago
    Exactly. The article is stating that there is nothing wrong with the move Prime Video did, and that it was a "serverless to containers" refactoring that somehow the Internet misrepresented as "microservices to monolith" and kicked up viral "even Amazon admits that microservices is overengineering!" meme discussions. Which was obvious to anyone who had read the original post that was not the case, but most people were commenting on what they thought the (incorrect) title meant. I am disappointed that the top comment and all of the discussion is once again not related to the article at hand in the slightest and just focused on the easy "complex is bad, guys!" dunk.
- klabb3 3y ago
  Definitely a real phenomenon. Obscure overengineering also serves another purpose: if you want to build a mini-empire, people leave you alone because they don’t have time to fully grok your design, so they can’t provide meaningful feedback. In these companies the burden-of-proof falls on the critic of a design, not the designer. Simply saying “this looks too complex” is treated as “oh, they couldn’t understand this sophisticated piece of engineering”. This is a cultural issue. If your peers don’t grok it quickly, you should return to the drawing board, or at least provide evidence for the extraordinary circumstances that warrants all that complexity.
retrac98 3y ago
Added to https://www.microservice-stories.com https://www.microservice-stories.com
kierank 3y ago
I do wonder if anyone actually read the Amazon post before creating their hot takes. The crux of the post is simple and obvious to anyone who's ever actually developed "low-level" (used loosely) video libraries. Microservices are not suited for video because exchange of video between microservices is substantially more costly than bits of JSON.
tamiral 3y ago
for a new project that I had to architect, we had a section that was web facing... and it was previously prototyped with React. When i spoke to the team none of them really had React experience but most had AngularJS experience. A lot of the benefits we saw was intermediate to expert knowledge in our stack and it was having an open conversation with the development team + organization to understand it instead of proposing something that's way out in left field!

512 points by mparnisari 3y ago

The points in the Prime Video article still stand. People overuse microservices and serverless and stuff gets expensive quickly.

j45 3y ago
It’s interesting that there are folks who think there must be only one architecture to be the best in all cases. It reeks of inexperience and being highly opinionated followers of popular opinion instead of playing with both personally. So few projects actually scale to benefit from microservices where its a benefit. One’s interpretation and benefit for microservices is fine. It’s just not guaranteed to be factual in many used cases. First, the line of where scaling benefits from microservices is also moving higher each year due to the same underlying reason: Whether Monolith or Microservices, the increased performance of Linux networking, availability of inexpensive ram and cpu along with much more optimized databases like Postgres or mysql from the past 10-15 years ago seems to have been missed by many people who seem to have set an interpretation in place and never looked back. When horsepower gets faster and cheaper each year the architectural best practices of microservices adoption was likely relative to its year of adoption. What to do? Keep your stack unapologetically simple. Don’t handcuff your development speed when the complexity arrives to rearchitect things anyways. Because the first version will have the wrong assumptions about what customers want. How? Consider monorepos to let you start as monolith for velocity and move to microservices as needed Monolithic apps can have a service based architecture internally (monorepos can be pretty handy), which in turn lets startups ship much more regularly than they ought think for monoliths. You can design folders in a monorepo at the start to fuel a monolith and keep a path to go to micro services, where and if needed. To know this enough experience (both good and bad) is needed with both microservices and monoliths.
- groestl 3y ago
  > Monolithic apps can have a service based architecture internally Not only that, but if you absolutely must (for example different scaling requirements), forking out a service from a monolith and deploying it as microservice is relatively straightforward. Provided source code interdependencies have been maintained cleanly before, which can be done with support from tools.
  - tonyarkles 3y ago
    I definitely agree. More than once I've taken a monolith and sliced out a small piece of it to run as a separate service. That's been a significantly less painful task than trying to aggregate multiple microservices into a less expensive monolith, especially when multiple microservices use the same underlying packages but different conflicting versions or different global configuration.
    - j45 3y ago
      The interesting thing about doing this is how things kind of end back at n-tier architecture, where one can have abstractions between each layers of the cake to more easily swap in/out technologies as well.
      - tonyarkles 3y ago
        Yeah, definitely. In my experience, the trick is that without doing analysis up front, you don't know exactly what the n-tiers are going to look like. Jumping right to microservices right off the bat is putting a lot of extra overhead at the start of the project (defining HTTP or gRPC or whatever APIs, working out the boilerplate for each project, etc.) without knowing whether the boundaries are appropriate or whether a given component is likely to end up being a hotspot that will need independent scaling. When you start with a monolith that isn't just a ball of spaghetti (e.g. clear module boundaries), you don't have to put as much work into the whole big picture and it's easier to evolve the internal architecture of one larger service than N smaller services. And if you do discover that one piece of it is appropriate to split off (hotspot, reasonably transactional instead of interactive calling, etc), then you just replace the guts of that module with whatever RPC mechanism you're using and the rest of the application doesn't really need to care about it.

zwischenzug 3y ago

I still don't really get it. There are so many simple, easy to build, cheap to run three tier frameworks out there so why bother with all the hassle of porting at all? If your system gets super busy then you can repurpose the code with higher performance and well tuned DBs later. I spent 15 years doing that for some of the busiest systems in the world, but those started on a single cpu.

gregoriol 3y ago
The point is: managers and keywords
gadflyinyoureye 3y ago
At my client, we go Serverless first in a bastardized state, NestJS. The benefit is quick development at a low cost. We have a series of calculations. Each read from a common S3 and write to it. This allows the teams to do their work with independent deployments. Each one costs about $5 a month. That’s rather hard to get in the AWS EC2 world. They are also easier to manage. We don’t have to manage keys or certain. There is no redeploy after 90s that comes with EC2. However these are low access, slowish apps. Maybe a 1,000 calls a day. They can take a minute to do their thing. Seems like a nice match for lambdas. Benefit of NestJS is if we ever needed to move out of lambda we have a tradition all Express app.
- zwischenzug 3y ago
  Useful insight, but whether it's 5$ or 50$ a month, compared to dev costs that's a rounding error pretty much everywhere.
  - gadflyinyoureye 3y ago
    That's why we went with NestJS. It's like Spring but for TypeScript. We can run everything locally. Boot up the NestJS service at localhost:8080 with LocalStack running on the side for access to S3 and DynamoDB compatible, working services.
- 3y ago

jpgvm 3y ago

Serverless first is a mistake. It should be "serverless where it's distinct advantages are actually useful" which is way less common than people think. Also generally the conception that building things with containers + k8s actually takes more time than serverless is a horseshit take IMO. It doesn't and if it's taking you longer you are probably doing something wrong. All the points noted in the slides etc just come from poor engineering leadership. i.e decisions/choices, etc. Constaints do speed up development but they can be enforced by leadership, not by putting yourself in an endless pit of technical debt. The only world where I can make a good case for serverless for small teams is if you have said poor leadership and serverless helps you escape their poor decision making. Otherwise assuming you have the mandate necessary to get stuff built the right way just stick with the Boring Tech stack of JVM + PG + k8s. It solves all problems regardless of size, there are minimal decisions to be made (really just Spring or Quarkus atm) and you won't spend time dicking around with half-baked serverless integration crap. Instead you can just use battle tested high performance libraries for literally everything. Don't get sucked into this serverless nonsense. It's always been bad and it probably always will be. When it's out of fashion Boring Tech will still be here.

foffoofof 3y ago
Is jvm pg and k8s boring enough? K8s is overkill. Jvm is heavy. Pg is fine but relational dbs generally suck.
- beckingz 3y ago
  elaborate on how relational databases suck?
  - vbezhenar 3y ago
    They're not web scale. https://youtu.be/b2F-DItXtZs https://youtu.be/b2F-DItXtZs
  - foffoofof 3y ago
    Try streaming updates to a query without polling…which is essentially all web apps. It’s fundamentally the wrong approach and people don’t realize. The relational model is good for analytics. Think about how often you need to know the internals or you have to debug perf by trying to get a black box to do what you want. Devs should be a layer deeper.

jack_squat 3y ago

Am I alone in feeling there is nothing left to be said on this topic? Correct application design, and balancing that against the constraints of getting a product out the door, the requirements of a particular application, and the resources on hand (skill-sets, developers, infrastructure) does not in my opinion boil down to "microservices vs monolith". Both strategies have tradeoffs, both have produced successes and failures, and choice of approach seems (obviously) too specific to a given context and application for general advice to apply.

deleted 3y ago
[deleted]
TeeWEE 3y ago
Exactly, tool vs hammer. Sometimes you made the wrong choice in tool, and then you switch tools.. Nothing wrong with that. A craftman just knows its tools better. There is no magic bullet here.
- matisseverduyn 3y ago
  > Exactly, tool vs hammer. Sometimes you made the wrong choice in tool, and then you switch tools.. Nothing wrong with that. A craftsman just knows its tools better. There is no magic bullet here. Rational take, but I see the debate similar to Roman vs Arabic numerals. Keeping a tally? Roman. Need to use operators? Arabic. Sometimes you can keep a tally in Arabic (not ideal), and sometimes you can do basic operations on Roman numerals (not ideal). However, when you want to start using variables, only one tool enables this easily. I can't architect the types of redundant and properly separated interoperable systems with a monolith that microservices otherwise enable. So the desire to move forward isn't the need to find a magic bullet, but the next evolution of an existing ability that unlocks new capabilities... (I don't think calculus would have been discovered using Roman numerals)
  - TeeWEE 3y ago
    No, but you can start a business with a singlesystem, keep things well decoupled (hard / software engineer is hard).. Then you gradually run some parts of the systems in different docker containers, then at some point you completely decoupled systems and have different teams handle different service boundaries. You could still have the core system be a big bigger, and sometimes referred to as monolith. But monolith vs microservices is a polarises the discussion, its a gradual scale. Its the engineering, and keeping things separated that matters.

balls187 3y ago

I don’t quite understand the point in all this. Design tradeoffs are still a thing; did we all just forget? “Abortions for some... miniature American flags for others!”

mattgreenrocks 3y ago
Software architecture is (incorrectly) seen as NP-hard by practitioners, so they avoid it, and instead focus on implementing the current "best practices." However, "best practices" can't be best if they're situational, so we arrive at very local maxima, such as microservices for all.

foffoofof 3y ago

The whole industry is stupid at this point. I don’t think anyone knows what they are doing. Everyone just wants to play with the new shiny thing. Half the engineers probably have ADHD which leads to this behavior of seeking novelty. Everyone is obsessed with their shitty stack they used at their last company, and this shit will keep going on and on. The only things that matters is how easy is something to debug and to change. Think about how often you have heard that as any kind of priority. So many shitty decisions would be avoided if people started thinking about that.

zwischenzug 3y ago
I've often mused about creating a language that put readability, maintainability, logging etc over, say, performance.
- Arbortheus 3y ago
  You might enjoy Go. Lots of emphasis has been put into the development tooling and ecosystem. There is an impressive commitment to avoiding breaking changes. We have a legacy Go microservices codebase that’s over a decade old and because the Go maintainers have paid so much attention to avoiding creating breaking changes and a guarantee to never create a “Go 2”, the codebase has aged much better than stuff written at the same time in Python (some of which is still in Python 2 because it is too risky to upgrade, and that functionality will be killed instead because we don’t know how to safely update it). Most of the time we don’t need to make any changes when bumping Go versions, just fixing up things our linter warns is deprecated.
- Nextgrid 3y ago
  Performance is absolutely a non-issue when modern "engineering" insists on running everything on underpowered VMs with high-latency network-based storage. Even if your language is truly, actually slow, you can get an instant performance boost just moving that workload to a modern bare-metal server (cloud providers basically ate all the performance improvements of modern hardware), or scale that (most likely stateless) workload horizontally if a single bare-metal machine isn't enough.

nforgerit 3y ago

I'm starting to be convinced that the EU should regulate acceptable software design patterns. This will prevent us all from heaving to implement techbro cargo cult pseudo-innovations like "Serverless for everything" /s

ipaddr 3y ago
EU should stop regulating what is acceptable and let society naturally come to conclusions. Limiting software designs seems like an area the government shouldn't be involved with. Regulate cloud providers if necessary not abstract patterns

lopkeny12ko 3y ago

The original, unedited title is "So many bad takes — What is there to learn from the Prime Video microservices to monolith story"

mellosouls 3y ago

The nth posting of this story. https://hn.algolia.com/?dateRange=pastWeek&page=0&prefix=false&query=Primevideo&sort=byDate&type=story https://hn.algolia.com/?dateRange=pastWeek&page=0&prefix=fal...

qwertywert_ 3y ago

This has been posted like 6 times already man.

iamleppert 3y ago

If you have a predictable workload (i.e. we ingest 100,000 videos a month with n size) you should be looking at it from that perspective -- how much compute you are going to need, and when. Serverless works well for predictably unpredictable, infrequent workloads that don't need to have perfect output ("good enough"). The big mistake I see people make is trying to be overly clever and predict the future workload needs. It never works out like that. Design for your current workload now, if your architecture can handle 10x of that great! Each time your scale from 10x workload you will likely need to redesign the system anyway, so you shouldn't pay that tax unless you absolutely have to. There are a lot of limitations of serverless, the big one I experienced was inability to control the host environment and limitations on the slice of memory/CPU you get, such that you must take that into consideration when designing your atomic work units. Also paying the distributed computing tax is real and occurs at development and runtime -- things like job tracking and monitoring are important when you have 10,000 processes running -- you start to get into the territory of things that basically never happen on a single machine or small cluster, but become problems with thousands of disparate machines / processes.

goostavos 3y ago
>Design for your current workload Please be my friend. The bulk of my job these days is sitting in design reviews trying to convince people that just making up scenarios where you need to scale to 100x isn't actually engineering. It's exhausting self-indulgence. Nothing is easier than inventing scenarios where you'll need to "scale". It's borderline impossible to get software folks to just live in the world that exists in front of their eyes. "Scale" should go back to its original meaning: change relative to some starting point. Slap a trend line on what you know. Do some estimation for what you don't. Design accordingly.
- jpgvm 3y ago
  Yeah unless you are designing the replacement for a system that has already reached it's scalability limits you shouldn't be worrying too much other than not doing very silly things architecturally. When you are designing replacements though you need to have an idea of what your scalability runway is before the next re-design will be possible. Sometimes that is just 2x, often times it's 10x, occasionally it's 100x but it's all situational.

mr_tristan 3y ago

What's funny to me, is that I focused on something completely different; how independent these teams at Amazon are. The move from pretty different infrastructure services, all at the same company. I can't think of a single place I've worked at in 20 years where these infra teams wouldn't be either: a.) mandated, i.e., "you will stick with lambda/kubernetes/whatever", or b.) review and be involved in the architectural decisionmaking of the video service approach, i.e., have to politick your way around any deviation. I guess the shift from "mono to micro" just isn't very interesting to me. You can usually change your definition of either concept to fit your architecture. This just seemed like the team did the math and revised their approach. Good for them!

iLoveOncall 3y ago
This is in part where the "It's always Day 1" mentality manifests at Amazon. There's only one programming language that is forbidden (PHP - for wrong reasons), and only one cloud vendor that is mandated (obvious). But beside that, teams are able to use literally whatever technology they want. There are some technologies that impose themselves because of the ecosystem, but you can always do your own sauce if needed. I've been in two teams that had basically 90% different tech stacks, but it's never a problem, and it doesn't really ever come up in design reviews (unless there's a real reason, not for personal preferences).
- jpgvm 3y ago
  Curious though, what is the reason for banning PHP? It must be pretty specific if they singled out just PHP.
  - iLoveOncall 3y ago
    I don't know the full history because it's from way before I joined, but it is honestly a terrible take (by a guy who's now Distinguished Engineer at Apple, no less!) explaining that it is impossible to write secure applications in PHP. Basically, they used "the historical number of vulnerabilities identified in applications developed using a technology" as metric to determine how insecure a technology is. For the argument here, they looked at the CVE list, where at the time (in 2012, with the list existing since 1999) 40% of all software vulnerabilities recorded were in PHP applications. This led to the conclusion that PHP is insecure by nature. Of course, he didn't mention that at the time, PHP was also used by 80% of all websites, because that would have made his argument worthless. That wiki page explaining that is still up. It's so baffling to me when the argument violates so many of Amazon's leadership principles.

ThalesX 3y ago

I completed my role as a CTO for a company several years ago. I chose a boring stack with plenty of developers available: MySQL, PHP with Laravel, and Angular 2. Since then, I've been a founding engineer at some startups using the latest, cutting-edge tech stacks. I'm proud to announce that the stack I managed as CTO is still running stable and smooth with minimal intervention, and it remains easy to modify. Meanwhile, the startups with flashy stacks and nearly impossible-to-debug architectures have gone out of business. It amazes me how some companies still adopt the "build it and they will come" approach, and when confronted with harsh reality, they double down on building (and especially increasing architectural complexity). CEOs, if you can't attract customers with an idea or don't have a vertical that people are eager to pay for, cramming 1000 half-baked features into a microservice architecture and using the latest programming language / paradigm won't save your company!

goodrubyist 3y ago
Were those startups that went out of business did so because of their stacks/architecture, or are you confusing correlation with causation? And, there is a good reason people shy away from PHP, and it has nothing to do with trying to be "flashy." There should be a name for this kind of fallacy.
- iLoveOncall 3y ago
  > there is a good reason people shy away from PHP Please share it. And don't rely on articles that are more than a decade and 3 or 4 versions of PHP old.
  - 5Qn8mNbc2FNCiVV 3y ago
    Afaik you cannot discern null and undefined. So, not there and "null"? Either that or I'm going to have to have a good talk with my colleague
    - iLoveOncall 3y ago
      Go have your talk [1]. But that's a pretty weak argument anyway given that: 1. Other languages also do not have this distinction (Python for example AFAIK), 2. It's relatively useless. I get the value of it in JavaScript which has a lot of asynchronous code and frontend where the behavior may indeed be very different between null and undefined, but in a server-side application I don't think the nuance is necessary at runtime, and potential programming mistakes involving that can be caught by a linter. [1] https://stackoverflow.com/questions/6905491/how-to-tell-whether-a-variable-is-null-or-undefined-in-php https://stackoverflow.com/questions/6905491/how-to-tell-whet...

enterthestartup 3y ago

Very interesting that they were able to reduce costs by 90%, though also agree with the opinions mentioned here.

amluto 3y ago

> they had some overly chatty calls between AWS lambda functions and S3. They were able to re-use most of their working code by combining it into a single long running microservice that is horizontally scaled using ECS, and which is invoked via a lambda function. This is only one of many microservices that make up the Prime Video application. The problem is that they called this refactoring a microservice to monolith transition, when it’s clearly a microservice refactoring step If I understood the post at all, I disagree. One can spend hundreds of HN comments discussing technology stacks, monoliths, etc, and this is important: it affects maintainability, developer hours, and money spent on orchestration. For some applications, almost the entire workload is orchestration, and this discussion makes sense. But for this workload, actual work is being done, and it can be quantified and priced. So let’s get to the basics: an uncompressed 1080p video stream is something like 3 Gbps. It costs a certain amount of CPU to decompress. It costs a certain amount of CPU to analyze. These have price tags. You can pay for that CPU at EC2 rates or on-prem rates or Lambda rates or whatever. You can calculate this! And you pay for that bandwidth. You can pay for 3Gbps per stream using S3: you pay S3 and you pay EC2 (or whatever) because that uses 1/10th of the practical maximum EC2 <-> S3 bandwidth per instance. Or you pay for the fraction of main memory bandwidth used if you decode and analyze on the same EC2 instance (or Lambda function or whatever). Or you pay for the fraction of cache bandwidth used if you decide partial frames and analyze without ever sending to memory. And you’ll find that S3 is not “chatty”. It’s catastrophic. In fact, any use of microservices here is likely catastrophic if not carefully bound to the same machine.

jayd16 3y ago
So it didn't make sense to separate some functionality. So what? No one who advocates microservices would say you can cut a service along any arbitrary line and expect good results.
jiggawatts 3y ago
This is called mechanical sympathy and most architects and developers do not have it. Computer architectures encompass 13 orders of magnitude of performance! That's roughly the difference between something like a trivial function call processing data in L1 cache to a remote call out to something in another region. People often make relatively "small" mistakes of 3 or 4 orders of magnitude, which is still crazy if you think about it, but that's considered to be a minor sin relative to making the architecture diagram look pretty.

joanne123 3y ago

[dead]

jbverschoor 3y ago

Cant' we just ask the "Application Deployer" to upload an EAR, published by the "Application Assembler", and run it on any of the spec-compatible servers? These problems have been solved decades ago.. 2.5 to be exact

th3h4mm3r 3y ago

What if Amazon take bad choices in "something" and my company is tied up with its services? I know that Prime is always "Amazon" and that's okay. But for a startup (but not only) imho it's better starts with something more "standard" than something as serverless that is obviously an abstraction above other tecnologies.

AndrewKemendo 3y ago

Look, the point is simple, AWS marketing says you can do everything all the time with microservices and their 'DevOps' infrastructure to "abstract away" all the complex you know - engineering - that you cannot automate effectively and have a long running robust system. So it's humorous to see an Amazon team that is doing things correct and holistically in contrast to the sales and marketing bullshit that AWS and Cloud evangelists spew. Us nerds have a great way of missing the point and bikeshedding on the technicals (which is also great sometimes!)

namaria 3y ago
It's amazing how cloud companies managed to convince so many people in software engineering that software engineering wasn't a core competence in producing software.
- AndrewKemendo 3y ago
  Yeah cause engineers aren’t ones typically doing the buying
  - intelVISA 3y ago
    Same reason MSFT SWEs use Mac over Windows -- they've seen too much.
- atulatul 3y ago
  I think the difference was simple vs easy. They made a few of the things very easy.
scarface74 3y ago
> Look, the point is simple, AWS marketing says you can do everything all the time with microservices and their 'DevOps' infrastructure to "abstract away" all the complex you know - engineering - that you cannot automate effectively and have a long running robust system. As someone who knows a little bit about what AWS recommends to customers when first hand, I can absolutely guarantee you that AWS does not recommend serverless to everyone. The vast majority of projects that consultants do by revenue at any cloud consulting company (including AWS’s own internal company) is regular old VMs.

bhauer 3y ago

Obviously, the article is microservice apologia, but... > They were able to re-use most of their working code by combining it into a single long running microservice that is horizontally scaled using ECS... No, it was no longer a microservice; it became a plain service, as in SOA. It was no longer micro. That's the whole point. They could have saved time and money had they just made the thing a plain service to begin with. This thing was not sufficiently complicated to warrant the complexity of starting with a bunch of microservices. The article says many hot takes have missed the point, but I think what we're seeing here is an example of the two sides talking past one another since the author hasn't appreciated the opposition's arguments at all.

m_mueller 3y ago
One thing to note is that the transition from smaller to larger services tends to be straight forward, while cutting into smaller can be tricky. Thus IMO there is some merit to keeping them small in the beginning until you can analyse them under production workloads. That being said, in this case here I think some very simple performance / cost modelling would have shown the issues with serverless already in the beginning. I do find serverless architecture useful, but not for such a case with heavy base load. Furthermore, data locality is also an important aspect to consider in anything with strong latency or throughput requirements - serverless or not.
- mirekrusin 3y ago
  It's the other way around - start with monolith (because it's easier to change things, it's just single pr addressing all places at once) and then, possibly few years later, whatever has crystalized and has clear boundaries with little to no changes coming in or changes contained within this boundary - can be potentially extracted. Just look how over-represented RoR is/was as bootstrap tech in known, successful companies. Microservices is not yes/no - it's a slider. You may find sweet spot at ie. 50/50 split like ie. github does, have more or less microservices while keeping core in monolith or services under single versioned monorepo. Microservices are good for satellite services like system integration, pre/post-processing, gateways etc. As a rule of thumb whatever can fit single (tech lead + team)'s "head" - can be monolithic (single monolith or set of services under single versioned monorepo managed by that team). Their job is to provide stable apis/uis - otherwise it can be seen as black box by other teams. This is natural way things evolve (teams are created around naturally bounded concepts) and the suggestion is - don't break it by creating mismatches, keep it in harmony. If something creates measurable problems, slowly form team around it with new tech lead from existing team and let it grow on it's own - it'll extract itself to separate subsystem by itself. It doesn't have to be done overnight. It's astounding how many people use "scale" as chupacabra to scare everybody in the meeting room without clearly defining what they mean by "scale". To make decision around changing architecture to scale you need to precisely define what it means, have metrics, have benchmarks showing current limits, proof it's a problem now or near future and focus just on those actual issues, if they even exist.

EMM_386 3y ago

> I do think microservices were over sold as the answer to everything, and I think this may have arisen from vendors who wanted to sell Kubernetes with a simple marketing message that enterprises needed to modernize by using Kubernetes to do cloud native microservices for everything Yes, they absolutely were oversold, Kubernetes marketing was part of it. There was also a lot of resume-driven development, and simply "look, new shiny object!". A lot of young engineers entered the workforce over the last couple of decades. That often comes along with "forget this old stuff, what's the new stuff?". I use KISS as much as possible with every project I'm involved in desinging. The microservices push got the point where a dev shops with 10 people had it all broken into microservices and running Kubernetes to manage low-traffic front ends, an API and a database. I would see some of these systems and the only thing I could think was "... but why?". Do microservices have a place? Yes. Have I seen them used where they shouldn't be? Yes. Have I seen more poorly implemented microservices architecture than good ones. Yes. Over-engineering the tech stack, or selecting one part of it vs. another based on which is trendier, only leads to more frusturation, difficulty reasoning about it, added development time ... all of which means it will cost more.

t0b1 3y ago
That reminds me of a somewhat funny story. I was listening to a presentation about a startup with about 10 people as you mentioned doing hospital digitalization (gist of it seemed to be sending patient data from one doctor to another). They explained that they wanted to choose a “robust” architecture so they chose microservices (kubernetes and all). I thought that was a bit odd for something hosted at a hospital locally especially because they had like 7 or so services (and maybe even two databases but not sure on that one). Well, we later asked them how much data they were even handling and (after a bit of side-stepping) they said it was around 200mb excluding images. That was my “but why?” moment. Apparently you need 7 services to have a web frontend for 200mb of data with most of it probably never accessed and hospitals can suddenly get three new floors overnight.

bravoteam5 3y ago

[dead]

lr1970 3y ago

Dup. Already discussed on HN less than a day ago [0]. [0] https://news.ycombinator.com/item?id=35847988 https://news.ycombinator.com/item?id=35847988

jlund-molfese 3y ago
No, it's a different link. The one in this topic is addressing the link you posted.

kennu 3y ago

Many comments seem to miss the point that nowadays native AWS cloud applications are meant to be built by combining various AWS services using Lambda as the glue between them. It's not about serverless vs non-serverless: it's about using cloud services for the purpose they are each good at. In many cases you might actually have more IaC code than Lambda handler code, especially if you can leverage services like AppSync and Step Functions. The application is no longer a set of microservices; it's a set of CDK / CloudFormation stacks and the resources that they define, some of which might be microservices.

bdcravens 3y ago

Serverless isn't "cheap". It's rightsized. If you aren't serving 24/7, it makes sense to move to an on-demand model. However if you compare the unit of compute pricing, Lambda is comparable to other services, and likely more expensive than buying compute wholesale. You can just buy it in smaller portions.

getcrunk 3y ago

Seeing as how a lot of players are going in on serverless (vercel, netlify, deno, cloudflare) the cynic in me wonders if this is a calculated move on part of Amazon to bring people back to using ec2 .. etc

distantsounds 3y ago

microservices? to sling video files over the internet? tell me you're over-engineering without telling me you're over-engineering

mlhpdx 3y ago

I am so baffled by this whole discussion. Isn’t serverless (aka event-driven distributed systems made up of small components, with no reserved capacity) just one of many options? I write code for 8-bit controllers that I’d never run on Lambda (even though I’m a Lambda/Step Functions freak). I need 90s server software that can only run in a VM. It’s a complicated world and your favorite tool/model just isn’t going to cut it everywhere. Move on maybe?

andy_ppp 3y ago

Nothing is ever truly a mistake, it's much more about understanding what you are trading off between real hardware, cloud servers, something like fly or heroku or vercel and purer serverless functions on AWS or whatever.

GabeIsko 3y ago

Yeah, I don't really understand why people are making a big deal about this. Two lambda functions passing data to step functions? Not exactly the best serverless architecture. And if you think about the problem of checking video streams, there is really no reason to scale different parts of it independently because you just need one process to watch the video and check for errors. Plus they still deployed the new solution on ECS, so it isn't really a monolith application that runs on a fixed number of servers. It's a real nothing story - the Prime Video team was just doing some engineering maintenance. Honestly the bigger story is that it was mischaracterized from the beginning, but I would imagine that is more a result of things coming together from multiple different systems and teams more than the way someone designed this stuff.

chrismsimpson 3y ago

Seeing all the acolytes of serverless get triggered by this has been enjoyable to watch

carabiner 3y ago

I think Agile workflow has a part to play in this. It embraces complexity and moving fast to build out buggy features, where every new bug becomes a new user story. It feels productive but it just creates more work than you need.

hnaccountme 3y ago

I'm surprised Amazon published what they did. AWS make a lot of money selling hyped up buzz word tech to incompetent companies.

aka878 3y ago
Prime Video, Lambda, EC2, everything in discussions still revolves around Amazon.

vaidhy 3y ago

I see a lot of about architecture, service design and complexity. One of the key missing piece is around Amazon culture. Promo-driven design is a real deal at Amazon. I am sure this design got someone promoted to principal engineer or higher. If you cannot build complexity into the design, how can you prove your caliber and get promoted.

theK 3y ago
The article is more about bundling together lambda functions into horizontally scalable containers. Paraphrasing the author: > I think the prime video team's presentation should have been called "Moving from lambda to container microservices" There is nothing particularly bad about this move. Prototype something in a FaaS, move it to a container based microservices when you know which features you want and how they perform. Not doing much FaaS prototyping myself but I agree with the author that FaaS beats containers in time to production (even if the difference might be negligible in some cases)
- disruptiveink 3y ago
  Exactly. The article is stating that there is nothing wrong with the move Prime Video did, and that it was a "serverless to containers" refactoring that somehow the Internet misrepresented as "microservices to monolith" and kicked up viral "even Amazon admits that microservices is overengineering!" meme discussions. Which was obvious to anyone who had read the original post that was not the case, but most people were commenting on what they thought the (incorrect) title meant. I am disappointed that the top comment and all of the discussion is once again not related to the article at hand in the slightest and just focused on the easy "complex is bad, guys!" dunk.
klabb3 3y ago
Definitely a real phenomenon. Obscure overengineering also serves another purpose: if you want to build a mini-empire, people leave you alone because they don’t have time to fully grok your design, so they can’t provide meaningful feedback. In these companies the burden-of-proof falls on the critic of a design, not the designer. Simply saying “this looks too complex” is treated as “oh, they couldn’t understand this sophisticated piece of engineering”. This is a cultural issue. If your peers don’t grok it quickly, you should return to the drawing board, or at least provide evidence for the extraordinary circumstances that warrants all that complexity.

retrac98 3y ago

Added to https://www.microservice-stories.com https://www.microservice-stories.com

kierank 3y ago

I do wonder if anyone actually read the Amazon post before creating their hot takes. The crux of the post is simple and obvious to anyone who's ever actually developed "low-level" (used loosely) video libraries. Microservices are not suited for video because exchange of video between microservices is substantially more costly than bits of JSON.

tamiral 3y ago

for a new project that I had to architect, we had a section that was web facing... and it was previously prototyped with React. When i spoke to the team none of them really had React experience but most had AngularJS experience. A lot of the benefits we saw was intermediate to expert knowledge in our stack and it was having an open conversation with the development team + organization to understand it instead of proposing something that's way out in left field!