8 ms·

Why intuitive troubleshooting has stopped working

71 points by kiyanwang 4y ago

aaronbrethorst 4y ago
The architecture of our systems has gotten a lot more sophisticated and complex over the past 20 years. We’re not running monoliths on a few beefy servers these days. I am running a monolith, and troubleshooting is still pretty easy. We’re operating distributed microservice ecosystems on top of a deep stack of frameworks, abstractions and runtimes that are all running on other people’s servers (aka 'the cloud'). Yikes, good luck. Snark aside, I just want to remind most people out there: you do not work for Google or Facebook, and your systems do not have to be architected as if you are the next Google or Facebook. Neither of these companies started existence with microservice architectures or whatever. Save yourself the pain. Build a product. Find some customers. Worry about the architecture astronautery later.
- johnthescott 4y ago
  i recall reading ken thompson's reply to what he thought of virtual machines. something like "they are such a good idea that i wrote unix".
- adrianN 4y ago
  Even without architecture astronautics you still run on a very deep stack of frameworks and abstractions that make troubleshooting difficult.
  - aaronbrethorst 4y ago
    My turtle stack is pretty well documented and 95%+ of problems have already been solved by other people. For everything else, I'm quite happy with https://sentry.io https://sentry.io and https://www.skylight.io https://www.skylight.io for my troubleshooting needs.
  - Gordonjcp 4y ago
    How do you know that?
    - adrianN 4y ago
      Because even if you run bare metal on a raspberry pi you rely on fairly complex binary blobs. Most software is too complicated for such a simple stack.
      - anonymoushn 4y ago
        You mean the Linux kernel and device drivers? Most of the time these things cause fewer problems than other things.
      - Gordonjcp 4y ago
        That seems a fairly stupid line to draw, to be honest. Do you know how the silicon your code runs on is made? Do you know how the fibreglass the CPU chip is stuck to is made? How do they get the copper to stick?
  - deleted 4y ago
    [deleted]
- QuadmasterXLII 4y ago
  Doctor, doctor, every time I touch my left knee to my right pinky I get a stabbing pain in my heart! Don’t worry, I know a physical therapy regime that will fix you right up: no medicines or scans required Wonderful! what is it? Every day before bed, don’t touch your left knee to your right pinky
- danwee 4y ago
  The problem is that if a company advertises a job that is all about maintaing one or two monolithic applications without K8s, distributed system topics, tracing, microservices, AWS/GCP, etc., well, then not many engineers will apply. That's actually a paradox in itself: my bet is that only a few group of (good) engineers would apply to such a job. Management otherwise usually thinks that the bigger the pool of applicants, the better for the company. Also, a lot of engineers out there are resume-driven: they do want to work with microservices, the cloud and k8s. It's a vicious circle.
  - raxxorraxor 4y ago
    > then not many engineers will apply I would. I don't run server farms and don't need to provision entire environments so K8s would be extreme overkill for anything I do. Docker makes sense to formally define system requirements but even that I use sparingly. I develop embedded systems though and usually only trace coffee machines. I do host stuff on AWS and gladly use all the services they provide. Neat, easy and less maintenance, but the most important thing for me personally is that I work with an account that doesn't use my own credit card. I can understand the applicants though, there is a fear to commit to solutions where any experience might not increase their market value. But from my experience that fear is highly exaggerated. There are people that reject candidates on these accounts but usually not the good one that know they need to invest in engineers. There are also quite a few bosses that are only interested in the newest tech. They have absolutely no idea what that means, but they will buy it. And a lot of cloud software hits the sweet spot for them.
    - tluyben2 4y ago
      > > then not many engineers will apply I would. Like you said ; one point is probably market value and another future relevance. I have had discussions with hiring managers lately and almost none that I talked to actually use most or any of the buzzwords they put in their ads. They just say they do to get more applications.
  - mwcampbell 4y ago
    How broadly is this true? Some of us developers do want to focus on things that are useful to people, that we can describe to non-tech people we meet in public. Are those of us who run bootstrapped, all-remote SaaS companies going to have trouble hiring?
- xixixao 4y ago
  Neither Facebook nor Google are built on microservices in this fashion. Amazon is (but it’s on its own servers).
- _8j50 4y ago
  Yes, but the sooner you build to scale the better. More costly later on.
  - wilde 4y ago
    Is it? You trade retrofit costs later for maintenance costs and feature development friction every day until scale is needed.
    - _8j50 4y ago
      You make a very good point.
  - VLM 4y ago
    Scalability is a two way street and those services don't scale small. That makes business opportunities for more flexible engineers.
  - tluyben2 4y ago
    Chances are quite high you will never need scale. If you knew 100% sure you needed it when building, you would build for scale from day 1. But most/all startups and also for many projects in large corps, they don't know if they need scale. And startup statistics definitely show it is a very high chance you don't need it ever because you will be out of business long before that. It's mostly a waste of effort, time and money.
- johnthescott 4y ago
  > astronautery gonna borrow that phrase.
  - twic 4y ago
    Surely "astronautics" is the right word though?
5350-uiop-1130 4y ago
It's odd that over time things don't get simpler, instead everything trends towards more and more layered complexity - where the higher layers are abstracted so as to appear simpler.
- HPsquared 4y ago
  Such is life (that is, biological evolution worked the same way).
  - high_5 4y ago
    Life implies death. When will these systems implode under their own complexity?
    - api 4y ago
      This is a well known cycle in software. Systems grow in complexity until they collapse under their own weight, then a new system is created that steals the most valuable ideas from the old system. Repeat. It happens in nature too. Your skull has fewer bones in it than the skulls of many fish and humans do not by any stretch have the largest genome in nature.
  - rightbyte 4y ago
    Zealous creationism in that case.
  - mwcampbell 4y ago
    But unlike nature, we can do intelligent design.
    - arein3 4y ago
      How is the evolutionary design not intelligent?
      - quickthrower2 4y ago
pontifier 4y ago
In the sci-fi future such as star trek, you often see systems being reprogrammed by swapping crystals around or something like that. I used to think it was really stupid, but as I deal with trying to build complicated systems I find myself wishing I could build something like that to set up systems and have them just work. Interestingly, 90's industrial automation sort of feels like it was trying to build systems like that. The control cabinets are filled with simple, discrete, computational blocks that control valves and actuators based on sensor inputs. Getting back to the sci-fi, I realized that the crystal swapping could just be the high level interface to all the complex code below. They really wouldn't need to do anything except light up when they are detected correctly, but still control complex code hiding below the surface... Maybe written on demand by AI that reads the crystals as prompts. I don't know where this is all heading, but maybe someday swapping databases will be as complicated as changing a lightbulb.
- valenterry 4y ago
  Not possible. Imagine the number of scenarios that require fixes. There are just too many of them. So you either end up with a gigantic amount of crystals (but still can't solve a problem in some exceptional cases) or you have a small number of crystals but a high number of situations where you can't resolve the problem with them. So maybe those crystals are more like app-configurations that we already have: you can tune things in a predictable way, but as soon as it goes beyond that you need to change the code.
- AndyMcConachie 4y ago
  Imagine 100 years from now when humanity's population has been reduced by 100s of millions because of climate catastrophe. Now imagine all the software society relies upon. Now imagine being one of the few people left alive being capable of working on that software. You probably won't have source code. One thing that has remained constant in our lives is the number of people who are capabable of working on complex software has steadily increased. What happens if that number suddenly decreases?
  - Ekaros 4y ago
    Now I started to think what is the actual ratio of software we need and software that is produced or distributed.
api 4y ago
The real solution to this is how we solved it last time we were here: abandon the mainframe and go back to local applications on PCs. The cloud really provides two things for most applications: (1) Data synchronization at scale. This could be provided by a much, much thinner cloud that basically just syncs files but using CRDTs to make that more robust. Applications could use these services to sync state for collaboration. The vast majority of the brains could be at the endpoint. (2) The cloud is copy protection and DRM. It gives software publishers an easy place to put a toll booth. This could be solved by a reboot of the concept of local licensing, or by just periodically calling out to a license server. Yes people will pirate your software but those people aren't going to pay anyway.
- mwcampbell 4y ago
  If I'm not mistaken, you overlooked another role of cloud services: coordination of real-time connections between endpoints, and in some cases actual relaying of data to work around NAT. I guess this is the space you're in with ZeroTier. But I'm not sure how much of a market there is for generalized solutions like ZeroTier and Tailscale. For the most part, people want solutions that connect their machines for a specific purpose, such as remote desktop access (e.g. TeamViewer/AnyDesk), and developers of such products don't want to require their users to have another networking product (ZeroTier/Tailscale) first.
- spion 4y ago
  Not sure this works. For example, how do you implement search? I'm not looking forward to having to download the entire search index catalog of a web store in order to find out whether they have something.
  - quickthrower2 4y ago
    Maybe not run everything locally, but most “web apps” could be local apps with something like dropbox to back up the data and make it accessible. The issue with this is the web is zero install cross platform and it is hard to compete with that.
  - api 4y ago
szundi 4y ago
Please someone tell me wether it is at all possible to deliver very complex/sophisticated b2b stuff via cloud stuff. For me it seems to be a nightmare to debug complex stuff, but without the “old” tooling it seems to me that it is quite impossible to do this when buried behind queues, pubsubs, cloud functions and all that. Any comment on this is quite welcome.
brianmcc 4y ago
I hate headlines like this pieces's full one - omitted here on HN so far - which is "Why Intuitive Troubleshooting Has Stopped Working For You". For me? Excuse me no, because I design systems to be supportable in production. Tsk. "For some"? "For many"? Sure! But please don't annoy me before I even start reading with this kind of headline.
- zamfi 4y ago
  Why the hate? This headline makes the audience clear, and it’s not you! Enjoy the freedom this gives you to skip reading the post.
  - brianmcc 4y ago
    Ha fair enough a strong term. Forgive me I am just venting a bit about a wider pattern of these sort of headlines: "Why YOU aren't saving enough for retirement", "Why YOU need to eat fewer avocados", etc. It's not just an audience indicator it's a challenge designed to engage, but it just winds me up. A stylistic technique I really dislike, that's all! :-)
    - bombcar 4y ago
      A/B testing has destroyed headlines and I hate it.
      - zamfi 4y ago
        Yeah. But, fortunately, it's a dynamic system! Learn to ignore the headlines that are written this way, because the noise is high. Some folks have to use ad blockers to avoid seeing ads, while others train themselves to just ignore ads entirely. Personally, I don't even see them most of the time. If you're in the latter camp, training yourself to ignore headlines like these doesn't seem much harder!
RedShift1 4y ago
Regardless of complexity: error messages, error messages, error messages. I can't emphasize enough how important decent error messages are. When you are having trouble with a file, include the full path in the error message. Put in the name of the function in the error message, what you expected and what you got. Bubble up lower level errors, don't swallow them. Include a stack trace, include the path of the log file if applicable. Please don't have your error message say "Something went wrong".
- enlyth 4y ago
  Agreed, a few times I've found myself in the unfortunate position of needing to push changes to production just to be able to diagnose an error, because it was being completely swallowed by a try-catch block with some generic "something went wrong" message.
nicbou 4y ago
I went to a serverless conference some time ago (great excuse to travel on the company's dime), and every single talk had a few minutes about how they were unable to test or debug their production infrastructure. They couldn't even run their code locally, just bits of it.
- quickthrower2 4y ago
  That seems odd to me. If the service can’t run or emulate locally then I wouldn’t use it.
nicbou 4y ago
I went to a serverless conference some time ago (great excuse to travel on the company's dime), and every single talk had a few minutes about how they were unable to test or debug their production infrastructure. They couldn't even run their code locally, just bits of it. Serverless code could not run without a server. A specific provider's servers. Call me old-fashioned, but I develop with a few docker containers locally, then release them to a plain old linux server, and it's working great. Sure, my humble business isn't Google, but neither are yours, right?
- ratww 4y ago
  I went to one too, and saw a talk about synchronising data between microservices and was astounded at how much work it was and the fact they had a whole team just to build tooling to debug issues on this specific company. Basically they store 100x more data than they need because of data duplication between microservices. The company didn't even have that much customers (<1000), data or even enough users (a handful per company), but duplication was still generating several terabytes. It's amazing how much complexity people are able to cram into simple system. (Interestingly enough there's another recent comment on another article here talking about this exact issue: https://news.ycombinator.com/item?id=32320660 https://news.ycombinator.com/item?id=32320660 )
- itsmemattchung 4y ago
  Agree that being unable to test serverless code locally is begging for pain. While not perfect, testing AWS Lambda has improved over the years. I recently deployed a Lambda function only because I was able to reproduce the environment locally, using their Lambda Runtime Interface Emulator (RIE): https://docs.aws.amazon.com/lambda/latest/dg/images-test.html https://docs.aws.amazon.com/lambda/latest/dg/images-test.htm...
  - lincler 4y ago
    Until you need to connect to another service that can not be run locally and it is only accesible within the same Cloud Provider. Sure it's not impossible to go around that, start mocking or run some alternative locally for debugging but I don't think this extra level of complexity is necessary (at least not always).
ianbooker 4y ago
I often feel that for example Python web stacks are overly reliant on conventions. Your work is expected to follow nested conventions, and if you do so, things work. Yet, some conventions imported from other underlying packages are not documented in the current package, so if you are new to the stack you are lost immediatly.
drewcoo 4y ago
tldr: Loose coupling means no intuitive troubleshooting. Which you can infer from the diagrams but isn't stated directly in this marketing puff piece.

71 points by kiyanwang 4y ago

The architecture of our systems has gotten a lot more sophisticated and complex over the past 20 years. We’re not running monoliths on a few beefy servers these days. I am running a monolith, and troubleshooting is still pretty easy. We’re operating distributed microservice ecosystems on top of a deep stack of frameworks, abstractions and runtimes that are all running on other people’s servers (aka 'the cloud'). Yikes, good luck. Snark aside, I just want to remind most people out there: you do not work for Google or Facebook, and your systems do not have to be architected as if you are the next Google or Facebook. Neither of these companies started existence with microservice architectures or whatever. Save yourself the pain. Build a product. Find some customers. Worry about the architecture astronautery later.

johnthescott 4y ago
i recall reading ken thompson's reply to what he thought of virtual machines. something like "they are such a good idea that i wrote unix".
adrianN 4y ago
Even without architecture astronautics you still run on a very deep stack of frameworks and abstractions that make troubleshooting difficult.
- aaronbrethorst 4y ago
  My turtle stack is pretty well documented and 95%+ of problems have already been solved by other people. For everything else, I'm quite happy with https://sentry.io https://sentry.io and https://www.skylight.io https://www.skylight.io for my troubleshooting needs.
- Gordonjcp 4y ago
  How do you know that?
  - adrianN 4y ago
    Because even if you run bare metal on a raspberry pi you rely on fairly complex binary blobs. Most software is too complicated for such a simple stack.
    - anonymoushn 4y ago
      You mean the Linux kernel and device drivers? Most of the time these things cause fewer problems than other things.
    - Gordonjcp 4y ago
      That seems a fairly stupid line to draw, to be honest. Do you know how the silicon your code runs on is made? Do you know how the fibreglass the CPU chip is stuck to is made? How do they get the copper to stick?
- deleted 4y ago
  [deleted]
QuadmasterXLII 4y ago
Doctor, doctor, every time I touch my left knee to my right pinky I get a stabbing pain in my heart! Don’t worry, I know a physical therapy regime that will fix you right up: no medicines or scans required Wonderful! what is it? Every day before bed, don’t touch your left knee to your right pinky
danwee 4y ago
The problem is that if a company advertises a job that is all about maintaing one or two monolithic applications without K8s, distributed system topics, tracing, microservices, AWS/GCP, etc., well, then not many engineers will apply. That's actually a paradox in itself: my bet is that only a few group of (good) engineers would apply to such a job. Management otherwise usually thinks that the bigger the pool of applicants, the better for the company. Also, a lot of engineers out there are resume-driven: they do want to work with microservices, the cloud and k8s. It's a vicious circle.
- raxxorraxor 4y ago
  > then not many engineers will apply I would. I don't run server farms and don't need to provision entire environments so K8s would be extreme overkill for anything I do. Docker makes sense to formally define system requirements but even that I use sparingly. I develop embedded systems though and usually only trace coffee machines. I do host stuff on AWS and gladly use all the services they provide. Neat, easy and less maintenance, but the most important thing for me personally is that I work with an account that doesn't use my own credit card. I can understand the applicants though, there is a fear to commit to solutions where any experience might not increase their market value. But from my experience that fear is highly exaggerated. There are people that reject candidates on these accounts but usually not the good one that know they need to invest in engineers. There are also quite a few bosses that are only interested in the newest tech. They have absolutely no idea what that means, but they will buy it. And a lot of cloud software hits the sweet spot for them.
  - tluyben2 4y ago
    > > then not many engineers will apply I would. Like you said ; one point is probably market value and another future relevance. I have had discussions with hiring managers lately and almost none that I talked to actually use most or any of the buzzwords they put in their ads. They just say they do to get more applications.
- mwcampbell 4y ago
  How broadly is this true? Some of us developers do want to focus on things that are useful to people, that we can describe to non-tech people we meet in public. Are those of us who run bootstrapped, all-remote SaaS companies going to have trouble hiring?
xixixao 4y ago
Neither Facebook nor Google are built on microservices in this fashion. Amazon is (but it’s on its own servers).
_8j50 4y ago
Yes, but the sooner you build to scale the better. More costly later on.
- wilde 4y ago
  Is it? You trade retrofit costs later for maintenance costs and feature development friction every day until scale is needed.
  - _8j50 4y ago
    You make a very good point.
- VLM 4y ago
  Scalability is a two way street and those services don't scale small. That makes business opportunities for more flexible engineers.
- tluyben2 4y ago
  Chances are quite high you will never need scale. If you knew 100% sure you needed it when building, you would build for scale from day 1. But most/all startups and also for many projects in large corps, they don't know if they need scale. And startup statistics definitely show it is a very high chance you don't need it ever because you will be out of business long before that. It's mostly a waste of effort, time and money.
johnthescott 4y ago
> astronautery gonna borrow that phrase.
- twic 4y ago
  Surely "astronautics" is the right word though?

5350-uiop-1130 4y ago

It's odd that over time things don't get simpler, instead everything trends towards more and more layered complexity - where the higher layers are abstracted so as to appear simpler.

HPsquared 4y ago
Such is life (that is, biological evolution worked the same way).
- high_5 4y ago
  Life implies death. When will these systems implode under their own complexity?
  - api 4y ago
    This is a well known cycle in software. Systems grow in complexity until they collapse under their own weight, then a new system is created that steals the most valuable ideas from the old system. Repeat. It happens in nature too. Your skull has fewer bones in it than the skulls of many fish and humans do not by any stretch have the largest genome in nature.
- rightbyte 4y ago
  Zealous creationism in that case.
- mwcampbell 4y ago
  But unlike nature, we can do intelligent design.
  - arein3 4y ago
    How is the evolutionary design not intelligent?
    - quickthrower2 4y ago

pontifier 4y ago

In the sci-fi future such as star trek, you often see systems being reprogrammed by swapping crystals around or something like that. I used to think it was really stupid, but as I deal with trying to build complicated systems I find myself wishing I could build something like that to set up systems and have them just work. Interestingly, 90's industrial automation sort of feels like it was trying to build systems like that. The control cabinets are filled with simple, discrete, computational blocks that control valves and actuators based on sensor inputs. Getting back to the sci-fi, I realized that the crystal swapping could just be the high level interface to all the complex code below. They really wouldn't need to do anything except light up when they are detected correctly, but still control complex code hiding below the surface... Maybe written on demand by AI that reads the crystals as prompts. I don't know where this is all heading, but maybe someday swapping databases will be as complicated as changing a lightbulb.

valenterry 4y ago
Not possible. Imagine the number of scenarios that require fixes. There are just too many of them. So you either end up with a gigantic amount of crystals (but still can't solve a problem in some exceptional cases) or you have a small number of crystals but a high number of situations where you can't resolve the problem with them. So maybe those crystals are more like app-configurations that we already have: you can tune things in a predictable way, but as soon as it goes beyond that you need to change the code.
AndyMcConachie 4y ago
Imagine 100 years from now when humanity's population has been reduced by 100s of millions because of climate catastrophe. Now imagine all the software society relies upon. Now imagine being one of the few people left alive being capable of working on that software. You probably won't have source code. One thing that has remained constant in our lives is the number of people who are capabable of working on complex software has steadily increased. What happens if that number suddenly decreases?
- Ekaros 4y ago
  Now I started to think what is the actual ratio of software we need and software that is produced or distributed.

api 4y ago

The real solution to this is how we solved it last time we were here: abandon the mainframe and go back to local applications on PCs. The cloud really provides two things for most applications: (1) Data synchronization at scale. This could be provided by a much, much thinner cloud that basically just syncs files but using CRDTs to make that more robust. Applications could use these services to sync state for collaboration. The vast majority of the brains could be at the endpoint. (2) The cloud is copy protection and DRM. It gives software publishers an easy place to put a toll booth. This could be solved by a reboot of the concept of local licensing, or by just periodically calling out to a license server. Yes people will pirate your software but those people aren't going to pay anyway.

mwcampbell 4y ago
If I'm not mistaken, you overlooked another role of cloud services: coordination of real-time connections between endpoints, and in some cases actual relaying of data to work around NAT. I guess this is the space you're in with ZeroTier. But I'm not sure how much of a market there is for generalized solutions like ZeroTier and Tailscale. For the most part, people want solutions that connect their machines for a specific purpose, such as remote desktop access (e.g. TeamViewer/AnyDesk), and developers of such products don't want to require their users to have another networking product (ZeroTier/Tailscale) first.
spion 4y ago
Not sure this works. For example, how do you implement search? I'm not looking forward to having to download the entire search index catalog of a web store in order to find out whether they have something.
- quickthrower2 4y ago
  Maybe not run everything locally, but most “web apps” could be local apps with something like dropbox to back up the data and make it accessible. The issue with this is the web is zero install cross platform and it is hard to compete with that.
- api 4y ago

szundi 4y ago

Please someone tell me wether it is at all possible to deliver very complex/sophisticated b2b stuff via cloud stuff. For me it seems to be a nightmare to debug complex stuff, but without the “old” tooling it seems to me that it is quite impossible to do this when buried behind queues, pubsubs, cloud functions and all that. Any comment on this is quite welcome.

brianmcc 4y ago

I hate headlines like this pieces's full one - omitted here on HN so far - which is "Why Intuitive Troubleshooting Has Stopped Working For You". For me? Excuse me no, because I design systems to be supportable in production. Tsk. "For some"? "For many"? Sure! But please don't annoy me before I even start reading with this kind of headline.

zamfi 4y ago
Why the hate? This headline makes the audience clear, and it’s not you! Enjoy the freedom this gives you to skip reading the post.
- brianmcc 4y ago
  Ha fair enough a strong term. Forgive me I am just venting a bit about a wider pattern of these sort of headlines: "Why YOU aren't saving enough for retirement", "Why YOU need to eat fewer avocados", etc. It's not just an audience indicator it's a challenge designed to engage, but it just winds me up. A stylistic technique I really dislike, that's all! :-)
  - bombcar 4y ago
    A/B testing has destroyed headlines and I hate it.
    - zamfi 4y ago
      Yeah. But, fortunately, it's a dynamic system! Learn to ignore the headlines that are written this way, because the noise is high. Some folks have to use ad blockers to avoid seeing ads, while others train themselves to just ignore ads entirely. Personally, I don't even see them most of the time. If you're in the latter camp, training yourself to ignore headlines like these doesn't seem much harder!

RedShift1 4y ago

Regardless of complexity: error messages, error messages, error messages. I can't emphasize enough how important decent error messages are. When you are having trouble with a file, include the full path in the error message. Put in the name of the function in the error message, what you expected and what you got. Bubble up lower level errors, don't swallow them. Include a stack trace, include the path of the log file if applicable. Please don't have your error message say "Something went wrong".

enlyth 4y ago
Agreed, a few times I've found myself in the unfortunate position of needing to push changes to production just to be able to diagnose an error, because it was being completely swallowed by a try-catch block with some generic "something went wrong" message.

nicbou 4y ago

quickthrower2 4y ago
That seems odd to me. If the service can’t run or emulate locally then I wouldn’t use it.

nicbou 4y ago

I went to a serverless conference some time ago (great excuse to travel on the company's dime), and every single talk had a few minutes about how they were unable to test or debug their production infrastructure. They couldn't even run their code locally, just bits of it. Serverless code could not run without a server. A specific provider's servers. Call me old-fashioned, but I develop with a few docker containers locally, then release them to a plain old linux server, and it's working great. Sure, my humble business isn't Google, but neither are yours, right?

ratww 4y ago
I went to one too, and saw a talk about synchronising data between microservices and was astounded at how much work it was and the fact they had a whole team just to build tooling to debug issues on this specific company. Basically they store 100x more data than they need because of data duplication between microservices. The company didn't even have that much customers (<1000), data or even enough users (a handful per company), but duplication was still generating several terabytes. It's amazing how much complexity people are able to cram into simple system. (Interestingly enough there's another recent comment on another article here talking about this exact issue: https://news.ycombinator.com/item?id=32320660 https://news.ycombinator.com/item?id=32320660 )
itsmemattchung 4y ago
Agree that being unable to test serverless code locally is begging for pain. While not perfect, testing AWS Lambda has improved over the years. I recently deployed a Lambda function only because I was able to reproduce the environment locally, using their Lambda Runtime Interface Emulator (RIE): https://docs.aws.amazon.com/lambda/latest/dg/images-test.html https://docs.aws.amazon.com/lambda/latest/dg/images-test.htm...
- lincler 4y ago
  Until you need to connect to another service that can not be run locally and it is only accesible within the same Cloud Provider. Sure it's not impossible to go around that, start mocking or run some alternative locally for debugging but I don't think this extra level of complexity is necessary (at least not always).

ianbooker 4y ago

I often feel that for example Python web stacks are overly reliant on conventions. Your work is expected to follow nested conventions, and if you do so, things work. Yet, some conventions imported from other underlying packages are not documented in the current package, so if you are new to the stack you are lost immediatly.

drewcoo 4y ago

tldr: Loose coupling means no intuitive troubleshooting. Which you can infer from the diagrams but isn't stated directly in this marketing puff piece.