7 ms·

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system and look at the data when you have issues, rather than grasping at straws.

sitkack 8y ago
> Reason through your system modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. abstractions make reasoning harder, what tools and techniques would you suggest for doing this? I'd probably run the application in some sort of sandbox and measure the outbound request load vs inbound request load, something a containerized deployment should be giving the end user (developer) as an affordance for application maintenance and visibility. Differential analysis and graphing built directly into the execution substrate. Judge away!
- cloakandswagger 8y ago
  Basic metrics around platform API calls would have done the trick, no need for fancy container solutions. edit: I would have also assumed they would get this for free from GCP's billing breakdown, but I'm not familiar with it. My first intuition when facing unexpected billing would be to figure out what the major contributor to the bill is (in this case, massive reads from FireStore), not update my frontend packages.
  - dkoston 8y ago
    Exactly. Not having proper tooling meant they didn’t know what to do. When your check engine light comes on, you don’t replace every system in your car, you get out a scanner and check the code. Google cloud has trace built in which could have shown them execution times and is dead simple to drop into most frameworks. The real story here is that hey didn’t have engineering leadership on the team who knew how to properly diagnose issues, put tooling in place before launch, and understand how their system is architected. Kudos to the engineers for solving this issue under pressure.
    - chrisweekly 8y ago
      >"When your check engine light comes on, you don’t replace every system in your car, you get out a scanner and check the code." ^ Best response, hands down.
  - themarkn 8y ago
    > My first intuition when facing unexpected billing would be to figure out what the major contributor to the bill is (in this case, massive reads from FireStore), not update my frontend packages. They didn’t upgrade packages to solve the mystery billing. They upgraded packages before they checked what was going on with the database. When they saw the high billing that pointed them to the problem and they fixed it. There was some questionable judgement shown by not checking db requests first, sure, but in no way did someone think “our google billing is high, we better upgrade angular”.
- deleted 8y ago
  [deleted]
- TeMPOraL 8y ago
  > modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. Also, lots of people are impatient and/or intellectually lazy. We have piled up a ton of abstraction layers, yes, but they aren't hard to pry apart. But people want immediate results without doing necessary cognitive work - understanding-guided exploration. It usually isn't hard to identify which component of your product is misbehaving. Before getting into complex magic of containers and sandboxes and stuff, I'd start with the easiest - looking at the Network tab, at your server's resource use, reading the logs, adding some log statements measuring times in suspected areas, actually profiling the backend code (e.g. with a statistical profiler). This should quickly help you identify where the problem is manifesting itself. Then the search for the cause begins.
  - cm2187 8y ago
    There is also a common sense problem. Not trying to play the blame game but the guy who ran a subtotal displayed on the front page by having to download every row in the database probably isn’t a great programmer. It’s easy to say don’t write bugs, but there are bugs and moronic design decisions. This to me doesn’t feel like a real bug.
    - dmurray 8y ago
      There's a good chance this is true, but this can happen even with competent developers. One person writes the unoptimized function figuring it can be improved later, someone else assumes this must be cheap and drops it on the front page. Or there was meant to be a caching layer, that didn't work because of some bug.
  - voltagex_ 8y ago
    >Also, lots of people are impatient and/or intellectually lazy. We have piled up a ton of abstraction layers Sorry, the guy that wrote/implemented abstraction layers 2 and 3 left 2 years ago and didn't document anything. We've been understaffed for a year and we've been told not to hire any more staff until the new financial year. I've got a "technical debt" item on the backlog but business drives the priorities and it'll never get done.
- gitgud 8y ago
  Well a browser is basically a sandbox for a webapp. To debug something complex, I would use chrome devtools, which can measure all kinds of metrics, and the function "Audit > LightHouse" automates the process and ranks a webapp in several key categories. This case would appear to be related to network requests, so that issue should be fairly obvious in LightHouse.
Lazare 8y ago
That jumped out at me too. Obviously they were under a lot of pressure and it's easy to stand here and criticise, but... ...if my site is slowing down with load or usage, I'm not sure how you make the jump to "I should update my UI libraries!". Angular 4 isn't getting any slower, so best case is you've got some unknown performance bottleneck in your UI that is somehow causing 30s page load times, and which just happens to be fixed in Angular 6, and you don't accidentally add any new issues when you upgrade. Conversely, it feels like if you're struggling with "slow load times" on a SPA, the first thing you'd do is open the network tab and see what requests are being made, to what, how often, and how long they're taking. Grasping at straws does seem to be the right metaphor. (Or maybe the old chestnut about the drunk dropping his car keys in a dark parking lot, then looking for them under a streetlight, since it's too dark to find them in the parking lot?) I'm happy for the team and it sounds like things are going great for them, but wow, that was an almost fatal bit of blindness. On the plus side, I bet everyone involved will check for inefficient database calls first next time. :)
- cpplinuxdude 8y ago
  That’s why you need good humans who’re expert at responding extremely well when things break. It’s one thing to prep for a whiteboard interview, it’s another to intuit that needle in the haystack.
  - tdumitrescu 8y ago
    It's only about finding needles in haystacks if you don't have sufficient instrumentation and monitoring in place. For most production issues your tools should be able to guide you at least the first 75% of the way to your issue, even if they can't usually offer a good fix (though sometimes they can, such as when new relic points out missing db indices causing unnecessary full table scans).
  - nevi-me 8y ago
    I agree. It's also about trusting developers' intuition when debugging problems. We recently (past few working days) went through something similar; where we had a problem blocking us from a release, and people scrambling to figure out the problem. We have some software that was returning different results from different environments, and we couldn't figure out the problem. There was a lot of panic in the room, from upgrading and downgrading Maven dependencies, building things inside and outside of Jenkins, and all sorts of random things. We kept telling the project leadership that we're poking at the wrong part (intuitively), but they kept pushing. I've had to explain how Maven works, how building on Jenkins doesn't differ to building from our IDE's, etc. It's only when we asked for isolation from the (human) elements, that we had the freedom to properly debug. In the end, an unstable sort was the cause of the issue. We were taking the last element from an array, but not sorting the array first. All of the stuff we did since last Thursday to Tuesday evening didn't help us. So, I agree, you need good humans who are good at responding well when things break.
thatjsguy 8y ago
I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refactoring was also pretty popular, although it usually just ended up making things more complicated. It’s silly to suggest that JavaScript itself is somehow responsible for this. It’s obvioisly just a tool. But I have to say, the most professional cluelessness I’ve ever encountered was in the JS ecosystem.
- ct520 8y ago
  I concur. I’m having a hard time not coming off as a dick while trying to opine on how amateurish I think this is. I like JS but the devs and ecosystem leaves me wanting.
  - radicalbyte 8y ago
    The sad thing is that this is in my experience the norm. Not just in frontend/js either. It's everywhere.
- deleted 8y ago
  [deleted]
- paulddraper 8y ago
  It has to do with the barrier to entry. PHP, Ruby on Rails, and jQuery are other technologies that had low barrier to entry received the attention of the "unwashed masses". This being 2018, JS has very low barrier to adoption (Have a web browser? You have a JS runtime.) and nature runs its course.
  - rspeer 8y ago
    Oh come on. Tech does not need barriers to entry. It just needs debugging and improving one's code to also have a low barrier to entry.
megaman22 8y ago
Scientific reasoning is a rare skill. I'm regularly met with bafflement and blank stares when I suggest that we measure something to determine performance, then perform the same workload, on the same hardware, after making a change. I've literally had people try to make conclusions on comparisons of test runs with completely different parameters, different data sets, different resources, different versions of code, absolutely everything varying. My head just explodes... I wamt to scream that this isn't how this works, it's not how any of this works.
- aflam 8y ago
  Managing those experiences takes effort, otherwise you're sure to end up with the issues you describe. Even when the data makes sense, sometimes people hand you over an unparsable csv with their results, while the conclusions comes from a gut feeling. Sadly not everybody understands standard tools like boxplots, and analyses are often hard to reproduce. The tooling to make it easy not there yet.
- notimetorelax 8y ago
  I think you have a very valid point there, but I would argue it is also important not to react to such situations too strongly. This is something April Wensel talks about here [1] . [1] bit.ly/2v37AzE
- michalprzy 8y ago
  Unfortunately I often see developers asking for complex diagnose in urgent emergencies. When it's literally like a 30 minutes reaction time frame. So from business pov there should be first quick and dirty unperfect analysis which is necessary for emergency decisions. Sometimes you can just prolong caching time as quick fix, but sometimes there is no such option. Programmers like nice and calm working environment but business often is very different place. So, please have in mind what business goals are. And yes, I see there is a lot of stupid managers in It and basically you're right, we should use precise data. Just one point: when available...
  - jon-wood 8y ago
planetjones 8y ago
Yep with performance like that it’s highly likely going to be a problem involving the persistent store of the application. The first instinct should have been to enable some tracing or logging to see where the time was being spent.
dokalanyi 8y ago
To be fair I recently upgraded an old PHP install from 5.5 to 7.2 and it sped up the site tremendously. My workflow while troubleshooting is now always going to involve upgrading software.
- aidos 8y ago
  While upgrading should be an option, just don’t do it as a first step. Changing any dependency comes with risks and moving from version 5.5 to 7.2 of anything screams danger to me. Sure, there are times when it’s going to work out for you but you should at least have narrowed down your issues before you go down that path.
  - zaarn 8y ago
    I mean, it can be fairly safe. When I moved from 5 to 7.0 I simply left the PHP-FPM configuration for 5 around and 5 itself installed. It still is. I can just spin up PHP5, 7.0, 7.1 in case anything goes wrong without delay.
Cthulhu_ 8y ago
Yeah I don't get it either, if you have performance problems you look at the network tab and see what's up, instead of yolo-upgrading your framework with fingers crossed and hope it works. Measure before fix.
overcast 8y ago
There has been MANY times, upgrading outdated libraries or software versions resolved similar performance issues for me. Should you just immediately run update all? No. But acting like that's not a viable solution is silly.
- macspoofing 8y ago
  >But acting like that's not a viable solution is silly. Usually you want to understand the problem before solving it. In this case, they wasted a bunch of time doing a bunch of things (upgrading all the dependencies, and refactoring the app) in the hope that something (ANYTHING) they're doing hopefully fixes a problem they don't understand. Smart move?
  - overcast 8y ago
    Everyone can sit back in hindsight and act like armchair developers. When you're in the shit, sometimes you make not so smart moves, now they've learned from their mistakes. Which is why they are talking about it.
    - macspoofing 8y ago
      >Everyone can sit back in hindsight and act like armchair developers. That's one perspective. But come on! I really don't understand that attitude that when presented with a problem, the first approach is to spend a few days blindingly refactoring code and upgrading all the underlying frameworks. Seriously? The problem is also obvious if you just stop and think about it for a second: - They are using Firebase. For the purposes of diagnosing our issue we can assume the backend will scale well (for trivial queries) and the pipe between server and client should be wide. Firebase could be the problem, but odds are Firebase didn't go down on you just as your go-live went ahead. - Because they are using Firebase, their app is completely client-side. You can go through the potential areas of concern: 1) UI has trouble rendering. That should largely be independent of the number of users. If this was only a UI issue you'd expect some users to have problems (maybe ones that created a large amount of artifacts) but not all users. Presumably before going live, the app worked well with their test datasets. 2) Some combination of UI or Network or Data model. They noticed their web-app got slower as the number of users grew. So question is why would individual user session slow down as the total number of users grow? It must be that a single-user view is somehow dependent on the total number of users in the system. WHY?!? We know Firebase is fast, but any fast system can choke if you have a bad data model. So it could be a slow query. Or it could be a too large of a response being sent down (again, why would a large response be sent down). Maybe it was a huge json object and the UI locked up. Or something like this. It really shouldn't have taken long to at least target potential areas to explore. HELL, you should be able to see the issue immediately if you open up the network tab. You'll see which requests are either taking forever, or lead to large amount of data being transferred or both. It really isn't about 'armchair developers'. I've been in situations where things are falling apart and you need to figure shit out. Our product is on-prem and used in hospitals and is connected to multitudes of other systems controlled by other vendors. When you're trying to diagnose issues, you have to have a rational approach based on some reasonable hypothesis.
macspoofing 8y ago
>When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. Yeah. That jumped out at me as well. They spent an inordinate amount of effort to solve a non-problem. It's great to stay on evergreen with versions, but probably not a good thing to do so while you're desperately trying to debug a problem. I suspect this was a hopeful but lazy attempt - in the spirit of "Maybe if we just do this, it will somehow fix the underlying problem". It's a lazy approach to solving problems. Debugging performance bottlenecks is hard and devs generally hate doing it. Upgrading version dependencies is a known factor and developers are comfortable with that.
pknopf 8y ago
> Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system and look at the data when you have issues, rather than grasping at straws. This is a good thought-process even when debugging issues during development. I've seen many developers attempt to "fix" issues by trying to figure out what dance/keystroke makes things work. Whenever you encounter an issue of any kind, anywhere, understand the issue before attempting to resolve it. It may require you to dig deep into things you don't currently understand, but your career is currently telling you that you need to understand it.

sitkack 8y ago

> Reason through your system modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. abstractions make reasoning harder, what tools and techniques would you suggest for doing this? I'd probably run the application in some sort of sandbox and measure the outbound request load vs inbound request load, something a containerized deployment should be giving the end user (developer) as an affordance for application maintenance and visibility. Differential analysis and graphing built directly into the execution substrate. Judge away!

cloakandswagger 8y ago
Basic metrics around platform API calls would have done the trick, no need for fancy container solutions. edit: I would have also assumed they would get this for free from GCP's billing breakdown, but I'm not familiar with it. My first intuition when facing unexpected billing would be to figure out what the major contributor to the bill is (in this case, massive reads from FireStore), not update my frontend packages.
- dkoston 8y ago
  Exactly. Not having proper tooling meant they didn’t know what to do. When your check engine light comes on, you don’t replace every system in your car, you get out a scanner and check the code. Google cloud has trace built in which could have shown them execution times and is dead simple to drop into most frameworks. The real story here is that hey didn’t have engineering leadership on the team who knew how to properly diagnose issues, put tooling in place before launch, and understand how their system is architected. Kudos to the engineers for solving this issue under pressure.
  - chrisweekly 8y ago
    >"When your check engine light comes on, you don’t replace every system in your car, you get out a scanner and check the code." ^ Best response, hands down.
- themarkn 8y ago
  > My first intuition when facing unexpected billing would be to figure out what the major contributor to the bill is (in this case, massive reads from FireStore), not update my frontend packages. They didn’t upgrade packages to solve the mystery billing. They upgraded packages before they checked what was going on with the database. When they saw the high billing that pointed them to the problem and they fixed it. There was some questionable judgement shown by not checking db requests first, sure, but in no way did someone think “our google billing is high, we better upgrade angular”.
deleted 8y ago
[deleted]
TeMPOraL 8y ago
> modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. Also, lots of people are impatient and/or intellectually lazy. We have piled up a ton of abstraction layers, yes, but they aren't hard to pry apart. But people want immediate results without doing necessary cognitive work - understanding-guided exploration. It usually isn't hard to identify which component of your product is misbehaving. Before getting into complex magic of containers and sandboxes and stuff, I'd start with the easiest - looking at the Network tab, at your server's resource use, reading the logs, adding some log statements measuring times in suspected areas, actually profiling the backend code (e.g. with a statistical profiler). This should quickly help you identify where the problem is manifesting itself. Then the search for the cause begins.
- cm2187 8y ago
  There is also a common sense problem. Not trying to play the blame game but the guy who ran a subtotal displayed on the front page by having to download every row in the database probably isn’t a great programmer. It’s easy to say don’t write bugs, but there are bugs and moronic design decisions. This to me doesn’t feel like a real bug.
  - dmurray 8y ago
    There's a good chance this is true, but this can happen even with competent developers. One person writes the unoptimized function figuring it can be improved later, someone else assumes this must be cheap and drops it on the front page. Or there was meant to be a caching layer, that didn't work because of some bug.
- voltagex_ 8y ago
  >Also, lots of people are impatient and/or intellectually lazy. We have piled up a ton of abstraction layers Sorry, the guy that wrote/implemented abstraction layers 2 and 3 left 2 years ago and didn't document anything. We've been understaffed for a year and we've been told not to hire any more staff until the new financial year. I've got a "technical debt" item on the backlog but business drives the priorities and it'll never get done.
gitgud 8y ago
Well a browser is basically a sandbox for a webapp. To debug something complex, I would use chrome devtools, which can measure all kinds of metrics, and the function "Audit > LightHouse" automates the process and ranks a webapp in several key categories. This case would appear to be related to network requests, so that issue should be fairly obvious in LightHouse.

Lazare 8y ago

That jumped out at me too. Obviously they were under a lot of pressure and it's easy to stand here and criticise, but... ...if my site is slowing down with load or usage, I'm not sure how you make the jump to "I should update my UI libraries!". Angular 4 isn't getting any slower, so best case is you've got some unknown performance bottleneck in your UI that is somehow causing 30s page load times, and which just happens to be fixed in Angular 6, and you don't accidentally add any new issues when you upgrade. Conversely, it feels like if you're struggling with "slow load times" on a SPA, the first thing you'd do is open the network tab and see what requests are being made, to what, how often, and how long they're taking. Grasping at straws does seem to be the right metaphor. (Or maybe the old chestnut about the drunk dropping his car keys in a dark parking lot, then looking for them under a streetlight, since it's too dark to find them in the parking lot?) I'm happy for the team and it sounds like things are going great for them, but wow, that was an almost fatal bit of blindness. On the plus side, I bet everyone involved will check for inefficient database calls first next time. :)

cpplinuxdude 8y ago
That’s why you need good humans who’re expert at responding extremely well when things break. It’s one thing to prep for a whiteboard interview, it’s another to intuit that needle in the haystack.
- tdumitrescu 8y ago
  It's only about finding needles in haystacks if you don't have sufficient instrumentation and monitoring in place. For most production issues your tools should be able to guide you at least the first 75% of the way to your issue, even if they can't usually offer a good fix (though sometimes they can, such as when new relic points out missing db indices causing unnecessary full table scans).
- nevi-me 8y ago
  I agree. It's also about trusting developers' intuition when debugging problems. We recently (past few working days) went through something similar; where we had a problem blocking us from a release, and people scrambling to figure out the problem. We have some software that was returning different results from different environments, and we couldn't figure out the problem. There was a lot of panic in the room, from upgrading and downgrading Maven dependencies, building things inside and outside of Jenkins, and all sorts of random things. We kept telling the project leadership that we're poking at the wrong part (intuitively), but they kept pushing. I've had to explain how Maven works, how building on Jenkins doesn't differ to building from our IDE's, etc. It's only when we asked for isolation from the (human) elements, that we had the freedom to properly debug. In the end, an unstable sort was the cause of the issue. We were taking the last element from an array, but not sorting the array first. All of the stuff we did since last Thursday to Tuesday evening didn't help us. So, I agree, you need good humans who are good at responding well when things break.

thatjsguy 8y ago

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refactoring was also pretty popular, although it usually just ended up making things more complicated. It’s silly to suggest that JavaScript itself is somehow responsible for this. It’s obvioisly just a tool. But I have to say, the most professional cluelessness I’ve ever encountered was in the JS ecosystem.

ct520 8y ago
I concur. I’m having a hard time not coming off as a dick while trying to opine on how amateurish I think this is. I like JS but the devs and ecosystem leaves me wanting.
- radicalbyte 8y ago
  The sad thing is that this is in my experience the norm. Not just in frontend/js either. It's everywhere.
deleted 8y ago
[deleted]
paulddraper 8y ago
It has to do with the barrier to entry. PHP, Ruby on Rails, and jQuery are other technologies that had low barrier to entry received the attention of the "unwashed masses". This being 2018, JS has very low barrier to adoption (Have a web browser? You have a JS runtime.) and nature runs its course.
- rspeer 8y ago
  Oh come on. Tech does not need barriers to entry. It just needs debugging and improving one's code to also have a low barrier to entry.

megaman22 8y ago

Scientific reasoning is a rare skill. I'm regularly met with bafflement and blank stares when I suggest that we measure something to determine performance, then perform the same workload, on the same hardware, after making a change. I've literally had people try to make conclusions on comparisons of test runs with completely different parameters, different data sets, different resources, different versions of code, absolutely everything varying. My head just explodes... I wamt to scream that this isn't how this works, it's not how any of this works.

aflam 8y ago
Managing those experiences takes effort, otherwise you're sure to end up with the issues you describe. Even when the data makes sense, sometimes people hand you over an unparsable csv with their results, while the conclusions comes from a gut feeling. Sadly not everybody understands standard tools like boxplots, and analyses are often hard to reproduce. The tooling to make it easy not there yet.
notimetorelax 8y ago
I think you have a very valid point there, but I would argue it is also important not to react to such situations too strongly. This is something April Wensel talks about here [1] . [1] bit.ly/2v37AzE
michalprzy 8y ago
Unfortunately I often see developers asking for complex diagnose in urgent emergencies. When it's literally like a 30 minutes reaction time frame. So from business pov there should be first quick and dirty unperfect analysis which is necessary for emergency decisions. Sometimes you can just prolong caching time as quick fix, but sometimes there is no such option. Programmers like nice and calm working environment but business often is very different place. So, please have in mind what business goals are. And yes, I see there is a lot of stupid managers in It and basically you're right, we should use precise data. Just one point: when available...
- jon-wood 8y ago

planetjones 8y ago

Yep with performance like that it’s highly likely going to be a problem involving the persistent store of the application. The first instinct should have been to enable some tracing or logging to see where the time was being spent.

dokalanyi 8y ago

To be fair I recently upgraded an old PHP install from 5.5 to 7.2 and it sped up the site tremendously. My workflow while troubleshooting is now always going to involve upgrading software.

aidos 8y ago
While upgrading should be an option, just don’t do it as a first step. Changing any dependency comes with risks and moving from version 5.5 to 7.2 of anything screams danger to me. Sure, there are times when it’s going to work out for you but you should at least have narrowed down your issues before you go down that path.
- zaarn 8y ago
  I mean, it can be fairly safe. When I moved from 5 to 7.0 I simply left the PHP-FPM configuration for 5 around and 5 itself installed. It still is. I can just spin up PHP5, 7.0, 7.1 in case anything goes wrong without delay.

Cthulhu_ 8y ago

Yeah I don't get it either, if you have performance problems you look at the network tab and see what's up, instead of yolo-upgrading your framework with fingers crossed and hope it works. Measure before fix.

overcast 8y ago

There has been MANY times, upgrading outdated libraries or software versions resolved similar performance issues for me. Should you just immediately run update all? No. But acting like that's not a viable solution is silly.

macspoofing 8y ago
>But acting like that's not a viable solution is silly. Usually you want to understand the problem before solving it. In this case, they wasted a bunch of time doing a bunch of things (upgrading all the dependencies, and refactoring the app) in the hope that something (ANYTHING) they're doing hopefully fixes a problem they don't understand. Smart move?
- overcast 8y ago
  Everyone can sit back in hindsight and act like armchair developers. When you're in the shit, sometimes you make not so smart moves, now they've learned from their mistakes. Which is why they are talking about it.
  - macspoofing 8y ago
    >Everyone can sit back in hindsight and act like armchair developers. That's one perspective. But come on! I really don't understand that attitude that when presented with a problem, the first approach is to spend a few days blindingly refactoring code and upgrading all the underlying frameworks. Seriously? The problem is also obvious if you just stop and think about it for a second: - They are using Firebase. For the purposes of diagnosing our issue we can assume the backend will scale well (for trivial queries) and the pipe between server and client should be wide. Firebase could be the problem, but odds are Firebase didn't go down on you just as your go-live went ahead. - Because they are using Firebase, their app is completely client-side. You can go through the potential areas of concern: 1) UI has trouble rendering. That should largely be independent of the number of users. If this was only a UI issue you'd expect some users to have problems (maybe ones that created a large amount of artifacts) but not all users. Presumably before going live, the app worked well with their test datasets. 2) Some combination of UI or Network or Data model. They noticed their web-app got slower as the number of users grew. So question is why would individual user session slow down as the total number of users grow? It must be that a single-user view is somehow dependent on the total number of users in the system. WHY?!? We know Firebase is fast, but any fast system can choke if you have a bad data model. So it could be a slow query. Or it could be a too large of a response being sent down (again, why would a large response be sent down). Maybe it was a huge json object and the UI locked up. Or something like this. It really shouldn't have taken long to at least target potential areas to explore. HELL, you should be able to see the issue immediately if you open up the network tab. You'll see which requests are either taking forever, or lead to large amount of data being transferred or both. It really isn't about 'armchair developers'. I've been in situations where things are falling apart and you need to figure shit out. Our product is on-prem and used in hospitals and is connected to multitudes of other systems controlled by other vendors. When you're trying to diagnose issues, you have to have a rational approach based on some reasonable hypothesis.

macspoofing 8y ago

>When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. Yeah. That jumped out at me as well. They spent an inordinate amount of effort to solve a non-problem. It's great to stay on evergreen with versions, but probably not a good thing to do so while you're desperately trying to debug a problem. I suspect this was a hopeful but lazy attempt - in the spirit of "Maybe if we just do this, it will somehow fix the underlying problem". It's a lazy approach to solving problems. Debugging performance bottlenecks is hard and devs generally hate doing it. Upgrading version dependencies is a known factor and developers are comfortable with that.

pknopf 8y ago

> Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system and look at the data when you have issues, rather than grasping at straws. This is a good thought-process even when debugging issues during development. I've seen many developers attempt to "fix" issues by trying to figure out what dance/keystroke makes things work. Whenever you encounter an issue of any kind, anywhere, understand the issue before attempting to resolve it. It may require you to dig deep into things you don't currently understand, but your career is currently telling you that you need to understand it.