4 ms·

When I see a regression, I look at what was recently updated (in the past hours, days), and it's usually one of those packages. Because frequent updates tend to

When I see a regression, I look at what was recently updated (in the past hours, days), and it's usually one of those packages. Because frequent updates tend to be independent, it's usually not difficult to revert that change (eg. if I update react and lodash at the same time, chances are really good that I can revert one of those changes independently without any issues)

actually_a_dog 5y ago
This is the way. Also, you should try to update as few things as possible at once, and let your changes "soak" into production for a while before going in all "upgrade ALL the things!" Why? Well, sometimes things that fail take a while to actually start showing you symptoms. Sometimes, the failure itself takes a while to propagate, simply due to the number of servers you have. And, one of these days, cosmic rays, or some other random memory/disk error is going to flip a critical bit in some crucial file in that neato library you depend on. And, oh, the fouled up build is only going to go out to a subset of your machines. You'll be glad you didn't "upgrade ALL the things!" when any of these things ends up happening to you.
- specialist 5y ago
  Totally agree. That's why I'm asking if compulsively updating modules is a JavaScript, nodejs, whatever pathos. This team pushed multiple changes to prod per day. And the load balancer with autoscaling is bouncing instances on its own. And, and, and... So resource leaks were being masked. It was only noticed during a lull in work. And then because so many releases had passed, delta debugging was a lot tougher. And then because nodejs & npm has a culture of one module per line of source (joking!), figuring out which module to blame took even more time. I think continuous deploys are kinda cool. But not without a safety net of some sort.
  - actually_a_dog 5y ago
    I'm gonna toss this grenade out here, just because I don't see a better place to do it lol... One of the companies I worked at had an incident a couple years before I started, where there were multiple malicious Python libraries being used in the code. For 3 months. Luckily, the libraries didn't do anything significant except to ping an IP address in China, actually did perform their advertised functionality, didn't exfiltrate any data other than the source IP address on the ping packets, and they were easy to replace once the situation was found out. But, for months, they had servers pinging or attempting to ping somewhere in China. Oh, and those libraries made it into our internal repos we were using to pin versions with, too. Beyond being a speecy spicy meatball of a story, the moral here is that you have to constantly be on your guard and verifying every single line of code that goes into your application somehow.
    - x0x0 5y ago
      Also, you really ought to not allow egress from your servers except to your load balancer. Just attempting should create alarms.
      - actually_a_dog 5y ago
        Correct. That's probably actually why it took so long to detect the malicious packages: when the got installed on machines running internal services, nothing much unexpected happened. Come to think of it, I actually don't remember how the packages were detected and identified to begin with.

actually_a_dog 5y ago

This is the way. Also, you should try to update as few things as possible at once, and let your changes "soak" into production for a while before going in all "upgrade ALL the things!" Why? Well, sometimes things that fail take a while to actually start showing you symptoms. Sometimes, the failure itself takes a while to propagate, simply due to the number of servers you have. And, one of these days, cosmic rays, or some other random memory/disk error is going to flip a critical bit in some crucial file in that neato library you depend on. And, oh, the fouled up build is only going to go out to a subset of your machines. You'll be glad you didn't "upgrade ALL the things!" when any of these things ends up happening to you.

specialist 5y ago
Totally agree. That's why I'm asking if compulsively updating modules is a JavaScript, nodejs, whatever pathos. This team pushed multiple changes to prod per day. And the load balancer with autoscaling is bouncing instances on its own. And, and, and... So resource leaks were being masked. It was only noticed during a lull in work. And then because so many releases had passed, delta debugging was a lot tougher. And then because nodejs & npm has a culture of one module per line of source (joking!), figuring out which module to blame took even more time. I think continuous deploys are kinda cool. But not without a safety net of some sort.
- actually_a_dog 5y ago
  I'm gonna toss this grenade out here, just because I don't see a better place to do it lol... One of the companies I worked at had an incident a couple years before I started, where there were multiple malicious Python libraries being used in the code. For 3 months. Luckily, the libraries didn't do anything significant except to ping an IP address in China, actually did perform their advertised functionality, didn't exfiltrate any data other than the source IP address on the ping packets, and they were easy to replace once the situation was found out. But, for months, they had servers pinging or attempting to ping somewhere in China. Oh, and those libraries made it into our internal repos we were using to pin versions with, too. Beyond being a speecy spicy meatball of a story, the moral here is that you have to constantly be on your guard and verifying every single line of code that goes into your application somehow.
  - x0x0 5y ago
    Also, you really ought to not allow egress from your servers except to your load balancer. Just attempting should create alarms.
    - actually_a_dog 5y ago
      Correct. That's probably actually why it took so long to detect the malicious packages: when the got installed on machines running internal services, nothing much unexpected happened. Come to think of it, I actually don't remember how the packages were detected and identified to begin with.