12 ms·

I would bet it's just the influx of traffic post holiday with systems that haven't been updated in so long maybe some annoying memory leaks have crept up and go

I would bet it's just the influx of traffic post holiday with systems that haven't been updated in so long maybe some annoying memory leaks have crept up and gone unnoticed or some other bad state that was exacerbated by return to work day for most NA folks. Code freezes were good at identifying bugs that only show up after long periods. Doubt anyone releasing big changes Monday morning.

hnlmorg 6y ago
That might be true but when you take the global usage of Slack and their respective time zones, more than half the world would have signed into Slack this morning before SV had and I certainly didn't notice any downtime this morning in my time zone.
- radicalbyte 6y ago
  It was ropey before SV woke up, I thought it was just my (normally rock solid thanks to using Ubiquity) network having issues. Guess it was Slack being Slack.
exhaze 6y ago
I haven't worked at Slack, so I can't speak with high confidence. A traffic spike is a possible reason, but I'm willing to bet that it's not the reason: > Doubt anyone releasing big changes Monday morning. This is definitely an engineering best practice, and by best practice, I mean something that Uber's, I mean Slack's SRE team strongly pushed for, and got politely overruled on. After a code freeze is lifted, it's quite common for lots of promotion-eager engineers to release big changes.
- agrippanux 6y ago
  In my experience it's not promotion-eager engineers that want to push after a code freeze, it's antsy product managers. YMMV tho.
  - glouwbug 6y ago
    What's there to change in Slack, though? It's arguably a messaging system, and that feature is tried and tested. That, and giphys, to be honest. EDIT: Guys it was a joke, chill
    - VectorLock 6y ago
      HN's tolerance for jokes and sarcasm is extremely low.
      - deleted 6y ago
        [deleted]
      - brlewis 6y ago
        I'm not sure about that. I feel like I get more upvotes from sarcasm and jokes than from insight. In this instance, I think it's because when people hear something dumb said seriously in real life, they're not going to readily recognize online that it's a joke.
        Cederfjard 6y ago
        Yeah, Poe’s law applies here. That’s definitely something someone less informed might say in earnest.
        mewpmewp2 6y ago
        Yeah, there was other thread about Uber, where similar sentiment was seriously debated there, so I didn't recognise this as sarcasm either.
  - godot 6y ago
    IMO it really doesn't have to be promotion-eager engineers or antsy product managers. I'm fairly satisfied with my role and comp and work type with where my career/life-stage is. I just did a code release first thing this morning, not because I am promotion-eager, but just because I'm picking back up where I left off, like any normal day. Granted I work at a much smaller company than Slack with orders of magnitude less traffic.
- Thaxll 6y ago
  You just don't deploy something major the first day after a 2 weeks vacation, it does not makes any sense.
  - matsemann 6y ago
    Why? I had a rewrite of some core logic the last day before Christmas that I didn'td deploy, as it wasn't time critical to get out and I didn't want to be disturbed during holidays. Today it was perfect to deploy, as I can watch it the whole week if needed.
    - devilduck 6y ago
      lol Good Luck!!
    - hacky_engineer 6y ago
      Yeah, I do this all the time. I don't want to be bothered on the weekend, so I push releases at the beginning of the week when possible.
      - kords 6y ago
        Same, I would rather release on a Monday than a Friday.
    - johnmaguire2013 6y ago
      Well, I think it probably depends on where you work. At my work, people just took 2-3 weeks of time off. It takes a moment to get your head back in the game.
  - beamatronic 6y ago
    It depends on the goal you’re trying to accomplish. Are you going for a promotion or bonus? Or instead is your goal to maximize uptime?
    - adambyrtek 6y ago
- Aperocky 6y ago
  Does Uber/Slack not release in CI/CD? At least in backend? I don't see any need to deploy a big change at once in the software world today. At worst feature gate the thing you want to do and run it in a beta environment, but still push the actual code down the pipeline.
  - exhaze 6y ago
    > run it in a beta environment Every Uber/ex-Uber engineer is nervously chuckling at this comment right now
    - aeyes 6y ago
      For those that don't know what this comment is about: https://eng.uber.com/multitenancy-microservice-architecture/ https://eng.uber.com/multitenancy-microservice-architecture/
      - xtracto 6y ago
        Aaah the wonders of not having to be PCI or SOC2 compliant...
      - yjftsjthsd-h 6y ago
        I'm actually more confused after reading that. I assumed that you meant that tested in production on purpose, but it sounds, at a skim, like they do non-prod testing environments - in fact, it looks like they've gone to having multiple beta environments of every service?
        aeyes 6y ago
        My understanding is that they have a "tenancy" variable in every service call which can take a different code path. They seem to only have one environment for everything and do tests/experiments at code level based on this variable.
        6y ago
- throwaway201103 6y ago
  Interesting, I've never worked anywhere where engineers decide when to release changes. That's a product decision, and there is a process of review and approval at both the code level and the functional/end-user-experience level that has to happen first. Did you mean that literally? E.g. is it common at Uber that engineers can release changes to production on their own?
  - pcl 6y ago
    At Cisco (Webex team), the engineers decide when to release code, and most features are enabled by configs or feature flags independently of the deploys. The engineering team is responsible for the mess caused by a bad deploy, so it's appropriate that those engineers should also choose the timing. Our team typically deploys between 10am and 4ish, local time, since that's when we're at our desks and ready to click through the approvals and monitor the changes as they go through our pipelines. The feature enablement happens through an EFT / beta process, and the final timing of GA enablement is a PM decision. But features are widely used by customers ahead of that time, as part of the rollout process. Our team usually rolls out non-feature changes to services via dynamic configuration switches, so that we can get new bits in place, and then enable new behavior without a redeploy. This also enables us to roll back the dynamic config quickly if something unexpected happens. (We generally don't do this for net new functionality; there's lower risk in adding a new REST endpoint etc. than in changing an existing query's behavior or implementation.)
bobthepanda 6y ago
What would make that strange? Where I work it is frowned upon to do releases on weekends and so bad changes due to buildups happen on Monday. Although, we also don’t close the pipeline for just any holiday break. In fact low holiday traffic is a good time to keep pipelines open, since changes will impact less people.
deleted 6y ago
[deleted]

hnlmorg 6y ago

That might be true but when you take the global usage of Slack and their respective time zones, more than half the world would have signed into Slack this morning before SV had and I certainly didn't notice any downtime this morning in my time zone.

radicalbyte 6y ago
It was ropey before SV woke up, I thought it was just my (normally rock solid thanks to using Ubiquity) network having issues. Guess it was Slack being Slack.

exhaze 6y ago

I haven't worked at Slack, so I can't speak with high confidence. A traffic spike is a possible reason, but I'm willing to bet that it's not the reason: > Doubt anyone releasing big changes Monday morning. This is definitely an engineering best practice, and by best practice, I mean something that Uber's, I mean Slack's SRE team strongly pushed for, and got politely overruled on. After a code freeze is lifted, it's quite common for lots of promotion-eager engineers to release big changes.

agrippanux 6y ago
In my experience it's not promotion-eager engineers that want to push after a code freeze, it's antsy product managers. YMMV tho.
- glouwbug 6y ago
  What's there to change in Slack, though? It's arguably a messaging system, and that feature is tried and tested. That, and giphys, to be honest. EDIT: Guys it was a joke, chill
  - VectorLock 6y ago
    HN's tolerance for jokes and sarcasm is extremely low.
    - deleted 6y ago
      [deleted]
    - brlewis 6y ago
      I'm not sure about that. I feel like I get more upvotes from sarcasm and jokes than from insight. In this instance, I think it's because when people hear something dumb said seriously in real life, they're not going to readily recognize online that it's a joke.
      - Cederfjard 6y ago
        Yeah, Poe’s law applies here. That’s definitely something someone less informed might say in earnest.
        mewpmewp2 6y ago
        Yeah, there was other thread about Uber, where similar sentiment was seriously debated there, so I didn't recognise this as sarcasm either.
- godot 6y ago
  IMO it really doesn't have to be promotion-eager engineers or antsy product managers. I'm fairly satisfied with my role and comp and work type with where my career/life-stage is. I just did a code release first thing this morning, not because I am promotion-eager, but just because I'm picking back up where I left off, like any normal day. Granted I work at a much smaller company than Slack with orders of magnitude less traffic.
Thaxll 6y ago
You just don't deploy something major the first day after a 2 weeks vacation, it does not makes any sense.
- matsemann 6y ago
  Why? I had a rewrite of some core logic the last day before Christmas that I didn'td deploy, as it wasn't time critical to get out and I didn't want to be disturbed during holidays. Today it was perfect to deploy, as I can watch it the whole week if needed.
  - devilduck 6y ago
    lol Good Luck!!
  - hacky_engineer 6y ago
    Yeah, I do this all the time. I don't want to be bothered on the weekend, so I push releases at the beginning of the week when possible.
    - kords 6y ago
      Same, I would rather release on a Monday than a Friday.
  - johnmaguire2013 6y ago
    Well, I think it probably depends on where you work. At my work, people just took 2-3 weeks of time off. It takes a moment to get your head back in the game.
- beamatronic 6y ago
  It depends on the goal you’re trying to accomplish. Are you going for a promotion or bonus? Or instead is your goal to maximize uptime?
  - adambyrtek 6y ago
Aperocky 6y ago
Does Uber/Slack not release in CI/CD? At least in backend? I don't see any need to deploy a big change at once in the software world today. At worst feature gate the thing you want to do and run it in a beta environment, but still push the actual code down the pipeline.
- exhaze 6y ago
  > run it in a beta environment Every Uber/ex-Uber engineer is nervously chuckling at this comment right now
  - aeyes 6y ago
    For those that don't know what this comment is about: https://eng.uber.com/multitenancy-microservice-architecture/ https://eng.uber.com/multitenancy-microservice-architecture/
    - xtracto 6y ago
      Aaah the wonders of not having to be PCI or SOC2 compliant...
    - yjftsjthsd-h 6y ago
      I'm actually more confused after reading that. I assumed that you meant that tested in production on purpose, but it sounds, at a skim, like they do non-prod testing environments - in fact, it looks like they've gone to having multiple beta environments of every service?
      - aeyes 6y ago
        My understanding is that they have a "tenancy" variable in every service call which can take a different code path. They seem to only have one environment for everything and do tests/experiments at code level based on this variable.
        6y ago
throwaway201103 6y ago
Interesting, I've never worked anywhere where engineers decide when to release changes. That's a product decision, and there is a process of review and approval at both the code level and the functional/end-user-experience level that has to happen first. Did you mean that literally? E.g. is it common at Uber that engineers can release changes to production on their own?
- pcl 6y ago
  At Cisco (Webex team), the engineers decide when to release code, and most features are enabled by configs or feature flags independently of the deploys. The engineering team is responsible for the mess caused by a bad deploy, so it's appropriate that those engineers should also choose the timing. Our team typically deploys between 10am and 4ish, local time, since that's when we're at our desks and ready to click through the approvals and monitor the changes as they go through our pipelines. The feature enablement happens through an EFT / beta process, and the final timing of GA enablement is a PM decision. But features are widely used by customers ahead of that time, as part of the rollout process. Our team usually rolls out non-feature changes to services via dynamic configuration switches, so that we can get new bits in place, and then enable new behavior without a redeploy. This also enables us to roll back the dynamic config quickly if something unexpected happens. (We generally don't do this for net new functionality; there's lower risk in adding a new REST endpoint etc. than in changing an existing query's behavior or implementation.)

bobthepanda 6y ago

What would make that strange? Where I work it is frowned upon to do releases on weekends and so bad changes due to buildups happen on Monday. Although, we also don’t close the pipeline for just any holiday break. In fact low holiday traffic is a good time to keep pipelines open, since changes will impact less people.

deleted 6y ago

[deleted]