3 ms·

I run a machine learning team in a high traffic ecommerce company (specializing in stock photography and digital assets). We see an even bigger seasonal spike t

by mlthoughts2018 6y ago

I run a machine learning team in a high traffic ecommerce company (specializing in stock photography and digital assets). We see an even bigger seasonal spike than Etsy.

throwanem 6y ago
With no code chill at all? Not even for artifacts which, presumably unlike what your team works on, lie in the critical path for revenue generation aka checkout? If so, that's very impressive, but also very atypical. Source: I work in high-volume ecommerce, have friends and colleagues at other companies who do likewise, and no one I've talked to about it works anywhere that doesn't implement at least some controls on at least checkout-path deployments during holiday prep and through holiday proper. It isn't a tech debt thing, it's just good business sense: the time of year when people give you by far the most money is the time of year when you least want to risk causing problems for people who want to do so. The problem isn't untrustworthy systems or bad engineering culture; the problem is that anything short of perfection has a measurable impact on revenue, and humans achieve perfection with less than perfect reliability. Even where I work, that revenue impact can easily amount to a major problem if we let a significant bug slip through. People buy our product for its own sake, and will do so from Amazon or Walmart or wherever if they can't get it direct - but our margin on sales through second-party vendors is nothing like as good as on direct sales, so we still take a hit. For Etsy, it's much worse; on sales through other storefronts, they get nothing. Too, a bad enough break during holiday will also risk some fraction of their subscriber revenue from sellers who, fed up at losing out on what would otherwise also be their highest revenue of the year, might go elsewhere. It's easy to talk shit about tech debt and bad engineering culture, I get that. It's also poorly founded in fact, and evinces an apparently questionable grasp on some fundamentals of the business, besides. I don't know whether you have represented yourself well or poorly by creating this impression, but it is the impression you've created, and if that bothers you then I might suggest trying to do otherwise next time.
- mlthoughts2018 6y ago
  The machine learning team is responsible for all search and discovery features that users engage with when searching our inventory, some of the highest traffic, uptime demanding services I’ve seen anywhere in my 15 year career. We also run all of the image and language processing that occurs in real time in the image editor post upload, along with a HA queue system that aggregates requests to route them to shared GPUs on the backend. In our case, many machine learning systems are on the critical path for customers, and we have a significant platform team within machine learning specifically to ensure we can support this, meet uptime demands, design auto-scaling solutions in our on-prem data centers, execute failover and disaster recovery (and drills to simulate these) and more. And so does every other team.
  - throwanem 6y ago
    Okay, that makes sense. Why do you assume, or at least give the very strong impression of assuming, that no one else does anything like this? Or even just that Etsy doesn't?
    - mlthoughts2018 6y ago
      Because they just wrote a big article about how they created a culture, developed special vocab (“slush”), and intentionally plan on a multi-year basis all around a code freeze that is not just a day, not just a weekend, but lasts for weeks and affects all systems (which either means no systems are in a state of proving their resiliency, or leadership doesn’t trust engineers, or both), and they are proud of this. That is a whole mess of giant neon red flags that something is wrong here regarding engineering for resilience and trusting your deployment system to catch errors. Additionally, for a freeze lasting weeks, unfreezing and resuming merges from some backlog of frozen changes would likely introduce far greater risk to revenue or to customers than managing a trustworthy deployment workflow during high traffic events. I would even question the fundamental premise that this is safer or beneficial to customers at all. It seems much, much more likely to be a “cover your ass” method to prevent any possible blowback from outages due to unremediated tech debt that the weak leadership is not capable of protecting as a worthwhile priority for quarterly goals.

I run a machine learning team in a high traffic ecommerce company (specializing in stock photography and digital assets). We see an even bigger seasonal spike than Etsy.

throwanem 6y ago

With no code chill at all? Not even for artifacts which, presumably unlike what your team works on, lie in the critical path for revenue generation aka checkout? If so, that's very impressive, but also very atypical. Source: I work in high-volume ecommerce, have friends and colleagues at other companies who do likewise, and no one I've talked to about it works anywhere that doesn't implement at least some controls on at least checkout-path deployments during holiday prep and through holiday proper. It isn't a tech debt thing, it's just good business sense: the time of year when people give you by far the most money is the time of year when you least want to risk causing problems for people who want to do so. The problem isn't untrustworthy systems or bad engineering culture; the problem is that anything short of perfection has a measurable impact on revenue, and humans achieve perfection with less than perfect reliability. Even where I work, that revenue impact can easily amount to a major problem if we let a significant bug slip through. People buy our product for its own sake, and will do so from Amazon or Walmart or wherever if they can't get it direct - but our margin on sales through second-party vendors is nothing like as good as on direct sales, so we still take a hit. For Etsy, it's much worse; on sales through other storefronts, they get nothing. Too, a bad enough break during holiday will also risk some fraction of their subscriber revenue from sellers who, fed up at losing out on what would otherwise also be their highest revenue of the year, might go elsewhere. It's easy to talk shit about tech debt and bad engineering culture, I get that. It's also poorly founded in fact, and evinces an apparently questionable grasp on some fundamentals of the business, besides. I don't know whether you have represented yourself well or poorly by creating this impression, but it is the impression you've created, and if that bothers you then I might suggest trying to do otherwise next time.

mlthoughts2018 6y ago
The machine learning team is responsible for all search and discovery features that users engage with when searching our inventory, some of the highest traffic, uptime demanding services I’ve seen anywhere in my 15 year career. We also run all of the image and language processing that occurs in real time in the image editor post upload, along with a HA queue system that aggregates requests to route them to shared GPUs on the backend. In our case, many machine learning systems are on the critical path for customers, and we have a significant platform team within machine learning specifically to ensure we can support this, meet uptime demands, design auto-scaling solutions in our on-prem data centers, execute failover and disaster recovery (and drills to simulate these) and more. And so does every other team.
- throwanem 6y ago
  Okay, that makes sense. Why do you assume, or at least give the very strong impression of assuming, that no one else does anything like this? Or even just that Etsy doesn't?
  - mlthoughts2018 6y ago
    Because they just wrote a big article about how they created a culture, developed special vocab (“slush”), and intentionally plan on a multi-year basis all around a code freeze that is not just a day, not just a weekend, but lasts for weeks and affects all systems (which either means no systems are in a state of proving their resiliency, or leadership doesn’t trust engineers, or both), and they are proud of this. That is a whole mess of giant neon red flags that something is wrong here regarding engineering for resilience and trusting your deployment system to catch errors. Additionally, for a freeze lasting weeks, unfreezing and resuming merges from some backlog of frozen changes would likely introduce far greater risk to revenue or to customers than managing a trustworthy deployment workflow during high traffic events. I would even question the fundamental premise that this is safer or beneficial to customers at all. It seems much, much more likely to be a “cover your ass” method to prevent any possible blowback from outages due to unremediated tech debt that the weak leadership is not capable of protecting as a worthwhile priority for quarterly goals.