3 ms·

How Etsy Prepared for Historic Volumes of Holiday Traffic in 2020

24 points by kiyanwang 6y ago

domano 6y ago
Does anyone know why the first deployment of the day needs to compile the whole codebase on all hosts? Stumbled over this part.
- jpgvm 6y ago
  I think it's simply a badly worded way of saying they would need to push the build to 1000+ hosts after they compile it.
_0o6v 6y ago
Let's hope they're not trying to log in with Google on their phones, because that hasn't worked for about a year now.
- leesalminen 6y ago
  Just tried it with my iPhone 12 Pro and worked fine. Logged in with Google on the first try.
- 0xcafecafe 6y ago
  I ordered quite a few items from etsy over the holidays and all by logging in with Google on my android phone
doctoboggan 6y ago
I run a small 3D printed jewelry business (https://lulimjewelry.com https://lulimjewelry.com) where I sell on my own website and also on Etsy.com. The Etsy team handled Christmas great this year (although it was my first Christmas in the platform so I don’t really have much to compare it to). Specifically they handled the communication around the mail delays very well, both with me and with my customers. It’s really interesting to see the tech decisions that had to be made to make those communications!
ckdarby 6y ago
The article was rather a disappointment compared to other articles Etsy has put up on their blog. It was overall just lackluster and the TL;DR can be entirely summed up as, more than doubled traffic without issue (and no technical insights really).
- victor106 6y ago
  I agree. I learnt nothing from it.
mlthoughts2018 6y ago
The need for code freezes around high traffic events is usually a sign of very poor engineering culture, where patching extreme tech debt is much higher priority than technical excellence or product improvements that help the customers. A code semi-freeze for weeks is incredibly alarming. How much unremediated tech debt must be causing that? That is a massive red flag that Etsy engineering culture must be something like the SRE tail wagging the product engineering dog. I would bet that you only get rewarded at Etsy for being a visible firefighter, stopping failures on nights or weekends, jumping in front of outage bullets, and that if you take the time to design systems robust enough and safe enough that they don’t need code freezes in the first place, you are never going to get recognized or rewarded. It’s baffling that not only did they have to resort to a long code freeze - but they even approved a public blog post bragging about it. Boy, that tells you a lot about the culture. Not going to be seeking jobs there any time soon. To put in context, I work for an ecommerce company that routinely sees 10-15x traffic around Black Friday -> Cyber Monday, and around Christmas to New Year. We don’t do any code freezes for the load, almost all of which is handled through auto-scaling in our on-prem data centers.
- throwanem 6y ago
  An opinion no doubt informed by extensive experience in high-volume ecommerce.
  - mlthoughts2018 6y ago
    I run a machine learning team in a high traffic ecommerce company (specializing in stock photography and digital assets). We see an even bigger seasonal spike than Etsy.
    - throwanem 6y ago
      With no code chill at all? Not even for artifacts which, presumably unlike what your team works on, lie in the critical path for revenue generation aka checkout? If so, that's very impressive, but also very atypical. Source: I work in high-volume ecommerce, have friends and colleagues at other companies who do likewise, and no one I've talked to about it works anywhere that doesn't implement at least some controls on at least checkout-path deployments during holiday prep and through holiday proper. It isn't a tech debt thing, it's just good business sense: the time of year when people give you by far the most money is the time of year when you least want to risk causing problems for people who want to do so. The problem isn't untrustworthy systems or bad engineering culture; the problem is that anything short of perfection has a measurable impact on revenue, and humans achieve perfection with less than perfect reliability. Even where I work, that revenue impact can easily amount to a major problem if we let a significant bug slip through. People buy our product for its own sake, and will do so from Amazon or Walmart or wherever if they can't get it direct - but our margin on sales through second-party vendors is nothing like as good as on direct sales, so we still take a hit. For Etsy, it's much worse; on sales through other storefronts, they get nothing. Too, a bad enough break during holiday will also risk some fraction of their subscriber revenue from sellers who, fed up at losing out on what would otherwise also be their highest revenue of the year, might go elsewhere. It's easy to talk shit about tech debt and bad engineering culture, I get that. It's also poorly founded in fact, and evinces an apparently questionable grasp on some fundamentals of the business, besides. I don't know whether you have represented yourself well or poorly by creating this impression, but it is the impression you've created, and if that bothers you then I might suggest trying to do otherwise next time.

24 points by kiyanwang 6y ago

Does anyone know why the first deployment of the day needs to compile the whole codebase on all hosts? Stumbled over this part.

jpgvm 6y ago
I think it's simply a badly worded way of saying they would need to push the build to 1000+ hosts after they compile it.

_0o6v 6y ago

Let's hope they're not trying to log in with Google on their phones, because that hasn't worked for about a year now.

leesalminen 6y ago
Just tried it with my iPhone 12 Pro and worked fine. Logged in with Google on the first try.
0xcafecafe 6y ago
I ordered quite a few items from etsy over the holidays and all by logging in with Google on my android phone

doctoboggan 6y ago

I run a small 3D printed jewelry business (https://lulimjewelry.com https://lulimjewelry.com) where I sell on my own website and also on Etsy.com. The Etsy team handled Christmas great this year (although it was my first Christmas in the platform so I don’t really have much to compare it to). Specifically they handled the communication around the mail delays very well, both with me and with my customers. It’s really interesting to see the tech decisions that had to be made to make those communications!

ckdarby 6y ago

The article was rather a disappointment compared to other articles Etsy has put up on their blog. It was overall just lackluster and the TL;DR can be entirely summed up as, more than doubled traffic without issue (and no technical insights really).

victor106 6y ago
I agree. I learnt nothing from it.

mlthoughts2018 6y ago

The need for code freezes around high traffic events is usually a sign of very poor engineering culture, where patching extreme tech debt is much higher priority than technical excellence or product improvements that help the customers. A code semi-freeze for weeks is incredibly alarming. How much unremediated tech debt must be causing that? That is a massive red flag that Etsy engineering culture must be something like the SRE tail wagging the product engineering dog. I would bet that you only get rewarded at Etsy for being a visible firefighter, stopping failures on nights or weekends, jumping in front of outage bullets, and that if you take the time to design systems robust enough and safe enough that they don’t need code freezes in the first place, you are never going to get recognized or rewarded. It’s baffling that not only did they have to resort to a long code freeze - but they even approved a public blog post bragging about it. Boy, that tells you a lot about the culture. Not going to be seeking jobs there any time soon. To put in context, I work for an ecommerce company that routinely sees 10-15x traffic around Black Friday -> Cyber Monday, and around Christmas to New Year. We don’t do any code freezes for the load, almost all of which is handled through auto-scaling in our on-prem data centers.

throwanem 6y ago
An opinion no doubt informed by extensive experience in high-volume ecommerce.
- mlthoughts2018 6y ago
  I run a machine learning team in a high traffic ecommerce company (specializing in stock photography and digital assets). We see an even bigger seasonal spike than Etsy.
  - throwanem 6y ago
    With no code chill at all? Not even for artifacts which, presumably unlike what your team works on, lie in the critical path for revenue generation aka checkout? If so, that's very impressive, but also very atypical. Source: I work in high-volume ecommerce, have friends and colleagues at other companies who do likewise, and no one I've talked to about it works anywhere that doesn't implement at least some controls on at least checkout-path deployments during holiday prep and through holiday proper. It isn't a tech debt thing, it's just good business sense: the time of year when people give you by far the most money is the time of year when you least want to risk causing problems for people who want to do so. The problem isn't untrustworthy systems or bad engineering culture; the problem is that anything short of perfection has a measurable impact on revenue, and humans achieve perfection with less than perfect reliability. Even where I work, that revenue impact can easily amount to a major problem if we let a significant bug slip through. People buy our product for its own sake, and will do so from Amazon or Walmart or wherever if they can't get it direct - but our margin on sales through second-party vendors is nothing like as good as on direct sales, so we still take a hit. For Etsy, it's much worse; on sales through other storefronts, they get nothing. Too, a bad enough break during holiday will also risk some fraction of their subscriber revenue from sellers who, fed up at losing out on what would otherwise also be their highest revenue of the year, might go elsewhere. It's easy to talk shit about tech debt and bad engineering culture, I get that. It's also poorly founded in fact, and evinces an apparently questionable grasp on some fundamentals of the business, besides. I don't know whether you have represented yourself well or poorly by creating this impression, but it is the impression you've created, and if that bothers you then I might suggest trying to do otherwise next time.