4 ms·

I’ve been sitting on a weird epic ramble about Etsy homepages from the middle 2000’s for the better part of 15 years now so I figured I would write it up as a t

I’ve been sitting on a weird epic ramble about Etsy homepages from the middle 2000’s for the better part of 15 years now so I figured I would write it up as a thread 1/ The story begins while I was still working in the financial industry in lower Manhattan, watching Visual Studio whitescreen for a half hour as it often did and questioning my life choices. 2/ One day while that was happening a woman I knew DM’d me a link and said “I want everything on this website.” 3/ I don’t know what she was trying to link me to because the site was broken, and it served me this error page. “Hey, sick error page,” I remember thinking. 4/ This kept happening though, and eventually I came to be dimly aware that there was a new site called Etsy with a reputation for being full of beautiful stuff. 5/ That reputation came from the homepage, which at the time was a grid of items that looked like this 6/ The homepage grids changed periodically, and they were always interesting. The items would be beautiful individually, or beautiful together, or it’d contain some oblique joke. They’d be a rainbow of color, or they’d be the same shade of eggshell white, whatever. 7/ It was this really wonderful handcrafted thing about handcrafted things, from the pre-personalization era. 8/ Anyway I became familiar enough with this via osmosis to recognize the name Etsy when I saw it on the python.org jobs page, so I jumped and applied. 9/ There were a few red flags in my Etsy interview which I was not conditioned to notice or emphasize at the time. It was scheduled for 3PM, but didn’t start until 3:30 because my interviewer was “really hung over.” 10/ The interview was also about Lisp. There was this Lisp moment in the middle 2000s. You had to be there. 11/ So after joining the company I eventually learned that the homepages were sourced from a feature called Treasury. Treasuries were a way for Etsy users to create collections, roughly the same as Pinterest these days. 12/ Treasuries were selected and promoted to homepages via human cron. There was one poor employee living the Leonardo Da Vinci lifestyle, waking up every hour around the clock to turn over the homepage. 13/ When I started she’d been at it for a while, and it took another six or eight months for that to end 14/ Obviously this is not the kind of thing a human being should be asked to endure for a startup. But you have to understand, there was a problem of approximately this magnitude everywhere you looked. 15/ ------- So the first version of Etsy (2005) was PHP talking to Postgres, written by someone learning PHP as he was doing it, which was all fine. Postgres was kind of an idiosyncratic choice for the time, but it didn’t matter yet and anyway that’s a different story. ------- The p95 of search in that era was several minutes. The site was down for a full hour each week or so, and not on purpose. The human tragedy of someone waking up every hour to change the homepage felt kind of contained I guess 16/ So like many of the frontend features of Etsy in those years, the Treasury was built in Flash by a visionary named Jared Tarbell, the “fourth founder.” He was more of a visual artist working in Flash as his medium than an engineer. 17/ He lived in New Mexico and was very difficult to get ahold of. Once I talked to him and he told me he was working on etching intricate patterns into rocks with a laser, so he could drop them in the desert for future raccoon civilizations to wonder about. 18/ Search ranking was a big problem for us, and we figured we could improve this if we knew things like “this or that listing is popular in popular treasuries.” 19/ Beyond that, the homepage was the same for everyone looking at it. We thought maybe we could iterate away from that to something less artificially scarce. Getting an item on the homepage was a big deal for sellers, but the math of the situation made it unattainable for most. 20/ The Flash architecture really held those ambitions back. There were only 250 Treasuries live on the site at any given time, which was a function of the RTMP server backend it was using. This was some kind of Flash chat server dingus. 21/ We didn’t even have access to it. There was a contractor managing a Windows box someplace. And there was no database backing it, so as far as we knew once Treasuries were off the site they were lost forever. 22/ In our heads this beautiful, wonderful thing feeding the public perception of Etsy was a small scale tragedy in terms of data loss and lost opportunity. 23/ Back in the constant chaos of daily goings on on the engineering team, we were trying to get our arms around things like security vulnerabilities. We hired some pen testers for the first time, and they found problems everywhere. 24/

UberFly 4y ago
SQL injection, cross site scripting, really just everywhere. If you name a category of security vulnerability, we definitely had it on every page. 25/ Flash was one of the systemic problems. Everywhere we had Flash, we had XSS. And there were worse issues with it, for example, you could impersonate the Etsy CEO in chat with users without authenticating if you wanted. 26/ There was an internal Flash endpoint you could use to exfiltrate all of our sales data, and in fact there was a whole other company that had figured this out and was selling our sales insights to Etsy sellers. 28/ Anyway we realized that, oh god, we are going to need to get our arms around this Flash stuff. And Jared had his hands full with the raccoons and the lasers, so we tried hiring somebody. 29/ We hired this guy that I guess I will call Steve. That’s not his real name. We let Jared interview Steve all by himself, which was a thing we never did again. 30/ Steve worked there for about a month, or a month and a half tops. A couple of vignettes I can remember about Steve: one day he freaked out at me and called me a pseudo-intellectual. Touché. Totally fair. 31/ Steve had this intricate conspiracy theory he believed in centered around Bertrand Russell, which I never figured out. 33/ There was one day when the site was down—the new CEO ran into the eng room yelling at everybody. Everyone was hair on fire freaking out about it. And then suddenly, we all just stopped as we gradually realized that Steve … 34/ … was at his desk watching an Infowars video at maximum volume. He’d been cranking the volume that whole time because he had his headphones on, but they were not plugged in. 35/ One day, I was walking back to the office after lunch and I see Steve coming out of the building. I hid, crouching between two parked cars, so he wouldn’t see me. 36/ When I got up to the office, the head of ops handed me Steve’s laptop. “Here you go, you’ve got to figure this Flash stuff out. We just kicked Steve out.” 37/ The first thing I did was make sure there were neither physical nor virtual booby traps on the laptop. After establishing that it wasn’t going to explode or send my passwords to Steve, I opened up Adobe tools and started fucking around 38/ I quickly surmised that he’d gotten nowhere, and was just watching Infowars videos that entire time. But luckily I got in touch with Jared who delivered a tarball of the Flash source (it wasn’t in source control anywhere) 39/ Somewhere in that tarball I found a set of credentials for an ftp server in a comment. After connecting to it, I do an ls and find a giant pile of binary files, format unknown. 40/ However the file timestamps are a clue: this is the schedule of when the treasuries turn over. It’s the treasury data! 41/ This happened on a Wednesday or a Thursday, and I barely slept through the weekend as I tried to reverse engineer and recover all of this data. Which remember, we all believed was going to be a gold mine that would change everything. 42/ I never figured out what the file format was, but I opened it up in hexl-mode in emacs and defocused my eyes and could figure out the basic size of the records and overall structure. 43/ By Sunday I had figured it out just enough to be able to write a Python script that downloaded all of the data and grepped through it to dump out the Etsy listing ID’s that were in them along with comment counts and other metadata. 44/ And I had reverse engineered Jared’s ranking formula, so I could emit the most popular (and therefore best) treasuries of all time. I hit enter on this script and let it crank for hours. 45/ When it was finally done, after days and days of effort I opened up MacOS finder and flipped through the results. It was almost entirely men with their shirts off, which surprised me at the time. 46/ I went back to the office and never mentioned it. I don’t think anyone even asked what happened with the whole treasury data thing, because again, the entire building was on fire. 47/ “Yeah, I don’t know, the treasury data thing didn’t pan out.” 48/ There apparently was this entire phenomenon of sellers putting up sexy pictures of themselves, and then people making treasuries out of them. But engineers and leadership never knew about this because the moderators were just filtering that stuff out. 49/ A few years later, we rebuilt the treasury and let people have as many treasuries as they wanted. And it turned out that what most of them wanted was to make treasuries full of penises and vote them to the top. 50/ This generated an entire protracted incident where I spent two weeks working on “class DickFilter” - and then pretty much the day that I was ready to deploy this the hivemind decided it wasn’t funny anymore and the meme just kind of dissolved. 51/ I guess it’s a testament to the power of compartmentalization that I rebuilt the entire treasury and didn’t forsee this crisis that was 100% predictable on the basis of my own experience. 52/ Evolution probably weeded out everyone that would have dwelled on the Shirtless Men Result too much and given up back on the Serengeti or something. 53/ There are probably several morals here I guess, but the main one for me was never to overlook the human in the loop. The magic in the system was the person yanking 24 beautiful treasuries a day out of the sea of shirtless dudes. 54/ I still think about this saga a lot, for example when someone suggests in a legal filing that you could find bots on Twitter using AI without worrying about the human labeling. 55/55

UberFly 4y ago

SQL injection, cross site scripting, really just everywhere. If you name a category of security vulnerability, we definitely had it on every page. 25/ Flash was one of the systemic problems. Everywhere we had Flash, we had XSS. And there were worse issues with it, for example, you could impersonate the Etsy CEO in chat with users without authenticating if you wanted. 26/ There was an internal Flash endpoint you could use to exfiltrate all of our sales data, and in fact there was a whole other company that had figured this out and was selling our sales insights to Etsy sellers. 28/ Anyway we realized that, oh god, we are going to need to get our arms around this Flash stuff. And Jared had his hands full with the raccoons and the lasers, so we tried hiring somebody. 29/ We hired this guy that I guess I will call Steve. That’s not his real name. We let Jared interview Steve all by himself, which was a thing we never did again. 30/ Steve worked there for about a month, or a month and a half tops. A couple of vignettes I can remember about Steve: one day he freaked out at me and called me a pseudo-intellectual. Touché. Totally fair. 31/ Steve had this intricate conspiracy theory he believed in centered around Bertrand Russell, which I never figured out. 33/ There was one day when the site was down—the new CEO ran into the eng room yelling at everybody. Everyone was hair on fire freaking out about it. And then suddenly, we all just stopped as we gradually realized that Steve … 34/ … was at his desk watching an Infowars video at maximum volume. He’d been cranking the volume that whole time because he had his headphones on, but they were not plugged in. 35/ One day, I was walking back to the office after lunch and I see Steve coming out of the building. I hid, crouching between two parked cars, so he wouldn’t see me. 36/ When I got up to the office, the head of ops handed me Steve’s laptop. “Here you go, you’ve got to figure this Flash stuff out. We just kicked Steve out.” 37/ The first thing I did was make sure there were neither physical nor virtual booby traps on the laptop. After establishing that it wasn’t going to explode or send my passwords to Steve, I opened up Adobe tools and started fucking around 38/ I quickly surmised that he’d gotten nowhere, and was just watching Infowars videos that entire time. But luckily I got in touch with Jared who delivered a tarball of the Flash source (it wasn’t in source control anywhere) 39/ Somewhere in that tarball I found a set of credentials for an ftp server in a comment. After connecting to it, I do an ls and find a giant pile of binary files, format unknown. 40/ However the file timestamps are a clue: this is the schedule of when the treasuries turn over. It’s the treasury data! 41/ This happened on a Wednesday or a Thursday, and I barely slept through the weekend as I tried to reverse engineer and recover all of this data. Which remember, we all believed was going to be a gold mine that would change everything. 42/ I never figured out what the file format was, but I opened it up in hexl-mode in emacs and defocused my eyes and could figure out the basic size of the records and overall structure. 43/ By Sunday I had figured it out just enough to be able to write a Python script that downloaded all of the data and grepped through it to dump out the Etsy listing ID’s that were in them along with comment counts and other metadata. 44/ And I had reverse engineered Jared’s ranking formula, so I could emit the most popular (and therefore best) treasuries of all time. I hit enter on this script and let it crank for hours. 45/ When it was finally done, after days and days of effort I opened up MacOS finder and flipped through the results. It was almost entirely men with their shirts off, which surprised me at the time. 46/ I went back to the office and never mentioned it. I don’t think anyone even asked what happened with the whole treasury data thing, because again, the entire building was on fire. 47/ “Yeah, I don’t know, the treasury data thing didn’t pan out.” 48/ There apparently was this entire phenomenon of sellers putting up sexy pictures of themselves, and then people making treasuries out of them. But engineers and leadership never knew about this because the moderators were just filtering that stuff out. 49/ A few years later, we rebuilt the treasury and let people have as many treasuries as they wanted. And it turned out that what most of them wanted was to make treasuries full of penises and vote them to the top. 50/ This generated an entire protracted incident where I spent two weeks working on “class DickFilter” - and then pretty much the day that I was ready to deploy this the hivemind decided it wasn’t funny anymore and the meme just kind of dissolved. 51/ I guess it’s a testament to the power of compartmentalization that I rebuilt the entire treasury and didn’t forsee this crisis that was 100% predictable on the basis of my own experience. 52/ Evolution probably weeded out everyone that would have dwelled on the Shirtless Men Result too much and given up back on the Serengeti or something. 53/ There are probably several morals here I guess, but the main one for me was never to overlook the human in the loop. The magic in the system was the person yanking 24 beautiful treasuries a day out of the sea of shirtless dudes. 54/ I still think about this saga a lot, for example when someone suggests in a legal filing that you could find bots on Twitter using AI without worrying about the human labeling. 55/55