4 ms·

Documenting what you're willing to support (and not)

98 points by zdw 1y ago

bravesoul2 1y ago
> At some point, I realized that if I wrote a wiki page and documented the things that we were willing to support, I could wait about six months and then it would be like it had always been there. Enough people went through the revolving doors of that place such that six months' worth of employee turnover was sufficient to make it look like a whole other company. All I had to do was write it, wait a bit, then start citing it when needed. Like!
paol 1y ago
> a giant social network. You know, the one with all of the cat pictures This really doesn't narrow it down. > and later the whole genocide thing and enabling fascism. Still not helping.
- leosanchez 1y ago
  > genocide thing I can only think of one social network when I hear that word. Are there others?
  - tehjoker 1y ago
    As of 2023, it's all of them promoting a pro-genocide narrative, particularly Twitter these days is promoting outright pro-genocide accounts in my feed from people I don't follow, but I think the author is referring to the Rohingya genocide case in Myanmar.
    - stdbrouw 1y ago
      See https://www.amnesty.org/en/latest/news/2022/09/myanmar-facebooks-systems-promoted-violence-against-rohingya-meta-owes-reparations-new-report/ https://www.amnesty.org/en/latest/news/2022/09/myanmar-faceb...
- zdc1 1y ago
  The funny thing with social networks is they all have both of these cohorts, so long as you go down the right rabbit-holes and engage with the right content creators
- jraph 1y ago
  Rachel worked at Facebook for some time a while ago. She's been consistently mentioning the cat pictures to refer to it in her blog. I suppose this way of referring to it is also meant to belittle it. I assume the "the whole genocide thing and enabling fascism" part is more a dig than something to narrow down.
- grishka 1y ago
  Yeah, might be Twitter, but might as well be Facebook. Though I'm leaning towards Twitter
  - stingraycharles 1y ago
    I interpreted it as the Rohingya / Myanmar genocide and Facebook’s role in it, and the CambridgeAnalytica scandal that (allegedly) enabled Trump to get elected.
- loloquwowndueo 1y ago
  Maybe she didn’t really want to tell you which one it is :)
- drjasonharrison 1y ago
  Rachel Kroll worked at Facebook. Her work history is not explicitly listed on her blog, but you can find it: https://news.ycombinator.com/item?id=13400687 https://news.ycombinator.com/item?id=13400687 https://medium.com/wogrammer/rachel-kroll-7944eeb8c692 https://medium.com/wogrammer/rachel-kroll-7944eeb8c692 https://www.usenix.org/conference/srecon16/speaker-or-organizer/rachel-kroll-facebook https://www.usenix.org/conference/srecon16/speaker-or-organi...
znpy 1y ago
> I used to be on a team that was responsible for the care and feeding of a great many Linux boxes which together constituted the "web tier" for a giant social network. You know, the one with all of the cat pictures... and later the whole genocide thing and enabling fascism. Yeah, them. I love how people are so willing to criticise companies that paid their salary but not their willingness to ignore the issues in the name of the fat paycheque. This is clearly a reference to Meta, and in that sense the writing has been on the wall for years. I wonder if the author feels or takes any responsibility in directly enabling that genocide and that fascism with their direct work. But hey, a fat paycheque is a fat paycheque.
- philipwhiuk 1y ago
  Maybe she left when it transitioned from cat pictures to fascism.
- ivape 1y ago
  I would say you were less of a douchebag for taking those gigs once upon a time than now days.
RhysU 1y ago
Also, document what "support" means. "That's supported" has no universal interpretation beyond physically describing a tabletop atop legs. Absent a clear/consistent definition, people interpret "support" in the most favorable way possible from the seats in which they sit. Then, all around sadness ensues.
- bombcar 1y ago
  This is really important - because "not supported" can range from "eh, we don't have to help you if it explodes" (which they often DO help with, but it's not official) to "this is literally against the license, the law, and the basics of human decency to even attempt".
simonw 1y ago
Related, the concept of the "golden path" advocated by Charity Majors: https://charity.wtf/2018/12/02/software-sprawl-the-golden-path-and-scaling-teams-with-agency/ https://charity.wtf/2018/12/02/software-sprawl-the-golden-pa... > 1. Assemble a small council of trusted senior engineers > 2. Task them with creating a recommended list of default components for developers to use when building out new services. This will be your Golden Path, the path of convergence (and the path of least resistance). > 3. Tell all your engineers that going forward, the Golden Path will be fully supported by the org. Upgrades, patches, security fixes; backups, monitoring, build pipeline; deploy tooling, artifact versioning, development environment, even tier 1 on call support. Pave the path with gold. Nobody HAS to use these components … but if they don’t, they’re on their own. They will have to support it themselves.
- hnthrow90348765 1y ago
  >or maybe it’s their executives who are afraid? I'd say yes. It seems like an unforced error to them to broadcast publicly how bad things might be, but that's because they want software to be an efficient factory that turns computer cycles into money.
- roryirvine 1y ago
  You do need to be careful that you don't end up with a "build it and they will come" mindset when creating that Golden Path, though - it needs to have early and continuous input from actual users. There's a tension between a theoretical Golden Path that leads someplace no-one actually wants to go, and simply paving every possible "desire line". Managing that is one of the trickiest parts of platform engineering.
- cogman10 1y ago
  Step 3 is where I've seen my org completely fall on its face. Yes, the golden path is "fully supported" yet after a year or two the company executive swoop in and say "why are we spending so much money on the golden path" and slowly, but surely, the support for is whittled away into nothing until the golden path is out of date and actively punishing anyone that chose to use it. For example, one of our golden paths was a UX framework built on top of standard web tech for the time. The team maintaining that framework is no more and it's now very far out of date. Adopting it means you are pulling in Angular circa 2016 and that you'll be dealing with incompatibilities between that and any new web component you want to start using.
Pxtl 1y ago
Reading this I'm mostly shocked that groups had the power to throw boxes onto their outward-facing infrastructure and she had to handle that reactively. Like, it's not "requisition a public network server" but rather "you just jam it out there and we'll either baby it or boot it". That's crazy loose for a major corp.
- deleted 1y ago
  [deleted]
- NewJazz 1y ago
  She mentioned experiments. Facebook is indeed crazy loose with how they handle experiments on the site's user base. https://journals.sagepub.com/doi/full/10.1177/1747016115599568 https://journals.sagepub.com/doi/full/10.1177/17470161155995...
  - Pxtl 1y ago
    Move fast and break people, eh?
    - busterarm 1y ago
      people, governments, political candidates, groups for old ladies trading knitting patterns...all of the above!
- yuliyp 1y ago
  Think "hardware groups with next generation experimental hardware". Facebook runs their own data centers, including designing their own servers. How do you verify that the servers actually perform well? One natural choice was to put them to work as web servers. Given its size, there was a lot of tooling there to be able to measure very precisely what the throughput of a web server was, so things like "How many of these new servers should we order" could be answered, in addition to the "does the rack catch fire" questions. One example of a source of tension that such standards were trying to deal with here was in a group trying to run web servers on machines with SSDs that were way too small: obviously for the bean counter saving a bunch of money on the SSDs was nice, but for the team trying to make sure the disk can fit all the code and logs on it, it was less nice.
BigChemical 1y ago
[dead]
kazinator 1y ago
You could write a program that will check most of the system requirements, and say you must run this program and have it pass.
- fn-mote 1y ago
  It helps, but it’s important to have a name and a green check mark for each result so people understand WHY the magic doesn’t permit them to use the machine.
- 12_throw_away 1y ago
  Yep, this is good. On one occasion, I saved myself a huge amount of back and forth by writing a dumb "test suite" for a software artifact to be shipped to a client; i.e. make sure it was compressed as a zipfile; it included an INSTALL.md; it was installable via [specific installer program]; it included [specific runtime libraries versions]; etc. etc. Turns out human software engineers are really bad at fulfilling even well-specified requirements, so you will always save time by automating as much of the conformance testing as possible.

98 points by zdw 1y ago

> At some point, I realized that if I wrote a wiki page and documented the things that we were willing to support, I could wait about six months and then it would be like it had always been there. Enough people went through the revolving doors of that place such that six months' worth of employee turnover was sufficient to make it look like a whole other company. All I had to do was write it, wait a bit, then start citing it when needed. Like!

paol 1y ago

> a giant social network. You know, the one with all of the cat pictures This really doesn't narrow it down. > and later the whole genocide thing and enabling fascism. Still not helping.

leosanchez 1y ago
> genocide thing I can only think of one social network when I hear that word. Are there others?
- tehjoker 1y ago
  As of 2023, it's all of them promoting a pro-genocide narrative, particularly Twitter these days is promoting outright pro-genocide accounts in my feed from people I don't follow, but I think the author is referring to the Rohingya genocide case in Myanmar.
  - stdbrouw 1y ago
    See https://www.amnesty.org/en/latest/news/2022/09/myanmar-facebooks-systems-promoted-violence-against-rohingya-meta-owes-reparations-new-report/ https://www.amnesty.org/en/latest/news/2022/09/myanmar-faceb...
zdc1 1y ago
The funny thing with social networks is they all have both of these cohorts, so long as you go down the right rabbit-holes and engage with the right content creators
jraph 1y ago
Rachel worked at Facebook for some time a while ago. She's been consistently mentioning the cat pictures to refer to it in her blog. I suppose this way of referring to it is also meant to belittle it. I assume the "the whole genocide thing and enabling fascism" part is more a dig than something to narrow down.
grishka 1y ago
Yeah, might be Twitter, but might as well be Facebook. Though I'm leaning towards Twitter
- stingraycharles 1y ago
  I interpreted it as the Rohingya / Myanmar genocide and Facebook’s role in it, and the CambridgeAnalytica scandal that (allegedly) enabled Trump to get elected.
loloquwowndueo 1y ago
Maybe she didn’t really want to tell you which one it is :)
drjasonharrison 1y ago
Rachel Kroll worked at Facebook. Her work history is not explicitly listed on her blog, but you can find it: https://news.ycombinator.com/item?id=13400687 https://news.ycombinator.com/item?id=13400687 https://medium.com/wogrammer/rachel-kroll-7944eeb8c692 https://medium.com/wogrammer/rachel-kroll-7944eeb8c692 https://www.usenix.org/conference/srecon16/speaker-or-organizer/rachel-kroll-facebook https://www.usenix.org/conference/srecon16/speaker-or-organi...

znpy 1y ago

> I used to be on a team that was responsible for the care and feeding of a great many Linux boxes which together constituted the "web tier" for a giant social network. You know, the one with all of the cat pictures... and later the whole genocide thing and enabling fascism. Yeah, them. I love how people are so willing to criticise companies that paid their salary but not their willingness to ignore the issues in the name of the fat paycheque. This is clearly a reference to Meta, and in that sense the writing has been on the wall for years. I wonder if the author feels or takes any responsibility in directly enabling that genocide and that fascism with their direct work. But hey, a fat paycheque is a fat paycheque.

philipwhiuk 1y ago
Maybe she left when it transitioned from cat pictures to fascism.
ivape 1y ago
I would say you were less of a douchebag for taking those gigs once upon a time than now days.

RhysU 1y ago

Also, document what "support" means. "That's supported" has no universal interpretation beyond physically describing a tabletop atop legs. Absent a clear/consistent definition, people interpret "support" in the most favorable way possible from the seats in which they sit. Then, all around sadness ensues.

bombcar 1y ago
This is really important - because "not supported" can range from "eh, we don't have to help you if it explodes" (which they often DO help with, but it's not official) to "this is literally against the license, the law, and the basics of human decency to even attempt".

simonw 1y ago

Related, the concept of the "golden path" advocated by Charity Majors: https://charity.wtf/2018/12/02/software-sprawl-the-golden-path-and-scaling-teams-with-agency/ https://charity.wtf/2018/12/02/software-sprawl-the-golden-pa... > 1. Assemble a small council of trusted senior engineers > 2. Task them with creating a recommended list of default components for developers to use when building out new services. This will be your Golden Path, the path of convergence (and the path of least resistance). > 3. Tell all your engineers that going forward, the Golden Path will be fully supported by the org. Upgrades, patches, security fixes; backups, monitoring, build pipeline; deploy tooling, artifact versioning, development environment, even tier 1 on call support. Pave the path with gold. Nobody HAS to use these components … but if they don’t, they’re on their own. They will have to support it themselves.

hnthrow90348765 1y ago
>or maybe it’s their executives who are afraid? I'd say yes. It seems like an unforced error to them to broadcast publicly how bad things might be, but that's because they want software to be an efficient factory that turns computer cycles into money.
roryirvine 1y ago
You do need to be careful that you don't end up with a "build it and they will come" mindset when creating that Golden Path, though - it needs to have early and continuous input from actual users. There's a tension between a theoretical Golden Path that leads someplace no-one actually wants to go, and simply paving every possible "desire line". Managing that is one of the trickiest parts of platform engineering.
cogman10 1y ago
Step 3 is where I've seen my org completely fall on its face. Yes, the golden path is "fully supported" yet after a year or two the company executive swoop in and say "why are we spending so much money on the golden path" and slowly, but surely, the support for is whittled away into nothing until the golden path is out of date and actively punishing anyone that chose to use it. For example, one of our golden paths was a UX framework built on top of standard web tech for the time. The team maintaining that framework is no more and it's now very far out of date. Adopting it means you are pulling in Angular circa 2016 and that you'll be dealing with incompatibilities between that and any new web component you want to start using.

Pxtl 1y ago

Reading this I'm mostly shocked that groups had the power to throw boxes onto their outward-facing infrastructure and she had to handle that reactively. Like, it's not "requisition a public network server" but rather "you just jam it out there and we'll either baby it or boot it". That's crazy loose for a major corp.

deleted 1y ago
[deleted]
NewJazz 1y ago
She mentioned experiments. Facebook is indeed crazy loose with how they handle experiments on the site's user base. https://journals.sagepub.com/doi/full/10.1177/1747016115599568 https://journals.sagepub.com/doi/full/10.1177/17470161155995...
- Pxtl 1y ago
  Move fast and break people, eh?
  - busterarm 1y ago
    people, governments, political candidates, groups for old ladies trading knitting patterns...all of the above!
yuliyp 1y ago
Think "hardware groups with next generation experimental hardware". Facebook runs their own data centers, including designing their own servers. How do you verify that the servers actually perform well? One natural choice was to put them to work as web servers. Given its size, there was a lot of tooling there to be able to measure very precisely what the throughput of a web server was, so things like "How many of these new servers should we order" could be answered, in addition to the "does the rack catch fire" questions. One example of a source of tension that such standards were trying to deal with here was in a group trying to run web servers on machines with SSDs that were way too small: obviously for the bean counter saving a bunch of money on the SSDs was nice, but for the team trying to make sure the disk can fit all the code and logs on it, it was less nice.

BigChemical 1y ago

[dead]

kazinator 1y ago

You could write a program that will check most of the system requirements, and say you must run this program and have it pass.

fn-mote 1y ago
It helps, but it’s important to have a name and a green check mark for each result so people understand WHY the magic doesn’t permit them to use the machine.
12_throw_away 1y ago
Yep, this is good. On one occasion, I saved myself a huge amount of back and forth by writing a dumb "test suite" for a software artifact to be shipped to a client; i.e. make sure it was compressed as a zipfile; it included an INSTALL.md; it was installable via [specific installer program]; it included [specific runtime libraries versions]; etc. etc. Turns out human software engineers are really bad at fulfilling even well-specified requirements, so you will always save time by automating as much of the conformance testing as possible.