4 ms·
A very good lesson I learned from a previous company (medium size, 1-2K employees) where a C-Level exec was hired in and needed to make some large scale changes
by hn_throwaway_99 4y ago
A very good lesson I learned from a previous company (medium size, 1-2K employees) where a C-Level exec was hired in and needed to make some large scale changes in the company with speed:
1. You can't "fake it" at this level. One of the most impressive things IMO about this exec was his ability to know, at a fairly low level, what every department and org in the company was doing, and to diagnose their strengths and weaknesses very quickly. We'd have quarterly day-long meetings where every department would present, and his ability to quickly hone in on critical low-level details was extremely impressive. I think he would have failed if he had just thought that his job was to "brush with broad strokes". (Aside, this did NOT mean he was a micromanager, it just meant that he had a very good understanding of the details across many departments).
2. It's important to put some structures in place where departments are forced to show some accountability for speed. For example, one metric that I actually hated at the time, but later learned to appreciate the purpose, was that individual departments were judged on the number of A/B tests they ran per month. I hated this metric at the time because I felt it was easily "gamed" - departments would run small little A/B tests like button color changes. However, after a while there were a couple of big cultural changes that had taken place: (1) the company built tools and processes that made it easier to deploy and run tests in the first place (better CI/CD pipelines, better analysis tools, etc.) which had the overall effect of letting us ship faster with higher quality, (2) while yes, there was a lot of "gaming" of count of A/B tests run in the beginning, it didn't take that long for teams to actually run out of tests to game, and people actually put in the hard work of thinking about better tests to run, and (3) it changed our culture to become much more data-driven - it wasn't perfect, and "data driven" can be a double-edged sword, but it was an improvement.
- the_watcher 4y agoThanks for sharing the second part! I’ve worked places as a DS where we were ramping up A/B tests, and it’s honestly one of the worst experiences a DS can have - everybody is asking why you need such large samples, then when you get null results, everyone wants an explanation for why (“this idea didn’t do what you wanted it to” isn’t an acceptable answer), etc. You’re entirely right that running a bunch of tests is a really effective way to advocate for better resources to run the tests. While I’ve never worked anywhere that ran out of ideas to game the testing incentives, it’s definitely true that people who initially fall into the trap of gaming the tests come around with some experience and in general the org builds better intuitions and culture around how to use tests. It may be that an org that _has_ a culture of testing just has them constantly running in the background but they’re minor in terms of time spent running them, but that building a culture of testing initially involves the whole org focusing on it to a seemingly ridiculous extent, as it’s the only way to generate enough momentum to get the infrastructure and institutional knowledge right.
- pyrolistical 4y agoWhat is DS?
- Thorondor 4y agoData scientist, I'd guess.
- tomnipotent 4y agoThis is an important take away. A lot of senior management isn't about specific outcomes, but building muscle memory within an organization that evolves into workflows and processes that level up the company. A favorite quote of mine is "leadership is holding a vision long enough for someone else to realize it for themselves."
- matthewsinclair 4y agoThis is excellent. I also like to use the phrase (to myself as much as anyone else) “keep repeating things until you hear your words used by others”. It’s not until people are saying things back to you as if they were their own ideas that they have fully internalised what you want.
- aledalgrande 4y agoAny books to recommend? Even better if audiobooks
- matthewsinclair 4y agoThat particular idiom is just something I’ve worked out myself, rather than coming from a particular book. But if you want a recommendation, then I’d start with this [0]. It’s the book about leading engineers that I always wanted to write. I can’t recommend it highly enough. And for a bit of shameless self-promotion, I also wrote a blog post [1] a while back on good books for engineers to read (from non-fiction to fiction). You might find a few good tips in there. [0]: https://press.stripe.com/an-elegant-puzzle https://press.stripe.com/an-elegant-puzzle [1]: https://link.medium.com/1FNBlU2H2qb https://link.medium.com/1FNBlU2H2qb
- aledalgrande 4y agowoa didn't know Stripe had a library, thank you so much!
- brightball 4y agoI watched an org limp into better A/B testing as slowly and painfully as possible. They knew why it was important but just weren’t in a hurry to get there. I can see all of the organizational side effects that would come from a metric like this and the upside far outweighs the downside.
- donw 4y agoMetrics will always be "gamed". The key is making the "game" the easiest way to get teams to do what you want done: A/B testing, TDD, etc. Once you've built in that institutional muscle memory, you can deprioritize and move on to other things.