3 ms·

I worked at a startup in the very early 2000s that was making online games in Java. Think applets with 2D sprite graphics. They developed their own custom data

I worked at a startup in the very early 2000s that was making online games in Java. Think applets with 2D sprite graphics. They developed their own custom database, a shitty key value store that didn’t even have transactions because “Oracle doesn’t scale and it’s too expensive”. They couldn’t make reporting work so they copied the data hourly to Oracle anyway. (It turned out that Oracle does scale.) They also invented their own RPC protocol, had their own threading library, used an obscure build system, an even more obscure SCM, etc… Nothing was standard. It was all bespoke, webscale, and in-house. When I started there as a junior they had me fix bugs in their code. Instead, I would simply delete thousands of lines of spaghetti and replace it with a call to a standard library. It was like spitting into a volcano to put out the fires of hell itself. There was no hope. They burned $10M and made $250K revenue and collapsed. Their competition used Oracle, ordinary tooling, and made tens of millions in profit.

canadianfella 3y ago
Which company was the competition?
tikhonj 3y ago
As a counterpoint, I interned at Jane Street about a decade ago—when they were much smaller than now—and they also built everything in-house. I'm not just talking about the trading/finance-specific systems (I didn't interact with those as an intern), but general-purpose internal tools: a code review system, a pub-sub system, custom binary and text protocols, lots of developer tooling, their own build system, their own standard library and async runtime... All with <100 developers total. And you know what? It worked amazingly well. I'm not just talking about scale, but also about how much any single developer seemed to be able to do. Their tools and APIs were tastefully designed, tailored to their own needs and fit together remarkably well. I remember that they were consistently building things with just one or a handful of people that would have taken multiple teams at other organizations I've seen. Looking back, that was a remarkably productive engineering environment because of how much they were willing to do themselves. The key insight is that building something for yourself is qualitatively different from building something for external consumption or using something general-purpose that already exists. You naturally tailor the code and tools to what you need, which is much easier than building something for what you think other people might need—and, surprisingly often, easier than bending a different tool to your own ends. But, perhaps more importantly, you develop a shared mental model of the system you're developing and how it fits in with everything else that you are doing. That shared mental model is, ultimately, worth far more than the code by itself... but it isn't legible to management, so it's hard to foster and preserve without a strong engineering culture backing it up. The other lesson I took away from that experience (and others!) is that the high-level questions people focus on ("buy vs build", "monorepo vs polyrepo"... etc) are nowhere near determinative. Whether or not any given approach works out comes down far more to the little details and nuances of what you're doing and how you're doing it, as well as a healthy dollop of circumstances beyond your immediate control. At the end of the day, if everything else was the same, but the startup you're talking about was using perfectly standard and popular tools, would you expect the outcome to be any different? I've seen far too many Enterprise-grade trash-fires to believe that. If the core dynamics that led to a mess don't change, you can have a "not invented here" mess or an "Enterprise Best Practices" mess or a "buy everything, build nothing" mess—but there's no simple strategic direction that won't get you some kind of mess.
- walterbell 3y ago
  Was Jane Street searching for product-market fit? > shared mental model of the system you're developing and how it fits in with everything else that you are doing. That shared mental model is, ultimately, worth far more than the code by itself... but it isn't legible to management, so it's hard to foster and preserve without a really strong engineering culture backing it up. Sounds awesome, did they have engineering leaders with long tenure?
  - andrewjl 3y ago
    > Was Jane Street searching for product-market fit? In a manner of speaking, yes.
  - zrail 3y ago
    Jane Street started as and remains a proprietary trading firm. They added consulting services as some point as well, but that was after they proved their competence.
  - EdwardDiego 3y ago
    I infer long term engineering leaders, just from how coherent their dev culture seems to be. They're big on FP and OCaml and do a lot of open source work around OCaml, and have done for years. Speaks to consistent leadership.
- benjaminwootton 3y ago
  Goldman Sachs are famously the same. Propriety language (Slang), proprietary object database (SecDB) and then a million and one proprietary apps on top of that where a commercial or open source alternative exists. It may have moved on since I worked there, but it seemed to be worth the investment in the early 2000s when I worked there.
  - tikhonj 3y ago
    Yeah. One of my friends and coworkers started his career at GS around that same time and had some (very!) positive stories about both the tools and the people building them. It sounded like the combination of Slang and SecDB was a real force multiplier for building and evaluating the kind of complex mathematical models he was working on. That's the kind of tool development that shows off just how high-leverage engineering work can be. My (100% outsider's) impression is that the focus, culture and teams at Goldman Sachs are totally different now thanks to market and regulatory changes, and that it is not a good place to work in a primarily technical sort of role any more.
deterministic 3y ago
The most successful companies I have worked for in my 30+ years career built most of their tools internally. It gave them a huge competitive advantage. The worst companies I have ever worked for didn’t have the skills to do it. So yet again it depends on the quality of your developers. Not how you do things.

canadianfella 3y ago

Which company was the competition?

tikhonj 3y ago

As a counterpoint, I interned at Jane Street about a decade ago—when they were much smaller than now—and they also built everything in-house. I'm not just talking about the trading/finance-specific systems (I didn't interact with those as an intern), but general-purpose internal tools: a code review system, a pub-sub system, custom binary and text protocols, lots of developer tooling, their own build system, their own standard library and async runtime... All with <100 developers total. And you know what? It worked amazingly well. I'm not just talking about scale, but also about how much any single developer seemed to be able to do. Their tools and APIs were tastefully designed, tailored to their own needs and fit together remarkably well. I remember that they were consistently building things with just one or a handful of people that would have taken multiple teams at other organizations I've seen. Looking back, that was a remarkably productive engineering environment because of how much they were willing to do themselves. The key insight is that building something for yourself is qualitatively different from building something for external consumption or using something general-purpose that already exists. You naturally tailor the code and tools to what you need, which is much easier than building something for what you think other people might need—and, surprisingly often, easier than bending a different tool to your own ends. But, perhaps more importantly, you develop a shared mental model of the system you're developing and how it fits in with everything else that you are doing. That shared mental model is, ultimately, worth far more than the code by itself... but it isn't legible to management, so it's hard to foster and preserve without a strong engineering culture backing it up. The other lesson I took away from that experience (and others!) is that the high-level questions people focus on ("buy vs build", "monorepo vs polyrepo"... etc) are nowhere near determinative. Whether or not any given approach works out comes down far more to the little details and nuances of what you're doing and how you're doing it, as well as a healthy dollop of circumstances beyond your immediate control. At the end of the day, if everything else was the same, but the startup you're talking about was using perfectly standard and popular tools, would you expect the outcome to be any different? I've seen far too many Enterprise-grade trash-fires to believe that. If the core dynamics that led to a mess don't change, you can have a "not invented here" mess or an "Enterprise Best Practices" mess or a "buy everything, build nothing" mess—but there's no simple strategic direction that won't get you some kind of mess.

walterbell 3y ago
Was Jane Street searching for product-market fit? > shared mental model of the system you're developing and how it fits in with everything else that you are doing. That shared mental model is, ultimately, worth far more than the code by itself... but it isn't legible to management, so it's hard to foster and preserve without a really strong engineering culture backing it up. Sounds awesome, did they have engineering leaders with long tenure?
- andrewjl 3y ago
  > Was Jane Street searching for product-market fit? In a manner of speaking, yes.
- zrail 3y ago
  Jane Street started as and remains a proprietary trading firm. They added consulting services as some point as well, but that was after they proved their competence.
- EdwardDiego 3y ago
  I infer long term engineering leaders, just from how coherent their dev culture seems to be. They're big on FP and OCaml and do a lot of open source work around OCaml, and have done for years. Speaks to consistent leadership.
benjaminwootton 3y ago
Goldman Sachs are famously the same. Propriety language (Slang), proprietary object database (SecDB) and then a million and one proprietary apps on top of that where a commercial or open source alternative exists. It may have moved on since I worked there, but it seemed to be worth the investment in the early 2000s when I worked there.
- tikhonj 3y ago
  Yeah. One of my friends and coworkers started his career at GS around that same time and had some (very!) positive stories about both the tools and the people building them. It sounded like the combination of Slang and SecDB was a real force multiplier for building and evaluating the kind of complex mathematical models he was working on. That's the kind of tool development that shows off just how high-leverage engineering work can be. My (100% outsider's) impression is that the focus, culture and teams at Goldman Sachs are totally different now thanks to market and regulatory changes, and that it is not a good place to work in a primarily technical sort of role any more.

deterministic 3y ago

The most successful companies I have worked for in my 30+ years career built most of their tools internally. It gave them a huge competitive advantage. The worst companies I have ever worked for didn’t have the skills to do it. So yet again it depends on the quality of your developers. Not how you do things.