7 ms·

The TailScale folks caught a bunch of flak for their "do things that don't scale" approach to databases. But, honestly, most startups would be better off follow

The TailScale folks caught a bunch of flak for their "do things that don't scale" approach to databases. But, honestly, most startups would be better off following that approach than what Fast did. Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had.

deleted 4y ago
[deleted]
randmeerkat 4y ago
> Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had. More like, sounds like management was trying to get as much money as quickly as they could from their VCs before it all imploded. The engineers aren’t at fault that management didn’t have a real product or vision.
- mrkurt 4y ago
  We all know plenty of engineers who want to build things for inappropriate scale. The company was a disaster, but part of building a disastrous company is hiring the engineers who want to make everything webscale and have no sense of pragmatism.
  - randmeerkat 4y ago
    > The company was a disaster, but part of building a disastrous company is hiring the engineers who want to make everything webscale and have no sense of pragmatism. I mean, the engineers held up their side of the bargain. They were hired for webscale, the company got webscale… Now, if this was a more strategic startup, that had tried to hire pragmatic engineers, that insisted on webscale, I would blame the engineers. However, in this case, I think it’s pretty clear that the engineers did exactly what they were hired to do.
hobs 4y ago
Pretty sure the reason they caught that flak is because they chose to reinvent the wheel instead of do a normal thing that would work until later - they're on their what - third rewrite of that now? Tootally worth the time /s
- lolsoftware 4y ago
  I don't think that's a fair characterization of what happened. First, they scaled a simple solution for a long time and migrated away from it before it exploded spectacularly. I don't recall them explicitly saying how much time they spent on it, so how is it possibly to say whether it was worth the time or not? I also don't remember if it was here or on Twitter, but one of the engineers listed the many features they built and shipped _instead_ messing with their database. I'd say that was a good business decision - they got more features to their customers sooner and dealt with a scaling issue before it impacted customers.
  - deleted 4y ago
    [deleted]
  - deleted 4y ago
    [deleted]
- deleted 4y ago
  [deleted]
enra 4y ago
> Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had. This is often the problem when hiring too fast. New employees don't have context or direction but generally people try to be productive, so they start inventing work. Management team is new too, and not built up either direct or handle the bandwidth and they also are lacking the direction. Senior leadership and founders are likely busy hiring more folks and potentially trying to sort through the chaos described before. IMO, pre-product market fit company shouldn't have 500 employees, maybe not even 10. These things don't become easier with more people, usually everything gets harder and slow. You also shouldn't have massive marketing or sales force before you actually have product to sell and have figured out your sales motion. It seemed that Fast was just trying to shortcut their way to be the next Stripe by hiring people instead of actually working on the fundamentals like the product and business.
- kevan 4y ago
  It's not just limited to startups. When my business unit in Amazon spun up we hired way too fast and ended up having a harder time shipping v1 of the product than if we would've started lean and grown organically. We ended up spending a lot of time on a modularized platform to enable ~5 matrixed feature teams to contribute across iOS and Android apps. Looking back, a small dedicated team for iOS and another team for Android probably could've shipped v1 in less than half the time with a lot less tech debt.
  - enra 4y ago
    Yeah for sure. I think even senior leaders forget the team culture/context piece, and try to scale up too fast because they are used to it in their current orgs or previous companies. If you have a 500 person org that has operated for years you can probably add 10%-20% new people each year without it breaking too bad. But it can be very hard to build a complete function from 0 to 100 people within a year because there is no infra, culture or understanding how things work. The new people cannot be absorbed/aligned because there is no central gravity so people easily end up pulling different directions and spending their time on internal coordination vs actual useful output.
xen0 4y ago
I wonder, how many engineers had full time jobs just developing and maintaining the various integrations with all their small clients?
- mywittyname 4y ago
  I have to imagine they were all stepping on each other and/or reinventing slightly different shaped wheels too. My experience has been that companies who will customize integrations for you have poorly designed products. By that, I mean, when we ask for a feature, and they bring on an engineer who does some requirements gathering then comes back with some invisible solution. That's a major redflag. The success of an integration, in my experience, is directly proportional to the amount of the work I'm able to handle as a client.
  - vimwizard 4y ago
    >My experience has been that companies who will customize integrations for you have poorly designed products. By that, I mean, when we ask for a feature, and they bring on an engineer who does some requirements gathering then comes back with some invisible solution. That's a major redflag. I'm feeling a bit called out right now (not an owner but that's my experience, too)
nonameiguess 4y ago
Staying employed for another few decades without losing seniority in an industry that expects you to stay current on all of the latest hottest fads is a problem the engineers actually had.
deleted 4y ago
[deleted]
zozbot234 4y ago
Tailscale's "database from 2022" is neither here nor there. What you should do in this situation is use the most boring tech, such as PostgreSQL with a simple HA setup. Thus avoiding the "play with nifty toys" trap and maximizing the chance of making appropriate choices.
- mwcampbell 4y ago
  Perhaps one problem with the current backlash against over-complicated infrastructure is that for some of us, doing everything in one process on one machine using a local embedded database is its own form of playing with nifty toys.
  - bryanrasmussen 4y ago
    yeah, I guess, I'm not going to get too specific here but I do think it would be nifty if I didn't go through a couple 10 minute periods every day where I have to basically shut everything down run a number of scripts, then restart individual processes that are problematic, to be able to work again. Note that this basically works 99% of the time now and is part of a learning process that had the whole thing taking 2 - 5 hours every day for several weeks, down to 2 hours a day for several weeks after that, and now down to the nearly inescapable 20 - 40 minutes a day I have now. Yeah, not doing that would be my own form of playing with nifty toys.
    - collinmanderson 4y ago
      Wow. I would find it super annoying to be interrupted twice a day to do a 10min task. Can’t you try to narrow down the underlying bug? Or somehow automate the restart?
- randomdata 4y ago
  The most boring tech imaginable would be a file. Perhaps containing JSON or similar structured data that can be handled using commonly available libraries. Which is exactly what they used for a long time. I've done something similar before and it saved countless hours wasted on configuring databases, writing queries, fighting impedance mismatches. Postgres brings a lot of work. With a file you just need a couple of lines to write your memory out to a file and a couple more to read it back. Simple and effective. Until it stops scaling, but at that point you've established that your business is successful enough that it warrants the effort of doing something shiny. Postgres has a place in this world, but no need to put the cart before the horse.
SkyPuncher 4y ago
I've done startup engineering for a while. I've seen exactly 1 issue that we couldn't solve easily (N+1) or with simply turning our server count up (often way cheaper than putting an engineer to a problem). Basically, to move development velocity faster we "load everything". Seems like a really bad idea on the surface. In practice, we only ran into an issue with a single customer that was literally 1000x as large as the other customers. It took about two weeks by 1 engineer to rework a select few calls that were egregiously slow and we were back at it. ---- In other words, even with some arguably inefficient technical decisions, I've rarely if ever seen performance issues at early stage startups.
- tomp 4y ago
  Even before using more servers, one should consider using (one) bigger server (and another as backup). Massive servers are laughably cheap these days. E.g. 16 cores, 128GB RAM is €112 from Hetzner.
  - mwcampbell 4y ago
    Damn, wish those dedicated servers were available in Hetzner's US data center. As far as I'm aware, the closest thing to a competitor in the US is OVH, and the server you get for that price is about a quarter of the capacity. That's plenty for my little company, but still...
    - hardwaresofton 4y ago
      Same here -- I'm eagerly awaiting them having a dedicated server offering In the meantime have you looked at leaseweb? https://www.leaseweb.com/dedicated-servers#US https://www.leaseweb.com/dedicated-servers#US I'm planning on running nimbus web services[0] there and there are a few other smaller providers I have earmarked depending on how brave you are: https://www.defendhosting.com/usa-unmanaged-servers/ https://www.defendhosting.com/usa-unmanaged-servers/ https://cc.delimiter.com/cart/dedicated-servers/ https://cc.delimiter.com/cart/dedicated-servers/ https://greenserver.io https://greenserver.io [0]: https://nimbusws.com https://nimbusws.com
  - 4y ago
lmeyerov 4y ago
It wasn't "doesn't scale to 10-100X bigger" but "doesn't burn themselves and their teammates out 6mo from now and needlessly risk a stressful & prolonged sev0 when one of the 1-2 people who did weird things gets COVID / goes on a honeymoon / leaves for netflix". Spending time on needless infra (scale you don't need, AI you don't need, infra you don't need, things could have been postgres as in this case, ...) both prevents building useful stuff ($-generating features/experiences/...) and increases operational cost (maintenance, debugging, ...). Both the lost revenue growth and the increased costs are compounding. Imagine a maniac steadily piling on a bit more technical debt every day and eat a bit more of the seed corn every day instead of growing it. A good phrase for this is "playing house": http://www.paulgraham.com/before.html http://www.paulgraham.com/before.html
wnevets 4y ago
> Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had. Programmers love to add the latest tech to their resume then move on to a new position within ~18 months
- CoastalCoder 4y ago
  That may be specific to early-mid career focus If/when a developer builds deep knowledge of a problem domain, things like framework-of-the-month may become an unwelcome distraction.
  - teaearlgraycold 4y ago
    Seeing someone using nothing but tried and true tech to solve new problems is a resume green flag.
    - Nextgrid 4y ago
      Is there a place where such "green flags" would net me money? It seems like as a software engineering employee, the roles that require me to waste time reinventing the wheel and battling self-inflicted complexity pay significantly more than those who require solving actual problems with boring tech.
      - teaearlgraycold 4y ago
        > Is there a place where such "green flags" would net me money? Well the best way to reap the benefits of your experience is to run your own company. Not screwing yourself over with BS tech is a competitive market advantage. However you might need to hire people so you can’t pick something too old. Working for other people it’ll mostly net you less on call headaches.
lupire 4y ago
> The TailScale folks caught a bunch of flak for their "do things that don't scale" approach to databases. I've seen this repeated on HN, but never saw the alleged flak.
- Aeolun 4y ago
  It’s a post from around a week ago. They went from embedded JSON, to something else (equally crazy), to embedded JSON again. Lots of laughs were had (and a lot of, just use postgres).
  - randomdata 4y ago
    > They went from embedded JSON, to something else (equally crazy), to embedded JSON again. They went from a JSON file, to etcd, to SQLite. etcd seems a little misplaced, but presumably it was already in their infrastructure and they thought they could save time leveraging it. The file-based approach seems appropriate for their particular use-case, though. It's not like it's a Rails app.
    - Aeolun 4y ago
      Crap, I’d forgotten about the sqlite part. You are of course correct. Apparently I’d only remembered the fact they went back to something file based :/ I’m still of the opinion that whatever you are storing, if your alternatives are JSON or Sqlite, etcd is a really strange/unconventional choice.
mooreds 4y ago
Yes! I've seen a few startups that seemed to overengineer things (or build custom solutions rather than use something off the shelf) in expectation of massive growth. I also thought it might have been due to resume driven development and the need to keep engineers engaged so they didn't leave. I've also seen successful startups with a monolithic rails or PHP application that ran on heroku with next to zero custom architecture components.

deleted 4y ago

[deleted]

randmeerkat 4y ago

> Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had. More like, sounds like management was trying to get as much money as quickly as they could from their VCs before it all imploded. The engineers aren’t at fault that management didn’t have a real product or vision.

mrkurt 4y ago
We all know plenty of engineers who want to build things for inappropriate scale. The company was a disaster, but part of building a disastrous company is hiring the engineers who want to make everything webscale and have no sense of pragmatism.
- randmeerkat 4y ago
  > The company was a disaster, but part of building a disastrous company is hiring the engineers who want to make everything webscale and have no sense of pragmatism. I mean, the engineers held up their side of the bargain. They were hired for webscale, the company got webscale… Now, if this was a more strategic startup, that had tried to hire pragmatic engineers, that insisted on webscale, I would blame the engineers. However, in this case, I think it’s pretty clear that the engineers did exactly what they were hired to do.

hobs 4y ago

Pretty sure the reason they caught that flak is because they chose to reinvent the wheel instead of do a normal thing that would work until later - they're on their what - third rewrite of that now? Tootally worth the time /s

lolsoftware 4y ago
I don't think that's a fair characterization of what happened. First, they scaled a simple solution for a long time and migrated away from it before it exploded spectacularly. I don't recall them explicitly saying how much time they spent on it, so how is it possibly to say whether it was worth the time or not? I also don't remember if it was here or on Twitter, but one of the engineers listed the many features they built and shipped _instead_ messing with their database. I'd say that was a good business decision - they got more features to their customers sooner and dealt with a scaling issue before it impacted customers.
- deleted 4y ago
  [deleted]
- deleted 4y ago
  [deleted]
deleted 4y ago
[deleted]

enra 4y ago

> Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had. This is often the problem when hiring too fast. New employees don't have context or direction but generally people try to be productive, so they start inventing work. Management team is new too, and not built up either direct or handle the bandwidth and they also are lacking the direction. Senior leadership and founders are likely busy hiring more folks and potentially trying to sort through the chaos described before. IMO, pre-product market fit company shouldn't have 500 employees, maybe not even 10. These things don't become easier with more people, usually everything gets harder and slow. You also shouldn't have massive marketing or sales force before you actually have product to sell and have figured out your sales motion. It seemed that Fast was just trying to shortcut their way to be the next Stripe by hiring people instead of actually working on the fundamentals like the product and business.

kevan 4y ago
It's not just limited to startups. When my business unit in Amazon spun up we hired way too fast and ended up having a harder time shipping v1 of the product than if we would've started lean and grown organically. We ended up spending a lot of time on a modularized platform to enable ~5 matrixed feature teams to contribute across iOS and Android apps. Looking back, a small dedicated team for iOS and another team for Android probably could've shipped v1 in less than half the time with a lot less tech debt.
- enra 4y ago
  Yeah for sure. I think even senior leaders forget the team culture/context piece, and try to scale up too fast because they are used to it in their current orgs or previous companies. If you have a 500 person org that has operated for years you can probably add 10%-20% new people each year without it breaking too bad. But it can be very hard to build a complete function from 0 to 100 people within a year because there is no infra, culture or understanding how things work. The new people cannot be absorbed/aligned because there is no central gravity so people easily end up pulling different directions and spending their time on internal coordination vs actual useful output.

xen0 4y ago

I wonder, how many engineers had full time jobs just developing and maintaining the various integrations with all their small clients?

mywittyname 4y ago
I have to imagine they were all stepping on each other and/or reinventing slightly different shaped wheels too. My experience has been that companies who will customize integrations for you have poorly designed products. By that, I mean, when we ask for a feature, and they bring on an engineer who does some requirements gathering then comes back with some invisible solution. That's a major redflag. The success of an integration, in my experience, is directly proportional to the amount of the work I'm able to handle as a client.
- vimwizard 4y ago
  >My experience has been that companies who will customize integrations for you have poorly designed products. By that, I mean, when we ask for a feature, and they bring on an engineer who does some requirements gathering then comes back with some invisible solution. That's a major redflag. I'm feeling a bit called out right now (not an owner but that's my experience, too)

nonameiguess 4y ago

Staying employed for another few decades without losing seniority in an industry that expects you to stay current on all of the latest hottest fads is a problem the engineers actually had.

deleted 4y ago

[deleted]

zozbot234 4y ago

Tailscale's "database from 2022" is neither here nor there. What you should do in this situation is use the most boring tech, such as PostgreSQL with a simple HA setup. Thus avoiding the "play with nifty toys" trap and maximizing the chance of making appropriate choices.

mwcampbell 4y ago
Perhaps one problem with the current backlash against over-complicated infrastructure is that for some of us, doing everything in one process on one machine using a local embedded database is its own form of playing with nifty toys.
- bryanrasmussen 4y ago
  yeah, I guess, I'm not going to get too specific here but I do think it would be nifty if I didn't go through a couple 10 minute periods every day where I have to basically shut everything down run a number of scripts, then restart individual processes that are problematic, to be able to work again. Note that this basically works 99% of the time now and is part of a learning process that had the whole thing taking 2 - 5 hours every day for several weeks, down to 2 hours a day for several weeks after that, and now down to the nearly inescapable 20 - 40 minutes a day I have now. Yeah, not doing that would be my own form of playing with nifty toys.
  - collinmanderson 4y ago
    Wow. I would find it super annoying to be interrupted twice a day to do a 10min task. Can’t you try to narrow down the underlying bug? Or somehow automate the restart?
randomdata 4y ago
The most boring tech imaginable would be a file. Perhaps containing JSON or similar structured data that can be handled using commonly available libraries. Which is exactly what they used for a long time. I've done something similar before and it saved countless hours wasted on configuring databases, writing queries, fighting impedance mismatches. Postgres brings a lot of work. With a file you just need a couple of lines to write your memory out to a file and a couple more to read it back. Simple and effective. Until it stops scaling, but at that point you've established that your business is successful enough that it warrants the effort of doing something shiny. Postgres has a place in this world, but no need to put the cart before the horse.

SkyPuncher 4y ago

I've done startup engineering for a while. I've seen exactly 1 issue that we couldn't solve easily (N+1) or with simply turning our server count up (often way cheaper than putting an engineer to a problem). Basically, to move development velocity faster we "load everything". Seems like a really bad idea on the surface. In practice, we only ran into an issue with a single customer that was literally 1000x as large as the other customers. It took about two weeks by 1 engineer to rework a select few calls that were egregiously slow and we were back at it. ---- In other words, even with some arguably inefficient technical decisions, I've rarely if ever seen performance issues at early stage startups.

tomp 4y ago
Even before using more servers, one should consider using (one) bigger server (and another as backup). Massive servers are laughably cheap these days. E.g. 16 cores, 128GB RAM is €112 from Hetzner.
- mwcampbell 4y ago
  Damn, wish those dedicated servers were available in Hetzner's US data center. As far as I'm aware, the closest thing to a competitor in the US is OVH, and the server you get for that price is about a quarter of the capacity. That's plenty for my little company, but still...
  - hardwaresofton 4y ago
    Same here -- I'm eagerly awaiting them having a dedicated server offering In the meantime have you looked at leaseweb? https://www.leaseweb.com/dedicated-servers#US https://www.leaseweb.com/dedicated-servers#US I'm planning on running nimbus web services[0] there and there are a few other smaller providers I have earmarked depending on how brave you are: https://www.defendhosting.com/usa-unmanaged-servers/ https://www.defendhosting.com/usa-unmanaged-servers/ https://cc.delimiter.com/cart/dedicated-servers/ https://cc.delimiter.com/cart/dedicated-servers/ https://greenserver.io https://greenserver.io [0]: https://nimbusws.com https://nimbusws.com
- 4y ago

lmeyerov 4y ago

It wasn't "doesn't scale to 10-100X bigger" but "doesn't burn themselves and their teammates out 6mo from now and needlessly risk a stressful & prolonged sev0 when one of the 1-2 people who did weird things gets COVID / goes on a honeymoon / leaves for netflix". Spending time on needless infra (scale you don't need, AI you don't need, infra you don't need, things could have been postgres as in this case, ...) both prevents building useful stuff ($-generating features/experiences/...) and increases operational cost (maintenance, debugging, ...). Both the lost revenue growth and the increased costs are compounding. Imagine a maniac steadily piling on a bit more technical debt every day and eat a bit more of the seed corn every day instead of growing it. A good phrase for this is "playing house": http://www.paulgraham.com/before.html http://www.paulgraham.com/before.html

wnevets 4y ago

> Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had. Programmers love to add the latest tech to their resume then move on to a new position within ~18 months

CoastalCoder 4y ago
That may be specific to early-mid career focus If/when a developer builds deep knowledge of a problem domain, things like framework-of-the-month may become an unwelcome distraction.
- teaearlgraycold 4y ago
  Seeing someone using nothing but tried and true tech to solve new problems is a resume green flag.
  - Nextgrid 4y ago
    Is there a place where such "green flags" would net me money? It seems like as a software engineering employee, the roles that require me to waste time reinventing the wheel and battling self-inflicted complexity pay significantly more than those who require solving actual problems with boring tech.
    - teaearlgraycold 4y ago
      > Is there a place where such "green flags" would net me money? Well the best way to reap the benefits of your experience is to run your own company. Not screwing yourself over with BS tech is a competitive market advantage. However you might need to hire people so you can’t pick something too old. Working for other people it’ll mostly net you less on call headaches.

lupire 4y ago

> The TailScale folks caught a bunch of flak for their "do things that don't scale" approach to databases. I've seen this repeated on HN, but never saw the alleged flak.

Aeolun 4y ago
It’s a post from around a week ago. They went from embedded JSON, to something else (equally crazy), to embedded JSON again. Lots of laughs were had (and a lot of, just use postgres).
- randomdata 4y ago
  > They went from embedded JSON, to something else (equally crazy), to embedded JSON again. They went from a JSON file, to etcd, to SQLite. etcd seems a little misplaced, but presumably it was already in their infrastructure and they thought they could save time leveraging it. The file-based approach seems appropriate for their particular use-case, though. It's not like it's a Rails app.
  - Aeolun 4y ago
    Crap, I’d forgotten about the sqlite part. You are of course correct. Apparently I’d only remembered the fact they went back to something file based :/ I’m still of the opinion that whatever you are storing, if your alternatives are JSON or Sqlite, etcd is a really strange/unconventional choice.

mooreds 4y ago

Yes! I've seen a few startups that seemed to overengineer things (or build custom solutions rather than use something off the shelf) in expectation of massive growth. I also thought it might have been due to resume driven development and the need to keep engineers engaged so they didn't leave. I've also seen successful startups with a monolithic rails or PHP application that ran on heroku with next to zero custom architecture components.