12 ms·

I think the most applicable warning for engineers when it comes to the actual work, as opposed to whether one should join a particular startup, is this: > Engi

I think the most applicable warning for engineers when it comes to the actual work, as opposed to whether one should join a particular startup, is this: > Engineers calculated the load Fast had in needing to serve their traffic. The Fast button was rendered less than 500,000 times per day - rarely needed to ever serve more than a few requests per second. > One of the few warning signs engineers noticed is how Fast spent far more on infrastructure than the scale of the operation would have called for. Engineers sometimes brought up suggestions to scale infra down, and save costs - given there was not much revenue generated. Sounds like the whole thing could have run on a single cheap VM, perhaps with a second one for redundancy.

lolsoftware 5y ago
The TailScale folks caught a bunch of flak for their "do things that don't scale" approach to databases. But, honestly, most startups would be better off following that approach than what Fast did. Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had.
- deleted 5y ago
  [deleted]
- randmeerkat 5y ago
  > Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had. More like, sounds like management was trying to get as much money as quickly as they could from their VCs before it all imploded. The engineers aren’t at fault that management didn’t have a real product or vision.
  - mrkurt 5y ago
    We all know plenty of engineers who want to build things for inappropriate scale. The company was a disaster, but part of building a disastrous company is hiring the engineers who want to make everything webscale and have no sense of pragmatism.
    - randmeerkat 5y ago
      > The company was a disaster, but part of building a disastrous company is hiring the engineers who want to make everything webscale and have no sense of pragmatism. I mean, the engineers held up their side of the bargain. They were hired for webscale, the company got webscale… Now, if this was a more strategic startup, that had tried to hire pragmatic engineers, that insisted on webscale, I would blame the engineers. However, in this case, I think it’s pretty clear that the engineers did exactly what they were hired to do.
- hobs 5y ago
  Pretty sure the reason they caught that flak is because they chose to reinvent the wheel instead of do a normal thing that would work until later - they're on their what - third rewrite of that now? Tootally worth the time /s
  - lolsoftware 5y ago
    I don't think that's a fair characterization of what happened. First, they scaled a simple solution for a long time and migrated away from it before it exploded spectacularly. I don't recall them explicitly saying how much time they spent on it, so how is it possibly to say whether it was worth the time or not? I also don't remember if it was here or on Twitter, but one of the engineers listed the many features they built and shipped _instead_ messing with their database. I'd say that was a good business decision - they got more features to their customers sooner and dealt with a scaling issue before it impacted customers.
    - deleted 5y ago
      [deleted]
    - deleted 5y ago
      [deleted]
  - deleted 5y ago
    [deleted]
- enra 5y ago
  > Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had. This is often the problem when hiring too fast. New employees don't have context or direction but generally people try to be productive, so they start inventing work. Management team is new too, and not built up either direct or handle the bandwidth and they also are lacking the direction. Senior leadership and founders are likely busy hiring more folks and potentially trying to sort through the chaos described before. IMO, pre-product market fit company shouldn't have 500 employees, maybe not even 10. These things don't become easier with more people, usually everything gets harder and slow. You also shouldn't have massive marketing or sales force before you actually have product to sell and have figured out your sales motion. It seemed that Fast was just trying to shortcut their way to be the next Stripe by hiring people instead of actually working on the fundamentals like the product and business.
  - kevan 5y ago
    It's not just limited to startups. When my business unit in Amazon spun up we hired way too fast and ended up having a harder time shipping v1 of the product than if we would've started lean and grown organically. We ended up spending a lot of time on a modularized platform to enable ~5 matrixed feature teams to contribute across iOS and Android apps. Looking back, a small dedicated team for iOS and another team for Android probably could've shipped v1 in less than half the time with a lot less tech debt.
    - enra 5y ago
      Yeah for sure. I think even senior leaders forget the team culture/context piece, and try to scale up too fast because they are used to it in their current orgs or previous companies. If you have a 500 person org that has operated for years you can probably add 10%-20% new people each year without it breaking too bad. But it can be very hard to build a complete function from 0 to 100 people within a year because there is no infra, culture or understanding how things work. The new people cannot be absorbed/aligned because there is no central gravity so people easily end up pulling different directions and spending their time on internal coordination vs actual useful output.
- xen0 5y ago
  I wonder, how many engineers had full time jobs just developing and maintaining the various integrations with all their small clients?
  - mywittyname 5y ago
    I have to imagine they were all stepping on each other and/or reinventing slightly different shaped wheels too. My experience has been that companies who will customize integrations for you have poorly designed products. By that, I mean, when we ask for a feature, and they bring on an engineer who does some requirements gathering then comes back with some invisible solution. That's a major redflag. The success of an integration, in my experience, is directly proportional to the amount of the work I'm able to handle as a client.
    - vimwizard 5y ago
      >My experience has been that companies who will customize integrations for you have poorly designed products. By that, I mean, when we ask for a feature, and they bring on an engineer who does some requirements gathering then comes back with some invisible solution. That's a major redflag. I'm feeling a bit called out right now (not an owner but that's my experience, too)
- nonameiguess 5y ago
  Staying employed for another few decades without losing seniority in an industry that expects you to stay current on all of the latest hottest fads is a problem the engineers actually had.
- deleted 5y ago
  [deleted]
- zozbot234 5y ago
  Tailscale's "database from 2022" is neither here nor there. What you should do in this situation is use the most boring tech, such as PostgreSQL with a simple HA setup. Thus avoiding the "play with nifty toys" trap and maximizing the chance of making appropriate choices.
  - mwcampbell 5y ago
    Perhaps one problem with the current backlash against over-complicated infrastructure is that for some of us, doing everything in one process on one machine using a local embedded database is its own form of playing with nifty toys.
    - bryanrasmussen 5y ago
      yeah, I guess, I'm not going to get too specific here but I do think it would be nifty if I didn't go through a couple 10 minute periods every day where I have to basically shut everything down run a number of scripts, then restart individual processes that are problematic, to be able to work again. Note that this basically works 99% of the time now and is part of a learning process that had the whole thing taking 2 - 5 hours every day for several weeks, down to 2 hours a day for several weeks after that, and now down to the nearly inescapable 20 - 40 minutes a day I have now. Yeah, not doing that would be my own form of playing with nifty toys.
      - collinmanderson 5y ago
        Wow. I would find it super annoying to be interrupted twice a day to do a 10min task. Can’t you try to narrow down the underlying bug? Or somehow automate the restart?
  - randomdata 5y ago
    The most boring tech imaginable would be a file. Perhaps containing JSON or similar structured data that can be handled using commonly available libraries. Which is exactly what they used for a long time. I've done something similar before and it saved countless hours wasted on configuring databases, writing queries, fighting impedance mismatches. Postgres brings a lot of work. With a file you just need a couple of lines to write your memory out to a file and a couple more to read it back. Simple and effective. Until it stops scaling, but at that point you've established that your business is successful enough that it warrants the effort of doing something shiny. Postgres has a place in this world, but no need to put the cart before the horse.
- SkyPuncher 5y ago
  I've done startup engineering for a while. I've seen exactly 1 issue that we couldn't solve easily (N+1) or with simply turning our server count up (often way cheaper than putting an engineer to a problem). Basically, to move development velocity faster we "load everything". Seems like a really bad idea on the surface. In practice, we only ran into an issue with a single customer that was literally 1000x as large as the other customers. It took about two weeks by 1 engineer to rework a select few calls that were egregiously slow and we were back at it. ---- In other words, even with some arguably inefficient technical decisions, I've rarely if ever seen performance issues at early stage startups.
  - tomp 5y ago
    Even before using more servers, one should consider using (one) bigger server (and another as backup). Massive servers are laughably cheap these days. E.g. 16 cores, 128GB RAM is €112 from Hetzner.
    - mwcampbell 5y ago
      Damn, wish those dedicated servers were available in Hetzner's US data center. As far as I'm aware, the closest thing to a competitor in the US is OVH, and the server you get for that price is about a quarter of the capacity. That's plenty for my little company, but still...
      - hardwaresofton 5y ago
        Same here -- I'm eagerly awaiting them having a dedicated server offering In the meantime have you looked at leaseweb? https://www.leaseweb.com/dedicated-servers#US https://www.leaseweb.com/dedicated-servers#US I'm planning on running nimbus web services[0] there and there are a few other smaller providers I have earmarked depending on how brave you are: https://www.defendhosting.com/usa-unmanaged-servers/ https://www.defendhosting.com/usa-unmanaged-servers/ https://cc.delimiter.com/cart/dedicated-servers/ https://cc.delimiter.com/cart/dedicated-servers/ https://greenserver.io https://greenserver.io [0]: https://nimbusws.com https://nimbusws.com
    - 5y ago
- lmeyerov 5y ago
  It wasn't "doesn't scale to 10-100X bigger" but "doesn't burn themselves and their teammates out 6mo from now and needlessly risk a stressful & prolonged sev0 when one of the 1-2 people who did weird things gets COVID / goes on a honeymoon / leaves for netflix". Spending time on needless infra (scale you don't need, AI you don't need, infra you don't need, things could have been postgres as in this case, ...) both prevents building useful stuff ($-generating features/experiences/...) and increases operational cost (maintenance, debugging, ...). Both the lost revenue growth and the increased costs are compounding. Imagine a maniac steadily piling on a bit more technical debt every day and eat a bit more of the seed corn every day instead of growing it. A good phrase for this is "playing house": http://www.paulgraham.com/before.html http://www.paulgraham.com/before.html
- wnevets 5y ago
  > Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had. Programmers love to add the latest tech to their resume then move on to a new position within ~18 months
  - CoastalCoder 5y ago
    That may be specific to early-mid career focus If/when a developer builds deep knowledge of a problem domain, things like framework-of-the-month may become an unwelcome distraction.
    - teaearlgraycold 5y ago
      Seeing someone using nothing but tried and true tech to solve new problems is a resume green flag.
      - Nextgrid 5y ago
        Is there a place where such "green flags" would net me money? It seems like as a software engineering employee, the roles that require me to waste time reinventing the wheel and battling self-inflicted complexity pay significantly more than those who require solving actual problems with boring tech.
        teaearlgraycold 5y ago
        > Is there a place where such "green flags" would net me money? Well the best way to reap the benefits of your experience is to run your own company. Not screwing yourself over with BS tech is a competitive market advantage. However you might need to hire people so you can’t pick something too old. Working for other people it’ll mostly net you less on call headaches.
- lupire 5y ago
  > The TailScale folks caught a bunch of flak for their "do things that don't scale" approach to databases. I've seen this repeated on HN, but never saw the alleged flak.
  - Aeolun 5y ago
    It’s a post from around a week ago. They went from embedded JSON, to something else (equally crazy), to embedded JSON again. Lots of laughs were had (and a lot of, just use postgres).
    - randomdata 5y ago
      > They went from embedded JSON, to something else (equally crazy), to embedded JSON again. They went from a JSON file, to etcd, to SQLite. etcd seems a little misplaced, but presumably it was already in their infrastructure and they thought they could save time leveraging it. The file-based approach seems appropriate for their particular use-case, though. It's not like it's a Rails app.
      - Aeolun 5y ago
        Crap, I’d forgotten about the sqlite part. You are of course correct. Apparently I’d only remembered the fact they went back to something file based :/ I’m still of the opinion that whatever you are storing, if your alternatives are JSON or Sqlite, etcd is a really strange/unconventional choice.
- mooreds 5y ago
  Yes! I've seen a few startups that seemed to overengineer things (or build custom solutions rather than use something off the shelf) in expectation of massive growth. I also thought it might have been due to resume driven development and the need to keep engineers engaged so they didn't leave. I've also seen successful startups with a monolithic rails or PHP application that ran on heroku with next to zero custom architecture components.
Spivak 5y ago
And if you think this is undersized for comedic effect I did ops for a company that went from <1mil to 15mil revenue (with the same sales strategy of targeting lots of small clients) on 4 1u white label supermicros. Two app servers and two MySQL dbs. Never in my life had such reliable infrastructure.
- jjav 5y ago
  This is a big lesson to heed. Sure, sometimes a startup really needs extravagantly scalable infrastructure. For the 99.9% of startup, no, you don't. So the odds are you don't. Keep it very simple, very small. I can't highlight this enough. A few boxes will get you very far. If you ever actually hit the limits, throw a big party to celebrate success and only then start to think about making your infrastructure a bit (just a bit) more complex.
- StillBored 5y ago
  Early in my career when in a discussion with my manager at the time, he threw out the "planes with two engines have twice as many failures as single engine planes" line. So in my now close to 30 years of experience later, working on systems expected to run 24/7/365, I can't tell you what percentage of failures and outages were caused by the redundancy/fail-over/etc software and hardware layered into a system, but its definitely a very significant part of the problem. Frequently because its just that a "layer" someone bolted on, even if that layer was some software engineer designing for "web scale". All the edge cases are frequently not completely thought out, and when you hit one its a lot harder to recover, than simply restarting a single service running on a single server.
  - amluto 5y ago
    I once managed a thoroughly non-critical server that lived in a closet. In the couple years I managed it, we had quite a few failures due to the UPS tripping and no unscheduled downtime at all from any other reason. I’ve had massive problems with fancy (HPE, IIRC, but it wasn’t called HPE then) raid arrays caused by the controller being a POS. Using plain mdraid would have been far more reliable. Sometimes I wonder if all the effort people put into protocols that run Paxos and Raft could be better spent running a small number of coordinator servers with manual failover. I’m suspicious that, in many use cases, leader elections cause failures more often than the leaders themselves fail.
jonas21 5y ago
It would not have made a difference. Maybe they'd be losing $9.9M a month instead of $10M a month.
- whateveracct 5y ago
  Heh maybe if they also didn't hire a bunch of engineers to scale out too, it would've helped..
- mwcampbell 5y ago
  Sure, their primary problem was extravagant hiring, but I think this is still an important warning for developers working in fledgling companies that are appropriately cautious on the hiring side.
  - mcphage 5y ago
    > Sure, their primary problem was extravagant hiring I think their primary problem was a huge valuation—with correspondingly huge expectations—but no real revenue. If you're pulling in $600K revenue, but valued at $500M... you're gonna have a bad time.
- weezin 5y ago
  The problem was that the engineers were building a scalable infrastructure rather than a scalable business. from the article: "The average volume of Fast customers was low because most customers were from small businesses. And yet, every small business needed custom engineering work to be done, making integration slow. Several engineers mentioned how they did not understand how spending lots of engineering effort for each small client resulting in little revenue would result in building a company that could be worth $12B one day."
EinsDueTresFour 5y ago
I would add that the "Sales vs engineering and sales winning" would be a very relevant warning also: > [sales] signed up a large number of smaller businesses on the platform. [...] However, integrating these smaller businesses was challenging thanks to several customizations needed for each new customer. The ratio of revenue per each small customer vs the total cost of integration (and very likely on-going maintenance) was probably at least an amber flag somewhere for those who had visibility of it. But maybe there was on-going hope that they would be able to sign up a big customer and integrate them before they ran out of money?
- lumost 5y ago
  At some point managers often fear the "red flag" of "we have a giant team with no work/customers". Hence, they will often push for dubious consulting(ish) work to at least ensure everyone remains busy. Unfortunately this can lead to the outcome that the company just loses more money.
- vosper 5y ago
  > The ratio of revenue per each small customer vs the total cost of integration (and very likely on-going maintenance) was probably at least an amber flag somewhere for those who had visibility of it. At the last company I worked at (won't call them a startup, just a 10-year old small business that somehow kept raising funding but has never made a profit) I was astonished to learn from a new CPO that in our tenth year we didn't know if we were making money from a customer. We had no idea how individual customers used the application, which was very data intensive, or what that was costing us. The information was there, but no-one ever looked. Every sale was considered a success, even if it turned out to cost us 3x in data costs and customer support. Often customer support alone would make a sale into a loss. We spent a relatively large portion of our revenue running very expensive servers to respond instantly to searches that no-one ever performed. I created detailed monitoring to show this. I talked about them to everyone, including Product and the CEO. No-one disputed those facts. But instead we had several people essentially dedicated to downsizing application servers and deleting unused S3 buckets, trimming three zeros a month when we could have trimmed five. I have learned not to assume that warnings are visible, that people pay attention to them if they are visible, or that anyone cares. They're still circling the drain, still raising money (somehow!) and I doubt anything's changed.
hinkley 5y ago
I would hope this problem doesn't exist at startups but I've seen it at entrenched institutions any number of time: If your boss's boss or peers signed a contract for $1m a year from Oracle, you're going to use Oracle even if Postgres is a better product. And the only way to use Postgres in this situation is if they don't know you're running it. Which puts you in the uncomfortable situation of wanting to say, "But Oracle sucks, look how successful we've been at using Postgres?". Basically if you don't already know that someone higher up the food chain is comfortable with saying, "I've made a huge mistake", you aren't going to 'help' them learn to do it by pointing out they fucked up. What they're going to hear is that you're asking them to say, "I just wasted a million dollars of company money on something everybody hates," which not only are they not going to do, but they're going to start thinking about how to shoot the messenger. As many conversations go in a couple of my hobbies and talking to friends and acquaintances about health concerns, the answer often starts with, "first, get a time machine" and ends with, "or learn to live with it," often with some useful compromises in between. One of the important things to do early in your tenure at a place is to bid various people in the management chain and those who seem to be on a management track to see how open they are to having their minds changed. Because you want to know this when the stakes are low (also the blowback on low stakes things seems to be less problematic). Then you know if or when a boondoggle is in the making whether you can stop it before it starts, or amplify all of the problems and kill it along the way, or whether you should keep your head down and conspire to keep the 'doggle at arm's length with those who agree with you.
- zozbot234 5y ago
  > If your boss's boss or peers signed a contract for $1m a year from Oracle, you're going to use Oracle even if Postgres is a better product. What you'd do in that situation is make the best of that $1M contract you're stuck with anyway, while being flexible enough to migrate away from that solution when it actually makes sense to do so. Just because you've made a huge mistake doesn't mean that making a different choice after-the-fact is more sensible. Hindsight is always 20/20.
  - Aeolun 5y ago
    Only if your $1M solution is already integrated. Otherwise it’s just a sunk cost fallacy.
rockostrich 5y ago
This is pretty hilarious to read. I did an interview screen with them at the end of last year for a data engineering position (former coworker was high up on the design side of things and he gave a referral). The first bit of the technical question was just "write a SQL query" and boiled down to doing a group by, count, and limit. The second was something like "what if we needed to return this query from a distributed system without going to the database?" without much context. As any sane interviewer would do, I asked for context around scale, infrastructure, and what "without going to the database" even meant. The interviewer just reiterated the question. I started talking about different approaches depending on context and all the interviewer did was tell me to program a solution. Turns out all he wanted was a time-windowed hash-map to cache values and the fact that it's a distributed system didn't matter at all. I think that just about sums up their approach to engineering solutions.
- teruakohatu 5y ago
  What is a time-windowed hash-map? Just a time based key-value store?
  - omalleyt 5y ago
    I think he means an LRU cache
  - agilob 5y ago
    Yes, a hash map that expires objects and loads new KV when key isn't present inside https://github.com/google/guava/wiki/CachesExplained https://github.com/google/guava/wiki/CachesExplained
    - lupire 5y ago
      No, it's a rolling buffer of last N hours/days of data, but compressed into a hash table of counters instead of a log array.
- deleted 5y ago
  [deleted]
fourseventy 5y ago
I run a company in the e-commerce tech space. We handle ~1M requests per day and run it all on a few medium sized VMs on Google cloud for like $1k/mo.
- whymauri 5y ago
  time to raise a $100M and go w e b s c a l e
mylons 5y ago
this sounds like Ridgeline, a startup most have never heard of, founded by the founder of Workday and some of its early employees.
- ammanley 5y ago
  I have actually heard of this startup amazingly, as well as the founder, but I wasn't aware it was/is? as much of a s.show that the article portrays at Fast. Do you have any context you might be willing to share, I am genuinely curious.
  - mylons 5y ago
    bizarre tech decisions have led to insane costs with 0 customers. launch is about 2 years behind. having a lot of trouble hiring senior engineers, when they do they don't listen to them. they have a strange platform that every product team must use. tech leadership has never launched a successful startup before. they have over 300 employees, maybe 400 now. 4 levels of management already, and a _lot_ of them. it's independently funded for now, and has the potential to keep on going like this as long as Dave D. doesn't pull the plug.
    - ammanley 5y ago
      I guess its not _that_ different than that one time I had 4 VPs of Engineering, for 30 engineers </s> Sounds like an absolute bonkers-fest, thank you for sharing.
deleted 5y ago
[deleted]
suresk 5y ago
It's always funny to read about crazy overspending on infrastructure and I have a few funny stories myself, but this was far from their biggest problem. For one, we don't know how big those costs were or how much time it would take to wind some of it down. I absolutely hate waste and was always proud of myself when I found ways to shave a thousand here and there off our AWS bill, but it was a bit depressing when I went through the thought exercise of "What would it be like for the company if I managed to get everything onto the AWS free tier?" And then compared that to what it would be like for the company if we found a way to make 1 or 2% more in revenue. Early on, the math heavily favors worrying less about infra spend and more about product enhancements that can build revenue. (Although often reducing the size of your infra for other reasons - usually around complexity for ops or ease of development can also have the side benefit of reducing cost and it seems worthwhile in those cases.)
- geoduck14 5y ago
  >Early on, the math heavily favors worrying less about infra spend and more about product enhancements that can build revenue. YES! This is VERY true
  - myrandomcomment 5y ago
    It is a careful balance. At my last startup we knew we had scale technical debt in our SW. Not something we could fix by tossing HW at the issue. It was all great until we landed that major customer that blew past our supported scale. 6 months of engineering work and a complete stop of feature process. A more careful consideration of the tradeoffs when making them would have saved a huge headache, but no one wanted to look at the issue preferring to ignore it.
    - mwcampbell 5y ago
      > Not something we could fix by tossing HW at the issue. I'm curious to learn why, if there's any more you can share. What about vertical scaling, i.e. bigger machines instead of more of them? Can't that go pretty far?
ec109685 5y ago
They had 20 user designers / researchers. Just massively over done for a company that was just getting traction: https://twitter.com/marianoaavila/status/1511410574327881728 https://twitter.com/marianoaavila/status/1511410574327881728
Nextgrid 5y ago
> Sounds like the whole thing could have run on a single cheap VM, perhaps with a second one for redundancy. You're describing most startups here.
- ejb999 5y ago
  very true

lolsoftware 5y ago

The TailScale folks caught a bunch of flak for their "do things that don't scale" approach to databases. But, honestly, most startups would be better off following that approach than what Fast did. Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had.

deleted 5y ago
[deleted]
randmeerkat 5y ago
> Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had. More like, sounds like management was trying to get as much money as quickly as they could from their VCs before it all imploded. The engineers aren’t at fault that management didn’t have a real product or vision.
- mrkurt 5y ago
  We all know plenty of engineers who want to build things for inappropriate scale. The company was a disaster, but part of building a disastrous company is hiring the engineers who want to make everything webscale and have no sense of pragmatism.
  - randmeerkat 5y ago
    > The company was a disaster, but part of building a disastrous company is hiring the engineers who want to make everything webscale and have no sense of pragmatism. I mean, the engineers held up their side of the bargain. They were hired for webscale, the company got webscale… Now, if this was a more strategic startup, that had tried to hire pragmatic engineers, that insisted on webscale, I would blame the engineers. However, in this case, I think it’s pretty clear that the engineers did exactly what they were hired to do.
hobs 5y ago
Pretty sure the reason they caught that flak is because they chose to reinvent the wheel instead of do a normal thing that would work until later - they're on their what - third rewrite of that now? Tootally worth the time /s
- lolsoftware 5y ago
  I don't think that's a fair characterization of what happened. First, they scaled a simple solution for a long time and migrated away from it before it exploded spectacularly. I don't recall them explicitly saying how much time they spent on it, so how is it possibly to say whether it was worth the time or not? I also don't remember if it was here or on Twitter, but one of the engineers listed the many features they built and shipped _instead_ messing with their database. I'd say that was a good business decision - they got more features to their customers sooner and dealt with a scaling issue before it impacted customers.
  - deleted 5y ago
    [deleted]
  - deleted 5y ago
    [deleted]
- deleted 5y ago
  [deleted]
enra 5y ago
> Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had. This is often the problem when hiring too fast. New employees don't have context or direction but generally people try to be productive, so they start inventing work. Management team is new too, and not built up either direct or handle the bandwidth and they also are lacking the direction. Senior leadership and founders are likely busy hiring more folks and potentially trying to sort through the chaos described before. IMO, pre-product market fit company shouldn't have 500 employees, maybe not even 10. These things don't become easier with more people, usually everything gets harder and slow. You also shouldn't have massive marketing or sales force before you actually have product to sell and have figured out your sales motion. It seemed that Fast was just trying to shortcut their way to be the next Stripe by hiring people instead of actually working on the fundamentals like the product and business.
- kevan 5y ago
  It's not just limited to startups. When my business unit in Amazon spun up we hired way too fast and ended up having a harder time shipping v1 of the product than if we would've started lean and grown organically. We ended up spending a lot of time on a modularized platform to enable ~5 matrixed feature teams to contribute across iOS and Android apps. Looking back, a small dedicated team for iOS and another team for Android probably could've shipped v1 in less than half the time with a lot less tech debt.
  - enra 5y ago
    Yeah for sure. I think even senior leaders forget the team culture/context piece, and try to scale up too fast because they are used to it in their current orgs or previous companies. If you have a 500 person org that has operated for years you can probably add 10%-20% new people each year without it breaking too bad. But it can be very hard to build a complete function from 0 to 100 people within a year because there is no infra, culture or understanding how things work. The new people cannot be absorbed/aligned because there is no central gravity so people easily end up pulling different directions and spending their time on internal coordination vs actual useful output.
xen0 5y ago
I wonder, how many engineers had full time jobs just developing and maintaining the various integrations with all their small clients?
- mywittyname 5y ago
  I have to imagine they were all stepping on each other and/or reinventing slightly different shaped wheels too. My experience has been that companies who will customize integrations for you have poorly designed products. By that, I mean, when we ask for a feature, and they bring on an engineer who does some requirements gathering then comes back with some invisible solution. That's a major redflag. The success of an integration, in my experience, is directly proportional to the amount of the work I'm able to handle as a client.
  - vimwizard 5y ago
    >My experience has been that companies who will customize integrations for you have poorly designed products. By that, I mean, when we ask for a feature, and they bring on an engineer who does some requirements gathering then comes back with some invisible solution. That's a major redflag. I'm feeling a bit called out right now (not an owner but that's my experience, too)
nonameiguess 5y ago
Staying employed for another few decades without losing seniority in an industry that expects you to stay current on all of the latest hottest fads is a problem the engineers actually had.
deleted 5y ago
[deleted]
zozbot234 5y ago
Tailscale's "database from 2022" is neither here nor there. What you should do in this situation is use the most boring tech, such as PostgreSQL with a simple HA setup. Thus avoiding the "play with nifty toys" trap and maximizing the chance of making appropriate choices.
- mwcampbell 5y ago
  Perhaps one problem with the current backlash against over-complicated infrastructure is that for some of us, doing everything in one process on one machine using a local embedded database is its own form of playing with nifty toys.
  - bryanrasmussen 5y ago
    yeah, I guess, I'm not going to get too specific here but I do think it would be nifty if I didn't go through a couple 10 minute periods every day where I have to basically shut everything down run a number of scripts, then restart individual processes that are problematic, to be able to work again. Note that this basically works 99% of the time now and is part of a learning process that had the whole thing taking 2 - 5 hours every day for several weeks, down to 2 hours a day for several weeks after that, and now down to the nearly inescapable 20 - 40 minutes a day I have now. Yeah, not doing that would be my own form of playing with nifty toys.
    - collinmanderson 5y ago
      Wow. I would find it super annoying to be interrupted twice a day to do a 10min task. Can’t you try to narrow down the underlying bug? Or somehow automate the restart?
- randomdata 5y ago
  The most boring tech imaginable would be a file. Perhaps containing JSON or similar structured data that can be handled using commonly available libraries. Which is exactly what they used for a long time. I've done something similar before and it saved countless hours wasted on configuring databases, writing queries, fighting impedance mismatches. Postgres brings a lot of work. With a file you just need a couple of lines to write your memory out to a file and a couple more to read it back. Simple and effective. Until it stops scaling, but at that point you've established that your business is successful enough that it warrants the effort of doing something shiny. Postgres has a place in this world, but no need to put the cart before the horse.
SkyPuncher 5y ago
I've done startup engineering for a while. I've seen exactly 1 issue that we couldn't solve easily (N+1) or with simply turning our server count up (often way cheaper than putting an engineer to a problem). Basically, to move development velocity faster we "load everything". Seems like a really bad idea on the surface. In practice, we only ran into an issue with a single customer that was literally 1000x as large as the other customers. It took about two weeks by 1 engineer to rework a select few calls that were egregiously slow and we were back at it. ---- In other words, even with some arguably inefficient technical decisions, I've rarely if ever seen performance issues at early stage startups.
- tomp 5y ago
  Even before using more servers, one should consider using (one) bigger server (and another as backup). Massive servers are laughably cheap these days. E.g. 16 cores, 128GB RAM is €112 from Hetzner.
  - mwcampbell 5y ago
    Damn, wish those dedicated servers were available in Hetzner's US data center. As far as I'm aware, the closest thing to a competitor in the US is OVH, and the server you get for that price is about a quarter of the capacity. That's plenty for my little company, but still...
    - hardwaresofton 5y ago
      Same here -- I'm eagerly awaiting them having a dedicated server offering In the meantime have you looked at leaseweb? https://www.leaseweb.com/dedicated-servers#US https://www.leaseweb.com/dedicated-servers#US I'm planning on running nimbus web services[0] there and there are a few other smaller providers I have earmarked depending on how brave you are: https://www.defendhosting.com/usa-unmanaged-servers/ https://www.defendhosting.com/usa-unmanaged-servers/ https://cc.delimiter.com/cart/dedicated-servers/ https://cc.delimiter.com/cart/dedicated-servers/ https://greenserver.io https://greenserver.io [0]: https://nimbusws.com https://nimbusws.com
  - 5y ago
lmeyerov 5y ago
It wasn't "doesn't scale to 10-100X bigger" but "doesn't burn themselves and their teammates out 6mo from now and needlessly risk a stressful & prolonged sev0 when one of the 1-2 people who did weird things gets COVID / goes on a honeymoon / leaves for netflix". Spending time on needless infra (scale you don't need, AI you don't need, infra you don't need, things could have been postgres as in this case, ...) both prevents building useful stuff ($-generating features/experiences/...) and increases operational cost (maintenance, debugging, ...). Both the lost revenue growth and the increased costs are compounding. Imagine a maniac steadily piling on a bit more technical debt every day and eat a bit more of the seed corn every day instead of growing it. A good phrase for this is "playing house": http://www.paulgraham.com/before.html http://www.paulgraham.com/before.html
wnevets 5y ago
> Sounds like the engineers were just entertaining themselves with shiny toys rather than solving the problems they actually had. Programmers love to add the latest tech to their resume then move on to a new position within ~18 months
- CoastalCoder 5y ago
  That may be specific to early-mid career focus If/when a developer builds deep knowledge of a problem domain, things like framework-of-the-month may become an unwelcome distraction.
  - teaearlgraycold 5y ago
    Seeing someone using nothing but tried and true tech to solve new problems is a resume green flag.
    - Nextgrid 5y ago
      Is there a place where such "green flags" would net me money? It seems like as a software engineering employee, the roles that require me to waste time reinventing the wheel and battling self-inflicted complexity pay significantly more than those who require solving actual problems with boring tech.
      - teaearlgraycold 5y ago
        > Is there a place where such "green flags" would net me money? Well the best way to reap the benefits of your experience is to run your own company. Not screwing yourself over with BS tech is a competitive market advantage. However you might need to hire people so you can’t pick something too old. Working for other people it’ll mostly net you less on call headaches.
lupire 5y ago
> The TailScale folks caught a bunch of flak for their "do things that don't scale" approach to databases. I've seen this repeated on HN, but never saw the alleged flak.
- Aeolun 5y ago
  It’s a post from around a week ago. They went from embedded JSON, to something else (equally crazy), to embedded JSON again. Lots of laughs were had (and a lot of, just use postgres).
  - randomdata 5y ago
    > They went from embedded JSON, to something else (equally crazy), to embedded JSON again. They went from a JSON file, to etcd, to SQLite. etcd seems a little misplaced, but presumably it was already in their infrastructure and they thought they could save time leveraging it. The file-based approach seems appropriate for their particular use-case, though. It's not like it's a Rails app.
    - Aeolun 5y ago
      Crap, I’d forgotten about the sqlite part. You are of course correct. Apparently I’d only remembered the fact they went back to something file based :/ I’m still of the opinion that whatever you are storing, if your alternatives are JSON or Sqlite, etcd is a really strange/unconventional choice.
mooreds 5y ago
Yes! I've seen a few startups that seemed to overengineer things (or build custom solutions rather than use something off the shelf) in expectation of massive growth. I also thought it might have been due to resume driven development and the need to keep engineers engaged so they didn't leave. I've also seen successful startups with a monolithic rails or PHP application that ran on heroku with next to zero custom architecture components.

Spivak 5y ago

And if you think this is undersized for comedic effect I did ops for a company that went from <1mil to 15mil revenue (with the same sales strategy of targeting lots of small clients) on 4 1u white label supermicros. Two app servers and two MySQL dbs. Never in my life had such reliable infrastructure.

jjav 5y ago
This is a big lesson to heed. Sure, sometimes a startup really needs extravagantly scalable infrastructure. For the 99.9% of startup, no, you don't. So the odds are you don't. Keep it very simple, very small. I can't highlight this enough. A few boxes will get you very far. If you ever actually hit the limits, throw a big party to celebrate success and only then start to think about making your infrastructure a bit (just a bit) more complex.
StillBored 5y ago
Early in my career when in a discussion with my manager at the time, he threw out the "planes with two engines have twice as many failures as single engine planes" line. So in my now close to 30 years of experience later, working on systems expected to run 24/7/365, I can't tell you what percentage of failures and outages were caused by the redundancy/fail-over/etc software and hardware layered into a system, but its definitely a very significant part of the problem. Frequently because its just that a "layer" someone bolted on, even if that layer was some software engineer designing for "web scale". All the edge cases are frequently not completely thought out, and when you hit one its a lot harder to recover, than simply restarting a single service running on a single server.
- amluto 5y ago
  I once managed a thoroughly non-critical server that lived in a closet. In the couple years I managed it, we had quite a few failures due to the UPS tripping and no unscheduled downtime at all from any other reason. I’ve had massive problems with fancy (HPE, IIRC, but it wasn’t called HPE then) raid arrays caused by the controller being a POS. Using plain mdraid would have been far more reliable. Sometimes I wonder if all the effort people put into protocols that run Paxos and Raft could be better spent running a small number of coordinator servers with manual failover. I’m suspicious that, in many use cases, leader elections cause failures more often than the leaders themselves fail.

jonas21 5y ago

It would not have made a difference. Maybe they'd be losing $9.9M a month instead of $10M a month.

whateveracct 5y ago
Heh maybe if they also didn't hire a bunch of engineers to scale out too, it would've helped..
mwcampbell 5y ago
Sure, their primary problem was extravagant hiring, but I think this is still an important warning for developers working in fledgling companies that are appropriately cautious on the hiring side.
- mcphage 5y ago
  > Sure, their primary problem was extravagant hiring I think their primary problem was a huge valuation—with correspondingly huge expectations—but no real revenue. If you're pulling in $600K revenue, but valued at $500M... you're gonna have a bad time.
weezin 5y ago
The problem was that the engineers were building a scalable infrastructure rather than a scalable business. from the article: "The average volume of Fast customers was low because most customers were from small businesses. And yet, every small business needed custom engineering work to be done, making integration slow. Several engineers mentioned how they did not understand how spending lots of engineering effort for each small client resulting in little revenue would result in building a company that could be worth $12B one day."

EinsDueTresFour 5y ago

I would add that the "Sales vs engineering and sales winning" would be a very relevant warning also: > [sales] signed up a large number of smaller businesses on the platform. [...] However, integrating these smaller businesses was challenging thanks to several customizations needed for each new customer. The ratio of revenue per each small customer vs the total cost of integration (and very likely on-going maintenance) was probably at least an amber flag somewhere for those who had visibility of it. But maybe there was on-going hope that they would be able to sign up a big customer and integrate them before they ran out of money?

lumost 5y ago
At some point managers often fear the "red flag" of "we have a giant team with no work/customers". Hence, they will often push for dubious consulting(ish) work to at least ensure everyone remains busy. Unfortunately this can lead to the outcome that the company just loses more money.
vosper 5y ago
> The ratio of revenue per each small customer vs the total cost of integration (and very likely on-going maintenance) was probably at least an amber flag somewhere for those who had visibility of it. At the last company I worked at (won't call them a startup, just a 10-year old small business that somehow kept raising funding but has never made a profit) I was astonished to learn from a new CPO that in our tenth year we didn't know if we were making money from a customer. We had no idea how individual customers used the application, which was very data intensive, or what that was costing us. The information was there, but no-one ever looked. Every sale was considered a success, even if it turned out to cost us 3x in data costs and customer support. Often customer support alone would make a sale into a loss. We spent a relatively large portion of our revenue running very expensive servers to respond instantly to searches that no-one ever performed. I created detailed monitoring to show this. I talked about them to everyone, including Product and the CEO. No-one disputed those facts. But instead we had several people essentially dedicated to downsizing application servers and deleting unused S3 buckets, trimming three zeros a month when we could have trimmed five. I have learned not to assume that warnings are visible, that people pay attention to them if they are visible, or that anyone cares. They're still circling the drain, still raising money (somehow!) and I doubt anything's changed.

hinkley 5y ago

I would hope this problem doesn't exist at startups but I've seen it at entrenched institutions any number of time: If your boss's boss or peers signed a contract for $1m a year from Oracle, you're going to use Oracle even if Postgres is a better product. And the only way to use Postgres in this situation is if they don't know you're running it. Which puts you in the uncomfortable situation of wanting to say, "But Oracle sucks, look how successful we've been at using Postgres?". Basically if you don't already know that someone higher up the food chain is comfortable with saying, "I've made a huge mistake", you aren't going to 'help' them learn to do it by pointing out they fucked up. What they're going to hear is that you're asking them to say, "I just wasted a million dollars of company money on something everybody hates," which not only are they not going to do, but they're going to start thinking about how to shoot the messenger. As many conversations go in a couple of my hobbies and talking to friends and acquaintances about health concerns, the answer often starts with, "first, get a time machine" and ends with, "or learn to live with it," often with some useful compromises in between. One of the important things to do early in your tenure at a place is to bid various people in the management chain and those who seem to be on a management track to see how open they are to having their minds changed. Because you want to know this when the stakes are low (also the blowback on low stakes things seems to be less problematic). Then you know if or when a boondoggle is in the making whether you can stop it before it starts, or amplify all of the problems and kill it along the way, or whether you should keep your head down and conspire to keep the 'doggle at arm's length with those who agree with you.

zozbot234 5y ago
> If your boss's boss or peers signed a contract for $1m a year from Oracle, you're going to use Oracle even if Postgres is a better product. What you'd do in that situation is make the best of that $1M contract you're stuck with anyway, while being flexible enough to migrate away from that solution when it actually makes sense to do so. Just because you've made a huge mistake doesn't mean that making a different choice after-the-fact is more sensible. Hindsight is always 20/20.
- Aeolun 5y ago
  Only if your $1M solution is already integrated. Otherwise it’s just a sunk cost fallacy.

rockostrich 5y ago

This is pretty hilarious to read. I did an interview screen with them at the end of last year for a data engineering position (former coworker was high up on the design side of things and he gave a referral). The first bit of the technical question was just "write a SQL query" and boiled down to doing a group by, count, and limit. The second was something like "what if we needed to return this query from a distributed system without going to the database?" without much context. As any sane interviewer would do, I asked for context around scale, infrastructure, and what "without going to the database" even meant. The interviewer just reiterated the question. I started talking about different approaches depending on context and all the interviewer did was tell me to program a solution. Turns out all he wanted was a time-windowed hash-map to cache values and the fact that it's a distributed system didn't matter at all. I think that just about sums up their approach to engineering solutions.

teruakohatu 5y ago
What is a time-windowed hash-map? Just a time based key-value store?
- omalleyt 5y ago
  I think he means an LRU cache
- agilob 5y ago
  Yes, a hash map that expires objects and loads new KV when key isn't present inside https://github.com/google/guava/wiki/CachesExplained https://github.com/google/guava/wiki/CachesExplained
  - lupire 5y ago
    No, it's a rolling buffer of last N hours/days of data, but compressed into a hash table of counters instead of a log array.
deleted 5y ago
[deleted]

fourseventy 5y ago

I run a company in the e-commerce tech space. We handle ~1M requests per day and run it all on a few medium sized VMs on Google cloud for like $1k/mo.

whymauri 5y ago
time to raise a $100M and go w e b s c a l e

mylons 5y ago

this sounds like Ridgeline, a startup most have never heard of, founded by the founder of Workday and some of its early employees.

ammanley 5y ago
I have actually heard of this startup amazingly, as well as the founder, but I wasn't aware it was/is? as much of a s.show that the article portrays at Fast. Do you have any context you might be willing to share, I am genuinely curious.
- mylons 5y ago
  bizarre tech decisions have led to insane costs with 0 customers. launch is about 2 years behind. having a lot of trouble hiring senior engineers, when they do they don't listen to them. they have a strange platform that every product team must use. tech leadership has never launched a successful startup before. they have over 300 employees, maybe 400 now. 4 levels of management already, and a _lot_ of them. it's independently funded for now, and has the potential to keep on going like this as long as Dave D. doesn't pull the plug.
  - ammanley 5y ago
    I guess its not _that_ different than that one time I had 4 VPs of Engineering, for 30 engineers </s> Sounds like an absolute bonkers-fest, thank you for sharing.

deleted 5y ago

[deleted]

suresk 5y ago

It's always funny to read about crazy overspending on infrastructure and I have a few funny stories myself, but this was far from their biggest problem. For one, we don't know how big those costs were or how much time it would take to wind some of it down. I absolutely hate waste and was always proud of myself when I found ways to shave a thousand here and there off our AWS bill, but it was a bit depressing when I went through the thought exercise of "What would it be like for the company if I managed to get everything onto the AWS free tier?" And then compared that to what it would be like for the company if we found a way to make 1 or 2% more in revenue. Early on, the math heavily favors worrying less about infra spend and more about product enhancements that can build revenue. (Although often reducing the size of your infra for other reasons - usually around complexity for ops or ease of development can also have the side benefit of reducing cost and it seems worthwhile in those cases.)

geoduck14 5y ago
>Early on, the math heavily favors worrying less about infra spend and more about product enhancements that can build revenue. YES! This is VERY true
- myrandomcomment 5y ago
  It is a careful balance. At my last startup we knew we had scale technical debt in our SW. Not something we could fix by tossing HW at the issue. It was all great until we landed that major customer that blew past our supported scale. 6 months of engineering work and a complete stop of feature process. A more careful consideration of the tradeoffs when making them would have saved a huge headache, but no one wanted to look at the issue preferring to ignore it.
  - mwcampbell 5y ago
    > Not something we could fix by tossing HW at the issue. I'm curious to learn why, if there's any more you can share. What about vertical scaling, i.e. bigger machines instead of more of them? Can't that go pretty far?

ec109685 5y ago

They had 20 user designers / researchers. Just massively over done for a company that was just getting traction: https://twitter.com/marianoaavila/status/1511410574327881728 https://twitter.com/marianoaavila/status/1511410574327881728

Nextgrid 5y ago

> Sounds like the whole thing could have run on a single cheap VM, perhaps with a second one for redundancy. You're describing most startups here.

ejb999 5y ago
very true