10 ms·

Startup CTO – Premature Scaling

159 points by mkarliner 9y ago

1ba9115454 9y ago
If I see premature scaling it's usually at the database side of things. Inexperienced devs can be quick to dismiss a SQL database and go with something based on key value pair caching. So I agree with the article keep it simple and easy to implement, worry about scaling later.
- mkarliner 9y ago
  Yep. Not long ago I asked a dev why he had suddenly introduced Riak into the system. 'Because it's cool' Oy.
  - mgkimsal 9y ago
    Similar experiences. Worked on a project, and a 'lead' guy left with .. a few weeks of a node/koa codebase in place that only he really knew. Between the rest of us we had loads of C#, PHP and Java experience (multiples years each over 3 people). "Let's rebuild something based on what the bulk of us already know," I suggested. (Keep in mind what we had was a few weeks of prototype code that was not in production.) "No, I really want to learn more node," was the only other input, and that 'won' the decision. 18 months later... "yeah, koa isn't really a goof fit for our use cases".
    - collyw 9y ago
      We have one of those guys on our team. He is friends with one of the founders so nothing he says seems to be wrong.
    - softawre 9y ago
      This is where as a tech manager you step in and push people the right way. Fun tech is fun, business is business. C# is a great language, JVM is great w/ Kotlin/Clojure, both of those have tons of web servers.
      - cutler 9y ago
        Whilst the JVM may be established, ie. boring technology Kotlin and Clojure certainly aren't. It's now 8 years since Clojure 1.0 and there is a grand total of 15 Clojure jobs listed on Indeed.co.uk, 12 of which are in London. Kotlin boasts 2.
    - mgkimsal 9y ago
      edit: ack - should be "good" fit, not "goof" fit, obviously. And yes, per another point in the replies, someone else should have stepped in to decide differently when there was a tie. I could possibly have pushed harder, but the 'bozo bit' basically was flipped on that project, and much of what I lobbied for was ignored, at least in part because I was the one saying it.
      - didip 9y ago
        If the LoC is low for that Koa project, you should rewrite it quickly using something that the team familiar with. Otherwise, it will grow into a monster that nobody want to deal with.
        mgkimsal 9y ago
        You'd think that, right? That was another part of my argument. And yes, it's ... not quite grown in to a monster, but few on the team really know how to deal with it. That actually came up a couple months ago - I heard a comment along the lines of "well, no one's really fully understood how to deal with that koa code since the beginning". Even the koa stuff, I think was done by someone who had used it a couple of times previously and liked it, but... no long term track record of using it in production. So... about 2 years later, there's some traction from folks to rebuild it in something else. The latest shiny object is AWS lambda.
  - mkosmo 9y ago
    I've asked many devs why they chose Mongo over a RDBMS. They can never give me a technical answer. It's always 'it's cool' or because they just read something about it on some non-technical blog. It usually only takes me a minute or two to ask them a future function question that's not reasonably attainable (and performant) with a NoSQL solution... so they wind up going back to whatever RDBMS is their standard.
    - zzzcpan 9y ago
      I don't think I ever heard a good technical reason to use a traditional RDBMS. And as a default choice it is not a very good one, when something distributed can deliver an easy to maintain resilient solution pretty much out of the box.
      - SOLAR_FIELDS 9y ago
        I'm sure many can be produced, but the first off of the top of my head is accessibility for quasi-developers. If you ever want to hire some analyst roles you are going to find that it's a lot easier to get people that have at least some familiarity with SQL instead of having to train them on whatever custom NoSQL flavor you are using. Of course you could solve that by having an ETL process dump your stuff into SQL (we keep our stuff in Cassandra and dump it into SQL for analysis), but that's another ETL you need to keep track of and in our case we use a quite complex data model and the ETL only supports a subset of it. Another one just off the top of my head is joins. Some data is just inherently better suited in normal form.
      - milquetoastaf 9y ago
        Relational databases can be distributed lol. You are clueless.
        sctb 9y ago
        Please stop violating the guidelines like this. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - michaelmior 9y ago
    For a side project, that's a fairly reasonable answer. For a business, rather scary.
- sbarre 9y ago
  Yup! MySQL + Redis (for query/object caching and queues) gets you the best of both worlds in a lot of scenarios, and is reasonably easy to set up and manage. And it will probably scale better than you expect at first!
  - zzzcpan 9y ago
    "MySQL + Redis" I think this would be 10x more expensive to set up, manage, maintain and use than something like Riak.
    - libertymcateer 9y ago
      I am using postgres and redis - they are both trivial to set up and cost about a few lattes a month for a fairly scalable solution. If I had huge traffic, a few hundred bucks a month would be totally adequate - and this is running on the too-expensive heroku. So, unless Riak costs pennies and takes literal minutes to set up, the 10x claim won't hold. Riak looks like it would be overpowered by my needs - but then again, I learn something new every day.
    - Posibyte 9y ago
      Can you explain why you think that? I would imagine using two rather popular systems in a somewhat common combination would be a boon to maintainability and set up, if at least for the large pool of developers and online resources already available for it. But then again, I could be super mistaken. I'm interested in seeing another side of it.
      - sbarre 9y ago
        This was also my thought when I made the comment... I am sure that for certain use cases, there are more tailored solutions, but "SQL + keystore" seems like a fairly universal, affordable and versatile stack.
        user5994461 9y ago
        They are different databases. Riak is a key value store, not unlike the core of redis. The SQL database is obviously relational. Ignoring the difference for a minute. If you can fit data in Redis, chances are that they can fit in riak and a RDBMS is not needed at all. Why Riak only then? It is one of the very few databases that can be clustered. It is sharded, multi-master, highly available, to talk in buzzwords. It is very reliable and very robust, you can add/remove nodes as you wish, replication is highly configurable, etc... Postgre/Mysql/MariaDB + Redis have no HA. They are both the disaster in disaster recovery.
- awinder 9y ago
  I always see this discussion cast as one between scalability (KV) and dev efficiency (SQL), but other reasons to adopt columnar databases include: 1. Difficulty in transitioning a system with SQL-Gone-Wild to a system that can be implemented with other dbs without complexity overdrive 2. Escalating costs to get any additional availability guarantees out of SQL (availability & cost are key business issues) In AWS & Google Cloud, you're going to need to pay outright for capacity as well as capacity devoted db recovery (a replica / set of replicas in another availability zone). On the other side with something like dynamo / cloud datastore, your payment model is based on requests / sec and the costs for replicated data. I'm just saying that there's nothing simple to deciding where price / availability / dev productivity balances will lead you to, and you can be an inexperienced developer who chooses either side of these technologies without really understanding the full story.
- hashkb 9y ago
  A potential client told me he was planning on having three SQL databases. I asked him why, what each would be for, and he didn't really know. Just a vague sense that his app was going to be too much for a single db.
- balls187 9y ago
  > Inexperienced devs can be quick to dismiss a SQL...So I agree with the article keep it simple and easy to implement If you're inexperienced with relational databases, perhaps a Key-value store with a simple API is much simpler and easier to implement.
zelias 9y ago
Wise words. In the earlier stages, you need to build a product that has room to pivot rapidly for the next iteration of the business.
bpicolo 9y ago
One of the big ones: For a small startup, granular microservices are often very premature, and will slow you down rather than speed you up.
- swsieber 9y ago
  This should get more emphasis. Microservices do require more overhead. Additionally, they require you to be familiar enough with you business logic to know where you boundaries lie - changing abstraction boundaries is much harder with microservices than with a monolith. I think that while they are great for scaling, they are a lot harder to use to successfully get off the ground (though I'm betting that there are places that have successfully done so)
  - ben_jones 9y ago
    You also have to know how to "do" microservices. Whether it's incorporating message queues like RabbitMQ, protocols like gRPC, etc., it's very hard to get right the first time by winging it without previous experience. Redesigning microservices after is a PITA.
    - mcginleyr1 9y ago
      100% agree. Your path is based on your teams experience and capabilities, not trends.
- vojant 9y ago
  That really depends on the startup, and sometimes starting with micro-services helps with hiring - In our case thanks to starting with micro-services we hired developers knowing different programming languages and they can work together on the project by exposing end-points/consumers. Of course I wouldn't recommend doing that for everyone but in our use-case we had to start with distributed system anyway due to quite complicated domain.
  - bpicolo 9y ago
gargarplex 9y ago
Yes, but also understand explicitly what the scaling limits of your initial solution are, communicate them to the CEO, and explain how long it will take to build a scalable solution for the next level and how much lead time will be necessary.
caseysoftware 9y ago
Needing scalability means you should have customers which means you have revenue. I tell every startup that I advise that "revenue solves many problems" because if you have cash flow, everything else becomes easier.. you can buy spot capacity in the short term. You can hire contractors to help refactor. You can rent coworking space for an office. There are many "good enough" solutions for now. Take advantage of them until you know what your next steps should be.. and then commit accordingly.
- aries1980 9y ago
  This approach really lacks the product vision. Although I agree with it to some extent, my experience is with semi-mature product, the code will stick around for ~2 years on average. You need to be experienced to make implementations that will be easy to replace when needed. E.g. one of your assumptions there are contractors who can help you. That can be true to a certain level, but even product vendors often lack the scale to support their own thing above the basic levels (this is true for post-IPO companies), now imagine this when you can't afford to incubate your product at an expensive tech hub location. Also, when you have cashflow, your business is under pressure to provide better customer service and marketing, and this is where the things getting tricky: you need to decide whether you want to grow your business or pay back the technical debts. Because the investors priority is always business over tech (rightly), you need to look 1-2 years ahead and make your key technical decisions accordingly.
halayli 9y ago
> For the truth is that even the most modest technology stack will be more than enough to handle the traffic from the vast majority of startups. It's not just the stack, the architecture matters more.
partycoder 9y ago
Just more advocates for hot potato engineering. Build a snappy prototype that doesn't scale then sell it to some clueless guy.
romanovcode 9y ago
In other words: Scalability is a good problem to have!
eddd 9y ago
Just don't scale, until you are ready. I highly recommend Eric's Schmidt take on this: https://youtu.be/hcRxFRgNpns?t=838 https://youtu.be/hcRxFRgNpns?t=838
- dboreham 9y ago
  Or "Only worry about solving problems you actually have". Spending your own money provides good discipline in pragmatism, I've found.
JMCQ87 9y ago
Is it me, or is there not really much in this post which is not obvious, i.e. common sense?
- JonMR 9y ago
  Common sense is often worth repeating.
- tedmiston 9y ago
  I was hoping for a little more substance or specific examples, but I think it's long enough to make the point that microservices have been cargo culted and don't make sense at every stage, combined with early startups can be prone to over-engineering beyond their current stage.
michaelfeathers 9y ago
There's a myth around scaling [1]. Actually, it's almost like a cognitive bias. We assume that we can usually use the same structure as we scale. Because of this bias, we haven't thought enough about how to make bigger structural transitions normal. The GP points at the transition between the quick & dirty architecture and the architecture that handles Google-scale, but those aren't the only good stable points for a system. [1] https://michaelfeathers.silvrback.com/the-myth-of-scaling https://michaelfeathers.silvrback.com/the-myth-of-scaling
neeleshs 9y ago
While it's true you don't need to design for scaling right off the bat, if you do become successful and keep pushing out the scaling pieces , the cost of scaling later increases nonlinearly. At some point it becomes so important that you start losing customers and/or new business opportunities. A CTO should be able to foresee this and act before scaling becomes a behemoth project and no amount of money can buy time-to-market.
- korzun 9y ago
  > the cost of scaling later increases nonlinearly At some point it becomes so important that you start losing customers and/or new business opportunities. That simply is your opinion. You are more likely to lose customers for not focusing on the core product first.
grandalf 9y ago
I've found that most startups fail to scale when it comes to making fast/responsive product decisions. This may not be obvious, since those startups are typically "agile" in the buzzword sense. The cost comes when the universe of possible change is constrained to the point where the right questions aren't being asked because the answer would be "no we can't do that". So the startup enters a dark age in which scientific thought and rational problem solving isn't really relevant. Often at this point the team starts using Scrum to force the business to plan better, when the core problem is that the existing infrastructure is not able to support the needed experimentation fast enough, making it useless. This can happen at various stages of funding, since sometimes a series A or B can occur without solid product market fit, or the company may be really good at growth in spite of having a promising but mediocre product. So I view the CTO's job as preserving the ability of the business to behave scientifically and boldly until product market fit is found.
- mkarliner 9y ago
  Very well put.
- bwilliams18 9y ago
  I'd submit it's important to always be able to behave scientifically and boldly. Product-Market fit isn't a one time solution. Markets move and change, Facebook, Apple and Google have been successful because they've changed their products as the markets have changed.
  - grandalf 9y ago
    Exactly. I totally agree, but typically with a startup achieving it at least once (the first time) is of particular importance. After that the role is more general as "CTO" than as "Startup CTO". But I'm just splitting hairs. You are totally right.
exabrial 9y ago
The problem is ego. "Scalability" is a sexy problem to have, when in fact, "simplicity" and low-risk "trailing edge" technologies is the prescription for success. Yep, i'm talking to you. You don't need an angular app backed by MongoDB to display static content. Static HTML will work just fine.
- korzun 9y ago
  Except that you don't need a CTO if you are building out static HTML pages. Sometimes you need to put a twist on the problem to make it a bit more interesting if you want to attract smart people.
  - exabrial 9y ago
    Are you actually advocating making a complicated architecture, just to "attract smart people?"
    - korzun 9y ago
      I never used the word complicated. If you are a technology company, the architecture should be interesting. Great engineers will not stick around if you call it a day and become a feature factory. Unless your goal is to retain sub-par seat fillers, who show up for a paycheck. Then you can do whatever you want.
  - nasalgoat 9y ago
    As a CTO I see it as part of my job to find the right tool for the right job. Part of that was when I told the team I wanted our splash page and everything on it to be just static HTML files that would be hand edited if and when it was needed. There was no need for fancy CMS tools for a landing page that gets updated maybe once a quarter. So, that's part of why you need a CTO - to know when to use static HTML!
    - korzun 9y ago
      > As a CTO I see it as part of my job to find the right tool for the right job. If that's a minuscule part of the job, then I agree with you here. My point is, you are not a CTO if the technology behind the product is a static web site. What do your VP and Directors do all day? What about the teams under them? There is a huge difference between a CTO and a front-end engineering.
deleted 9y ago
[deleted]
bhauer 9y ago
I am biased because in my work, I have indeed seen many projects that are struggling or outright failing because of performance problems. The underlying sentiment of this article may be (mis)interpreted by some as "don't worry about performance." I feel that is not the right guidance, at least in many cases. That said, there is an important premise presented by the article. If I can reframe it slightly: don't over-complicate your system architecture, especially not needlessly. If, by selecting some fancy and trendy new technology, you will complicate your architecture or development process for unknown or modest performance gains, it's probably wise to postpone such adoption until the need is more palpable. My advise, again showing my bias, is quite simple: just select foundation technologies that are mature and high-performance. Using high-performance foundation software gives your application a high performance ceiling, which in turn provides breathing room to build your application in a more "brute-force" fashion. With this approach, you can defer performance tuning and optimization for some time, and perhaps indefinitely. I have seen plenty of small and small-medium businesses operate well with very modest technology platforms. But importantly, they are modest in architecture (few servers, few processes, low system complexity, simple deployments) because they used high-performance platforms (e.g., JVM, Go, so on). On the flip-side, I've seen many small businesses struggling and prematurely adopting complex system architecture because their software platform (e.g., Ruby, PHP) presents a bottleneck early in the business' life, which creates a challenging scale event before the company has even found their market sweet spot. When I've written about this in the past, I've pointed out that many people call this situation a "good problem to have," which is of course valid in important ways. But it's also crude and demeaning to your development team. You're basically saying, "Fantastic, our business is scaling. Yes, our servers are on fire and my developers are panicking, but that's great!" It's better, in my opinion, to have used a high-performance foundation and (potentially) avoid that unnecessary pain. All else being equal (e.g., importantly, your development team is comfortable with a JVM language, for example), I recommend keeping your application simple by using platforms that give you a high ceiling to grow.
- mkarliner 9y ago
  All fair points. I think the thrust of what I was saying is simply: resist the pressures to provide huge scale before there is reasonable evidence that the product has some traction. Some huge percentage of companies will have to do a major pivot before they get that traction, so the balance should be heavily on simplicity, speed and agility, and if that's at the expense of scale, so be it. Obviously, performance does matter, and so does good architecture, but staying in business matter most :-)
jlebrech 9y ago
if you need to scale so much to be profitable you're doing something wrong. except when you're selling an idea to a VC and scaling is all they think about.
Ensorceled 9y ago
The issue I keep encountering, again and again, is that you really have no idea what to scale until you need to. For instance, I had a client where a very large online retailer wanted a RESTful API into my client's data that they could call directly from their customer's browser. Before the Black Friday Freeze. This was a completely unforeseen requirement and the solution was NOT to scale the client's overall website to handle the 5-10K requests per second but create a micro service on a new stack that could. Your startup will pivot, sign weird deals and change your target customer. Optimizing for scale early doesn't help, getting to the point where you know you need to pivot does.
avenoir 9y ago
While i absolutely agree with the author, how does this line of thinking change when you start using AWS or something similar? I'm building a SAAS app at the moment and a lot of these scaling decisions are incredibly easy to implement without imposing any significant costs in terms of maintenance (and money, a lot of times). This would've been a different story had i been using my own infrastructure.
- theptip 9y ago
  I think there's a key difference between building your architecture with plenty of headroom to scale (leaving plenty of low-hanging fruit), and actually implementing high-scale function before it's required. In a cloud environment, it's quite easy to be comfortable that you _could_ scale out to 10-100x of your current capacity, if you have to. But it's probably not worth proving that fact until you have some customers. Or put another way, being in a very flexible environment makes it even more of a no-brainer to delay building for scale, since you have so many free or nearly-free options there. Now of course, if you really do hit scale you'll have the fun problem of needing to optimize your spend, but you can probably delay thinking about that until well after you've proven product-market fit.
venture_lol 9y ago
There is nothing wrong with planning for scale and for microservices or whatever appropriate architecture from the start. It does not take more time to design properly from the get go. That, of course, assumes you have a CTO and a team capable of their job :)
- dasil003 9y ago
  Yes, if and only if you know exactly what you need to build and how big the scale will be from day 1. What usually happens is the first thing you build is wrong, and you need to iterate aggressively on not just the software but the entire business concept. Once you have product-market fit, then you are in a position to design an optimal architecture. If you go microservices before that, you are adding unjustifiable overhead, which will inarguably slow you down compared to a monolith when you are in the 1-3 engineer phase, and don't fool yourself about that.
  - venture_lol 9y ago
    I frankly have no idea what you are talking about. You are making wild assumptions that monolith are better just because... It will all come down the the experience of the team, esp. the leadership - the managers, the leads. A strong, experienced, mature and effective leadeship will enable a team to perform "magic" and set the right technology foundation that will prove its worth as the business grows over time
    - taurath 9y ago
      The point he's making is the technology foundation can be built for the wrong product or the wrong market. After 2 full product reconfigurations or pivots requiring a TON of code being dumped you will wish you didn't have the overhead of building infrastructure with the first product in mind. In the end, the base reasoning for the foundations being built the way they didn't hold up in production. Building something quickly that can be broken up over time is the best way to an MVP, and you can deal with scalability issues when you HAVE scalability issues. I consider scalability to be a good problem to have, because by that time you have actual traction. If you are a bigco with a big marketing budget in a field in which you already have experience and business then by all means scale to millions of users at day one. Don't take on the cost of it in a startup, because the problem is TRACTION not technology with regards to scaling.
chaostheory 9y ago
Are people, who are just starting, still building complex chef and puppet systems when Heroku and Elastic Beanstalk exist? .i.e scalability is easy now. It's a click away or even automatic which these types of services. It's also affordable.
- wink 9y ago
  How do you think generic configuration management systems are replacable by a hosting provider and some autoscaling system? Honestly curious, I've never seen anyone scale just by "using puppet/chef".
  - chaostheory 9y ago
    I didn't say they were replaceable for everyone and for every scenario. I'm just making the point that small startups no longer need to use them in the beginning if all they want is a way to auto-scale. Yes they made sys admin stuff easier but now it's even easier with scaling services that are already preconfigured. I just think it's strange that people are still discouraging founders to account for scaling when today it's cheap from both a time and money perspective to do from the start.
- korzun 9y ago
  > scalability is easy now Dumb, often limited scalability is easy now. Elastic Beanstalk approach is only as good as it's the weakest link, which you have no control over. EC2 is having issues with recovering from standby? Good luck getting anywhere. You can sit on the chat with the useless AWS 'technicians' that are paid to waste your time until the issue is corrected.
  - chaostheory 9y ago
    I'm not sure if I'm making myself clear enough. I'm not arguing that this is great for everything. In the context of the post, that's good enough for a lot of apps just starting out WITHOUT the work of setting up something custom and durable ala Netflix's infrastructure.
bedros 9y ago
Simply, to counter the flood of new users, build a Dam called waiting list, the slowly activate users as you build capacity Put priority on paid users vs free, to cover server costs
peterjlee 9y ago
My answer to most scaling question is: by the time we have that problem, we'll have a lot of customers and money so we'll hire smarter engineers with experience in scaling to figure that out. I still make sure we don't do anything stupid right now so the future engineers have more options on how to scale. Also it helps that I'm not working on a zero-revenue consumer app.
HodGreeley 9y ago
I think I can give one concrete example where a NoSQL choice (often) makes sense. Common modern day scenario in mobile is to build an app and iterate. Migration in RDBMS rapidly becomes a real pain. Worse, you have no good sync solution in case you need it. Contrast with something like Couchbase Mobile. NoSQL embedded db is much easier to get going with. (Just check the Android docs on using SQLite and see how it compares to "here's my JSON, done".) Sync doesn't get in the way, but if/when you need it, you're going to be really happy it's there. (Disclosure: I work for Couchbase.)

159 points by mkarliner 9y ago

If I see premature scaling it's usually at the database side of things. Inexperienced devs can be quick to dismiss a SQL database and go with something based on key value pair caching. So I agree with the article keep it simple and easy to implement, worry about scaling later.

mkarliner 9y ago
Yep. Not long ago I asked a dev why he had suddenly introduced Riak into the system. 'Because it's cool' Oy.
- mgkimsal 9y ago
  Similar experiences. Worked on a project, and a 'lead' guy left with .. a few weeks of a node/koa codebase in place that only he really knew. Between the rest of us we had loads of C#, PHP and Java experience (multiples years each over 3 people). "Let's rebuild something based on what the bulk of us already know," I suggested. (Keep in mind what we had was a few weeks of prototype code that was not in production.) "No, I really want to learn more node," was the only other input, and that 'won' the decision. 18 months later... "yeah, koa isn't really a goof fit for our use cases".
  - collyw 9y ago
    We have one of those guys on our team. He is friends with one of the founders so nothing he says seems to be wrong.
  - softawre 9y ago
    This is where as a tech manager you step in and push people the right way. Fun tech is fun, business is business. C# is a great language, JVM is great w/ Kotlin/Clojure, both of those have tons of web servers.
    - cutler 9y ago
      Whilst the JVM may be established, ie. boring technology Kotlin and Clojure certainly aren't. It's now 8 years since Clojure 1.0 and there is a grand total of 15 Clojure jobs listed on Indeed.co.uk, 12 of which are in London. Kotlin boasts 2.
  - mgkimsal 9y ago
    edit: ack - should be "good" fit, not "goof" fit, obviously. And yes, per another point in the replies, someone else should have stepped in to decide differently when there was a tie. I could possibly have pushed harder, but the 'bozo bit' basically was flipped on that project, and much of what I lobbied for was ignored, at least in part because I was the one saying it.
    - didip 9y ago
      If the LoC is low for that Koa project, you should rewrite it quickly using something that the team familiar with. Otherwise, it will grow into a monster that nobody want to deal with.
      - mgkimsal 9y ago
        You'd think that, right? That was another part of my argument. And yes, it's ... not quite grown in to a monster, but few on the team really know how to deal with it. That actually came up a couple months ago - I heard a comment along the lines of "well, no one's really fully understood how to deal with that koa code since the beginning". Even the koa stuff, I think was done by someone who had used it a couple of times previously and liked it, but... no long term track record of using it in production. So... about 2 years later, there's some traction from folks to rebuild it in something else. The latest shiny object is AWS lambda.
- mkosmo 9y ago
  I've asked many devs why they chose Mongo over a RDBMS. They can never give me a technical answer. It's always 'it's cool' or because they just read something about it on some non-technical blog. It usually only takes me a minute or two to ask them a future function question that's not reasonably attainable (and performant) with a NoSQL solution... so they wind up going back to whatever RDBMS is their standard.
  - zzzcpan 9y ago
    I don't think I ever heard a good technical reason to use a traditional RDBMS. And as a default choice it is not a very good one, when something distributed can deliver an easy to maintain resilient solution pretty much out of the box.
    - SOLAR_FIELDS 9y ago
      I'm sure many can be produced, but the first off of the top of my head is accessibility for quasi-developers. If you ever want to hire some analyst roles you are going to find that it's a lot easier to get people that have at least some familiarity with SQL instead of having to train them on whatever custom NoSQL flavor you are using. Of course you could solve that by having an ETL process dump your stuff into SQL (we keep our stuff in Cassandra and dump it into SQL for analysis), but that's another ETL you need to keep track of and in our case we use a quite complex data model and the ETL only supports a subset of it. Another one just off the top of my head is joins. Some data is just inherently better suited in normal form.
    - milquetoastaf 9y ago
      Relational databases can be distributed lol. You are clueless.
      - sctb 9y ago
        Please stop violating the guidelines like this. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- michaelmior 9y ago
  For a side project, that's a fairly reasonable answer. For a business, rather scary.
sbarre 9y ago
Yup! MySQL + Redis (for query/object caching and queues) gets you the best of both worlds in a lot of scenarios, and is reasonably easy to set up and manage. And it will probably scale better than you expect at first!
- zzzcpan 9y ago
  "MySQL + Redis" I think this would be 10x more expensive to set up, manage, maintain and use than something like Riak.
  - libertymcateer 9y ago
    I am using postgres and redis - they are both trivial to set up and cost about a few lattes a month for a fairly scalable solution. If I had huge traffic, a few hundred bucks a month would be totally adequate - and this is running on the too-expensive heroku. So, unless Riak costs pennies and takes literal minutes to set up, the 10x claim won't hold. Riak looks like it would be overpowered by my needs - but then again, I learn something new every day.
  - Posibyte 9y ago
    Can you explain why you think that? I would imagine using two rather popular systems in a somewhat common combination would be a boon to maintainability and set up, if at least for the large pool of developers and online resources already available for it. But then again, I could be super mistaken. I'm interested in seeing another side of it.
    - sbarre 9y ago
      This was also my thought when I made the comment... I am sure that for certain use cases, there are more tailored solutions, but "SQL + keystore" seems like a fairly universal, affordable and versatile stack.
      - user5994461 9y ago
        They are different databases. Riak is a key value store, not unlike the core of redis. The SQL database is obviously relational. Ignoring the difference for a minute. If you can fit data in Redis, chances are that they can fit in riak and a RDBMS is not needed at all. Why Riak only then? It is one of the very few databases that can be clustered. It is sharded, multi-master, highly available, to talk in buzzwords. It is very reliable and very robust, you can add/remove nodes as you wish, replication is highly configurable, etc... Postgre/Mysql/MariaDB + Redis have no HA. They are both the disaster in disaster recovery.
awinder 9y ago
I always see this discussion cast as one between scalability (KV) and dev efficiency (SQL), but other reasons to adopt columnar databases include: 1. Difficulty in transitioning a system with SQL-Gone-Wild to a system that can be implemented with other dbs without complexity overdrive 2. Escalating costs to get any additional availability guarantees out of SQL (availability & cost are key business issues) In AWS & Google Cloud, you're going to need to pay outright for capacity as well as capacity devoted db recovery (a replica / set of replicas in another availability zone). On the other side with something like dynamo / cloud datastore, your payment model is based on requests / sec and the costs for replicated data. I'm just saying that there's nothing simple to deciding where price / availability / dev productivity balances will lead you to, and you can be an inexperienced developer who chooses either side of these technologies without really understanding the full story.
hashkb 9y ago
A potential client told me he was planning on having three SQL databases. I asked him why, what each would be for, and he didn't really know. Just a vague sense that his app was going to be too much for a single db.
balls187 9y ago
> Inexperienced devs can be quick to dismiss a SQL...So I agree with the article keep it simple and easy to implement If you're inexperienced with relational databases, perhaps a Key-value store with a simple API is much simpler and easier to implement.

zelias 9y ago

Wise words. In the earlier stages, you need to build a product that has room to pivot rapidly for the next iteration of the business.

bpicolo 9y ago

One of the big ones: For a small startup, granular microservices are often very premature, and will slow you down rather than speed you up.

swsieber 9y ago
This should get more emphasis. Microservices do require more overhead. Additionally, they require you to be familiar enough with you business logic to know where you boundaries lie - changing abstraction boundaries is much harder with microservices than with a monolith. I think that while they are great for scaling, they are a lot harder to use to successfully get off the ground (though I'm betting that there are places that have successfully done so)
- ben_jones 9y ago
  You also have to know how to "do" microservices. Whether it's incorporating message queues like RabbitMQ, protocols like gRPC, etc., it's very hard to get right the first time by winging it without previous experience. Redesigning microservices after is a PITA.
  - mcginleyr1 9y ago
    100% agree. Your path is based on your teams experience and capabilities, not trends.
vojant 9y ago
That really depends on the startup, and sometimes starting with micro-services helps with hiring - In our case thanks to starting with micro-services we hired developers knowing different programming languages and they can work together on the project by exposing end-points/consumers. Of course I wouldn't recommend doing that for everyone but in our use-case we had to start with distributed system anyway due to quite complicated domain.
- bpicolo 9y ago

gargarplex 9y ago

Yes, but also understand explicitly what the scaling limits of your initial solution are, communicate them to the CEO, and explain how long it will take to build a scalable solution for the next level and how much lead time will be necessary.

caseysoftware 9y ago

Needing scalability means you should have customers which means you have revenue. I tell every startup that I advise that "revenue solves many problems" because if you have cash flow, everything else becomes easier.. you can buy spot capacity in the short term. You can hire contractors to help refactor. You can rent coworking space for an office. There are many "good enough" solutions for now. Take advantage of them until you know what your next steps should be.. and then commit accordingly.

aries1980 9y ago
This approach really lacks the product vision. Although I agree with it to some extent, my experience is with semi-mature product, the code will stick around for ~2 years on average. You need to be experienced to make implementations that will be easy to replace when needed. E.g. one of your assumptions there are contractors who can help you. That can be true to a certain level, but even product vendors often lack the scale to support their own thing above the basic levels (this is true for post-IPO companies), now imagine this when you can't afford to incubate your product at an expensive tech hub location. Also, when you have cashflow, your business is under pressure to provide better customer service and marketing, and this is where the things getting tricky: you need to decide whether you want to grow your business or pay back the technical debts. Because the investors priority is always business over tech (rightly), you need to look 1-2 years ahead and make your key technical decisions accordingly.

halayli 9y ago

> For the truth is that even the most modest technology stack will be more than enough to handle the traffic from the vast majority of startups. It's not just the stack, the architecture matters more.

partycoder 9y ago

Just more advocates for hot potato engineering. Build a snappy prototype that doesn't scale then sell it to some clueless guy.

romanovcode 9y ago

In other words: Scalability is a good problem to have!

eddd 9y ago

Just don't scale, until you are ready. I highly recommend Eric's Schmidt take on this: https://youtu.be/hcRxFRgNpns?t=838 https://youtu.be/hcRxFRgNpns?t=838

dboreham 9y ago
Or "Only worry about solving problems you actually have". Spending your own money provides good discipline in pragmatism, I've found.

JMCQ87 9y ago

Is it me, or is there not really much in this post which is not obvious, i.e. common sense?

JonMR 9y ago
Common sense is often worth repeating.
tedmiston 9y ago
I was hoping for a little more substance or specific examples, but I think it's long enough to make the point that microservices have been cargo culted and don't make sense at every stage, combined with early startups can be prone to over-engineering beyond their current stage.

michaelfeathers 9y ago

There's a myth around scaling [1]. Actually, it's almost like a cognitive bias. We assume that we can usually use the same structure as we scale. Because of this bias, we haven't thought enough about how to make bigger structural transitions normal. The GP points at the transition between the quick & dirty architecture and the architecture that handles Google-scale, but those aren't the only good stable points for a system. [1] https://michaelfeathers.silvrback.com/the-myth-of-scaling https://michaelfeathers.silvrback.com/the-myth-of-scaling

neeleshs 9y ago

While it's true you don't need to design for scaling right off the bat, if you do become successful and keep pushing out the scaling pieces , the cost of scaling later increases nonlinearly. At some point it becomes so important that you start losing customers and/or new business opportunities. A CTO should be able to foresee this and act before scaling becomes a behemoth project and no amount of money can buy time-to-market.

korzun 9y ago
> the cost of scaling later increases nonlinearly At some point it becomes so important that you start losing customers and/or new business opportunities. That simply is your opinion. You are more likely to lose customers for not focusing on the core product first.

grandalf 9y ago

I've found that most startups fail to scale when it comes to making fast/responsive product decisions. This may not be obvious, since those startups are typically "agile" in the buzzword sense. The cost comes when the universe of possible change is constrained to the point where the right questions aren't being asked because the answer would be "no we can't do that". So the startup enters a dark age in which scientific thought and rational problem solving isn't really relevant. Often at this point the team starts using Scrum to force the business to plan better, when the core problem is that the existing infrastructure is not able to support the needed experimentation fast enough, making it useless. This can happen at various stages of funding, since sometimes a series A or B can occur without solid product market fit, or the company may be really good at growth in spite of having a promising but mediocre product. So I view the CTO's job as preserving the ability of the business to behave scientifically and boldly until product market fit is found.

mkarliner 9y ago
Very well put.
bwilliams18 9y ago
I'd submit it's important to always be able to behave scientifically and boldly. Product-Market fit isn't a one time solution. Markets move and change, Facebook, Apple and Google have been successful because they've changed their products as the markets have changed.
- grandalf 9y ago
  Exactly. I totally agree, but typically with a startup achieving it at least once (the first time) is of particular importance. After that the role is more general as "CTO" than as "Startup CTO". But I'm just splitting hairs. You are totally right.

exabrial 9y ago

The problem is ego. "Scalability" is a sexy problem to have, when in fact, "simplicity" and low-risk "trailing edge" technologies is the prescription for success. Yep, i'm talking to you. You don't need an angular app backed by MongoDB to display static content. Static HTML will work just fine.

korzun 9y ago
Except that you don't need a CTO if you are building out static HTML pages. Sometimes you need to put a twist on the problem to make it a bit more interesting if you want to attract smart people.
- exabrial 9y ago
  Are you actually advocating making a complicated architecture, just to "attract smart people?"
  - korzun 9y ago
    I never used the word complicated. If you are a technology company, the architecture should be interesting. Great engineers will not stick around if you call it a day and become a feature factory. Unless your goal is to retain sub-par seat fillers, who show up for a paycheck. Then you can do whatever you want.
- nasalgoat 9y ago
  As a CTO I see it as part of my job to find the right tool for the right job. Part of that was when I told the team I wanted our splash page and everything on it to be just static HTML files that would be hand edited if and when it was needed. There was no need for fancy CMS tools for a landing page that gets updated maybe once a quarter. So, that's part of why you need a CTO - to know when to use static HTML!
  - korzun 9y ago
    > As a CTO I see it as part of my job to find the right tool for the right job. If that's a minuscule part of the job, then I agree with you here. My point is, you are not a CTO if the technology behind the product is a static web site. What do your VP and Directors do all day? What about the teams under them? There is a huge difference between a CTO and a front-end engineering.

deleted 9y ago

[deleted]

bhauer 9y ago

I am biased because in my work, I have indeed seen many projects that are struggling or outright failing because of performance problems. The underlying sentiment of this article may be (mis)interpreted by some as "don't worry about performance." I feel that is not the right guidance, at least in many cases. That said, there is an important premise presented by the article. If I can reframe it slightly: don't over-complicate your system architecture, especially not needlessly. If, by selecting some fancy and trendy new technology, you will complicate your architecture or development process for unknown or modest performance gains, it's probably wise to postpone such adoption until the need is more palpable. My advise, again showing my bias, is quite simple: just select foundation technologies that are mature and high-performance. Using high-performance foundation software gives your application a high performance ceiling, which in turn provides breathing room to build your application in a more "brute-force" fashion. With this approach, you can defer performance tuning and optimization for some time, and perhaps indefinitely. I have seen plenty of small and small-medium businesses operate well with very modest technology platforms. But importantly, they are modest in architecture (few servers, few processes, low system complexity, simple deployments) because they used high-performance platforms (e.g., JVM, Go, so on). On the flip-side, I've seen many small businesses struggling and prematurely adopting complex system architecture because their software platform (e.g., Ruby, PHP) presents a bottleneck early in the business' life, which creates a challenging scale event before the company has even found their market sweet spot. When I've written about this in the past, I've pointed out that many people call this situation a "good problem to have," which is of course valid in important ways. But it's also crude and demeaning to your development team. You're basically saying, "Fantastic, our business is scaling. Yes, our servers are on fire and my developers are panicking, but that's great!" It's better, in my opinion, to have used a high-performance foundation and (potentially) avoid that unnecessary pain. All else being equal (e.g., importantly, your development team is comfortable with a JVM language, for example), I recommend keeping your application simple by using platforms that give you a high ceiling to grow.

mkarliner 9y ago
All fair points. I think the thrust of what I was saying is simply: resist the pressures to provide huge scale before there is reasonable evidence that the product has some traction. Some huge percentage of companies will have to do a major pivot before they get that traction, so the balance should be heavily on simplicity, speed and agility, and if that's at the expense of scale, so be it. Obviously, performance does matter, and so does good architecture, but staying in business matter most :-)

jlebrech 9y ago

if you need to scale so much to be profitable you're doing something wrong. except when you're selling an idea to a VC and scaling is all they think about.

Ensorceled 9y ago

The issue I keep encountering, again and again, is that you really have no idea what to scale until you need to. For instance, I had a client where a very large online retailer wanted a RESTful API into my client's data that they could call directly from their customer's browser. Before the Black Friday Freeze. This was a completely unforeseen requirement and the solution was NOT to scale the client's overall website to handle the 5-10K requests per second but create a micro service on a new stack that could. Your startup will pivot, sign weird deals and change your target customer. Optimizing for scale early doesn't help, getting to the point where you know you need to pivot does.

avenoir 9y ago

While i absolutely agree with the author, how does this line of thinking change when you start using AWS or something similar? I'm building a SAAS app at the moment and a lot of these scaling decisions are incredibly easy to implement without imposing any significant costs in terms of maintenance (and money, a lot of times). This would've been a different story had i been using my own infrastructure.

theptip 9y ago
I think there's a key difference between building your architecture with plenty of headroom to scale (leaving plenty of low-hanging fruit), and actually implementing high-scale function before it's required. In a cloud environment, it's quite easy to be comfortable that you _could_ scale out to 10-100x of your current capacity, if you have to. But it's probably not worth proving that fact until you have some customers. Or put another way, being in a very flexible environment makes it even more of a no-brainer to delay building for scale, since you have so many free or nearly-free options there. Now of course, if you really do hit scale you'll have the fun problem of needing to optimize your spend, but you can probably delay thinking about that until well after you've proven product-market fit.

venture_lol 9y ago

There is nothing wrong with planning for scale and for microservices or whatever appropriate architecture from the start. It does not take more time to design properly from the get go. That, of course, assumes you have a CTO and a team capable of their job :)

dasil003 9y ago
Yes, if and only if you know exactly what you need to build and how big the scale will be from day 1. What usually happens is the first thing you build is wrong, and you need to iterate aggressively on not just the software but the entire business concept. Once you have product-market fit, then you are in a position to design an optimal architecture. If you go microservices before that, you are adding unjustifiable overhead, which will inarguably slow you down compared to a monolith when you are in the 1-3 engineer phase, and don't fool yourself about that.
- venture_lol 9y ago
  I frankly have no idea what you are talking about. You are making wild assumptions that monolith are better just because... It will all come down the the experience of the team, esp. the leadership - the managers, the leads. A strong, experienced, mature and effective leadeship will enable a team to perform "magic" and set the right technology foundation that will prove its worth as the business grows over time
  - taurath 9y ago
    The point he's making is the technology foundation can be built for the wrong product or the wrong market. After 2 full product reconfigurations or pivots requiring a TON of code being dumped you will wish you didn't have the overhead of building infrastructure with the first product in mind. In the end, the base reasoning for the foundations being built the way they didn't hold up in production. Building something quickly that can be broken up over time is the best way to an MVP, and you can deal with scalability issues when you HAVE scalability issues. I consider scalability to be a good problem to have, because by that time you have actual traction. If you are a bigco with a big marketing budget in a field in which you already have experience and business then by all means scale to millions of users at day one. Don't take on the cost of it in a startup, because the problem is TRACTION not technology with regards to scaling.

chaostheory 9y ago

Are people, who are just starting, still building complex chef and puppet systems when Heroku and Elastic Beanstalk exist? .i.e scalability is easy now. It's a click away or even automatic which these types of services. It's also affordable.

wink 9y ago
How do you think generic configuration management systems are replacable by a hosting provider and some autoscaling system? Honestly curious, I've never seen anyone scale just by "using puppet/chef".
- chaostheory 9y ago
  I didn't say they were replaceable for everyone and for every scenario. I'm just making the point that small startups no longer need to use them in the beginning if all they want is a way to auto-scale. Yes they made sys admin stuff easier but now it's even easier with scaling services that are already preconfigured. I just think it's strange that people are still discouraging founders to account for scaling when today it's cheap from both a time and money perspective to do from the start.
korzun 9y ago
> scalability is easy now Dumb, often limited scalability is easy now. Elastic Beanstalk approach is only as good as it's the weakest link, which you have no control over. EC2 is having issues with recovering from standby? Good luck getting anywhere. You can sit on the chat with the useless AWS 'technicians' that are paid to waste your time until the issue is corrected.
- chaostheory 9y ago
  I'm not sure if I'm making myself clear enough. I'm not arguing that this is great for everything. In the context of the post, that's good enough for a lot of apps just starting out WITHOUT the work of setting up something custom and durable ala Netflix's infrastructure.

bedros 9y ago

Simply, to counter the flood of new users, build a Dam called waiting list, the slowly activate users as you build capacity Put priority on paid users vs free, to cover server costs

peterjlee 9y ago

My answer to most scaling question is: by the time we have that problem, we'll have a lot of customers and money so we'll hire smarter engineers with experience in scaling to figure that out. I still make sure we don't do anything stupid right now so the future engineers have more options on how to scale. Also it helps that I'm not working on a zero-revenue consumer app.

HodGreeley 9y ago

I think I can give one concrete example where a NoSQL choice (often) makes sense. Common modern day scenario in mobile is to build an app and iterate. Migration in RDBMS rapidly becomes a real pain. Worse, you have no good sync solution in case you need it. Contrast with something like Couchbase Mobile. NoSQL embedded db is much easier to get going with. (Just check the Android docs on using SQLite and see how it compares to "here's my JSON, done".) Sync doesn't get in the way, but if/when you need it, you're going to be really happy it's there. (Disclosure: I work for Couchbase.)