29 ms·

Does it scale? Who cares (2011)

431 points by ne01 10y ago

amelius 10y ago
A better title would be: scalability is a luxury problem.
uptownfunk 10y ago
I like the overall idea here. Focus on building something quality first then worry about scaling later. Most servers can handle a decent amount of traffic. Seems like common sense to me. I guess some people can get too hung up on engineering to make their site scale before actually deploying or innovating on the product. Wonder if people have encountered this in the workplace before?
- CodeWriter23 10y ago
  I think the point is to make something that makes money first, then use the money to make something quality.
tyingq 10y ago
He makes the valid point that performance for each individual user, like page load time, does matter. Just that building for an audience size you don't yet have is mostly wasted time. Seems reasonable. I wonder, though, if PHP feels like an anchor to the average Facebook developer. I realize they architected around it, but it must have some effect on recruiting, retention, etc. I use PHP myself, and don't hate it, but the stigma is there.
- toomuchtodo 10y ago
  Any stigma should be prioritizing tech over the business. Your business exists to turn a profit, stack be damned.
  - tyingq 10y ago
    That works to motivate some employees, assuming their comp is relative to your business success. Facebook has been able to do that for some time, but there's probably a plateau or two in their future.
  - milesrout 10y ago
    The business isn't your concern if you don't work there yet. i.e. it's not bad to avoid working somewhere because they use PHP. That's not 'prioritising tech over business', it's just choosing to avoid toxic crap technology.
- qaq 10y ago
  1) FB uses a lot of languages other than Hack 2) Hack is fairly reasonable language even has pipe operator :) 3) While it is prevailing sentiment that PHP sux I think PHP 7 is fairly reasonable language
  - milesrout 10y ago
    Even ignoring the innumerable flaws PHP (yes including 7) has as a language, its standard library is among the worst standard libraries of any language.
    - qaq 10y ago
      Everything has flaws but PHP is a reasonable choice for many problems. Python pip is crap compared to composer for example. While my favorite language is Elixir I mostly do Node and Python at work can't say experience is significantly better with either of them compared to PHP 7.
      - milesrout 10y ago
        >Everything has flaws False equivalence. PHP has many, many more flaws to a much deeper and more serious level than any other mainstream programming language. It's insecure, buggy, full of broken behaviour for legacy systems, slow and easy to misuse. Its standard library is inconsistent, hard to learn, easy to misuse, full of legacy behaviour and slow. >but PHP is a reasonable choice for many problems. PHP is an unreasonable choice for every problem unless you have already solved your problem with PHP. I'm not saying that Facebook should rewrite in something else, obviously, but nobody should be starting new work in PHP. Nobody. >Python pip is crap compared to composer for example. There's nothing at all wrong with it. >While my favorite language is Elixir I mostly do Node and Python at work can't say experience is significantly better with either of them compared to PHP 7. Well you're quite objectively wrong. Yes Node.js is a pile of shit comparable to PHP, but it's Javascript, what do you expect? Javascript is basically the second worst mainstream programming language right behind PHP. Yes they've both had superficial changes that make them more enjoyable to write recently, but none of those changes fix the underlying inherent problems with those languages. Python, on the other hand, is a well-built, well-designed, much more sane language.
- pbreit 10y ago
  Facebook doesn't want those developers anyway.
cabaalis 10y ago
I liked the article and agree with its premise. But as a side question, why do developers use so many parenthetical expressions? Those ideas (like this one, which happens to add nothing) are often either throwaway statements (like this one) or are by themselves complete thoughts that should be a separate sentence. (I see this so often in posts written by devs.)
- the_unknown 10y ago
  I'd place it in the hands of habit. We're used to dealing with strange keyboard symbols (like %@*# - and having them mean something). Personally I find that I use dashes far more than any human should while writing (but like others) I do engage in a bit of parenthetical shenanigans as well.
- drblast 10y ago
  Those are s-expressions. It's an in-joke among the Common Lisp crowd.
- soneca 10y ago
  Nice writing :) I would blame it as commenting habit. Need to explain everything twice. "What if the reader doesnt understand exactly what I mean?"
  - 2muchcoffeeman 10y ago
    So use a separate sentence or formulate your ideas more clearly? I see this so often when developers write. From tutorials to official documentation etc. Extra words where a simpler sentence would have sufficed, poor structuring of ideas, not splitting text into appropriate paragraphs and sentences. I always feel dumb until I figure out what was trying to be said and then you realise just how poor the writing is.
    - camel_Snake 10y ago
      You should write a post on the parallels between legible code/english.
xyzzy4 10y ago
Ok but please don't do things like using nested array searches with bad runtime when you can use hashmaps instead. I hate seeing code or using programs that are extremely unoptimized.
- hartator 10y ago
  Hashes are more convenient too. It's just code smell by poor developers.
- tyingq 10y ago
  That seems to be the real story behind most "We switched from stack X to stack Y and gained an Z-fold increase in performance" stories. They tend to leave out the "and we also fixed some generically bad things we found when porting".
devduderino 10y ago
I care because it usually goes like this: Product manager > "Niche sass app {x} will never need to support more than 10-20 users" Two weeks after launch > "We have 10k users and counting, why didn't you architect this for scale?" Always assume you underestimated the scope of the project.
- virmundi 10y ago
  I've literally been waffling back and forth between UUID and BigInt for the last week. The reason for this is I need to handle eventually distributed systems. Do I need UUIDs? Maybe not, but after much debt, I've decided that the storage requirements of such an index in memory is worth the ability to move from Postgres to Postgresql-xl.
  - qaq 10y ago
    The problem is not with memory problem is that after checkpoint postgres will do full page writes for serial you will be touching way fewer pages compared to UUID so write amplification for UUID vs bigint can be 60X+
    - virmundi 10y ago
      Can you go into a bit more detail? I've seen amplification referenced before. My estimation is that the database will be pretty much read heavy. I will need to split a given database across multiple instances.
      - qaq 10y ago
        https://blog.2ndquadrant.com/on-the-impact-of-full-page-writes/ https://blog.2ndquadrant.com/on-the-impact-of-full-page-writ... this is fairly detailed analysis of this issue
        virmundi 10y ago
        Thank you. I'm reading over it now and for the next few days (to really grok it). I do wonder how bad it will be for low write situations. At the same time I'm open to switching to BigInt and using Postgresql-xl.
timewarrior 10y ago
Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic product over a weekend and launched. Timeline after that whenever the web server exhausted all the JVM threads trying to serve requests: 1. 1 month - 20k users - learnt about indexes and created indexes. 2. 3 months - 500k users - Realized MyISAM is a bad fit for mutable tables. Converted the tables to InnoDB. Increased number of JVM threads to tomcat 3. 9 months - 5M users - Realized that the default MySQL config is for a desktop and allocates just 64MB RAM to the database. Setup the mysql configs. 2 application servers now. 4. 18 months - 15M users - Tuned MySQL even more. Optimized JDBC connector to cache MySQL prepared statements. 5. 36 months - 45M users - Split database by having different tables on different machines. I had no idea or previous experience about any of these issues. However I always had enough notice to fix issues. Worked really hard, learnt along the way and was always able to find a way to scale the service. I know of absolutely no service which failed because it couldn't scale. First focus on building what people love. If people love your product, they will put up with the growing pains (e.g. Twitter used to be down a lot!). Because of my previous experience, I can now build and launch a highly scalable service at launch. However the reason I do this is that it is faster for me to do it - not because I am building it for scale. Launch as soon as you can. Iterate as fast as you can. Time is the only currency you have which can't be earned and only spent. Spend it wisely. Edited: formatting
- qume 10y ago
  I built a service (~10mm users at peak) which was designed from the ground up to scale, around 2002. When the numbers did grow we just sat back and watched it pretty much. Even given this, I completely agree with you. That's why I now develop in ruby. I'll take developer productivity over performance any day. It's the old saying - 'nice problem to have'.
  - logicallee 10y ago
    >I built a service (~10mm users at peak) which was designed from the ground up to scale, around 2002. When the numbers did grow we just sat back and watched it pretty much. >Even given this, I completely agree with you. That's why I now develop in ruby. Ouch that is a pretty big burn.
deleted 10y ago
[deleted]
debt 10y ago
i concur. it's fun to dream, but sadly, statistically most of us will never have to worry about scaling! so save yourself the energy and don't worry about it.
beefsack 10y ago
Taking a completely blasé approach to efficiency is potentially as dangerous as becoming hyper-focused on it. Not all businesses become roaring successes, and those who achieve moderate success often don't get the resources to fix deep-seated performance or architectural issues (either via engineering and/or throwing hardware at it.) Eventually these technical woes can completely halt momentum and I've seen it even drown some businesses who just aren't able to dig theirselves out of the hole the find themselves in. People always seem to be arguing for extremes, but the most sensible approach for most tends to be somewhere in the middle.
- eridius 10y ago
  I agree with your comment. BTW, the phrase you're looking for is "deep seated", not "deep seeded".
  - beefsack 10y ago
    Updated my post, cheers.
- timewarrior 10y ago
  Agreed that completely blasé is pretty bad. However in this case being blasé would mean, not working hard to scale when you users are getting a bad experience. A basic stack these days node.js+mongo, go+*sql can easily handle more than 100k users even with one of the worst implementations. Most products don't reach that point!
  - beefsack 10y ago
    > A basic stack these days node.js+mongo, go+*sql can easily handle more than 100k users... That is making some very large assumptions about application workload. For an application which is purely a CRUD interface to a database, yes.
    - timewarrior 10y ago
      Agreed on this point. Many a times there is heavy lifting - recommendations, machines learning etc. However that is don't using specialized technologies in a non user facing process and the results are then dumped into a DB available for a CRUD app. I am hoping that products with such requirements will have some obvious tools for such tasks (Hadoop, Hive etc) and they would find a way to scale such processes with time. So their stack might be go+*sql+Hadoop. Can you please suggest some use-cases which don't fit the above pattern and maybe we can brainstorm. Seems like a fun exercise!
janwillemb 10y ago
In general: don't fix a non-existing problem. You don't know beforehand what the problems of the future look like. Fancy technology X of today is technical debt in 10 years. So do invest in solving technical debt along the way in products you keep.
innocentoldguy 10y ago
I agree with this article; however, there are considerations that can be made early on, to ensure an easy path for future scalability, that don't waste any time or money during the project's nascency. For example, if I know I want my app to scale at some point in the future, I may opt to build it in Elixir, or I may choose to use a Riak cluster rather than of MySQL.
the_arun 10y ago
I agree with this article only for launching new products. But if you already have a product which is serving millions of customers, you better worry about scale while you change anything.
dlwdlw 10y ago
Flexibility vs efficiency. Agile vs high momentum. As a rule of thumb, start-ups need to be more agile as they are mostly exploring new territory, trying to create new value or re-scope valueless things into valuable things. Larger companies operate at a scale where minor efficiency improvements can mean millions of dollars and thus require more people to do the same thing, but better. Individualistic thinking on new directions to go is not needed nor appreciated. Of course there are excepttions. The question boils down to whether or not the ladder is on the right walk before charging up it. In rare circumstances you can do both. Either the problem is trivial, or the problem becomes trivial because you have a super expert. 10x programmers who habitually write efficient code without needing to think too much have more bandwidth for things like strategy and direction. The car they drive is both more agile, accelerates faster, has a higher max speed, etc...but even this can't move mountains. The problem an individual can solve, no matter the level of genius, is still small in scope conpared to the power of movements and collective action and intention. The most poweful skill is to seed these movements and direct them. Abstractly, this is what VCs look for in founders and also a reason why very smart and technical people feel short-changed that they are not appreciated for their 10x skills. (Making 500k instead of millions/billions) They may have 10x skills, but there are whole orders of magnitude they can be blind to.
sbuttgereit 10y ago
The overall premise of the blog is exactly correct; though I would say some areas you probably need to consider more than others. Spending a lot of time figuring out what exact microservice/sharding/etc strategy you need to serve a zillion visits a day and building it before you've even got customer/visitor one is overkill out of the gate. But that shouldn't mean you shouldn't think about how you'll scale over the short term or medium term at all. When I approach scaling, I'll tend to spend much more time on the data retention strategy than anything else: databases (or other stores), being stateful, means that it's a harder problem to deal with later than earlier as compared to the stateless parts of the system. Even so, I'm typically not developing the data services for the Unicorn I wish the client will become, I'm just putting a lot more thought into optimizing the data services I am building so it won't hit the breaking point as early as it might if I were designing for functionality alone. I do expect there to be a breaking point and a need to change direction at some point in these early stage designs. But in that short to medium term period, the simpler designs are regularly easier to maintain than the fully "scalable" approaches that might be tried otherwise, and rarely do those companies ever need anything more.
namanyayg 10y ago
Can we have (2011) in the title?
salman89 10y ago
I agree with the general premise of avoiding premature optimizations, but designing systems that scale is important for several reasons: - Startups grow exponentially, if you're playing catchup as you're growing you are focusing on keeping the lights on and hanging on for the ride. Important for a growing company to focus on vision. - Software that scales in traffic is easier to scale in engineering effort. For example, harder for a 100 engineers to work on a single monolith vs 10 services. - Service infrastructure cost is high on the list of cash burn. Scalable systems are efficient, allow startups to live longer. - If the product you are selling directly correlates to computing power, important to make sure you are selling something that can be done profitably. For example, if you are selling video processing as a service, you absolutely need to validate that you can do this at scale in a profitable manner. I also don't agree with the premise that speed of development and scalable systems are always in contention. After a certain point, scalable systems go hand and hand with your ability to execute quickly.
- mk89 10y ago
  I totally agree with you on all the points. I think it takes experienced managers and developers with business-oriented mindset to achieve the best compromise. Saying "there is no need to overengineer now" can be as bad as months of overengineering. It all depends on the business.
- Michielvv 10y ago
  In the projects that I was involved in in the past years, the ones that decided to build a scalable infrastructure before gaining traction were exactly those that were burning cash on infrastructure. There is just a reasonable minimum of servers you need to run such an infrastructure and then you need something similar as a staging environment to make sure it all works together correctly. That being said, there are of course choices that can be made early on that will help you in case you need to scale at some time. Spending a bit of time thinking through what would happen if you do need to split up and scale out your system will help you avoid pitfalls such as relying too much on the local filesystem being shared across pageviews. In general I believe there is more benefit on spending time on performance early on. (not caching, that is just postponing the problem) as it benefits not just your ability to run on a single system for much longer, it will also make life better for your users.
deleted 10y ago
[deleted]
ninjakeyboard 10y ago
I agree BUT it's not that hard to ensure your app scales. It's more about using appropriate tools for the job.
hopfog 10y ago
I'm in the unfortunate position where this question actually matters from day 1. I learnt the hard way a few days ago when I hit the bottleneck (about 50-100 concurrent users) and I'm not sure how to proceed. It's a multiplayer drawing site built with Node.js/socket.io. I'm already on the biggest Heroku dyno my budget can allow and it's too big of a task to rewrite the back end to support load balancing (and I wouldn't know where to start). Bear in mind that this is a side-project I'm not making any money of. I had a lot of new features planned but now I've put development on hold. It's not fun to work on something you can't allow to get popular since it would kill it.
- Everhusk 10y ago
  Hey, you might want to check out https://realm.io/ https://realm.io/ , the demo is exactly what you said. Good luck!
  - hopfog 10y ago
    Looks cool but seems like it's aimed at mobile and not websites? Also, how do they differ from Firebase? My previous experience with Firebase makes me hesitant to use a BaaS. It works great for simple stuff but as soon as you want more control it becomes a mess compared to just having your own server. It might just be me who haven't wrapped my head around the mindset correctly though.
- yxhuvud 10y ago
  What is it your processor spends its time on? Perhaps those parts could be rethought and redesigned? Perhaps heroku is the wrong solution for those tasks even if it is good for the rest of the app. Would it be possible to find something cheaper for just those parts?
  - hopfog 10y ago
    To be honest I haven't really benchmarked it. Maybe that's something I should invest time in to see if there's some obvious bottleneck I can fix. Thanks for the suggestion.
shoefly 10y ago
Evolution is a beautiful thing. I once worked for a monolith who decided to invent a new way of programming. They built something massive and ready to scale even larger. And then they discovered no one wanted the product.
Jach 10y ago
No one else bothered by "End-to-end tracking of customers" as the primary concern? Ok then. On the subject of scaling, I think it's good to have an idea in your head about a path to scalability. One server, using PHP and MySQL? Ok. Just be aware you might have to load balance either or both the server and DB in the future, and that's assuming you've gotten the low hanging fruit of making them faster on their own. But as this thread's top comment illustrates, learning that stuff on the fly isn't too hard. So maybe it's better to make sure you're going with technology you sort of know has had big successes elsewhere (like Java, PHP, or MySQL) and even if you're not quite sure how you might scale it beyond the defaults you know others have solved that problem and you can learn later if/when needed.
andromeda__ 10y ago
Fundamentally disagree with the ethos of this article. What about ambition? What happened to that? > You won’t make the cover of Time Magazine and you won’t be ordering a private jet but that Ferrari is a definite possibility, if that’s the thing you are hurting for. (college education for your kids is probably a better idea ;) ). I'd like to be on the cover of fortune or as Russ Hanneman might say, "I wanna make a fuckton of money all at once". I don't see any problem with being ambitious or wanting that private jet.
addicted 10y ago
Healthcare.gov was a site that failed and suffered due to scaling issues. However, it was an anomaly because unlike a product someone this article is intended towards would be building, that site had an immediate audience of millions of users from the get go. Also, the fact that it took a few weeks to be rewritten to handle the load at which point it became extremely successful, strengthens the original article's point. By the time scalability becomes a problem, you will have enough resources to tackle the scalability problem.
- user5994461 10y ago
  This is not an anomaly. It's the norm for all government projects, and most projects started in established companies that already have millions of users. To be working in these domains, the "doesn't need to scale" mentality" is very inappropriate and I find it to be doing a lot of damages. The guys at healthcare handled most the scaling problems, accounting for the means at their disposal and the time frame they had, judging by what was published in the news. That it took a few weeks to smooth showed that they handled scaling beforehand. A launch from 10 users to 10 million is always a bit bumpy for the initial weeks.
shadowmint 10y ago
I care. It's easy to brush off scaling concerns as not important, but I've had personal experience where it's mattered, and if you want a high profile example, look at twitter. Yes, premature optimization is a bad thing, and so is over engineering; but that's easy to say if you have the experience to make the right initial choices that mean you have a meaningful path forward to scale when you do need it. For example, lets say you build a typical business app and push something out quickly that doesn't say, log when it fails, or provide an auto-update mechanism, or have any remote access. Now you have it deployed at 50 locations and its 'not working' for some reason. Not only do you physically have to go out to see whats wrong, you have to organize a reinstall at 50 locations. Bad right? yes. It's very bad. (<---- Personal experience) Or, you do a similar ruby or python app when your domain is something that involves bulk processing massive loads of data. It works fine and you have a great 'platform' until you have 3 users, and then it starts to slow down for everyone; and it turns out, you need a dedicated server for each customer because doing your business logic in a slow language works when you only need to do 10 items a second, not 10000. Bad right? yes. Very. Bad. (<---- Personal experience) It's not premature optimization to not pick stupid technology choices for your domain, or ship prototypes. ...but sometimes you don't have someone on the team with the experience to realize that, and the push from management is to just get it out, and not worry about the details; but trust me, if you have someone who is sticking their neck out and go, hey wait, this isn't going to scale... Maybe you should listen to what they have to say, not quote platitudes. Ecommerce is probably one of those things where the domain is well enough known you can get away with it; heck, just throw away all your rubbish and use an off-the-shelf solution if you hit a problem; but I'm going to suggest that the majority of people aren't building that sort of platform, because its largely a solved problem.
- std_throwaway 10y ago
  I don't like opinionated pieces like the one presented here because while they are right about some things they miss other things and present half-truths as full-truths. In my experience (and I have little of that) it's important to know the upgrade path and adjust your planning accordingly. How many users can you serve with your solution? How big do you expect the market to be in that stage? What technology would be the next step? How do you get there? How much more work would it be? Always be one step ahead with technology, but not two. Most markets are surprisingly small. Most use cases scale surprisingly well. You can probably push your solution by an order of one magnitude if you need it quickly. In order to answer the questions you need people who know the product, the (potential) technologies and the market. When you start you probably won't know any of that. See the first prototype you deploy to the customers as a means of collecting data for the first production version. Your first product is not your first product. Your funding should respect that. Get it done quickly with the aim of answering the critical questions. Then go back and design the next version "good-enough" for the second scaling step with the upgrade path in mind.
partycoder 10y ago
While a point the article tries to make (fix your leaky funnel before acquiring users) is true, I disagree with the article. If your application is converting well, scalability problems are not acceptable. I have seen applications that convert very well, but were limited by scalability problems. That meant that the business had to hold on on marketing and user acquisition, missed their financial targets, and that cascaded into breaching contracts. The phrase that nobody wants to hear in that situation is "who cares about scalability". Now, if you did not have a lot of problems scaling in your particular case, that just means it was not an obstacle for you. e.g: you had good intuition around performance/scalability, or the problem was coincidentally a good fit for your technological choices. Unfortunately not everyone has a good intuition about scalability, not everyone is risk averse and not everyone is good at picking a good technology for their use case. So I disagree with this article in the sense that it is not in the best interest of a random reader to not care about scalability.
tianlins 10y ago
It really depends the type of growth. Organic growth is driven by the quality of product therefore scaling issue comes later. But most venture-backed startup services such as O2O need quickly dominate market by throwing cash to get users so scaling is an issue from day one.
seajones 10y ago
I do agree, but being at the "we need to scale up asap" stage atm makes it harder to. There's a balance to be struck. Maybe about approach of "who cares" with the POC, MVP etc stages, then keep it more and more in mind at each stage after that would be best.
etnos 10y ago
Mhee, We built a rest API with Laravel, after a heated conversation with a junior dev about PHP-Laravel being slow. +20k petitions average a day no problem at all. Digital ocean bill is like 20dlls/month ; ) Kids these days need to stop watching "The Social Network", chill out Mr. next Zuckerberg, you ain't scaling anything but a headache
iveqy 10y ago
Having working on an app that we just throwed more hardware at, to the point where the azure subscription cost could be lowered by my whole anually salary, by 3 months work of optimization. I believe performance does matter. We where a 4 person team and could have added a fifth if we had a cheaper design.
nomercy400 10y ago
Once worked at a startup where we expected 'some activity' in our webshop at product launch, and didn't think about scaling for that. Well, some activity turned out to be 450mbit/s for five hours, which our unscalable application/webshop didn't handle very well. It became overloaded in the first minute, and took us more than an hour to get remote access again. It's one of those things we did better for our next big event (major sharding, basically replicated the application 32 times of the largest VM instance we could get. It was needed and it survived).
mannykannot 10y ago
There is a similar argument with regard to making code reusable. I have seen inordinately complex code come from a desire to make it reusable, even if the prospects for it being reused were slim to nonexistent.
shanecleveland 10y ago
Came across a service last week with a free trial and paid plan. How to upgrade? Contact by email! Why spend time and resources on a payment process if you don't have any paying customers yet? Obviously not right for everyone, and I'm not saying it doesn't have its own challenges, but the core product deserves the most attention early on.
didibus 10y ago
The reason scale isn't so important today is because most DBs can actually scale vertically to really high numbers. The tipping point is high enough that if you have this problem, you probably can also afford to fix it. What matters though is performance and availability. No matter what scale you work at, you can't be slow, that will drive people away. You also can't be unavailable. This means that you might have to handle traffic spikes. Depending on your offering, you probably also want to be secure and reliable. Losing customer data or leaking it will drive customers away too. So, I'd mostly agree, in 2016, scale isn't a big problem. Better to focus on functionality, performance, security, reliability and availability. These things will impact all your customers, even when you only have one. They'll also be much harder to fix. Where scale matters is at big companies. When you already have a lot of customers, you’re first version of any new feature or product must already be at scale. Amazon couldn't have launched a non scalable prime now, or echo. Google can't launch a non scalable chat service, etc.
- hamburglar 10y ago
  > Where scale matters is at big companies. When you already have a lot of customers, you’re first version of any new feature or product must already be at scale. Amazon couldn't have launched a non scalable prime now, or echo. Google can't launch a non scalable chat service, etc. This is the point I was going to make. When you are designing the very platforms on which all these "does it scale? who cares" startups are going to be built, you do not have the luxury of having that attitude. Definitely don't take that mindset into an interview at google or amazon. :) I do agree with the OP's sentiment for startup projects in general. I have had the experience of worrying about scale too early and over-engineering a system which never had more than a few dozen users, and the opposite experience of tossing something together that wouldn't scale and then going through the hairy scrambling at every order of magnitude of scale through tens of millions of users. The latter was definitely a better strategy.
therealdrag0 10y ago
I started working at a company 6 months ago, and in that time I've fixed a half dozen performance problems. Some of them didn't matter THAT much, but some of them did. My team manages an authentication -service. Some of the calls were averaging hundreds of ms which added terrible overhead to other services' calls. In other places had CICD builds running tests twice, logs logging twice, inefficient algorithms, single HTTP transactions making dozens of DB calls. You name it. Some of these were simple things that if you stopped to smell the roses you could notice and fix. Let's all be boy scouts and make code better while we're in it :)
Vkkan2016 10y ago
IMHO until you hit scalability issue don't try to spend your time and money there yet

431 points by ne01 10y ago

A better title would be: scalability is a luxury problem.

I like the overall idea here. Focus on building something quality first then worry about scaling later. Most servers can handle a decent amount of traffic. Seems like common sense to me. I guess some people can get too hung up on engineering to make their site scale before actually deploying or innovating on the product. Wonder if people have encountered this in the workplace before?

CodeWriter23 10y ago
I think the point is to make something that makes money first, then use the money to make something quality.

tyingq 10y ago

He makes the valid point that performance for each individual user, like page load time, does matter. Just that building for an audience size you don't yet have is mostly wasted time. Seems reasonable. I wonder, though, if PHP feels like an anchor to the average Facebook developer. I realize they architected around it, but it must have some effect on recruiting, retention, etc. I use PHP myself, and don't hate it, but the stigma is there.

toomuchtodo 10y ago
Any stigma should be prioritizing tech over the business. Your business exists to turn a profit, stack be damned.
- tyingq 10y ago
  That works to motivate some employees, assuming their comp is relative to your business success. Facebook has been able to do that for some time, but there's probably a plateau or two in their future.
- milesrout 10y ago
  The business isn't your concern if you don't work there yet. i.e. it's not bad to avoid working somewhere because they use PHP. That's not 'prioritising tech over business', it's just choosing to avoid toxic crap technology.
qaq 10y ago
1) FB uses a lot of languages other than Hack 2) Hack is fairly reasonable language even has pipe operator :) 3) While it is prevailing sentiment that PHP sux I think PHP 7 is fairly reasonable language
- milesrout 10y ago
  Even ignoring the innumerable flaws PHP (yes including 7) has as a language, its standard library is among the worst standard libraries of any language.
  - qaq 10y ago
    Everything has flaws but PHP is a reasonable choice for many problems. Python pip is crap compared to composer for example. While my favorite language is Elixir I mostly do Node and Python at work can't say experience is significantly better with either of them compared to PHP 7.
    - milesrout 10y ago
      >Everything has flaws False equivalence. PHP has many, many more flaws to a much deeper and more serious level than any other mainstream programming language. It's insecure, buggy, full of broken behaviour for legacy systems, slow and easy to misuse. Its standard library is inconsistent, hard to learn, easy to misuse, full of legacy behaviour and slow. >but PHP is a reasonable choice for many problems. PHP is an unreasonable choice for every problem unless you have already solved your problem with PHP. I'm not saying that Facebook should rewrite in something else, obviously, but nobody should be starting new work in PHP. Nobody. >Python pip is crap compared to composer for example. There's nothing at all wrong with it. >While my favorite language is Elixir I mostly do Node and Python at work can't say experience is significantly better with either of them compared to PHP 7. Well you're quite objectively wrong. Yes Node.js is a pile of shit comparable to PHP, but it's Javascript, what do you expect? Javascript is basically the second worst mainstream programming language right behind PHP. Yes they've both had superficial changes that make them more enjoyable to write recently, but none of those changes fix the underlying inherent problems with those languages. Python, on the other hand, is a well-built, well-designed, much more sane language.
pbreit 10y ago
Facebook doesn't want those developers anyway.

cabaalis 10y ago

I liked the article and agree with its premise. But as a side question, why do developers use so many parenthetical expressions? Those ideas (like this one, which happens to add nothing) are often either throwaway statements (like this one) or are by themselves complete thoughts that should be a separate sentence. (I see this so often in posts written by devs.)

the_unknown 10y ago
I'd place it in the hands of habit. We're used to dealing with strange keyboard symbols (like %@*# - and having them mean something). Personally I find that I use dashes far more than any human should while writing (but like others) I do engage in a bit of parenthetical shenanigans as well.
drblast 10y ago
Those are s-expressions. It's an in-joke among the Common Lisp crowd.
soneca 10y ago
Nice writing :) I would blame it as commenting habit. Need to explain everything twice. "What if the reader doesnt understand exactly what I mean?"
- 2muchcoffeeman 10y ago
  So use a separate sentence or formulate your ideas more clearly? I see this so often when developers write. From tutorials to official documentation etc. Extra words where a simpler sentence would have sufficed, poor structuring of ideas, not splitting text into appropriate paragraphs and sentences. I always feel dumb until I figure out what was trying to be said and then you realise just how poor the writing is.
  - camel_Snake 10y ago
    You should write a post on the parallels between legible code/english.

xyzzy4 10y ago

Ok but please don't do things like using nested array searches with bad runtime when you can use hashmaps instead. I hate seeing code or using programs that are extremely unoptimized.

hartator 10y ago
Hashes are more convenient too. It's just code smell by poor developers.
tyingq 10y ago
That seems to be the real story behind most "We switched from stack X to stack Y and gained an Z-fold increase in performance" stories. They tend to leave out the "and we also fixed some generically bad things we found when porting".

devduderino 10y ago

I care because it usually goes like this: Product manager > "Niche sass app {x} will never need to support more than 10-20 users" Two weeks after launch > "We have 10k users and counting, why didn't you architect this for scale?" Always assume you underestimated the scope of the project.

virmundi 10y ago
I've literally been waffling back and forth between UUID and BigInt for the last week. The reason for this is I need to handle eventually distributed systems. Do I need UUIDs? Maybe not, but after much debt, I've decided that the storage requirements of such an index in memory is worth the ability to move from Postgres to Postgresql-xl.
- qaq 10y ago
  The problem is not with memory problem is that after checkpoint postgres will do full page writes for serial you will be touching way fewer pages compared to UUID so write amplification for UUID vs bigint can be 60X+
  - virmundi 10y ago
    Can you go into a bit more detail? I've seen amplification referenced before. My estimation is that the database will be pretty much read heavy. I will need to split a given database across multiple instances.
    - qaq 10y ago
      https://blog.2ndquadrant.com/on-the-impact-of-full-page-writes/ https://blog.2ndquadrant.com/on-the-impact-of-full-page-writ... this is fairly detailed analysis of this issue
      - virmundi 10y ago
        Thank you. I'm reading over it now and for the next few days (to really grok it). I do wonder how bad it will be for low write situations. At the same time I'm open to switching to BigInt and using Postgresql-xl.

timewarrior 10y ago

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic product over a weekend and launched. Timeline after that whenever the web server exhausted all the JVM threads trying to serve requests: 1. 1 month - 20k users - learnt about indexes and created indexes. 2. 3 months - 500k users - Realized MyISAM is a bad fit for mutable tables. Converted the tables to InnoDB. Increased number of JVM threads to tomcat 3. 9 months - 5M users - Realized that the default MySQL config is for a desktop and allocates just 64MB RAM to the database. Setup the mysql configs. 2 application servers now. 4. 18 months - 15M users - Tuned MySQL even more. Optimized JDBC connector to cache MySQL prepared statements. 5. 36 months - 45M users - Split database by having different tables on different machines. I had no idea or previous experience about any of these issues. However I always had enough notice to fix issues. Worked really hard, learnt along the way and was always able to find a way to scale the service. I know of absolutely no service which failed because it couldn't scale. First focus on building what people love. If people love your product, they will put up with the growing pains (e.g. Twitter used to be down a lot!). Because of my previous experience, I can now build and launch a highly scalable service at launch. However the reason I do this is that it is faster for me to do it - not because I am building it for scale. Launch as soon as you can. Iterate as fast as you can. Time is the only currency you have which can't be earned and only spent. Spend it wisely. Edited: formatting

qume 10y ago
I built a service (~10mm users at peak) which was designed from the ground up to scale, around 2002. When the numbers did grow we just sat back and watched it pretty much. Even given this, I completely agree with you. That's why I now develop in ruby. I'll take developer productivity over performance any day. It's the old saying - 'nice problem to have'.
- logicallee 10y ago
  >I built a service (~10mm users at peak) which was designed from the ground up to scale, around 2002. When the numbers did grow we just sat back and watched it pretty much. >Even given this, I completely agree with you. That's why I now develop in ruby. Ouch that is a pretty big burn.

deleted 10y ago

[deleted]

debt 10y ago

i concur. it's fun to dream, but sadly, statistically most of us will never have to worry about scaling! so save yourself the energy and don't worry about it.

beefsack 10y ago

Taking a completely blasé approach to efficiency is potentially as dangerous as becoming hyper-focused on it. Not all businesses become roaring successes, and those who achieve moderate success often don't get the resources to fix deep-seated performance or architectural issues (either via engineering and/or throwing hardware at it.) Eventually these technical woes can completely halt momentum and I've seen it even drown some businesses who just aren't able to dig theirselves out of the hole the find themselves in. People always seem to be arguing for extremes, but the most sensible approach for most tends to be somewhere in the middle.

eridius 10y ago
I agree with your comment. BTW, the phrase you're looking for is "deep seated", not "deep seeded".
- beefsack 10y ago
  Updated my post, cheers.
timewarrior 10y ago
Agreed that completely blasé is pretty bad. However in this case being blasé would mean, not working hard to scale when you users are getting a bad experience. A basic stack these days node.js+mongo, go+*sql can easily handle more than 100k users even with one of the worst implementations. Most products don't reach that point!
- beefsack 10y ago
  > A basic stack these days node.js+mongo, go+*sql can easily handle more than 100k users... That is making some very large assumptions about application workload. For an application which is purely a CRUD interface to a database, yes.
  - timewarrior 10y ago
    Agreed on this point. Many a times there is heavy lifting - recommendations, machines learning etc. However that is don't using specialized technologies in a non user facing process and the results are then dumped into a DB available for a CRUD app. I am hoping that products with such requirements will have some obvious tools for such tasks (Hadoop, Hive etc) and they would find a way to scale such processes with time. So their stack might be go+*sql+Hadoop. Can you please suggest some use-cases which don't fit the above pattern and maybe we can brainstorm. Seems like a fun exercise!

janwillemb 10y ago

In general: don't fix a non-existing problem. You don't know beforehand what the problems of the future look like. Fancy technology X of today is technical debt in 10 years. So do invest in solving technical debt along the way in products you keep.

innocentoldguy 10y ago

I agree with this article; however, there are considerations that can be made early on, to ensure an easy path for future scalability, that don't waste any time or money during the project's nascency. For example, if I know I want my app to scale at some point in the future, I may opt to build it in Elixir, or I may choose to use a Riak cluster rather than of MySQL.

the_arun 10y ago

I agree with this article only for launching new products. But if you already have a product which is serving millions of customers, you better worry about scale while you change anything.

dlwdlw 10y ago

Flexibility vs efficiency. Agile vs high momentum. As a rule of thumb, start-ups need to be more agile as they are mostly exploring new territory, trying to create new value or re-scope valueless things into valuable things. Larger companies operate at a scale where minor efficiency improvements can mean millions of dollars and thus require more people to do the same thing, but better. Individualistic thinking on new directions to go is not needed nor appreciated. Of course there are excepttions. The question boils down to whether or not the ladder is on the right walk before charging up it. In rare circumstances you can do both. Either the problem is trivial, or the problem becomes trivial because you have a super expert. 10x programmers who habitually write efficient code without needing to think too much have more bandwidth for things like strategy and direction. The car they drive is both more agile, accelerates faster, has a higher max speed, etc...but even this can't move mountains. The problem an individual can solve, no matter the level of genius, is still small in scope conpared to the power of movements and collective action and intention. The most poweful skill is to seed these movements and direct them. Abstractly, this is what VCs look for in founders and also a reason why very smart and technical people feel short-changed that they are not appreciated for their 10x skills. (Making 500k instead of millions/billions) They may have 10x skills, but there are whole orders of magnitude they can be blind to.

sbuttgereit 10y ago

The overall premise of the blog is exactly correct; though I would say some areas you probably need to consider more than others. Spending a lot of time figuring out what exact microservice/sharding/etc strategy you need to serve a zillion visits a day and building it before you've even got customer/visitor one is overkill out of the gate. But that shouldn't mean you shouldn't think about how you'll scale over the short term or medium term at all. When I approach scaling, I'll tend to spend much more time on the data retention strategy than anything else: databases (or other stores), being stateful, means that it's a harder problem to deal with later than earlier as compared to the stateless parts of the system. Even so, I'm typically not developing the data services for the Unicorn I wish the client will become, I'm just putting a lot more thought into optimizing the data services I am building so it won't hit the breaking point as early as it might if I were designing for functionality alone. I do expect there to be a breaking point and a need to change direction at some point in these early stage designs. But in that short to medium term period, the simpler designs are regularly easier to maintain than the fully "scalable" approaches that might be tried otherwise, and rarely do those companies ever need anything more.

namanyayg 10y ago

Can we have (2011) in the title?

salman89 10y ago

I agree with the general premise of avoiding premature optimizations, but designing systems that scale is important for several reasons: - Startups grow exponentially, if you're playing catchup as you're growing you are focusing on keeping the lights on and hanging on for the ride. Important for a growing company to focus on vision. - Software that scales in traffic is easier to scale in engineering effort. For example, harder for a 100 engineers to work on a single monolith vs 10 services. - Service infrastructure cost is high on the list of cash burn. Scalable systems are efficient, allow startups to live longer. - If the product you are selling directly correlates to computing power, important to make sure you are selling something that can be done profitably. For example, if you are selling video processing as a service, you absolutely need to validate that you can do this at scale in a profitable manner. I also don't agree with the premise that speed of development and scalable systems are always in contention. After a certain point, scalable systems go hand and hand with your ability to execute quickly.

mk89 10y ago
I totally agree with you on all the points. I think it takes experienced managers and developers with business-oriented mindset to achieve the best compromise. Saying "there is no need to overengineer now" can be as bad as months of overengineering. It all depends on the business.
Michielvv 10y ago
In the projects that I was involved in in the past years, the ones that decided to build a scalable infrastructure before gaining traction were exactly those that were burning cash on infrastructure. There is just a reasonable minimum of servers you need to run such an infrastructure and then you need something similar as a staging environment to make sure it all works together correctly. That being said, there are of course choices that can be made early on that will help you in case you need to scale at some time. Spending a bit of time thinking through what would happen if you do need to split up and scale out your system will help you avoid pitfalls such as relying too much on the local filesystem being shared across pageviews. In general I believe there is more benefit on spending time on performance early on. (not caching, that is just postponing the problem) as it benefits not just your ability to run on a single system for much longer, it will also make life better for your users.

deleted 10y ago

[deleted]

ninjakeyboard 10y ago

I agree BUT it's not that hard to ensure your app scales. It's more about using appropriate tools for the job.

hopfog 10y ago

I'm in the unfortunate position where this question actually matters from day 1. I learnt the hard way a few days ago when I hit the bottleneck (about 50-100 concurrent users) and I'm not sure how to proceed. It's a multiplayer drawing site built with Node.js/socket.io. I'm already on the biggest Heroku dyno my budget can allow and it's too big of a task to rewrite the back end to support load balancing (and I wouldn't know where to start). Bear in mind that this is a side-project I'm not making any money of. I had a lot of new features planned but now I've put development on hold. It's not fun to work on something you can't allow to get popular since it would kill it.

Everhusk 10y ago
Hey, you might want to check out https://realm.io/ https://realm.io/ , the demo is exactly what you said. Good luck!
- hopfog 10y ago
  Looks cool but seems like it's aimed at mobile and not websites? Also, how do they differ from Firebase? My previous experience with Firebase makes me hesitant to use a BaaS. It works great for simple stuff but as soon as you want more control it becomes a mess compared to just having your own server. It might just be me who haven't wrapped my head around the mindset correctly though.
yxhuvud 10y ago
What is it your processor spends its time on? Perhaps those parts could be rethought and redesigned? Perhaps heroku is the wrong solution for those tasks even if it is good for the rest of the app. Would it be possible to find something cheaper for just those parts?
- hopfog 10y ago
  To be honest I haven't really benchmarked it. Maybe that's something I should invest time in to see if there's some obvious bottleneck I can fix. Thanks for the suggestion.

shoefly 10y ago

Evolution is a beautiful thing. I once worked for a monolith who decided to invent a new way of programming. They built something massive and ready to scale even larger. And then they discovered no one wanted the product.

Jach 10y ago

No one else bothered by "End-to-end tracking of customers" as the primary concern? Ok then. On the subject of scaling, I think it's good to have an idea in your head about a path to scalability. One server, using PHP and MySQL? Ok. Just be aware you might have to load balance either or both the server and DB in the future, and that's assuming you've gotten the low hanging fruit of making them faster on their own. But as this thread's top comment illustrates, learning that stuff on the fly isn't too hard. So maybe it's better to make sure you're going with technology you sort of know has had big successes elsewhere (like Java, PHP, or MySQL) and even if you're not quite sure how you might scale it beyond the defaults you know others have solved that problem and you can learn later if/when needed.

andromeda__ 10y ago

Fundamentally disagree with the ethos of this article. What about ambition? What happened to that? > You won’t make the cover of Time Magazine and you won’t be ordering a private jet but that Ferrari is a definite possibility, if that’s the thing you are hurting for. (college education for your kids is probably a better idea ;) ). I'd like to be on the cover of fortune or as Russ Hanneman might say, "I wanna make a fuckton of money all at once". I don't see any problem with being ambitious or wanting that private jet.

addicted 10y ago

Healthcare.gov was a site that failed and suffered due to scaling issues. However, it was an anomaly because unlike a product someone this article is intended towards would be building, that site had an immediate audience of millions of users from the get go. Also, the fact that it took a few weeks to be rewritten to handle the load at which point it became extremely successful, strengthens the original article's point. By the time scalability becomes a problem, you will have enough resources to tackle the scalability problem.

user5994461 10y ago
This is not an anomaly. It's the norm for all government projects, and most projects started in established companies that already have millions of users. To be working in these domains, the "doesn't need to scale" mentality" is very inappropriate and I find it to be doing a lot of damages. The guys at healthcare handled most the scaling problems, accounting for the means at their disposal and the time frame they had, judging by what was published in the news. That it took a few weeks to smooth showed that they handled scaling beforehand. A launch from 10 users to 10 million is always a bit bumpy for the initial weeks.

shadowmint 10y ago

I care. It's easy to brush off scaling concerns as not important, but I've had personal experience where it's mattered, and if you want a high profile example, look at twitter. Yes, premature optimization is a bad thing, and so is over engineering; but that's easy to say if you have the experience to make the right initial choices that mean you have a meaningful path forward to scale when you do need it. For example, lets say you build a typical business app and push something out quickly that doesn't say, log when it fails, or provide an auto-update mechanism, or have any remote access. Now you have it deployed at 50 locations and its 'not working' for some reason. Not only do you physically have to go out to see whats wrong, you have to organize a reinstall at 50 locations. Bad right? yes. It's very bad. (<---- Personal experience) Or, you do a similar ruby or python app when your domain is something that involves bulk processing massive loads of data. It works fine and you have a great 'platform' until you have 3 users, and then it starts to slow down for everyone; and it turns out, you need a dedicated server for each customer because doing your business logic in a slow language works when you only need to do 10 items a second, not 10000. Bad right? yes. Very. Bad. (<---- Personal experience) It's not premature optimization to not pick stupid technology choices for your domain, or ship prototypes. ...but sometimes you don't have someone on the team with the experience to realize that, and the push from management is to just get it out, and not worry about the details; but trust me, if you have someone who is sticking their neck out and go, hey wait, this isn't going to scale... Maybe you should listen to what they have to say, not quote platitudes. Ecommerce is probably one of those things where the domain is well enough known you can get away with it; heck, just throw away all your rubbish and use an off-the-shelf solution if you hit a problem; but I'm going to suggest that the majority of people aren't building that sort of platform, because its largely a solved problem.

std_throwaway 10y ago
I don't like opinionated pieces like the one presented here because while they are right about some things they miss other things and present half-truths as full-truths. In my experience (and I have little of that) it's important to know the upgrade path and adjust your planning accordingly. How many users can you serve with your solution? How big do you expect the market to be in that stage? What technology would be the next step? How do you get there? How much more work would it be? Always be one step ahead with technology, but not two. Most markets are surprisingly small. Most use cases scale surprisingly well. You can probably push your solution by an order of one magnitude if you need it quickly. In order to answer the questions you need people who know the product, the (potential) technologies and the market. When you start you probably won't know any of that. See the first prototype you deploy to the customers as a means of collecting data for the first production version. Your first product is not your first product. Your funding should respect that. Get it done quickly with the aim of answering the critical questions. Then go back and design the next version "good-enough" for the second scaling step with the upgrade path in mind.

partycoder 10y ago

While a point the article tries to make (fix your leaky funnel before acquiring users) is true, I disagree with the article. If your application is converting well, scalability problems are not acceptable. I have seen applications that convert very well, but were limited by scalability problems. That meant that the business had to hold on on marketing and user acquisition, missed their financial targets, and that cascaded into breaching contracts. The phrase that nobody wants to hear in that situation is "who cares about scalability". Now, if you did not have a lot of problems scaling in your particular case, that just means it was not an obstacle for you. e.g: you had good intuition around performance/scalability, or the problem was coincidentally a good fit for your technological choices. Unfortunately not everyone has a good intuition about scalability, not everyone is risk averse and not everyone is good at picking a good technology for their use case. So I disagree with this article in the sense that it is not in the best interest of a random reader to not care about scalability.

tianlins 10y ago

It really depends the type of growth. Organic growth is driven by the quality of product therefore scaling issue comes later. But most venture-backed startup services such as O2O need quickly dominate market by throwing cash to get users so scaling is an issue from day one.

seajones 10y ago

I do agree, but being at the "we need to scale up asap" stage atm makes it harder to. There's a balance to be struck. Maybe about approach of "who cares" with the POC, MVP etc stages, then keep it more and more in mind at each stage after that would be best.

etnos 10y ago

Mhee, We built a rest API with Laravel, after a heated conversation with a junior dev about PHP-Laravel being slow. +20k petitions average a day no problem at all. Digital ocean bill is like 20dlls/month ; ) Kids these days need to stop watching "The Social Network", chill out Mr. next Zuckerberg, you ain't scaling anything but a headache

iveqy 10y ago

Having working on an app that we just throwed more hardware at, to the point where the azure subscription cost could be lowered by my whole anually salary, by 3 months work of optimization. I believe performance does matter. We where a 4 person team and could have added a fifth if we had a cheaper design.

nomercy400 10y ago

Once worked at a startup where we expected 'some activity' in our webshop at product launch, and didn't think about scaling for that. Well, some activity turned out to be 450mbit/s for five hours, which our unscalable application/webshop didn't handle very well. It became overloaded in the first minute, and took us more than an hour to get remote access again. It's one of those things we did better for our next big event (major sharding, basically replicated the application 32 times of the largest VM instance we could get. It was needed and it survived).

mannykannot 10y ago

There is a similar argument with regard to making code reusable. I have seen inordinately complex code come from a desire to make it reusable, even if the prospects for it being reused were slim to nonexistent.

shanecleveland 10y ago

Came across a service last week with a free trial and paid plan. How to upgrade? Contact by email! Why spend time and resources on a payment process if you don't have any paying customers yet? Obviously not right for everyone, and I'm not saying it doesn't have its own challenges, but the core product deserves the most attention early on.

didibus 10y ago

The reason scale isn't so important today is because most DBs can actually scale vertically to really high numbers. The tipping point is high enough that if you have this problem, you probably can also afford to fix it. What matters though is performance and availability. No matter what scale you work at, you can't be slow, that will drive people away. You also can't be unavailable. This means that you might have to handle traffic spikes. Depending on your offering, you probably also want to be secure and reliable. Losing customer data or leaking it will drive customers away too. So, I'd mostly agree, in 2016, scale isn't a big problem. Better to focus on functionality, performance, security, reliability and availability. These things will impact all your customers, even when you only have one. They'll also be much harder to fix. Where scale matters is at big companies. When you already have a lot of customers, you’re first version of any new feature or product must already be at scale. Amazon couldn't have launched a non scalable prime now, or echo. Google can't launch a non scalable chat service, etc.

hamburglar 10y ago
> Where scale matters is at big companies. When you already have a lot of customers, you’re first version of any new feature or product must already be at scale. Amazon couldn't have launched a non scalable prime now, or echo. Google can't launch a non scalable chat service, etc. This is the point I was going to make. When you are designing the very platforms on which all these "does it scale? who cares" startups are going to be built, you do not have the luxury of having that attitude. Definitely don't take that mindset into an interview at google or amazon. :) I do agree with the OP's sentiment for startup projects in general. I have had the experience of worrying about scale too early and over-engineering a system which never had more than a few dozen users, and the opposite experience of tossing something together that wouldn't scale and then going through the hairy scrambling at every order of magnitude of scale through tens of millions of users. The latter was definitely a better strategy.

therealdrag0 10y ago

I started working at a company 6 months ago, and in that time I've fixed a half dozen performance problems. Some of them didn't matter THAT much, but some of them did. My team manages an authentication -service. Some of the calls were averaging hundreds of ms which added terrible overhead to other services' calls. In other places had CICD builds running tests twice, logs logging twice, inefficient algorithms, single HTTP transactions making dozens of DB calls. You name it. Some of these were simple things that if you stopped to smell the roses you could notice and fix. Let's all be boy scouts and make code better while we're in it :)

Vkkan2016 10y ago

IMHO until you hit scalability issue don't try to spend your time and money there yet