12 ms·

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic product over a weekend and launched. Timeline after that whenever the web server exhausted all the JVM threads trying to serve requests: 1. 1 month - 20k users - learnt about indexes and created indexes. 2. 3 months - 500k users - Realized MyISAM is a bad fit for mutable tables. Converted the tables to InnoDB. Increased number of JVM threads to tomcat 3. 9 months - 5M users - Realized that the default MySQL config is for a desktop and allocates just 64MB RAM to the database. Setup the mysql configs. 2 application servers now. 4. 18 months - 15M users - Tuned MySQL even more. Optimized JDBC connector to cache MySQL prepared statements. 5. 36 months - 45M users - Split database by having different tables on different machines. I had no idea or previous experience about any of these issues. However I always had enough notice to fix issues. Worked really hard, learnt along the way and was always able to find a way to scale the service. I know of absolutely no service which failed because it couldn't scale. First focus on building what people love. If people love your product, they will put up with the growing pains (e.g. Twitter used to be down a lot!). Because of my previous experience, I can now build and launch a highly scalable service at launch. However the reason I do this is that it is faster for me to do it - not because I am building it for scale. Launch as soon as you can. Iterate as fast as you can. Time is the only currency you have which can't be earned and only spent. Spend it wisely. Edited: formatting

qume 10y ago
I built a service (~10mm users at peak) which was designed from the ground up to scale, around 2002. When the numbers did grow we just sat back and watched it pretty much. Even given this, I completely agree with you. That's why I now develop in ruby. I'll take developer productivity over performance any day. It's the old saying - 'nice problem to have'.
- logicallee 10y ago
  >I built a service (~10mm users at peak) which was designed from the ground up to scale, around 2002. When the numbers did grow we just sat back and watched it pretty much. >Even given this, I completely agree with you. That's why I now develop in ruby. Ouch that is a pretty big burn.
- hasenj 10y ago
  > That's why I now develop in ruby. I'll take developer productivity over performance any day. In my experience, dynamically typed languages don't do well for developer productivity.
  - petters 10y ago
    They do well for single-person code bases, but velocity scales badly with the number of developers in my experience.
  - abritinthebay 10y ago
    And in my experience the exact opposite is true. That's why it's called an anecdote; not data.
    - luord 10y ago
      Exactly, I never understand the static vs dynamic flamewars. Most of the issues usually presented stem from poor developers or poor development processes/environments/tools. Poor code written in JavaScript won't become pretty just by translating to Java, and vice versa. That I've seen anyway.
      - sidlls 10y ago
        Many of the "poor development processes/environments/tools" issues are related to choosing dynamic languages in a "ship first, design later (maybe)" paradigm, though.
        10y ago
doctorcroc 10y ago
Awesome story and takeaways. What happened to your product and what did you learn you could not keep up with or grow into?
- timewarrior 10y ago
  I built this product as a side project within another startup I was working in. We had lot of money, had been around for 3 years and did not have a product. Because of this I had multiple levels of bosses above me in the company. They all got really interested in starting to manage me and the product once we hit 5M users. They had a different vision, attachment and passion about the product. Around 2008, we were spending a lot of money on text messaging. Most people where using the product to send messages for cheap to their friends and family (essentially like WhatsApp vs Twitter). I wanted to slowly phase out text messaging in favor of Data (essentially become WhatsApp!). CEO didn't believe that Mobile Data will get adoption in India for a long time. He instead wanted to focus on monetizing - by sending ads on text messages, making people pay for premium content etc. We tried 8-10 things - nothing worked (this can be a really big post in itself). Soon I quit and moved to US. It was like a bad breakup! My biggest learnings were: 1. If you do not have ownership like a founder, do not that take responsibility like a cofounder. 2. Personally for me: do not work where I don't have veto rights. But bills need to be paid and family has to be supported. Took me 5 years of really hard work to get there!
  - ccmonnett 10y ago
    Thanks for sharing your experience. People are getting caught up in Ruby v Java v whatever and not seeing the forest for the trees - you have a valuable perspective and an effective way of communicating it. Appreciate it! May I ask you about your first 'biggest learning': how do you eschew cofounder responsibilities in an early company? Someone who genuinely cares and/or is really interested in the core business (pretty common among first hires of startups) may have a hard time turning work away or willfully not participating in major discussions/planning when they believe their contributions could be valuable. In other words, situations when you know you can contribute but the company will get the 'better end of the deal' as you take on more responsibility without taking on more compensation. It sounds difficult to ride that out - were you able to do that successfully or are you saying "This happened to me, don't let it happen to you?"
CodeWriter23 10y ago
At step 3, how many messages per day were you pumping with 64MB allocated to MySQL?
- timewarrior 10y ago
  We were using MySQL like a persistent queue. As soon as a message was spawned, we would send it to downstream SMS gateways and delete them. So essentially that volume didn't use a lot of memory. At stage 3 we had 50% daily active users who would receive on an average of 8 messages per day. So at step 3, we would send 20M messages per day.
hliyan 10y ago
This is probably because implicitly made available decisions from the get go. For example, if you had chosen MongoDB instead of MySQL, the story would be different. Ditto if say, Meteor instead of Java.
- hliyan 10y ago
  Available --> scalable (typo)
- timewarrior 10y ago
  I am guessing you point is that if I had chosen Meteor and MongoDB - we wouldn't have been able to scale. My initial DB schema was pretty bad. Had to do quite a few DB migrations - which took weeks of work to execute. I have used MongoDB extensively since, and I am confident that it would have helped us scale to 5M users comfortably. I might have migrated to something else at that point. There are very few products who reach even 5M users. Which is why developers should focus on launching fast. Another thing is that best and most successful product have a really simple core product (remember Facebook, Twitter, Instagram etc when they had 5M users). It is not that much work to migrate and rewrite. When products are not getting traction - is when they start getting overcomplicated.
- the_arun 10y ago
  Isn't Meteor a framework? I'm assuming you meant Javascript instead of Meteor
  - hliyan 10y ago
    Yes, I should have said JavaEE vs JavaScript
Alex3917 10y ago
> I know of absolutely no service which failed because it couldn't scale. If by scaling you mean increasing the number of page views that a given version of a web app can serve, then this is mostly true. But things like conceptual simplicity, unit economics, codebase readability, strategy, etc. are also dimensions of scaleability. E.g. the only reason that startups can even exist at all is because large companies have a lower marginal output per employee.
- pbreit 10y ago
  "scale" here, as with all conversations on HN about "scale", refers to the technology required to service a large number of users or usage.
  - ajdlinux 10y ago
    I think you'll find there are plenty of conversations here where "scale" is used in reference to stuff like the business models of "gig economy" apps, or at what point your startup has enough scale to need an HR department, and so on and so forth.
iamgopal 10y ago
need much longer story if you could share. I's also Indian and Mine died at about 500k due to my focus on scalability.
- timewarrior 10y ago
  Would love to know more about your story. My biggest learning is Functional->Fast->Pretty from the product POV and Launch as soon as you can from an Engineer's POV. Some more details on my blog: https://anandprakash.net/2016/10/28/how-to-build-a-business-iterate-fast/ https://anandprakash.net/2016/10/28/how-to-build-a-business-... Most of this was written in sleepless nights when we had a baby. So do not expect a super artistic flair ;)
carbocation 10y ago
This is a delightful retelling; thanks for sharing. > I know of absolutely no service which failed because it couldn't scale. Genuine question: what about Friendster?
- timewarrior 10y ago
  I remember reading that Friendster founder was complaining that there were facing a lot of technical issues and investors were not letting him invest effort in fixing them. IMHO, if your investors need to be involved in seeking approval for handling scaling pains and on top of that if they reject it - there are deeper issues within the company (management issues, politics, technical incompetence, leadership issues). So to clarify my point - any team could still screw up technical execution. However there is no reason that given a good team and support from within the company, a product can't be scaled.
- pbreit 10y ago
  Friendster's problem was that instead of fixing its problems, it decided to re-write.
- jssmith 10y ago
  Friendster definitely failed because they couldn't pull off scale. I was CTO of the Tagged social network at the time and the issue loomed large during our 2005 fundraising. Investors had seen $50m go down the drain when Friendster got slow. Despite network effects, users moved over to products that were up and running (mostly MySpace but also hi5, etc.). I seem to recall that the Friendster team had more than one aborted rewrite before they got it right, and by then it was too late. Scaling social networks required a different bag of tricks than was popular at the time (e.g., database replication was well accepted as a best practice, but sharding worked much better). Talented people could still get it wrong. Still, social networking illustrates how failing to scale is rare, and should practically never be the concern early on. Almost every social network that gained traction struggled to scale at times, but the vast majority of them overcame their challenges. We poured sweat and tears and many sleepless nights into scaling the Tagged back-end from 10 machines to over 1,000, over time serving hundreds of millions of users. It was a tremendous amount of work, much more than building the initial product, and I don't think there's much we could have done early on to make later scaling easier. More on the story: http://highscalability.com/blog/2011/8/8/tagged-architecture-scaling-to-100-million-users-1000-server.html http://highscalability.com/blog/2011/8/8/tagged-architecture...
metalmanac 10y ago
Can you share a link to the site?
- swatkat 10y ago
  SMSGupShup, now known as GupShup. http://techcircle.vccircle.com/2013/02/06/mobile-group-messaging-service-sms-gupshup-rebrands-to-gupshup-launches-messenger-app/ http://techcircle.vccircle.com/2013/02/06/mobile-group-messa...
cyberferret 10y ago
> I know of absolutely no service which failed because it couldn't scale. I would say this is because of the simple fact of visibility and adoption. You've probably never heard of these services probably because they ground to a halt with a mere 1000 users, so they never got mainstream enough to be recognised as a viable service. It is a bit like how no one remembers the dozens of people who failed to achieve sustained powered flight before the Wright brothers. Doesn't mean there weren't any, and scalability, technical or planning issues killed those efforts before anyone knew of it. Some 'scalability' issues are inherent to your initial design, and not just the choice or configuration of your hardware/software platforms. For instance, what if you were building a contact database of some sort. At first, you may have things like 'Phone Number' and 'Email Address' as part of the 'Person' database. Then, as your service gets popular, you notice people asking for extra contact like Twitter handles, LinkedIn pages etc. So you start adding those to your Person table as extra columns. Eventually, you realise that you should have though more about this at the outset and have contact details stored in another data table altogether, linked to a 'Contact Type' table and related back to the Person table. This would have been mitigated at the start via better database design and catering for eventualities that you might never have foreseen. Migrating the original database to normalise it is a massive effort in its own right, and probably will take more time, cause more outage time, and cause more bugs in existing code than designing for that eventuality in the first place. Even if 99% of your users only ever enter Phone and Email contact details, the second option, designed for scalability, will still handle that without a sweat, and 'scaling' to meet additional demands later is merely a matter of adding new contact types in the 'Contact Type' data table so that they become an extra option for all your users. I am willing to bet that 9 out of 10 'weekend projects' have had to be thrown out completely and redeveloped from scratch when the number of users became significant. Of those rebuilds, I would be interested to see some research into how many users abandoned the said platform when (a) the original one started to grind to a halt or constantly fell over with errors and (b) the new platform came out with new features or a different UX that broke the 'look and feel' of the original.
- timewarrior 10y ago
  I mentioned something similar in a separate comment. My initial DB schema was pretty bad. We did at 2 schema rewrites and migrations from the launch to 5M users. Each time it took 2 weeks of sleep less nights. The machines today are really powerful. You can do a lot with 244 GB RAM machines backed by SSD. Someone who doesn't have the skill set to be able to scale once they get traction - it's likely they will not have the skills to design for scale at start. My recommendation to everyone would be pick a language and db you are most comfortable with and get started as soon as you can. You will fail on the product side a lot more times before you will fail on the technical side. And if you are failing on technical side, reach out to me. I will definitely be able to help you find a way out. I am not sure if there is any product guy in the world who an make a similar claim on the product front. However there are at least dozens of technical guys in the world who can make a claim like I did. So focus on launching the product as soon as possible. Work hard, reach out for help if needed. You will eventually get success.
deleted 10y ago
[deleted]
bbcbasic 10y ago
Obligatory "Mongo DB Is Web Scale" https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
gpm 10y ago
> I know of absolutely no service which failed because it couldn't scale. First focus on building what people love. If people love your product, they will put up with the growing pains (e.g. Twitter used to be down a lot!). They haven't failed (yet), but I think gitlab.com would be a lot bigger if they scaled faster. Lots of people have been rejected using it because it was too slow.
- LanceH 10y ago
  Would it be as big if they had stopped to consider all scaling issues at the beginning?
  - hooph00p 10y ago
    You could argue it would be bigger. Instead of working around Travis to connect to repositories, the repositories are there. That's the biggest sell for me, anyway.
  - sytse 10y ago
    In 2015 we only had the people to focus on one thing. We opted to focus on the downloadable product. This grew our revenue much faster than GitLab.com. We started to focus on .Com in 2016 but in hindsight not aggressive enough. Now it is one of our top three priorities.
- EduardoBautista 10y ago
  gitlab.com is not their core business. It's mostly a demo.
  - stingraycharles 10y ago
    That's not how people perceive it, though.
    - kiriakasis 10y ago
      How people perceive it is not relevant if it actually is a side business
holri 10y ago
> Time is the only currency you have which can't be earned and only spent. I would argue that you can earn lifetime with a healthy life style. I remember to have read a study that measured the average difference between healthy and unhealthy life styles with 14 years. If you spend 1 day a week more for a healthy lifestyle (exercise, self cooking, enough sleep, little stress, enough recreation time) for 40 years you have earned ~9 years life time. And I would argue that those 40 years are more enjoyable.
- timewarrior 10y ago
  You make a great point. Health should be the topmost priority and I need to get better at it. However to my original point - you can lose fitness (within reasonable limits) and then gain it back and not notice a difference later on. However time lost is lost forever.
  - holri 10y ago
    Yes you can have a life style debt. But do not overload it or forget to pay it back.
apersona 10y ago
> I know of absolutely no service which failed because it couldn't scale. Bullshit. What about the site (voat something?) that tried to be an alternative to reddit when there was some scandal there but couldn't keep users because they kept crashing?
- godzillabrennus 10y ago
  It's still going and they post about the scaling issues being due to lack of funding available that would allow them to continue their business model. http://voat.co http://voat.co
  - Dylan16807 10y ago
    But is that a true scaling problem like "2x servers can't handle 2x users" or is it a business model problem like "every user means losing money and we can't afford more even if the cost was O(n^0.9)"? I can't find the posts you're talking about.
- henriquemaia 10y ago
  I don't think that's a fair analysis of what happened. Though it seems obvious that they have lost some potencial users by being down so many times, ultimately what has obstructed Voat’s success is its core community: the fringe of the fringe of reddit.
- Trundle 10y ago
  Bullshit. You can go on voat right now just fine and there are people using it just fine. It hasn't failed, it's just not a big deal because it's an almost exact replica of a site that already exists and has traction. The difference being one allows you to hate on fat people all the time and the other only lets you hate on fat people most of the time. It's just too niche of a USP for most people.
narrator 10y ago
>I know of absolutely no service which failed because it couldn't scale. These days there would be hardly any way to fail to scale given a large budget. The technology to do so is all so good. However, back in the day it wasn't always so. I'd say Friendster was an example of a company that failed due to scaling issues.
- graphememes 10y ago
  Depends on what you're doing, easy to fail when you do analytics or something like datadog.
- frik 10y ago
  Some things simply don't scale. You can't do some things in real-time. You have to do it in batch-jobs or avoid them. If you know about the O-notation, you will probably know certain algos don't scale. Friendster insisted to calculate the friends-of-friends-of-friends relationship in real-time and was not able to scale it. MySpace and Facebook did the same calc in the early days too, but both scrapped the feature as they were not able to scale it. Other examples are big data (e.g. log analysis), if you choose the wrong software stack (database) or wrong indexes you won't be able to scale it (especially if it's real time. You will bleed money in no time for cloud servers with TBs of RAM, when you could do it with a couple of servers with a normal sharded SQL database and one hand tuned index.
- user5994461 10y ago
  > The technology to do so is all so good. Mistake #1: Took the wrong technologies. For every technology that can scale, there is a bunch of other that will give a lot of troubles.
iamd3vil 10y ago
Since I am an Indian I know of a service which is hated by most people - IRCTC because of it's scaling issues. It's a lot better now but I remember those horrible days you have to do a tatkal ticket. I also know of some government services (my state has paperless administration so everything is digital) which are used in day to day basis (but only hundreds of users) by government employees goes down for almost 2 hours per day everyday which stops work for everyone (both the employees and the people came for that service). These services are developed by companies like TCS and Infosys. Only if they thought of scaling before.
- tajen 10y ago
  In case of government, it's often another issue. While the social network of the parent was probably a lean stack, the government one was probably over-engineered. Some webapps are front-ends that communicate with a back-end in XML, with massive flaws like serialization and lack of transactions, and the backend syncs with a business solution backed by dBase, all of that for 11 web pages (I know, I've been on govt apps before – France). Scaling was a major requirement from the beginning, but those engineers just don't know what they're doing and follow the XML frontend/ESB bus patterns, assuming that's what gives performance. And committee design (12 to 24 ppl) does the rest of the bloat. 11 screens, $2m, sluggish result. Had they started with only $300 like the parent comment, their webapp would have often performed much better. Big projects are a difficult curse, our most difficult question in IT engineering.
- acdha 10y ago
  > These services are developed by companies like TCS and Infosys This is the real problem: lack of in-house expertise and ongoing incentives to maintain performance, while contractors are usually paid for features and, if they also do hosting, even have a financial disincentive for efficiency. I would bet it's at least as likely that had someone thought of scaling it would have added a significant cost and delay to the actual project and the first major scalability bottleneck would still have been something unanticipated.
jv22222 10y ago
> I know of absolutely no service which failed because it couldn't scale. Well, Friendster failed for exactly that reason. But, those were the early days. p.s Completely agree with the article.
yummyfajitas 10y ago
I can name one: Darcs. It was my favorite certain control system, apart from the lack of scalability.
- stephenr 10y ago
  Darcs is a distributed version control system. It is inherently not a hosted solution that "needs to scale" the way the article is talking about.
  - manveru 10y ago
    I'd definitely say that Darcs failed to catch on because it didn't scale with the size of the repo, the larger your history was the more likely you were to run into performance issues, and there was little you could do about it because the underlying theory is flawed. The scaling problem with Darcs wasn't about hosting, but in the implementation of their theory of patches and how it handled conflicts. See http://darcs.net/FAQ/ConflictsDarcs1#problems-with-conflicts http://darcs.net/FAQ/ConflictsDarcs1#problems-with-conflicts for an overview of the issues in Darcs 1 and 2. This seems to be also one of the driving forces behind the development of Pijul, which is also patch-based instead of snapshot-based, which makes it easier to understand and use, but all implementations so far had major performance issues once repositories grow. For more on that, see https://pijul.org/faq.html https://pijul.org/faq.html I was one of the early users of Darcs 1, back before Git existed. I wanted to use version control, but the alternatives were pretty hard to understand and use. While Darcs was really nice to use (, and fast on small repos, after a few years I had to convert everything to git because exponential times on most operations was just not sustainable, and fixing that required constant vigilance and altering history, not very friendly for new contributors. Even GHC moved from Darcs to git in 2011 because of this: https://mail.haskell.org/pipermail/glasgow-haskell-users/2011-January/019752.html https://mail.haskell.org/pipermail/glasgow-haskell-users/201...
joe5644774 10y ago
What application did you built ?
partycoder 10y ago
I think in your particular case, you had a good intuition about performance/scalability and this is why it was not hard for you to scale. However this is not the case for everyone and I have seen many counterexamples in my career.
robk 10y ago
Arguably Friendster failed due to its interminable scaling issues
baby 10y ago
How much money did Twitter waste in its migration to Java? I think it's an important factor to consider.
- user5994461 10y ago
  They have thousands of people sitting around doing nothing. They don't really optimize for expenses.
edmccard 10y ago
>I know of absolutely no service which failed because it couldn't scale So what are some examples of services which failed because they tried too hard to scale too early? For example, when your network could handle 20k users, was there another network that could have already handled 500k, but they failed because they started a month after yours?
jaimex2 10y ago
You started with Java... a language designed to handle credit card transactions world wide. Your story would be very different if you started with say Rails.
- timewarrior 10y ago
  If you build stateless horizontally scalable service (and you should) - you are almost always going to be blocked on DB. On the app server front you just keep adding more machines behind load balancer. As I look back, if we had used RoR or Python we might have moved faster with no negative impact on scale. We would have spent more money on extra servers though!
user5994461 10y ago
There is scaling and there is scaling. A messaging service is the most trivial service one can make, it's a well known easy problem that's been solved for decades with current technologies. There is no challenge in that. For comparison, WhatsApp had people with experience and they could handle 1 billion users with 50 people. The fact that you can handle 10M users with a single untuned MySQL database is not a demonstration that scaling is overrated. It's an expression than you are running a trivial service that doesn't do much. Almost any problems will be more challenging than that. There are endless companies that have 1/100th the customer base and yet require 100 times the data volume and engineering.
stephenhuey 10y ago
I love this example. Thanks for sharing! It's very encouraging to remind me to just make something and get it out there. One prominent counterexample that comes to mind is Friendster which was a big social network before MySpace and Facebook. They had terrible performance issues but to be fair to your point, their decline was more mismanagement than a true inability to scale. https://www.inc.com/magazine/20070601/features-how-to-kill-a-great-idea.html https://www.inc.com/magazine/20070601/features-how-to-kill-a... https://en.wikipedia.org/wiki/Friendster https://en.wikipedia.org/wiki/Friendster
timewarrior 10y ago
There have been a bunch of discussions about language choices and impact on productivity. Wanted to touch upon this here based on my experience. TLDR: Depends a lot on individual situation. Pick the language you are fastest and most comfortable with. Background: SMS Gupshup - 6 years - Java and some C++ - Built distributed filesystems. Crawled 1B pages with early 2000 hardware. Built map-reduce framework before Hadoop came. Built a search engine on top of it. Built an infra which would send 1B+ text messages with prioritization and monitoring. Social network with 50M+ users. LinkedIn main stack - ~ 1 year - Java LinkedIn mobile stack - ~ 1 year - Javascript - Was one of the early engineers. Founder Startup - 1.5 years - Clojure Dropbox - ~ 1 year - Python Head Technology Tech incubator - 1.5 years - Node.js As you can see my experience has been all over the spectrum. While primarily using one language, I would keep experimenting with other languages like Scala, Go. Following are my learnings and the reasonings: I love Clojure. Love it so much that I feel like quitting everything go to a mountain cabin and just code Clojure. However I would most likely not use Clojure for my startup. The learning curve is really-really steep. It would make it very difficult to me to build a team. It's really fun if you understand it. But first few weeks for a new person might be a nightmare and very emasculating. So even though I really love it, my current strategy is to bring Clojure patterns to other languages. User facing logic I would not use a statically typed Object Oriented language for any user facing logic. User facing logic and behavior tends to be very fluid and I feel that it gets strangled by the constraints of statically typed languages. One you start building things with Class, Inheritance and Polymorphism etc - soon these patterns start driving business logic rather than the other way around. Because of this reason you will see most arcane and slow moving popular sites today are Java based - (LinkedIn, Yahoo, Amazon, Ebay) vs (Facebook, Twitter, Instagram, Pinterest). For this purpose I like using Dynamically typed languages building code in as functional way as possible. Pick Node.js, Python, Ruby - whatever you are comfortable with. In some cases people recommend Statically types languages for catching errors because of type safety etc. However I would solve that problem completely with test suite rather than solving it part of the way using language features while being forced to pick a statically types language. Another big benefit of dynamically types languages is ability to just create a dictionary and start using it. Most CRUD apps work with JSON request and response and I would rather just dynamically pass objects, rather than build a class for the request and response objects for each and every route. This was a nightmare to create and maintain in Scala and Java. Fast developing Infrastructure E.g. Hadoop, Hive, Distributed FS or Queue. 5 years back I would have built the in Java. However I would use Go for this today. However you couldn't go wrong either way. Pick what you are most comfortable with. There is decent amount of talent available. These languages are easy to learn. And you get close to C level performance if you built it right. Development is fast paced. Mission critical performance - File System or Database Most likely C, because you want to squeeze out last of the performance. However be ready for really slow development cycles. And most smart kids out of college wouldn't know C. So you will need to set aside some time till they are really productive.
mgkimsal 10y ago
> However I always had enough notice to fix issues. Thank you. This is the one major point I try to get across to people when 'scaling' comes up. "Oh, that won't be scalable" or "yeah, but when we have 2 million users, XYZ won't work". I've been on projects where weeks and months (calendar time, part time efforts) were spent on things that were 'scalable' instead of just shipping something that worked earlier. I keep trying to tell folks on these projects (have been involved in a couple now) - man, we're not going to go from 20 users to 2 million users overnight. We'll notice the problems and can adapt. Trying to whet their appetite, I've even argued that a new 'flavor of the month' will be out before our real scaling needs hit, and we can then waste time chasing that fad, which will be even cooler than the current fad we're chasing (although... I try to be slightly more diplomatic than that).
sidlls 10y ago
You're omitting a lot of details. The number of users isn't as important, for example, as the number of users simultaneously using services and which services they were using concurrently. Also, no offense here, but the way you describe your experience makes it seem like you were very inexperienced in general. Your "not noticing" services that failed could mean, for example, that you weren't measuring properly and probably never learned to. In which case your anecdote isn't necessarily a useful one.
- timewarrior 10y ago
  It might be a good idea to read my post again. When I talk about 'service which failed' it was other products (e.g. Facebook, Twitter etc) - not components of my product. My post says 50M+ users and 1B+ message on a day. Almost 50% daily active users. And this was early 2000 hardware. Last year WhatsApp was doing 60 billion messages per day (60x of what we did). However our messages had a lot more logic, because we delivered on unreliable Telco texting pipes. Pipes had a set throughput. Some pipes delivered to just some geographies. They would randomly fail downstream so you had to do health monitoring and management. Some pipes cost per message and some cost based on throughput. So pick in real time for the lowest cost. Different messages had different priority. In summary - there was a lot more business logic per message sent. I was of course very inexperienced when I started. I was just two years out of college. However in the end the product was 50+ services running on 200+ servers. This experience helped me scale user communication infra at Dropbox to 100x scale within 3 months. Your message was flame-bait at best and didn't seem to contribute to the discussion. You were passing judgements based on incomplete information. You could have asked nicely and still gotten a response. Edit: fixed typo
  - sidlls 10y ago
    > It might be a good idea to read my post again. When I talk about 'service which failed' it was other products (e.g. Facebook, Twitter etc) - not components of my product. This wasn't clear at all from your comment. > Your message was flame-bait at best and didn't seem to contribute to the discussion. You were passing judgements based on incomplete information. You could have asked nicely and still gotten a response. My message was not flame bait; that's your interpretation. I'm sorry you took offense, but please do consider that your original comment does omit lots of important details, as you apparently (should) know from your experience at Dropbox. Also, per your quoted numbers WhatsApp was going 60x what your app was, not 1/60. Typo?
xiaoma 10y ago
> "I know of absolutely no service which failed because it couldn't scale." Zooomr did. They came out of nowhere in 2006 and were a real threat to Flickr. They had AJAX-powered editss, geotagging, various other unique features and they were starting to pull in some highly followed followed photographers. But, the site kept crashing as traffic grew and some scaling problems even lead to data loss. I wanted to see them win but they just couldn't keep up with traffic and eventually Yahoo cloned their features and they became irrelevant. https://techcrunch.com/tag/zooomr/ https://techcrunch.com/tag/zooomr/
- timewarrior 10y ago
  Main question here is that would they have done better if they had started with scale in mind. When you start you don't know how you product will look like when it has millions of users. You will likely do major product changes quite a few times. If someone cannot scale your product which has adoption, they likely don't have insights, problem definition, capabilities, skills and resources to do that when they launch. If I could edit my original post, I would put a rider related to capable team for service not dying.
nasalgoat 10y ago
> I know of absolutely no service which failed because it couldn't scale. I know HN skews young when no one remembers Friendster.
mrbtie 9y ago
> I know of absolutely no service which failed because it couldn't scale. You must not be looking around at all, right?

qume 10y ago

I built a service (~10mm users at peak) which was designed from the ground up to scale, around 2002. When the numbers did grow we just sat back and watched it pretty much. Even given this, I completely agree with you. That's why I now develop in ruby. I'll take developer productivity over performance any day. It's the old saying - 'nice problem to have'.

logicallee 10y ago
>I built a service (~10mm users at peak) which was designed from the ground up to scale, around 2002. When the numbers did grow we just sat back and watched it pretty much. >Even given this, I completely agree with you. That's why I now develop in ruby. Ouch that is a pretty big burn.
hasenj 10y ago
> That's why I now develop in ruby. I'll take developer productivity over performance any day. In my experience, dynamically typed languages don't do well for developer productivity.
- petters 10y ago
  They do well for single-person code bases, but velocity scales badly with the number of developers in my experience.
- abritinthebay 10y ago
  And in my experience the exact opposite is true. That's why it's called an anecdote; not data.
  - luord 10y ago
    Exactly, I never understand the static vs dynamic flamewars. Most of the issues usually presented stem from poor developers or poor development processes/environments/tools. Poor code written in JavaScript won't become pretty just by translating to Java, and vice versa. That I've seen anyway.
    - sidlls 10y ago
      Many of the "poor development processes/environments/tools" issues are related to choosing dynamic languages in a "ship first, design later (maybe)" paradigm, though.
      - 10y ago

doctorcroc 10y ago

Awesome story and takeaways. What happened to your product and what did you learn you could not keep up with or grow into?

timewarrior 10y ago
I built this product as a side project within another startup I was working in. We had lot of money, had been around for 3 years and did not have a product. Because of this I had multiple levels of bosses above me in the company. They all got really interested in starting to manage me and the product once we hit 5M users. They had a different vision, attachment and passion about the product. Around 2008, we were spending a lot of money on text messaging. Most people where using the product to send messages for cheap to their friends and family (essentially like WhatsApp vs Twitter). I wanted to slowly phase out text messaging in favor of Data (essentially become WhatsApp!). CEO didn't believe that Mobile Data will get adoption in India for a long time. He instead wanted to focus on monetizing - by sending ads on text messages, making people pay for premium content etc. We tried 8-10 things - nothing worked (this can be a really big post in itself). Soon I quit and moved to US. It was like a bad breakup! My biggest learnings were: 1. If you do not have ownership like a founder, do not that take responsibility like a cofounder. 2. Personally for me: do not work where I don't have veto rights. But bills need to be paid and family has to be supported. Took me 5 years of really hard work to get there!
- ccmonnett 10y ago
  Thanks for sharing your experience. People are getting caught up in Ruby v Java v whatever and not seeing the forest for the trees - you have a valuable perspective and an effective way of communicating it. Appreciate it! May I ask you about your first 'biggest learning': how do you eschew cofounder responsibilities in an early company? Someone who genuinely cares and/or is really interested in the core business (pretty common among first hires of startups) may have a hard time turning work away or willfully not participating in major discussions/planning when they believe their contributions could be valuable. In other words, situations when you know you can contribute but the company will get the 'better end of the deal' as you take on more responsibility without taking on more compensation. It sounds difficult to ride that out - were you able to do that successfully or are you saying "This happened to me, don't let it happen to you?"

CodeWriter23 10y ago

At step 3, how many messages per day were you pumping with 64MB allocated to MySQL?

timewarrior 10y ago
We were using MySQL like a persistent queue. As soon as a message was spawned, we would send it to downstream SMS gateways and delete them. So essentially that volume didn't use a lot of memory. At stage 3 we had 50% daily active users who would receive on an average of 8 messages per day. So at step 3, we would send 20M messages per day.

hliyan 10y ago

This is probably because implicitly made available decisions from the get go. For example, if you had chosen MongoDB instead of MySQL, the story would be different. Ditto if say, Meteor instead of Java.

hliyan 10y ago
Available --> scalable (typo)
timewarrior 10y ago
I am guessing you point is that if I had chosen Meteor and MongoDB - we wouldn't have been able to scale. My initial DB schema was pretty bad. Had to do quite a few DB migrations - which took weeks of work to execute. I have used MongoDB extensively since, and I am confident that it would have helped us scale to 5M users comfortably. I might have migrated to something else at that point. There are very few products who reach even 5M users. Which is why developers should focus on launching fast. Another thing is that best and most successful product have a really simple core product (remember Facebook, Twitter, Instagram etc when they had 5M users). It is not that much work to migrate and rewrite. When products are not getting traction - is when they start getting overcomplicated.
the_arun 10y ago
Isn't Meteor a framework? I'm assuming you meant Javascript instead of Meteor
- hliyan 10y ago
  Yes, I should have said JavaEE vs JavaScript

Alex3917 10y ago

> I know of absolutely no service which failed because it couldn't scale. If by scaling you mean increasing the number of page views that a given version of a web app can serve, then this is mostly true. But things like conceptual simplicity, unit economics, codebase readability, strategy, etc. are also dimensions of scaleability. E.g. the only reason that startups can even exist at all is because large companies have a lower marginal output per employee.

pbreit 10y ago
"scale" here, as with all conversations on HN about "scale", refers to the technology required to service a large number of users or usage.
- ajdlinux 10y ago
  I think you'll find there are plenty of conversations here where "scale" is used in reference to stuff like the business models of "gig economy" apps, or at what point your startup has enough scale to need an HR department, and so on and so forth.

iamgopal 10y ago

need much longer story if you could share. I's also Indian and Mine died at about 500k due to my focus on scalability.

timewarrior 10y ago
Would love to know more about your story. My biggest learning is Functional->Fast->Pretty from the product POV and Launch as soon as you can from an Engineer's POV. Some more details on my blog: https://anandprakash.net/2016/10/28/how-to-build-a-business-iterate-fast/ https://anandprakash.net/2016/10/28/how-to-build-a-business-... Most of this was written in sleepless nights when we had a baby. So do not expect a super artistic flair ;)

carbocation 10y ago

This is a delightful retelling; thanks for sharing. > I know of absolutely no service which failed because it couldn't scale. Genuine question: what about Friendster?

timewarrior 10y ago
I remember reading that Friendster founder was complaining that there were facing a lot of technical issues and investors were not letting him invest effort in fixing them. IMHO, if your investors need to be involved in seeking approval for handling scaling pains and on top of that if they reject it - there are deeper issues within the company (management issues, politics, technical incompetence, leadership issues). So to clarify my point - any team could still screw up technical execution. However there is no reason that given a good team and support from within the company, a product can't be scaled.
pbreit 10y ago
Friendster's problem was that instead of fixing its problems, it decided to re-write.
jssmith 10y ago
Friendster definitely failed because they couldn't pull off scale. I was CTO of the Tagged social network at the time and the issue loomed large during our 2005 fundraising. Investors had seen $50m go down the drain when Friendster got slow. Despite network effects, users moved over to products that were up and running (mostly MySpace but also hi5, etc.). I seem to recall that the Friendster team had more than one aborted rewrite before they got it right, and by then it was too late. Scaling social networks required a different bag of tricks than was popular at the time (e.g., database replication was well accepted as a best practice, but sharding worked much better). Talented people could still get it wrong. Still, social networking illustrates how failing to scale is rare, and should practically never be the concern early on. Almost every social network that gained traction struggled to scale at times, but the vast majority of them overcame their challenges. We poured sweat and tears and many sleepless nights into scaling the Tagged back-end from 10 machines to over 1,000, over time serving hundreds of millions of users. It was a tremendous amount of work, much more than building the initial product, and I don't think there's much we could have done early on to make later scaling easier. More on the story: http://highscalability.com/blog/2011/8/8/tagged-architecture-scaling-to-100-million-users-1000-server.html http://highscalability.com/blog/2011/8/8/tagged-architecture...

metalmanac 10y ago

Can you share a link to the site?

swatkat 10y ago
SMSGupShup, now known as GupShup. http://techcircle.vccircle.com/2013/02/06/mobile-group-messaging-service-sms-gupshup-rebrands-to-gupshup-launches-messenger-app/ http://techcircle.vccircle.com/2013/02/06/mobile-group-messa...

cyberferret 10y ago

> I know of absolutely no service which failed because it couldn't scale. I would say this is because of the simple fact of visibility and adoption. You've probably never heard of these services probably because they ground to a halt with a mere 1000 users, so they never got mainstream enough to be recognised as a viable service. It is a bit like how no one remembers the dozens of people who failed to achieve sustained powered flight before the Wright brothers. Doesn't mean there weren't any, and scalability, technical or planning issues killed those efforts before anyone knew of it. Some 'scalability' issues are inherent to your initial design, and not just the choice or configuration of your hardware/software platforms. For instance, what if you were building a contact database of some sort. At first, you may have things like 'Phone Number' and 'Email Address' as part of the 'Person' database. Then, as your service gets popular, you notice people asking for extra contact like Twitter handles, LinkedIn pages etc. So you start adding those to your Person table as extra columns. Eventually, you realise that you should have though more about this at the outset and have contact details stored in another data table altogether, linked to a 'Contact Type' table and related back to the Person table. This would have been mitigated at the start via better database design and catering for eventualities that you might never have foreseen. Migrating the original database to normalise it is a massive effort in its own right, and probably will take more time, cause more outage time, and cause more bugs in existing code than designing for that eventuality in the first place. Even if 99% of your users only ever enter Phone and Email contact details, the second option, designed for scalability, will still handle that without a sweat, and 'scaling' to meet additional demands later is merely a matter of adding new contact types in the 'Contact Type' data table so that they become an extra option for all your users. I am willing to bet that 9 out of 10 'weekend projects' have had to be thrown out completely and redeveloped from scratch when the number of users became significant. Of those rebuilds, I would be interested to see some research into how many users abandoned the said platform when (a) the original one started to grind to a halt or constantly fell over with errors and (b) the new platform came out with new features or a different UX that broke the 'look and feel' of the original.

timewarrior 10y ago
I mentioned something similar in a separate comment. My initial DB schema was pretty bad. We did at 2 schema rewrites and migrations from the launch to 5M users. Each time it took 2 weeks of sleep less nights. The machines today are really powerful. You can do a lot with 244 GB RAM machines backed by SSD. Someone who doesn't have the skill set to be able to scale once they get traction - it's likely they will not have the skills to design for scale at start. My recommendation to everyone would be pick a language and db you are most comfortable with and get started as soon as you can. You will fail on the product side a lot more times before you will fail on the technical side. And if you are failing on technical side, reach out to me. I will definitely be able to help you find a way out. I am not sure if there is any product guy in the world who an make a similar claim on the product front. However there are at least dozens of technical guys in the world who can make a claim like I did. So focus on launching the product as soon as possible. Work hard, reach out for help if needed. You will eventually get success.

deleted 10y ago

[deleted]

bbcbasic 10y ago

Obligatory "Mongo DB Is Web Scale" https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs

gpm 10y ago

> I know of absolutely no service which failed because it couldn't scale. First focus on building what people love. If people love your product, they will put up with the growing pains (e.g. Twitter used to be down a lot!). They haven't failed (yet), but I think gitlab.com would be a lot bigger if they scaled faster. Lots of people have been rejected using it because it was too slow.

LanceH 10y ago
Would it be as big if they had stopped to consider all scaling issues at the beginning?
- hooph00p 10y ago
  You could argue it would be bigger. Instead of working around Travis to connect to repositories, the repositories are there. That's the biggest sell for me, anyway.
- sytse 10y ago
  In 2015 we only had the people to focus on one thing. We opted to focus on the downloadable product. This grew our revenue much faster than GitLab.com. We started to focus on .Com in 2016 but in hindsight not aggressive enough. Now it is one of our top three priorities.
EduardoBautista 10y ago
gitlab.com is not their core business. It's mostly a demo.
- stingraycharles 10y ago
  That's not how people perceive it, though.
  - kiriakasis 10y ago
    How people perceive it is not relevant if it actually is a side business

holri 10y ago

> Time is the only currency you have which can't be earned and only spent. I would argue that you can earn lifetime with a healthy life style. I remember to have read a study that measured the average difference between healthy and unhealthy life styles with 14 years. If you spend 1 day a week more for a healthy lifestyle (exercise, self cooking, enough sleep, little stress, enough recreation time) for 40 years you have earned ~9 years life time. And I would argue that those 40 years are more enjoyable.

timewarrior 10y ago
You make a great point. Health should be the topmost priority and I need to get better at it. However to my original point - you can lose fitness (within reasonable limits) and then gain it back and not notice a difference later on. However time lost is lost forever.
- holri 10y ago
  Yes you can have a life style debt. But do not overload it or forget to pay it back.

apersona 10y ago

> I know of absolutely no service which failed because it couldn't scale. Bullshit. What about the site (voat something?) that tried to be an alternative to reddit when there was some scandal there but couldn't keep users because they kept crashing?

godzillabrennus 10y ago
It's still going and they post about the scaling issues being due to lack of funding available that would allow them to continue their business model. http://voat.co http://voat.co
- Dylan16807 10y ago
  But is that a true scaling problem like "2x servers can't handle 2x users" or is it a business model problem like "every user means losing money and we can't afford more even if the cost was O(n^0.9)"? I can't find the posts you're talking about.
henriquemaia 10y ago
I don't think that's a fair analysis of what happened. Though it seems obvious that they have lost some potencial users by being down so many times, ultimately what has obstructed Voat’s success is its core community: the fringe of the fringe of reddit.
Trundle 10y ago
Bullshit. You can go on voat right now just fine and there are people using it just fine. It hasn't failed, it's just not a big deal because it's an almost exact replica of a site that already exists and has traction. The difference being one allows you to hate on fat people all the time and the other only lets you hate on fat people most of the time. It's just too niche of a USP for most people.

narrator 10y ago

>I know of absolutely no service which failed because it couldn't scale. These days there would be hardly any way to fail to scale given a large budget. The technology to do so is all so good. However, back in the day it wasn't always so. I'd say Friendster was an example of a company that failed due to scaling issues.

graphememes 10y ago
Depends on what you're doing, easy to fail when you do analytics or something like datadog.
frik 10y ago
Some things simply don't scale. You can't do some things in real-time. You have to do it in batch-jobs or avoid them. If you know about the O-notation, you will probably know certain algos don't scale. Friendster insisted to calculate the friends-of-friends-of-friends relationship in real-time and was not able to scale it. MySpace and Facebook did the same calc in the early days too, but both scrapped the feature as they were not able to scale it. Other examples are big data (e.g. log analysis), if you choose the wrong software stack (database) or wrong indexes you won't be able to scale it (especially if it's real time. You will bleed money in no time for cloud servers with TBs of RAM, when you could do it with a couple of servers with a normal sharded SQL database and one hand tuned index.
user5994461 10y ago
> The technology to do so is all so good. Mistake #1: Took the wrong technologies. For every technology that can scale, there is a bunch of other that will give a lot of troubles.

iamd3vil 10y ago

Since I am an Indian I know of a service which is hated by most people - IRCTC because of it's scaling issues. It's a lot better now but I remember those horrible days you have to do a tatkal ticket. I also know of some government services (my state has paperless administration so everything is digital) which are used in day to day basis (but only hundreds of users) by government employees goes down for almost 2 hours per day everyday which stops work for everyone (both the employees and the people came for that service). These services are developed by companies like TCS and Infosys. Only if they thought of scaling before.

tajen 10y ago
In case of government, it's often another issue. While the social network of the parent was probably a lean stack, the government one was probably over-engineered. Some webapps are front-ends that communicate with a back-end in XML, with massive flaws like serialization and lack of transactions, and the backend syncs with a business solution backed by dBase, all of that for 11 web pages (I know, I've been on govt apps before – France). Scaling was a major requirement from the beginning, but those engineers just don't know what they're doing and follow the XML frontend/ESB bus patterns, assuming that's what gives performance. And committee design (12 to 24 ppl) does the rest of the bloat. 11 screens, $2m, sluggish result. Had they started with only $300 like the parent comment, their webapp would have often performed much better. Big projects are a difficult curse, our most difficult question in IT engineering.
acdha 10y ago
> These services are developed by companies like TCS and Infosys This is the real problem: lack of in-house expertise and ongoing incentives to maintain performance, while contractors are usually paid for features and, if they also do hosting, even have a financial disincentive for efficiency. I would bet it's at least as likely that had someone thought of scaling it would have added a significant cost and delay to the actual project and the first major scalability bottleneck would still have been something unanticipated.

jv22222 10y ago

> I know of absolutely no service which failed because it couldn't scale. Well, Friendster failed for exactly that reason. But, those were the early days. p.s Completely agree with the article.

yummyfajitas 10y ago

I can name one: Darcs. It was my favorite certain control system, apart from the lack of scalability.

stephenr 10y ago
Darcs is a distributed version control system. It is inherently not a hosted solution that "needs to scale" the way the article is talking about.
- manveru 10y ago
  I'd definitely say that Darcs failed to catch on because it didn't scale with the size of the repo, the larger your history was the more likely you were to run into performance issues, and there was little you could do about it because the underlying theory is flawed. The scaling problem with Darcs wasn't about hosting, but in the implementation of their theory of patches and how it handled conflicts. See http://darcs.net/FAQ/ConflictsDarcs1#problems-with-conflicts http://darcs.net/FAQ/ConflictsDarcs1#problems-with-conflicts for an overview of the issues in Darcs 1 and 2. This seems to be also one of the driving forces behind the development of Pijul, which is also patch-based instead of snapshot-based, which makes it easier to understand and use, but all implementations so far had major performance issues once repositories grow. For more on that, see https://pijul.org/faq.html https://pijul.org/faq.html I was one of the early users of Darcs 1, back before Git existed. I wanted to use version control, but the alternatives were pretty hard to understand and use. While Darcs was really nice to use (, and fast on small repos, after a few years I had to convert everything to git because exponential times on most operations was just not sustainable, and fixing that required constant vigilance and altering history, not very friendly for new contributors. Even GHC moved from Darcs to git in 2011 because of this: https://mail.haskell.org/pipermail/glasgow-haskell-users/2011-January/019752.html https://mail.haskell.org/pipermail/glasgow-haskell-users/201...

joe5644774 10y ago

What application did you built ?

partycoder 10y ago

I think in your particular case, you had a good intuition about performance/scalability and this is why it was not hard for you to scale. However this is not the case for everyone and I have seen many counterexamples in my career.

robk 10y ago

Arguably Friendster failed due to its interminable scaling issues

baby 10y ago

How much money did Twitter waste in its migration to Java? I think it's an important factor to consider.

user5994461 10y ago
They have thousands of people sitting around doing nothing. They don't really optimize for expenses.

edmccard 10y ago

>I know of absolutely no service which failed because it couldn't scale So what are some examples of services which failed because they tried too hard to scale too early? For example, when your network could handle 20k users, was there another network that could have already handled 500k, but they failed because they started a month after yours?

jaimex2 10y ago

You started with Java... a language designed to handle credit card transactions world wide. Your story would be very different if you started with say Rails.

timewarrior 10y ago
If you build stateless horizontally scalable service (and you should) - you are almost always going to be blocked on DB. On the app server front you just keep adding more machines behind load balancer. As I look back, if we had used RoR or Python we might have moved faster with no negative impact on scale. We would have spent more money on extra servers though!

user5994461 10y ago

There is scaling and there is scaling. A messaging service is the most trivial service one can make, it's a well known easy problem that's been solved for decades with current technologies. There is no challenge in that. For comparison, WhatsApp had people with experience and they could handle 1 billion users with 50 people. The fact that you can handle 10M users with a single untuned MySQL database is not a demonstration that scaling is overrated. It's an expression than you are running a trivial service that doesn't do much. Almost any problems will be more challenging than that. There are endless companies that have 1/100th the customer base and yet require 100 times the data volume and engineering.

stephenhuey 10y ago

I love this example. Thanks for sharing! It's very encouraging to remind me to just make something and get it out there. One prominent counterexample that comes to mind is Friendster which was a big social network before MySpace and Facebook. They had terrible performance issues but to be fair to your point, their decline was more mismanagement than a true inability to scale. https://www.inc.com/magazine/20070601/features-how-to-kill-a-great-idea.html https://www.inc.com/magazine/20070601/features-how-to-kill-a... https://en.wikipedia.org/wiki/Friendster https://en.wikipedia.org/wiki/Friendster

timewarrior 10y ago

There have been a bunch of discussions about language choices and impact on productivity. Wanted to touch upon this here based on my experience. TLDR: Depends a lot on individual situation. Pick the language you are fastest and most comfortable with. Background: SMS Gupshup - 6 years - Java and some C++ - Built distributed filesystems. Crawled 1B pages with early 2000 hardware. Built map-reduce framework before Hadoop came. Built a search engine on top of it. Built an infra which would send 1B+ text messages with prioritization and monitoring. Social network with 50M+ users. LinkedIn main stack - ~ 1 year - Java LinkedIn mobile stack - ~ 1 year - Javascript - Was one of the early engineers. Founder Startup - 1.5 years - Clojure Dropbox - ~ 1 year - Python Head Technology Tech incubator - 1.5 years - Node.js As you can see my experience has been all over the spectrum. While primarily using one language, I would keep experimenting with other languages like Scala, Go. Following are my learnings and the reasonings: I love Clojure. Love it so much that I feel like quitting everything go to a mountain cabin and just code Clojure. However I would most likely not use Clojure for my startup. The learning curve is really-really steep. It would make it very difficult to me to build a team. It's really fun if you understand it. But first few weeks for a new person might be a nightmare and very emasculating. So even though I really love it, my current strategy is to bring Clojure patterns to other languages. User facing logic I would not use a statically typed Object Oriented language for any user facing logic. User facing logic and behavior tends to be very fluid and I feel that it gets strangled by the constraints of statically typed languages. One you start building things with Class, Inheritance and Polymorphism etc - soon these patterns start driving business logic rather than the other way around. Because of this reason you will see most arcane and slow moving popular sites today are Java based - (LinkedIn, Yahoo, Amazon, Ebay) vs (Facebook, Twitter, Instagram, Pinterest). For this purpose I like using Dynamically typed languages building code in as functional way as possible. Pick Node.js, Python, Ruby - whatever you are comfortable with. In some cases people recommend Statically types languages for catching errors because of type safety etc. However I would solve that problem completely with test suite rather than solving it part of the way using language features while being forced to pick a statically types language. Another big benefit of dynamically types languages is ability to just create a dictionary and start using it. Most CRUD apps work with JSON request and response and I would rather just dynamically pass objects, rather than build a class for the request and response objects for each and every route. This was a nightmare to create and maintain in Scala and Java. Fast developing Infrastructure E.g. Hadoop, Hive, Distributed FS or Queue. 5 years back I would have built the in Java. However I would use Go for this today. However you couldn't go wrong either way. Pick what you are most comfortable with. There is decent amount of talent available. These languages are easy to learn. And you get close to C level performance if you built it right. Development is fast paced. Mission critical performance - File System or Database Most likely C, because you want to squeeze out last of the performance. However be ready for really slow development cycles. And most smart kids out of college wouldn't know C. So you will need to set aside some time till they are really productive.

mgkimsal 10y ago

> However I always had enough notice to fix issues. Thank you. This is the one major point I try to get across to people when 'scaling' comes up. "Oh, that won't be scalable" or "yeah, but when we have 2 million users, XYZ won't work". I've been on projects where weeks and months (calendar time, part time efforts) were spent on things that were 'scalable' instead of just shipping something that worked earlier. I keep trying to tell folks on these projects (have been involved in a couple now) - man, we're not going to go from 20 users to 2 million users overnight. We'll notice the problems and can adapt. Trying to whet their appetite, I've even argued that a new 'flavor of the month' will be out before our real scaling needs hit, and we can then waste time chasing that fad, which will be even cooler than the current fad we're chasing (although... I try to be slightly more diplomatic than that).

sidlls 10y ago

You're omitting a lot of details. The number of users isn't as important, for example, as the number of users simultaneously using services and which services they were using concurrently. Also, no offense here, but the way you describe your experience makes it seem like you were very inexperienced in general. Your "not noticing" services that failed could mean, for example, that you weren't measuring properly and probably never learned to. In which case your anecdote isn't necessarily a useful one.

timewarrior 10y ago
It might be a good idea to read my post again. When I talk about 'service which failed' it was other products (e.g. Facebook, Twitter etc) - not components of my product. My post says 50M+ users and 1B+ message on a day. Almost 50% daily active users. And this was early 2000 hardware. Last year WhatsApp was doing 60 billion messages per day (60x of what we did). However our messages had a lot more logic, because we delivered on unreliable Telco texting pipes. Pipes had a set throughput. Some pipes delivered to just some geographies. They would randomly fail downstream so you had to do health monitoring and management. Some pipes cost per message and some cost based on throughput. So pick in real time for the lowest cost. Different messages had different priority. In summary - there was a lot more business logic per message sent. I was of course very inexperienced when I started. I was just two years out of college. However in the end the product was 50+ services running on 200+ servers. This experience helped me scale user communication infra at Dropbox to 100x scale within 3 months. Your message was flame-bait at best and didn't seem to contribute to the discussion. You were passing judgements based on incomplete information. You could have asked nicely and still gotten a response. Edit: fixed typo
- sidlls 10y ago
  > It might be a good idea to read my post again. When I talk about 'service which failed' it was other products (e.g. Facebook, Twitter etc) - not components of my product. This wasn't clear at all from your comment. > Your message was flame-bait at best and didn't seem to contribute to the discussion. You were passing judgements based on incomplete information. You could have asked nicely and still gotten a response. My message was not flame bait; that's your interpretation. I'm sorry you took offense, but please do consider that your original comment does omit lots of important details, as you apparently (should) know from your experience at Dropbox. Also, per your quoted numbers WhatsApp was going 60x what your app was, not 1/60. Typo?

xiaoma 10y ago

> "I know of absolutely no service which failed because it couldn't scale." Zooomr did. They came out of nowhere in 2006 and were a real threat to Flickr. They had AJAX-powered editss, geotagging, various other unique features and they were starting to pull in some highly followed followed photographers. But, the site kept crashing as traffic grew and some scaling problems even lead to data loss. I wanted to see them win but they just couldn't keep up with traffic and eventually Yahoo cloned their features and they became irrelevant. https://techcrunch.com/tag/zooomr/ https://techcrunch.com/tag/zooomr/

timewarrior 10y ago
Main question here is that would they have done better if they had started with scale in mind. When you start you don't know how you product will look like when it has millions of users. You will likely do major product changes quite a few times. If someone cannot scale your product which has adoption, they likely don't have insights, problem definition, capabilities, skills and resources to do that when they launch. If I could edit my original post, I would put a rider related to capable team for service not dying.

nasalgoat 10y ago

> I know of absolutely no service which failed because it couldn't scale. I know HN skews young when no one remembers Friendster.

mrbtie 9y ago

> I know of absolutely no service which failed because it couldn't scale. You must not be looking around at all, right?