18 ms·

Optimizing Your Web App 100x Is Like Adding 99 Servers

• hrishi 6y ago
  I feel like AWS and the massive amounts of credit they give out just for asking has only made this worse. Don't get me wrong - I have been a beneficiary of their startup programs, and it's amazing for companies not to worry about compute for a year or two (you're never really using 100K anyway). That said, I have seen multiple companies get into the hole of not worrying about optimization (at all - who needs to when you have a quarter of a million in credits), and locking themselves into a massively complex microarchitecture by the time that first real bill rolls around. AWS knows what they're doing, do your best to work them rather than the other way around.
• t-writescode 6y ago
  One rule that should be added here, but it might be a bit late: * choose a language that supports async operations so that you aren't waiting for external webrequests or database calls to get back to local code. A synchronous language can turn your big, beefy machine into a single concurrent transaction language and _that_, beyond any other pain, can really, really, really reduce your total throughput. Pick languages that can do other things while they're waiting for IO.
  • knuthsat 6y ago
    I was quite surprised when I realized that Python failed a lot at being async.
    • t-writescode 6y ago
      Python 3.7 and friends do have async support; and, Django has been recently gaining it, finally. Prior to the new async stuff, though, Python was a horrible language if you had internal microservices bouncing requests off each other - the edge layer would wait a very long time for all those secondary requests to finish before it could respond.
    • deleted 6y ago
      [deleted]
  • lostcolony 6y ago
    Well, most any language you'd use for web will have the capability. I think more pick a language that makes it -easy-.
    • t-writescode 6y ago
      Ruby doesn't. Python, until recently, didn't.
      • zo1 6y ago
        Does 10 years ago count as "until recently"? Below is Tornado, an async-based web server. https://github.com/tornadoweb/tornado/tree/branch1.2/tornado https://github.com/tornadoweb/tornado/tree/branch1.2/tornado That's the earliest I could find off the top of my head - But the other comment responding to you found an even earlier one where they mention the "select" feature that was added sometime around 1998 (22 years ago). The twisted async networking library seems to have been released/started at least 13-14 years ago as well. I think we all (myself included) kinda forget or underestimate how old Python really is.
      • lostcolony 6y ago
        Ruby does via libraries ( https://github.com/socketry/async https://github.com/socketry/async for instance). Which is my point. Pick a language that makes it easy. Even then, it's a bit misleading; make it so concurrency, in general, is easy and efficient. Async is immaterial if you have green threads or similar (a la Go or Erlang). Point is, you shouldn't risk a model that ends up unnecessarily blocking/synchronizing execution.
        • t-writescode 6y ago
          This is news to me. Very nice :) Do you have any examples of this being used with Rails? I'm having a bit of trouble finding examples of both Socketry and Rails being used concurrently; and, I will admit, my Rails knowledge is a bit .... intermediate or less.
          • 6y ago
  • QuadrupleA 6y ago
    Multithreaded python sorta works like async in web situations - the GIL still limits to one thread actually interpreting python code, but a thread can run while another one is blocking waiting for IO.
    • stjohnswarts 6y ago
      I still don't know why people are afraid of multiprocess python. Sure it's a bit more complicated, but not that much more. Use those processor cores. I use it all the day for doing things in parallel on testing hardware and deployments. Is it really that much harder for web?
    • t-writescode 6y ago
      I've still run into issues where an external service took approx 1 second per call. We had 20 threads on each docker container running Python. We had a lot of web requests. It regularly took down the multithreaded python instance. The solution we went with was to put all the different web requests into 2 flask servers and then take the output of all those different web requests and make one big request tot the django instance that had all the business logic. If memory serves, we only had 2 flask instances handling all the traffic that the old array of django servers. Async pays dividends.
      • djstein 6y ago
        have you tried with async Django yet?
        • t-writescode 6y ago
          I've since moved on to a different company that does not use Python for its main services. Django async came out after I left. I was very excited to see it arrive, though, as I expected it would improve the areas of concern I had and reduce the need for the extra layer. edit: wording
        • ZephyrBlu 6y ago
          Is async Django completed? I thought the initial async functionality wasn't 100% of Django being async, only specific parts (Like views?). Maybe I'm remembering wrongly though.
  • siscia 6y ago
    Pick an ARCHITECTURE that make simple to do things in parallel. People are saying that python and ruby can't do things in parallel in an easy way. While technically true, it is also pretty much irrelevant, since we have multithreading.
    • t-writescode 6y ago
      It's very relevant if the external webservice calls take long enough. Multithreading just increases your technical max concurrent request up to the size of your multithreading maximum. Async lets the computer do _anything else_ while it's waiting for the request. If you're handling thousands of concurrent requests, you need thousands of threads available ... or, 10s or hundreds of threads with async available (your mileage may vary).
    • speedgoose 6y ago
      I think it's relevant. Let's say you write an API that only needs to call another API that takes 1 second to respond. A golang or nodejs server would have no problem to serve thousands of parallel requests. You can't really justify having 5000 ruby threads, they are heavy and they will use too much memory.
      • Lio 6y ago
        If you wanted to run 5000 parallel async requests in MRI ruby surely you'd just use a simple thread pool wouldn't you? If you prefer some other paradigm then Concurrent Ruby[1] has a swiss army knife of them including event loops and Go style SCP. I think things will improve in ruby 3 but even without ractors it's still possible to do either async IO in MRI or use JRuby for fully parallel ruby. 1. https://github.com/ruby-concurrency/concurrent-ruby https://github.com/ruby-concurrency/concurrent-ruby
        • speedgoose 6y ago
          It's good to see that Ruby is getting some async IO. Hopefully Rails and the other frameworks will use it.
  • charrondev 6y ago
    Are there good articles on the benefits of async over something like PHP-FPM? At $WORK, we run a lot of 1000s of php servers with 10-20 processes each using nginx. External calls we don’t need the result of immediately get kicked out to an external queue, but we haven’t seen any issues with DB calls blocking the whole server. Long or slow DB connections are an issue for individual request time and DB load though (the biggest DBs we run handle 1000-2000 concurrent connections).
    • xemdetia 6y ago
      I think one of the forgotten blessings of PHP/CGI was that requests had to finish sometime so as to free the PHP process for the next request. This alone I feel ended up making decisions easy around when it felt right to offload stuff when slow requests started to affect index.php from rendering in a timely fashion but now that's often masked (rightly) by caches and other tricks. Right now in a microservice model often you have to remind developers that it just isn't ok to have a ~1ms response in a basic case and a 1 second to infinitely blocking case during errors or slow remote services.
  • marsdepinski 6y ago
    You don't know what you're talking about.
    • dang 6y ago
      OK, but please don't post unsubstantive/flamebait comments to HN, and certainly not personal attacks. If you know more, the thing to do is to share some of what you know, so the rest of us can learn. If you don't have time or don't want to do that, that's fine of course, but then please don't post. In case you're interested, a more in-depth explanation of this principle is https://news.ycombinator.com/item?id=25130956 https://news.ycombinator.com/item?id=25130956, from yesterday. Edit: it looks like you've unfortunately been posting a lot of unsubstantive/flamebait comments. We ban that sort of account because we're trying for curious conversation here. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and sticking to the rules when posting, we'd be grateful.
  • treis 6y ago
    It's pretty rare for a web app to be cpu bound. The constraint is usually memory consumption.
    • CyberDildonics 6y ago
      There is no such thing as "memory consumption". There is memory bandwidth and memory latency. If something is CPU bound it is likely to be memory latency unless it is very optimized and running on multiple cores. If something is not CPU bound neither memory latency or memory bandwidth will be the current bottleneck because both show up as CPU time.
      • seedless-sensat 6y ago
        I think the parent comment was suggesting a common scaling limit for web severs is that inflight requests exhaust the host's memory capacity.
        • CyberDildonics 6y ago
          I don't think I have even seen memory use be a benchmark for server frameworks. When is that ever actually a problem? Wouldn't that mean hundreds of thousands of in flight requests?
  • hansvm 6y ago
    At the level of a programming language, the win from async isn't one of performance (at any given time you're operating at most one process per core, and you'd generally get better performance if it were always the same process for each core doing exactly the right thing at the right time, including choosing when to interact with your asynchronous network card), but rather one of developer ergonomics; it can be a lot easier to interact with async things (those network events) by making everything that calls into it async as well and relying on some framework to turn your async code into something that actually matches the machine's execution model.
  • IshKebab 6y ago
    That's only required for languages that are single threaded. Your advice would be better stated as: * Don't pick languages with no concurrency support.
• fareesh 6y ago
  > Make sure your web server and database are set up and tuned right - that they have enough memory, threads, etc. Anyone have a link to good write-up on tuning Apache/nginx and Postgres/MySQL for a web app running on a $10-20 typical VM?
  • davidkell 6y ago
    For Postgres https://pgtune.leopard.in.ua/#/ https://pgtune.leopard.in.ua/#/
  • jeffbee 6y ago
    Don't. Just use your cloud's hosted database instead.
    • ZephyrBlu 6y ago
      I don't really understand why people _want_ to deal with all the nitty gritty stuff when it's so cheap and easy to have everything work perfectly and be maintained out of the box.
  • charrondev 6y ago
    Step one if you haven’t already would be adding caching. Something like varnish in front for, if nothing else guest traffic, and memcached or redis for commonly fetched data. For a lot of cases varnish (or a different reverse-proxy cache) will shave off 50% of your traffic from ever hitting the servers. It depends on your percentage of guests/sessioned users and and read/writes though. After that using a cache like redis or memcached for things like user sessions/permissions and commonly fetched data significantly reduce the load on your database.
  • dig1 6y ago
    Unless you are expecting >1k req/sec with similar amount of writes, default nginx/postgres/mysql settings should be fine. Focus on your app instead :)
• jchw 6y ago
  > 1/10 or 1/100 the carbon footprint I feel compelled to bring this article up, especially since last time it was brought up on HN people seemed sure in the comments that it was not a common sentiment. https://buttondown.email/hillelwayne/archive/no-your-clean-code-wont-save-the-planet/ https://buttondown.email/hillelwayne/archive/no-your-clean-c... I agree with the gist of this article, but I think if you are not a FAANG as the article suggests, then you probably also are not at a scale where your carbon impact is actually measurable...
  • mlyle 6y ago
    > I agree with the gist of this article, but I think if you are not a FAANG as the article suggests, then you probably also are not at a scale where your carbon impact is actually measurable... It's not hard to have a startup that does use 100kW of servers and maybe about 150kW total-- HVAC, networking, UPS inefficiencies, etc. There's about 450 grams of CO2 emissions per kilowatt-hour from typical electrical generation. Therefore this is about 600 tonnes of CO2-- not counting other lifecycle costs. On the one hand, this is a tiny smidgen of overall CO2 emissions. On the other hand, this is about 80 households worth of CO2 emissions. As a developer, making this kind of thing 1% more efficient is the same magnitude as completely eliminating your home carbon footprint, and is a whole hell of a lot more plausible to do. I know offsets are "cheap" at $10 per tonne or whatever as the article says. On the other hand, please don't really assume that buying $6k of offsets really does as much good as eliminating 600 tonnes of emissions for realsies.
    • jeffbee 6y ago
      A startup that cares at all about energy efficiency of compute will use a public cloud service. They don't have inefficient (or any) UPS, some of them don't even have traditional HVAC, their power overheads are only about 10% compared to 100-300% for standard corporate datacenters, and whatever fixed overhead they suffer is amortized over all their customers.
      • mlyle 6y ago
        OK, uh... Slice 25% off the numbers I said, which assumed an overhead of 50% instead of your suggested 10%. It doesn't change the point at all.
• deleted 6y ago
  [deleted]
• kazinator 6y ago
  Optimizing an application is better than adding servers, because it's faster for any one user, not just scaling better. > optimizing your app to fulfill a request in 1/10 the time is like adding 9 servers to a cluster. Not if the request executes on one server. Because then it's like saying that optimizing baby gestation to take place in one month is like adding 8 uteri.
  • jtsiskin 6y ago
    It would be like adding 8 uteri, if your requirements are “give me x babies per second” In both cases you can get 9 babies in 9 months, and can support the same amount of baby throughput. But in the 1 month gestation case, you only need to feed one mother.
    • comex 6y ago
      I think what kazinator is trying to say is that request latency is also important, in addition to throughout, because users will notice when requests are faster.
  • hinkley 6y ago
    The mean number of in-flight requests is calculated off of the arrival rate and time per request. If you can retire requests faster, you're available for the next one.
  • ghego1 6y ago
    I would argue that optimizing the server, in the uteri comparison, would translate to having twins or triplets (and so on). With one uteri you'd get more babies with no additional uteri. In real life you would also be in for some real "fun" after nine months ;-)
• jeffbee 6y ago
  They're kinda glossing over the difficulty in removing 99% of your runtime cost. Unless you've done an insanely bad job of that first draft, you won't be able to do it. Finding a 10% improvement in a system that has enjoyed even the slightest design work will be a challenge.
  • t-writescode 6y ago
    A lot of the improvements I've seen in systems have come from adding indexes. Sometimes, they're just missed over time. "Oh, we never thought that this database would be searched for this way". Other times, it's batch calling the DB ahead of time, rather than calling it for each entry in an array. edit: these are easy mistakes to make over time.
  • golergka 6y ago
    If you've optimized something 2-5 times, you've done something smart. But if you've optimized something 100 times, you've stopped doing something really stupid.
    • beagle3 6y ago
      “Stupid” is relative. Often, when designing a system, I did not have enough information about how it will actually be used (and in some cases - neither did anyone else; when you create a market all you have is guesses). Things I was sure weren’t important turned out to be important, and vice versa. E.g. in one of the systems, prior business analysis showed that legally valid “chain of custody” would be an issue, which dictated very specific (and often costly, performance wise) decisions. And then after the 3rd customer deployment, it turned out that they don’t care - they’d rather pay less for everything, and lose every 20th claim. And after the 20th customer it was proved beyond the shadow of a doubt. It was stupid to believe what customers said ahead of time, yes. But I don’t think that’s the kind of stupid you are Referring to.
  • mlyle 6y ago
    Maybe. Building things to scale horizontally tends to have a lot of inherent overhead: a lot of fine locking, serialization, consensus, messaging, intermediate representations and storage, etc. Sure it increases the top-end of where you can go, but it can be a huge penalty.
• Scotrix 6y ago
  We’re a startup and focussed on easy horizontal scalability early on. From a management point of view premature optimizations are cost intensive, especially while we’re still figuring out what our product is and how it works. At the end there are a lot of changes all the time to functionality and implementation and optimizations early get wasted very quickly. We rather purchase new servers. A powerful server cost 100 USD/month and wehe development team can focus on implementing features and functionality which moves our product forwar d, optimizations are opportunity losses at the beginning of a product. Of course, as soon as bad performance impacts user experience it’s a different story and also if you start to have hundreds or thousands of servers but then you’ll know what your product is doing and changes become smaller, focused and code optimizations start to make sense.
  • loukrazy 6y ago
    Not to mention the cost of X more servers for a year is probably less costly than the developer time of optimization and the opportunity cost.
    • Scotrix 6y ago
      Indeed, that’s basically the point as well. I have seen developers hunting for the „right“ way and best performance even to the microsecond level for no good business reason while ignoring the state of a project/product and the required value they should focus on.
    • peteradio 6y ago
      On the other end of the spectrum, really poorly performing code can affect development, testing and delivery times which eats developer time as well. Some developers or testers might sit oh welling waiting for 1hr job to finish when 1 day of development time could turn that job into a 5min run. That kind of optimization pays for itself before it even makes it to production.
      • abraae 6y ago
        Adding poorly performing code is a bit like pissing in a swimming pool. A little bit is ok. If you have to. But too much of it and there's no way back other than draining the pool and starting again.
• smarx007 6y ago
  Yes, that's true but beware of the fact it only works this way if your code is 0%-serial, see https://en.wikipedia.org/wiki/Amdahl%27s_law https://en.wikipedia.org/wiki/Amdahl%27s_law for the math. By the way, this law is why supercomputers like Fugaku https://www.youtube.com/watch?v=WVsFFojdq3c https://www.youtube.com/watch?v=WVsFFojdq3c simply have no other way than optimizing their code. 0.1% serial code means that a code running on 160000 CPUs will be just 1000x faster than the code running on a single CPU. Further, https://en.wikipedia.org/wiki/Gustafson%27s_law https://en.wikipedia.org/wiki/Gustafson%27s_law may be more appropriate here as it is common for the app requirements grow with the performance increase of the compute hardware. Edit: bottom line is that the conclusions of the article are dangerous. Reducing the controller method exec time by 10x is not likely to allow your system to process 10x RPS by itself, the cost and product backlog delay caused by a perf optimisation project may not be acceptable etc. But beware that slow code in the critical path of your distributed/parallel system such as kernel code, event loop code, transaction commit code, distributed consensus protocol implementation can make your system unscalable beyond N factor (whether it is cores or servers). You can surely add 100xN nodes but it won't help.
  • CyberDildonics 6y ago
    This is already a gross misunderstanding of Amdahl's Law and what it means pragmatically, but you have it things completely reversed. Optimizing something to run 100x faster avoids needing more servers, thus avoiding the synchronization from splitting things up to them. That synchronization by the way is not actually done by the servers, it would be done by dedicated networking hardware. Even on a single computer, for web server programs the synchronization is not really happening in the user space programs, it is happening in the in kernel for the IO to and from disk and to/from the network.
    • smarx007 6y ago
      We are actually talking about orthogonal aspects. You are right that being able to run the system on a smaller number of nodes reduces inefficiencies and reduces the "hit" of diminishing returns stemming from Amdahl's law. However, to do that, you need to speed up your code and I argue that unless you speed up the non-parallelizable part together with the parallelizable part, you won't be able to achieve that system speedup that allows you to remove a number of servers from your cluster in the first place. However, I think you are right that on small number of servers speeding up parallel part can have a more immediate effect. I did a small calculation just to check the numbers. Assume you have a server that can do 100 RPS. You can either speed up the parallel code 2x (let's be realistic here and stop this 100x nonsense) and bring the RPS almost to 200 or you can take that serial code that is just a fraction and reduce it by half, so that f=0.995 and not 0.99 (and adding a marginal RPS increase to 102 RPS). Here is the plot: https://imgur.com/a/7w1hms9 https://imgur.com/a/7w1hms9 Edit: read http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf as recommended in the upvoted comment above!
• renewiltord 6y ago
  > it may sound obvious, but - optimizing your app to fulfill a request in 1/10 the time is like adding 9 servers to a cluster. Optimizing to 1/100 the time (reducing requests from say 1.5 sec to 15ms) is like adding 99 server Who the hell is casually optimizing away 90% of the latency in the time it takes to spin up 9 more pods? That's insane. Like the latter is an operation on the scale of minutes. Also, this is a throughput/latency conflation. 90% of the cost a startup incurs is in the things it could be doing instead, not in the infra. Opportunity cost dominates.
  • greggyb 6y ago
    From a data analytics perspective, it is surprising to me if I don't find multiple opportunities for order of magnitude improvements in data processing infrastructure at my clients. At one client where I was doing more IC dev work, it got to the point where they'd just chuck reports over the wall at me and ask me to find efficiencies in these. This was supporting >15K internal users in a massive shared reporting and BI infrastructure, so it was well worth it to have someone spending 10s of hours on optimization on a regular basis. I could routinely find order of magnitude improvements in reporting queries that were already running in production. If bringing things from POC/end user mockup to production, I could frequently find two orders of magnitude of improvement in query time, which reflected directly in latency to report rendering for end users. That client was not special. I wouldn't call this sort of work casual, and it's a very different domain than optimizing serving web apps. I would spend days to weeks on such optimization tasks at the client I mentioned above. It might take a few weeks to do serious optimization work on large ETL pipelines, but I routinely find big gains in performance on such projects - usually in the range of 1/2-1/10 run time compared to baseline.
  • jtsiskin 6y ago
    Yes, the key is to at least be aware of it. I bet there is a non-insignificant amount of existing endpoints out there that could easily be improved a large percent if someone actually took the time to profile it. Just from the shifting codebase, build up of complexity, “legacy code”, etc. that no one is probably looking at. If there was, for example, easy to use developer tooling that automatically identified bottlenecks, the opportunity cost scale could tip towards optimization vs having to pay for and manage more servers. But the opportunity cost insight is great; optimization only wins if the cost to optimize is low and the amount that gets optimized is high.
• mwcampbell 6y ago
  I was agreeing with this article until: > Consider a fast single-file database like SQLite Sure, it's nice if your application is optimized enough that it could run on a single server. But it seems to me that actually tying it to a single server, with local storage on that same server, in production, is irresponsible. I sure wouldn't want to have to explain why the application went down, and will come back running a possibly out-of-date DB backup, if that single server suddenly disappears. SQLite may be faster and simpler, but to sleep well at night, there's no substitute for stateless application servers in front of a managed database.
  • zozbot234 6y ago
    SQLite is not a substitute for a proper database; it's a substitute for fopen().
    • bob1029 6y ago
      I strongly disagree with this in practice. I understand the philosophical motivations to encourage deeper thought regarding the engineering, but we have been using SQLite as our only datastore for almost 5 years now without a single DB-related incident. We probably dont even spend 30 minutes per month screwing with SQL/database-related concerns. I will say that our strategy probably would not work at a larger technology organization. It is more advanced and requires disciplined, motivated engineers to keep it on rails. In retrospect, we were probably only able to get away with this because we were willing to accept a ridiculous amount of up-front risk. We got lucky that it played out so well. Now that we are on the other side of that journey, I can assure everyone that the new world is incredible (and much more stable). To manage scalability and maintenance, we run multiple SQLite databases - one per logical type of persistent business entity (i.e. Users.db, Sessions.db, Customers.db, etc). This allows for us to manage schema versioning for each type independent of any others. We have ~25 types that each get their own separate DB. Our migrator is a simple for-loop, but somehow our approach seems even more elegant than Entity Framework because we don't need special unicorn tables to track migration metadata - see: pragma user_version. The part that requires discipline is that we have no hard referential integrity constraints. This is where developers have to make the right choices when designing related entities & data stores. We do not rely on the database/ORM to clean up our modeling for us. Our backup strategy is to snapshot the entire VM. The biggest motivation for having your application fit on a single box is that you can synchronously snapshot the whole system with a single click. This is far simpler than maintaining a completely separate SQL server instance and worrying about all of the added complexity of backing up 2 (or more) machines. We have yet to encounter a customer who did not have the ability and willingness to use this strategy. If your business application can run on 1 server (and is forecast to do so forever) and you have a RTO/RPO that permits using VM snapshots as backup/restore, then I would strongly recommend considering this type of approach from an engineering perspective (assuming you have the team/skills for it). Thinking more broadly, since we have committed to this idea of the datastore living on the application server, we could hypothetically build up clustering at the application-level by adding multiple application nodes. This would probably be better for us anyways, because we really only have 2-3 entities that we absolutely must have synchronously replicated across all nodes. A heavy-handed SQL Server cluster approach is way overkill when we can just swap to GUID keys on our sessions/state and pull consensus to update important settings, transactions or permissions. And at the broadest scope, I still feel like most developers vastly underestimate just how fast a computer can do things in a well-optimized domain. SQLite is substantially faster than SQL Server in the single-node case. You will never get lower latency than by having your database running in the same process as your business application. Latency is the biggest devil when it comes to dealing with transaction throughput. If I can get a user request out the door in 100 microseconds vs 5 milliseconds, it makes a shitload of difference when I am pushing thousands of these per second.
• mike503 6y ago
  Isn’t the example akin to saying cooking something at 600 degrees takes half the time as 300?
• eznzt 6y ago
  Don't use Python. Poof, free servers.
• speakspokespok 6y ago
  A more meaningful comparison is if you can optimize your code so that it runs 10x faster that means it will also run just as fast as it is now on something 10x slower. That's like going from needing a small server to something you could wear as a watch. (So to speak.)
• kortilla 6y ago
  >Setting up FAANG-level infrastructure won’t make your company into a FAANG, any more than a cargo cult can summon supply-bearing planes by building fake runways and wooden ATC towers. This sentence is strange. It uses the term cargo cult as part of an analogy to cargo culting. Wtf? Anyone who knows what “cargo cult” means doesn’t need the rest of the comparison. If they don’t, then the comparison is senseless.
  • quickthrower2 6y ago
    Also it’s a bit of a straw man. No one thinks Google is successful just because of their infrastructure and nothing else.
  • kamikaz1k 6y ago
    I think it's a little clever actually. But maybe only because I knew "cargo cult" as typically used here in HN (blindly copy). Reading this made me look into the origination. For lurkers, this is an excerpt from the wiki page [1] "A cargo cult is a millenarian belief system in which adherents practice rituals which they believe will cause a more technologically advanced society to deliver goods." [1] https://en.wikipedia.org/wiki/Cargo_cult https://en.wikipedia.org/wiki/Cargo_cult
    • redis_mlc 6y ago
      Ironically, if the islanders had built good runways instead of mockups, they would get more airplane arrivals. :) In Indonesia today, local tribes have a habit of making the jungle and hill runways for aid too short, soft and tree-ringed, resulting in perpetual accidents affecting the long-term viability of cargo and passenger flights. (Their huts are often made of wrecked airplane wings and fuselages.) In WW2, the Chinese government charged the US a fortune in gold to build the Chengdu B-29 runway using 70,000 laborers with hand tools, which had to be about twice as wide and twice as long as any previous runways. The Chinese runways turned out to be too far from Japan, so they were largely abandoned after nearer islands were captured. Period Photo of Chengdu Runway Construction Using Hand Tools https://www.youtube.com/watch?v=vqaEb7GxZ3A https://www.youtube.com/watch?v=vqaEb7GxZ3A (The infamous Gusdalcanal runway was built years earlier with modern equipment captured by surprise from the Japanese.) https://en.wikipedia.org/wiki/Boeing_B-29_Superfortress#Forward_base_in_China https://en.wikipedia.org/wiki/Boeing_B-29_Superfortress#Forw... Source: commercially-rated airplane pilot, WW2 student.
• nn3 6y ago
  I call it resume driven engineering. Just making one server run well won't do much to your resume. Creating some distributed micro service buzz word bingo monstrosity on AWS on the other hand looks impressive. It's all a question of incentives.
  • gopalv 6y ago
    > Just making one server run well won't do much to your resume. That's pretty much true, because doing that and that only will also mostly get you hired at a place which has a single server (like say "use SQLite" instead of RDS). On the other hand, if you can justify what you did to make a single request better (as in, "what I did meant that a single request came back in 40ms instead of 120", like that mixed API example - like if you bypassed the standard 1+N problems), then I'd be very keen to pick out what that person did. A lot of work involved for the latter is also the same for the first, but it is a difference in attitude towards user relevant functionality which I've noticed in the best people I've worked with. That is not to say don't use SQLite, but more along the lines of this is a simplicity hack, I can make a complex working system out of this simple system, because I used the right abstractions for that bit (like say, JDBC or PDO - maybe not a full on ORM).
  • ineedasername 6y ago
    It's the mechanical engineering equivalent of building a Rube Goldberg device and then showing it off as "Look how great of an engineer I am!"
  • rubyist5eva 6y ago
    I'm not sure about that. I've got a pretty good track record in interviews where I get right into the technical nitty gritty of reducing operating costs by 90%.
    • tartoran 6y ago
      You're splitting hairs. Yes, technically knowledgeable and capable people are still in demand but on average the trend that the OP is mentioning is quite true. And due to human limitations one cannot go both in depth and breadth. And with the proliferation of technologies and the rapid pace of cycles and recycles, the ones who keep up with the trend are more likely to be better rewarded.
• hinkley 6y ago
  The biggest thing we're going to regret looking back on the early Cloud era is the foolish notion that you need 100x as many servers to do 100x as much work. Servers don't scale linearly. It's more likely you'll need, at a minimum, 107-110x as many servers. Between 100 + log(100) and 100 + sqrt(100). So making your code 100x faster saves you more than 100 servers. I think the arguments never end because it's all fuzzy math. Get too attached to running the whole thing on one server, and your architecture starts making irreversible decisions about where the source of truth is, and you get locked into a single server. Give up too early, and you spend a lot of energy herding cattle instead of building features. And speed is something you outsource to the guys who write the checks. That's fine if you lock in your vertical, but I've worked on projects that lost out to a more responsive or financially efficient competitor. It really, really sucks.
  • missblit 6y ago
    Servers can and do scale effectively linearly depending on the workload. The broader point stands though. The more servers you have, the more money a 1% cpu optimization saves you in raw dollar amounts. At a certain scale it always makes sense to optimize your code.
    • hinkley 6y ago
      The moment the servers need any consensus at all, you've opened the door to Amdahl's law. That could be sharing caches, running Raft, or talking to a transactional store. Even if it's just to keep my UI in sync between the desktop and mobile site, you have costs. But since so many sites have social aspects now, you need to be able to communicate between accounts, which means interprocess communication. The Amdahl's law fraction is at least logarithmic to the number of users, but more likely to be proportional to the 'surface area' - sqrt(n). Every new server gets you a little bit less oomph than the one before.
      • hinkley 6y ago
        And remember, even if we aren't sending more and more data between our users, we're collecting more and more telemetry data from our servers all the time. Coalescing logs and gathering stats requires funneling data from everywhere down to one machine (the one that built the graph)
• austincheney 6y ago
  > A lot of tech discussion these days is focused around scaling web app infrastructures to handle huge traffic. It most cases this the wrong way of thinking. It isn't that its bad or wrong, but rather that it's dated and expensive. This is the 2000 era .bomb logic of get big fast and data is king. It's like thinking in terms of Facebook instead of BitTorrent. At this point if you are not already established or working on something extremely original you have probably missed your shot and crowded out from the incumbents. Current programming paradigms indicate two paths forward of which one is substantially less expensive than the other: distribution and concurrency. In the distribution model most of your costs are up front in the application. This is a service oriented approach but without a central service, rather a pool of nodes that intelligently communicate task, file, and event queues. The cost to scale is divested from the cost of application, which is the biggest difference between this approach and thinking in terms of huge traffic or server/data hoarding. Since the application is divested from cost to scale all that is required to compete with the large incumbents, at scale, is market penetration. Marketing is cheaper than a data center. The concurrency model requires a central service but does not operate as a central server. Each connection/session is a parallel child processing unit, such as an event loop. This approach requires less costs up front since there is still a central connection point to manage, but there is still a cost to scale even though much of that cost is offloaded from data management to increased processing overhead.
• didibus 6y ago
  For a fault tolerant system, even if all your traffic could be handled by a single machine, you'd still want some amount of redundancy. So you'd at least have a second node.
• haberman 6y ago
  > It may sound obvious, but - optimizing your app to fulfill a request in 1/10 the time is like adding 9 servers to a cluster. Optimizing to 1/100 the time (reducing requests from say 1.5 sec to 15ms) is like adding 99 servers. This cannot be correct. If you optimize 1/10, and then optimize to 1/10 again, that's the same as optimizing to 1/100. Does this "add" 18 servers or 99? If you optimize to 1/10, then your cluster can handle 10x the load, which is the same effect you would get by simply having 10x the number of servers (assuming your app is perfectly parallelizable). The article's quote is only correct if the "cluster" starts as just a single machine.
  • ivegotnoaccount 6y ago
    If you keep track of how many "virtual servers" your cluster has, and take into account the fact that the text obviously implies "per server" (let's not play dumb simply to be able to spit on the article), the math works correctly: - I have x servers. I optimize to 1/10th of the time. I still have x servers but now have (1+9)x = 10x virtual servers. - Now, I optimize again to 1/10th of the time. I have 10x virtual servers, so (1+9)(10x) = (1+99)x = 100x virtual servers.
    • haberman 6y ago
      The unstated assumption that these figures are per server was not obvious to me. Your assumptions about my motives are incorrect and inappropriate.
      • ivegotnoaccount 6y ago
        You're right. My bad, I went way too far with this message.
• darksaints 6y ago
  I get that there are many out there that have fully bought into the idea that developer productivity is more valuable than efficiency. But I think it is many times a false and very misleading tradeoff, or at least they got their cost function drastically wrong. 1) There are plenty of fast languages out there that are suitable for high productivity environments. Go, Kotlin, Scala, F#, C#, even Java, are all great languages for developing web apps and services. And they'll all beat the pants off of your PHP, Ruby, or Python framework of choice. 2) Many of the frameworks that get you up and running extremely quickly really only save you a day or two at the very beginning before they level out and are more comparable with other frameworks. Sure, `rails new myapp` and your first few migrations will save you hours over something like Play Framework, but after that, you don't really have any more jawdropping productivity tools left to use. I once had a conversation with a die-hard Rails enthusiast tell me about how he decided he would never go back after using his first ActiveRecord migration, and realizing how much code was generated for him. I asked him how much time he saved, and he said, with pride, "LIKE 4 HOURS!". Meanwhile, he probably spent probably 2-3 hours every day writing tests and debugging errors that C# or Scala would never even let compile in the first place. 3) It's not too long before you have to start thinking about performance anyway because your high latencies are a terrible user experience. This goes for client-side software as well. My 8 year old doesn't know shit about software, but he knows that he absolutely hates Microsoft Teams and would much rather his school use Zoom...because the bloat and terrible performance are a massive disappointment in user experience, especially on cheap school-provided laptops. 4) People really undervalue the quality of life and productivity improvement that comes from managing one server versus ten. Even with the cloud, which supposedly takes care of those headaches, you still have to worry about load balancing, health monitoring, auto scaling, complex caching, etc. And even if you could autoscale your web server worry-free, you're soon gonna have so many connections to your database open that you're gonna have to worry about setting up connection pooling, read replicas and logical replication, and configure all your servers to partition their queries into reads vs writes, etc. And 95% of stack overflow answers that you google when you hit a snag are now irrelevant to you because they don't take into account the complexity of your environment. Performance is absolutely an improvement in productivity and quality of life. It's one that pays dividends every single day, as opposed to a handful of days at the very beginning of a project.
• beagle3 6y ago
  It’s possibly much more. http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf Demonstrates, and it is totally in line with my experience, that using common scaling mechanisms has an immediate x10-x100 cost before you start to realize the (almost linear) scale in the number of machines. Just adding Hadoop costs you x10; to go 100 times faster (compared to one machine properly written without Hadoop) you need 1000. Common web app scaling is not quite as bad, but it is quite bad. Philip Greenspan had shown back in 1997 with the original arsDigita system that one machine hosting db+web could do better than the then-prevalent multi-machine deployment. That’s still correct (and still mostly ignored in scalability discussions cause it is boring and requires thinking ahead).
  • deleted 6y ago
    [deleted]
• kumarvvr 6y ago
  I think stacks that run on Java / .NET provide the best value in terms of productivity and costs when chosen to develop large web apps. Sure, Django might be easier to start off with, but if you want to optimize your app, there is only so much you can do. However, the non-optimized performance of an ASP.NET core web app (developed as per Microsoft recommendations) is orders of magnitude faster than any Django App, without any explicit optimizations. It also helps if you can split your app into an API and a client app, as APIs consume a lot less resources than traditional web apps. It makes sense, going forward in a cloud dominant world, that the traditional web app development process, where HTML is passed around over the wire, a new set of HTML for every page, has to die. Its imperative to replace it with modern stacks, with the client separated from the API.
  • smolder 6y ago
    It doesn't take a lot of consideration to arrive at the conclusion "modern = good", but I assure you, it's not imperative nor plainly better to make everything into a bundled js webapp. The boilerplate code for basic web apps often dwarfs the actual content of a basic web page. You could also make the argument that many things designed as web apps have no business running in a browser and should be written as standalone applications, but what's "best" in some sense isn't always the most pragmatic thing.
  • mwcampbell 6y ago
    > It also helps if you can split your app into an API and a client app, as APIs consume a lot less resources than traditional web apps. We shouldn't focus so much on what's best for us (at least in operating costs) that we sacrifice what's best for the end-user. In many cases, a conventional server-rendered web application is what's best for the end-user. If we develop one of those by combining an API backend with a separate server-rendered application consuming it, we've likely added extraneous complexity and resource consumption to get back to what we used to have in an integrated server-rendered web app. Also, as DHH has written [1] [2], an integrated system, a.k.a. monolith, is best for programmer productivity and especially for getting the most out of a small team of generalists. I want to use a framework whose authors and community are opinionated about this. It seems to me that the Phoenix framework and the broader Elixir community fall into this category, while still delivering pretty good performance. I've also taken a hard look at ASP.NET Core, but I don't think it's as focused on the integrated approach. So, after being undecided for too long, I'm using Phoenix for my new project. [1]: https://m.signalvnoise.com/integrated-systems-for-integrated-programmers/ https://m.signalvnoise.com/integrated-systems-for-integrated... [2]: https://m.signalvnoise.com/the-majestic-monolith/ https://m.signalvnoise.com/the-majestic-monolith/
• auganov 6y ago
  Optimizing execution and designing for horizontal scalability are both cases of premature optimization. Scaling vertically via hardware as much as possible is IMO the most developer-time saving way. Once that's exhausted you'll have to decide which one is easier. The biggest mistake I see is people spending a lot of time designing for horizontal scalability and then running a cluster of under-powered servers when a single powerful machine would do just fine. There are very few things that are easier in a cluster (uptime perhaps?) and many that are easier on a single machine. Once you add the requirement of having to work in a cluster, every new feature will have to respect that. If your system is doing anything interesting you'll be hitting all the distributed computing dilemmas pretty fast.
• fefe23 6y ago
  I recommend against optimizing. If you optimize after the fact, you probably already made bad design decisions earlier. Those usually cannot be corrected by another optimization pass over the code. It is much more effective and efficient to have a good plan before you start, and make sure not to write inefficient code in the beginning. It is also much more effective to take the lessons you learned from the first round, throw the code from that round away, and write a new version. However that is only an option if you modularized your project well, so that small modules are replacable without endangering the rest of the project. Writing the new version will go much faster because you now understand the problem better, and it will produce better code because you get rid of questionable design decisions that hold you back in the long term. Think about what your legacy in this world will be. Leave the world with better software than you found it with!
  • hackerfromthefu 6y ago
    I agree, I've found these are often a good strategies. As a subtext I think your general perspective is as an experienced developer and not all developers have the experience to write runtime efficient code or components with clean modularized interfaces, so it won't work for everyone.
  • archerx 6y ago
    You have probably never worked in game dev where there is an art to optimizing to get every last fps out. I feel a lot web apps could use a good optimization pass because there is no reason that so many sites and web apps are so laggy/janky while doing incredibly simple tasks compared to games runinng on the same system.
    • fefe23 6y ago
      I didn't say optimizing code doesn't work or does not yield speed-ups. In fact I am optimizing my own code, too. But I only get ~10% speedup, tops. Because perf is decided at the design level, in the architecture, data structures and algorithms. In fact, if you get massive speedups in your optimization phase, I would view that as a sign that your process has issues. There should not be that much potential left after you wrote the code down. Elite fit on one floppy disk. Do you think they got there by taking a slow implementation and optimizing it? Have you ever used Turbo Pascal? You think you could get that kind of speed by taking an existing pascal compiler and optimizing it a bit?
• H8crilA 6y ago
  Many apps/things can run with 100x overhead and nobody should care. Look at your salary, add ~50% for corporate overhead and compare this to the savings you'd generate using market pricing of cloud resources (even if you don't use cloud). Can you beat this with, say, a 3 years discounting plan for the savings? In a lot of cases the answer actually is "clearly not". Sometimes you can eek out very sizeable gains, but always do the capitalist homework first (and the engineering homework of profiling second) before trying any fixes. Also, if it's performance that you need look into "stupid" solutions first. Can you move the DB to flash with just a few config options? Can you mlock() it in RAM? You'd be surprised how many things fit in RAM, especially if your DB is a managed distributed solution, and still be very cheap compared to engineering salaries.
• lmilcin 6y ago
  Golden rule to follow: always optimize first before you attempt scaling. Working for large banks I see this error made every day. Nobody knows how to optimize, but everybody knows how to request for couple more servers. Then there is a talk of budgets, hiring more people for maintaining the behemoth, then introducing some pretty exotic and expensive technology to "optimize operations". I have seen yesterday a huge piece of infrastructure with literally terabytes of RAM and a number of GPUs costing at least 250k a year for a batch job that should fit a single server with no problems.
  • mattlondon 6y ago
    But how many engineering-hours would be needed to optimise it to fit into a single server? And what would the risk be that there is a screw up? Sure there might be a few low hanging things to clean up ( i.e. a few places where a data structure could be better picked), but if it needs that much iron then it is probably not a trivial refactor of a few hotspots, but likely a more extensive rewrite. Probably cheaper to just pay 250k than for a couple of engineers taking risks modifying a battle tested bit of code... If it is important enough for the business to justify paying that much for then you'll need to "do things properly" and not do a hatchet-job of a cowboy refactor, but plan things out and test them and have rollout/rollback plans, project management etc.
    • lmilcin 6y ago
      Optimisation is typically one time cost. You make a feature A and you spend a bit more time to ensure it works couple times faster or couple times less resources than initial/naive version. Maintaining larger infrastructure is an ongoing cost as you need to provide x times more resources throughout the life of the application. Not only that, but now future development will have to be done in context of large application. Maybe writing a process for a single node could be easy and now you have 100 nodes and it is complex to design your functionality to work correctly on multiple nodes, complex to deploy, complex to resolve issues. Maybe you decide to employ new technology to solve some of those problems but at the cost of creating other problems and also additional complexity. Then maybe you need more people -- and this comes with its own inefficiency. The point really is, most applications could be orders of magnitude more efficient if you just don't make stupid decisions.
• andmarios 6y ago
  I was under the impression a Web App runs on my browser, so it's more like edge computing. :p To be fair, a 100x optimization is not something easy, and if it was, I would probably blame the development team. Your Web App is a whole lot of code and components in series. An 100x optimization would mean that every single part in there can get 100 times faster, or there is a component that takes maybe 95% of the time and this can be optimized 200x, and the rest of the stack takes 5% of the time and can be optimized 10x. I do believe that often it's the other way around, instead of a very fast first iteration, people go and build the distributed version that can scale 100x but takes the performance hit that all distributed systems have for network, syncing, etc. Most of us are using off-the-shelf components, so the space for optimization in our code isn't that big —unless you are a novice developer but compilers go a long way in our time. Thus it may come to choosing the correct off-the-shelf components, like maybe SQLite instead a NoSQL external datastore.
  • fabian2k 6y ago
    I’d argue that in many cases a 100x optimization is much easier than a 10% speedup. Very often the 100x speedup isn’t because you’re now doing something particularly clever, but because you were doing something stupid before. It’s very easy to do something horribly inefficient if important parts are hidden behind some abstractions. An ORM is maybe the best example of this, if you’re not careful you can easily generate queries with it that are several orders of magnitude slower than the straightforward and efficient version. Optimizing parts that are already reasonably efficient is much harder, but unless you spent a lot of time optimizing already there are likely plenty of low hanging fruit.
• tristor 6y ago
  The comments in this overall thread are pretty much echoing the types of discussions and lessons one learns in System Operations. I wonder if part of the challenge is that so many companies have decided to ditch "systems" people for developers, assuming that software developers have the same knowledge, interests, experiences, and perspectives as systems folks, and that systems folks are no longer needed. The question isn't "optimize my code" or "add more servers", it's both, and determining when it's appropriate to do one or the other at a given point in time based on current load and expected future trends. But generally, "optimize my code" can be pretty low hanging fruit in complex systems, because if you lack systems people, you end up with everyone focused only on their specific component and nothing else.