25 ms·

A lot of complex “scalable” systems can be done with a simple, single C++ server

398 points by Impossible 7y ago

kitsuac 7y ago
I've been programming C++ and assembly for 23 years. Few years ago I became a huge fan of Python. In my opinion Python is amazingly well suited for rapid first revision and can then be swapped out for C++ / asm.
- davidcbc 7y ago
  This is fine as long as you can convince management to spend the money to rewrite your software. That's usually a hard sell though. In my experience this plan usually ends up with a python monstrosity that everyone hates but is forced to deal with forever.
  - j88439h84 7y ago
    Type hints and dataclasses are a game-changer for Python. Much easier to reason about programs that use them.
    - adev_ 7y ago
      > Type hints and dataclasses are a game-changer for Python. Much easier to reason about programs that use them. Type hints made large python codebase switch from "Any change will blow up to my face" to "I can touch it carefully with protection gloves". Through, it is still pretty easy to fool mypy and very far of the compile time guarantees that provide most static typed AOT compiled language. And unfortunately, it also does not bring any advantage in term of performance.
    - insulanian 7y ago
      > Type hints... At that point it makes more sense to use a statically typed language.
  - kitsuac 7y ago
    I'm speaking in terms of the reality of what is effective in development. Not in terms of what management at particular companies will approve.
- Fazel94 7y ago
  You just have to write a tiny part that uses a lot of CPU in C++/asm or anything else. Much of code's performance isn't really reflected on to the scalability since mostly a tiny part of code is really ran a lot of time, and the other parts are just glues or management stuff or rarely used(not used in scale) features.
  - kitsuac 7y ago
    It depends. You aren't going to make a very fast modern codec encoder or decoder using Python. The hotspot ends up being the vast majority of the process. That management/glue layer becomes very thin, amounting only to feeding in the bitstream and reading back the raw video frames.
fatbird 7y ago
Has Carmack looked at Rust? I'd be very curious for someone like him to look at it in light of his experience with C/C++ and high performance systems.
- Ygg2 7y ago
  Pretty sure he's aware of it. And honestly I see the logic, Rust has been top of Tech Empower Benchmarks for a while. That said, I think the benchmarks could use some tweaking. I'm not sure they are representative.
  - throwthisaway2 7y ago
    Those benchmarks matter little. If by server this topic is a webserver the backend scripts do little. Its your database, caching, and disk manipulation that slows down performance. Most people write bad queries or have a shallow caching strategy. A router redirecting to a controller and parsing a template is hardly different across languages that use the same queries. If you truly need speed for post processong the off load on a c script like queries are off loaded.
- jsty 7y ago
  He's had a look: https://www.reddit.com/r/rust/comments/ap2047/john_carmack_writing_rust_code_feels_very/ https://www.reddit.com/r/rust/comments/ap2047/john_carmack_w...
  - pizlonator 7y ago
    Yeah but I didn’t see anything about perf there. I think it’s a valid point that the reasons why C++ is good for perf - compiled, statically typed, low-level, manual memory management, high degree of control over memory layout - would also apply to Rust. If they don’t then that would really call into question positive claims being made about Rust.
- pizlonator 7y ago
  Good question, not sure why you were voted down. And I’m not even a Rust fan.
slx26 7y ago
the title is misleading. sure, it advocates for C++, or Java, or C#, or others above Python, but clarifying the context: "a lot (not all!) of complex “scalable” systems can be done with a simple, single C++ server.", which I take as: "sometimes it's better to write a simple server in a low level language, than a complex server in a higher level language".
- dang 7y ago
  Submitted title was "John Carmack advocates C++ for server development". We've replaced it with a shortened version of what he said.
kyledrake 7y ago
The analogy here is one of the best ice climbers in the world proposing an ascent of the Matterhorn. Please use testable, easy to prototype with, memory managed languages for production servers unless you are solving a very specific problem and really know what you are doing.
- mberning 7y ago
  Very apt. I think people are also forgetting “the bad old days” where you worked on a large-ish cpp or java app for a year, nothing worked right, schedules slipped, and then the whole thing was scrapped and teams disbanded to work on other stuff. That was very common. You can’t count on having a team of Carmacks work on your blub app.
  - otabdeveloper4 7y ago
    Replace "cpp or java" with "Python or Ruby" and how is it any different today?
BalinKing 7y ago
From the next tweet in the thread: “JAVA [sic] or C# would also be close, and there are good reasons to prefer those over C++ for servers.”
- paulddraper 7y ago
  Yep. C++ is the nuclear option, when you know you need to keep memory reasonable, or capture the last bit of compute performance. Neither of these are usually the case for web application servers.
fxtentacle 7y ago
Many people don't know about Python's GIL https://wiki.python.org/moin/GlobalInterpreterLock https://wiki.python.org/moin/GlobalInterpreterLock That's the reason why you need to go multi-process if you want to reach a similar level of concurrency in Python as multi-thread in C++. And that surely adds a lot of complexity. As a very practical example of this, TensorFlow has a dedicated page with advice on how to make the Python part that reads the files from disk less slow. Think about that: The bottleneck for training a highly advanced AI with millions of parameters is in the 20 lines of Python code to read in a binary file... https://www.tensorflow.org/guide/data_performance https://www.tensorflow.org/guide/data_performance
- lewis1028282 7y ago
  Not really IO is always going to be slow compared to the CPU. The Python interpreter will be blocked 90% of that time waiting for IO anyway.
  - SaxonRobber 7y ago
    Percentages are misleading, what matters is the actual time it takes. Even small gains will have an effect as you increase the traffic and add additional stages to the system. Small delays accumulate and can have a surprising effect on queue sizes and latency.
  - postpawl 7y ago
    Exactly, and I’m surprised none of the other comments mentioned this so far. Often web app endpoints are bottlenecked by IO because they’re spending most of their time talking to a database or cache server. Python is probably not the right tool for a CPU intensive endpoint that needs to serve up hundreds of thousands of requests per minute and can’t be cached. There are a lot of ways to handle the IO intensive scenario in python: * Threading - Works with python libraries written in C, but now you need to add locks to your code to prevent race conditions. Not good for CPU heavy work because it’s switching context every ~100 instructions. * Gevent - Requires minimal code changes, but it usually blocks when running code from python libraries written in C. This uses event loop style concurrency and automatically patches internal python libraries to switch context when IO occurs. This means less time spent unnecessarily switching context, so it can scale better than python’s threading. * multiprocessing - Better for CPU intensive work, but requires more memory than other solutions. You don’t need to worry about race conditions since the processes are separate. * asyncio - Requires code changes and using compatible libraries. It does event loop style concurrency that allows you to specify specifically when to switch context. Like with gevent, this means less time spent unnecessarily switching context. * ...And there are probably a bunch of other ways I’m missing. I’ve heard that this sort of stuff is simpler in Go.
blondin 7y ago
not how i read his tweet. he was lamenting that python is too slow for some server side development use cases. and he gave cpp as an alternative that would be simpler and faster. he even followed up citing java and csharp. totally agree. if all your backend server is mostly complex serialization & de-serialization, and pushing bytes to other sub-systems, i think many other languages have advantages over python. wondering why no mention of go or rust though...
rvz 7y ago
> ...My bias is that a lot (not all!) of complex “scalable” systems can be done with a simple, single C++ server. The second tweet of the discussion: > JAVA or C# would also be close, and there are good reasons to prefer those over C++ for servers. Many other languages would also be up there, the contrast is with Really Slow (but often productive and fun) languages. I'm afraid that Carmack has sided with his own anecdotal experiences of the 1990s to justify the use of C++ for server-side development for the 21st Century. This probably made sense at the time due to the availability of more C++ devs and less language choices, but today in the 2020s? I remain totally unconvinced by his argument. He goes on to suggest Java or C# which still makes sense for many companies for generic server-side development if you are after a more secure backend. Kotlin is pretty much the most sensible for this. But for the sake of Carmack's engineering background however, it is unsurprising why Java/C#/Kotlin are technically unsuitable for high-performance gaming platforms if one was to create one. So what credible languages could be used to compete with C++? I hear Discord is having a great time with using Elixir (Erlang could also be used) and another gaming platform called 'Hadean' is using Rust for their platform.
- jariel 7y ago
  "But for the sake of Carmack's engineering background however, it is unsurprising why Java/C#/Kotlin are technically unsuitable for high-performance gaming platforms if one was to create one." This really is just not true. Financial, real-time style High Frequency Trading apps are often written in Java - not C++. Much of the JVM is not a VM, it compiles to machine code - in an optimised manner. For starters. Given how difficult it is to develop safely in C++ I can hardly think of a reason to ever use it on the backend.
  - SaxonRobber 7y ago
    This and more opinions from the oracle certified java guru.
    - kitd 7y ago
      Which part are opinions?
jandrewrogers 7y ago
Many developers severely underestimate how much workload can be served by a single modern server and high-quality C++ systems code. I've scaled distributed workloads 10x by moving them to a single server and a different software architecture more suited for scale-up, dramatically reducing system complexity as a bonus. The number of compute workloads I see that actually need scale-out is vanishingly small even in industries known for their data intensity. You can often serve millions of requests per second from a single server even when most of your data model resides on disk. We've become so accustomed to extremely inefficient software systems that we've lost all perspective on what is possible.
- pleasecalllater 7y ago
  Two. The number is two. You always need a backup server :)
  - twic 7y ago
    I used to think that. But there are a couple of notable exceptions at either end of the latency spectrum. If your latency requirements are slack, then you can get away with one machine, because you can reboot or reprovision it and carry on processing without meet your requirements. If your latency requirements are tight, you don't have time to fail over anyway, so you might as well run one machine and make sure you can deal with failing to meet your requirements.
    - chucky_z 7y ago
      This is why I ran Redis on a single server with no failover strategy. Our team spent maybe 15 hours workshopping a handful of other things (redis cluster, sentinels + replicas) before realizing we could spend an hour to have everything sit in a degraded state working around it while Redis itself was fixed. Redis only failed once ever and it took all of an hour to fix it, all during non-peak hours.
  - dboreham 7y ago
    Three. So you can do maintenance on one while still having HA.
8fingerlouie 7y ago
As someone who has implemented a complex system in C++ in this decade, I’d say he’s not wrong, but you need to carefully weight the pros and cons. In our case latency and real time demands mattered a lot (NASDAQ feed parser), to the point of the (potential) slowdown of a garbage collector kicking in was enough to rule out Java and .NET. It runs entirely in memory and on 64+ cores. We implemented our own reference counting system to keep some of our sanity, and to at least try to avoid the worst memory leaks. This was an edge case, and for almost everything else you’re probably better off implementing it in something that handles memory for you. If performance is an issue, least try it in Go or Rust with a quick PoC before jumping the C++ wagon.
- choeger 7y ago
  But aren't the edge cases the things that satisfy your pay? The standard cases of today are the introductory examples of tomorrow and will be automated or abstracted away by the end of next week. In JavaScript.
- CoolGuySteve 7y ago
  If you're parsing market data then you shouldn't really be allocating at all once the initial setup is complete. So there shouldn't be a need for ref counts. This is because the allocator itself can have an unbounded runtime that takes milliseconds, causing you to drop. In the past I've replaced malloc with an implementation that asserts if called on certain threads after init time.
  - tom_ 7y ago
    Milliseconds??! What causes that to occur? Though I suppose if they can take that long, avoiding them is an even better idea than it is normally... For constant time allocations, try TLSF: http://www.gii.upv.es/tlsf/ http://www.gii.upv.es/tlsf/
    - cogman10 7y ago
      I'm not the op, but I'm guessing large amounts of fragmented memory is what causes that. It's one benefit of a gced language with compaction, allocations are typically bounded (except when they trigger a gc).
wruza 7y ago
Morals of this story: 1. Always use the most performant language available to you. (What if your program gets a few million users?) 2. Horizontal scalability is too much complexity/work. Just apply the correct amount of optimization when you initially write the code. If you, like me, read this on mobile and did not click [more answers] link under that, then do it. That may save you a minute or two of derealization time.
- choeger 7y ago
  I really had a hard time to decide whether I agree with that statement or not. If you design a whole service it is obviously not true. But if you develop something like a specialized Backend, say a database, you might want to reconsider the complexity of horizontal scalability.
- Aperocky 7y ago
  2 Is almost no longer true because there are simple templates you follow that will scale you till infinity with cloud (which also means your cost is going to scale, but hey we saved engineer time from scaling)
api 7y ago
This has been a repeated point since "enterprise" Java (a.k.a. Jabba) became a thing in the late 90s and early 2000s. A ton of enterprise code is comically inefficient and held together with scotch tape and used chewing gum. It ends up boiling down to the fact that compute power is a lot cheaper than developer time and really good developers are more expensive and harder to find than inexperienced ones.
ummonk 7y ago
Yes, I’m always shocked by just how much performance overhead most languages have compared to C and similar lower level languages. It is a price worth paying for better language ergonomics, but I do wonder whether Rust might be able to give us the best of both worlds here.
- _bxg1 7y ago
  He did say that Java/C# are also up there, and Go is in that family, so it probably remains the best balance for lots of cases. I do think there's also territory to be explored writing hot paths in Rust and interoping from Python/JS.
- twic 7y ago
  I semi-seriously think the entire modern shape of the cloud is a result of Ruby being really slow. Back when people were writing their backend business apps in C++, COBOL, Java, etc, if there was ever a performance problem, you could usually just get a slightly bigger machine and grow your thread pools a bit. But once the web took off and Ruby exploded onto it, you couldn't do that, because it's an order of magnitude slower, and doesn't really do multithreading. But, as long as you follow twelve-factor discipline, it scales horizontally like a champ. So, we took to horizontal scaling over multiple VMs (and caching things in Redis instead of local memory or Hazelcast or whatever), and that's been the unquestioned way to do scaling ever since.
  - celticmusic 7y ago
    I don't necessarily think you're right here, but I do know the number of horror stories that have come out of Heroku over the years certainly validates the opinion. I remember reading about their routing debacle and realizing just how much work went into trying to get ruby to scale.
  - ww520 7y ago
    The push for the need of scaling out started with Ruby and Python's lack of performance. The reason being pushed at the time was, "developer time was more expensive than hardware." Well, that didn't count the amortization of developer time over the lifetime of the product once the product was developed.
rossmohax 7y ago
Carmack is moving to AI and inevitably has to deal with a lot of Python, which still bottlenecks process here an there despite all the effort to move computation to C extensions. I have a really high hope that he detours a bit and creates very-very good non-python tooling for ML.
- ospider 7y ago
  Maybe he could make Python fast once and for all :p
  - stefano 7y ago
    Python has some fundamental language semantics that make it really hard, if not impossible, to create an implementation that can match Java. Pypy is probably the best you can do to optimize python, and it shines in tight numerical loops, but it gets less effective as code gets more complex.
zem 7y ago
vibe.d is well worth a look: https://vibed.org/ https://vibed.org/
- zerr 7y ago
  Isn't D dead practically? https://news.ycombinator.com/item?id=21902953 https://news.ycombinator.com/item?id=21902953
  - kal31dic 7y ago
    And yet dicebot continues to work in D, including for me for a while. I'm hiring 25 D programmers, so I suppose it very much depends on what you mean by practically!
    - zerr 7y ago
      What company is it?
  - zem 7y ago
    I have watched the ocaml community go through a renaissance when for a while it looked like it was moribund, and the D community looks like it is developing the same undercurrent of momentum. given that D is a great language and the implementation looks solid, I don't think it is in danger of dying.
    - zerr 7y ago
      Ocaml always enjoined institutional support such as from Inria.
  - vips7L 7y ago
    I really like a lot of the ideas in D, but I feel like the language suffered from having too many ideas/features.
IndrekR 7y ago
A site for proof. It keeps amusing me on what hardware/software Stack Overflow/Stack Exchange is running on: https://stackexchange.com/performance https://stackexchange.com/performance This is way less in HW than most people in the trade (from web devs to devops) seem to think when asked about it. SO ranks #36 in Alexa right now: https://www.alexa.com/siteinfo/stackoverflow.com https://www.alexa.com/siteinfo/stackoverflow.com
- panpanna 7y ago
  Note that they use C# and ASP.Net... (SO and DailyWTF are very much into Microsofts ecosystem)
  - tyuioyjj 7y ago
    AFAIK they're using ASP.NET Core There's significant difference between ASP.NET and ASP.NET Core https://meta.stackexchange.com/questions/316278/the-road-to-net-core-please-help-stack-exchange-test-ef-core https://meta.stackexchange.com/questions/316278/the-road-to-...
    - Merad 7y ago
      They use Core today, but SO was using .Net long before Core existed.
  - FpUser 7y ago
    Anything wrong about that?
    - panpanna 7y ago
      No, I'm just pointing out that they are different from your average SV company.
      - pjmlp 7y ago
        However they are quite average regarding typical European corporate stacks, either Java or .NET.
bsenftner 7y ago
I'm 15 years in writing high performance Internet servers in C++, and I can confirm higher level languages provide an illusion of capability, but once you're talking high performance with high compute requirements and scaling your service, the cost efficiency of C++ is exponential better than any other language. The higher level language ecosystems are bloated beyond repair. I was able to use one 32-core physical server running a C++ http server I wrote providing a rich media web service and replace an AWS server stack that cost my client 120K per month. The client purchased one $8K 32-core server and co-located it behind a firewall at a cost of $125 per month. And the C++ server ran at 30% utilization, plenty of room for user growth. Their AWS stack of a dozen C#, Python, PHP and Node apps were peaking their capacity too. If course, my solution caused existential questioning by the non-geek CEO and the CTO, but they were in crisis and needed to radically revise how they provided their service or close.
- willhslade 7y ago
  This sounds like an amazing war story that I just want to hear more of. Is there any more? What's your c++ stack like?
  - bsenftner 7y ago
    Currently using Restbed (https://github.com/Corvusoft/restbed https://github.com/Corvusoft/restbed) as the server core, wxWidgets as a server side gui, with Boost, Curl, SQLite and Standard Lib. It's not that complex, beyond using lambdas in a few places. It has extremely high performance, and can run on an Intel Compute Stick, but I tend to use an Intel Nuc at minimum, with clients typically using whatever they have, gaining over redundancy and an ability to pair down. The memory management in a C++ application is just another resource one manages with whatever level of algorithm support you feel comfortable. There are ref-counting systems and complete garbage collectors available one can integrate into their business logic, unlike in a high level language that "transparently" manages memory outside application control. Have you ever thought about how much processing a typical 3D video game performs every frame? What if that caliber of optimized algorithm logic were handling a rich media, non-3D game server hosted business application? It would have pretty amazing performance and scale very economically. That's what I do. Before doing this, I wrote 3D video games and their production environments.
twic 7y ago
What's that, Frank McSherry? For horizontally scaled systems, we should ask what the Configuration that Outperforms a Single Thread is? https://blog.acolyer.org/2015/06/05/scalability-but-at-what-cost/ https://blog.acolyer.org/2015/06/05/scalability-but-at-what-...
- Silhouette 7y ago
  IMO, the linked article is much more insightful than the flippant comment here might suggest. It's a solid argument, backed by real world data, about how easy it is to make bad assumptions equating better scalability with better performance.
amelius 7y ago
The catch is: until you can't.
buboard 7y ago
people will bike to work to "save the environment" but won't use C direct on metal to reduce the carbon footprint of their code. For a guy like Carmack, it may be quite frustrating working with the constraints of pytorch etc. He ll probably end up making his own pytorh frontend in C, which as a bonus people will use to deploy models.
- Aperocky 7y ago
  I don’t think it’s that big of an language issue. Java isn’t that much slower (logarithmically speaking). But design choices have arbitrarily incorporated huge amount of bloat and inefficiency.
  - buboard 7y ago
    yep, logarithmic. Well, the question is what's the motive about those design choices? Today's e.g. web app choices seem to be basically aesthetic with an eye for extreme novelty, and total disregard for how many times a buffer is being copied back and forth before reaching the user.
    - Aperocky 7y ago
      That is compared to python/ruby which would be multiple logs slower or scaling choices that makes the service resemble the microkernels. Usually dedicated, simple engineering can be very fast using either C or java
- FpUser 7y ago
  Hypocrisy is not such a rare animal. There is however extra factor here: biking to work if possible is your personal choice. Using your favorite tools at work: often not as much
  - buboard 7y ago
    probably not so much hypocrisy as lack of consideration. Even though the energy benefit from making apps e.g. twice as fast isn't so big, it's worth giving a try for the time savings.
tyuioyjj 7y ago
What if those "scalable" systems were designed to "just work" and then after it succeed it was changed into pseudo ""scalable""?
fnord77 7y ago
wow, so much speculation, zero empirical evidence.
- nbevans 7y ago
  His entire comment was based on empirical evidence. Which is why people are disputing it - because there is room to do so.
arcticbull 7y ago
I'm not sure there's any such thing as a simple C++ server ;)
lmm 7y ago
Horizontal scalability carries a lot of overhead. Probably a factor of 10, easily. But the clue is in the name: eventually you'll get to a point where you have to scale. Back in 2010 I worked for a company whose system, in Java, ran on a single web server (with one identical machine for failover). We laughed at our nearest rivals, who were using Ruby, and apparently needed 60(!) machines to run their system, which had about 5x the average request latency of ours. Then traffic doubled, and suddenly we were having to buy six-figure servers and carefully performance-optimize our code, and our rivals with the 120 Ruby servers didn't look so funny any more. And then traffic doubled again.
- steev 7y ago
  This sounds like an architecture problem, not a language problem. Can you elaborate?
  - lmm 7y ago
    Of course it's possible to write a horizontally scaled application in Java or C++. But once you have to deal with horizontal scaling anyway, language performance is much less of an advantage: as Carmack says, the difference between 100 servers and 10 is just accounting.
    - betaby 7y ago
      And the difference between 1000 servers and 100 is not just accounting. In essence, 1 full rack is easy to reason about while N>1 is not, for myriads of non accounting reasons.
    - adrianN 7y ago
      The Internet uses on the order of 10% of the world's electricity. A 10x difference is huge.
      - jsty 7y ago
        Out of curiosity - what's the source on that? Most figures I've seen put datacenter usage an order of magnitude lower: e.g. https://www.iea.org/reports/tracking-buildings/data-centres-and-data-transmission-networks https://www.iea.org/reports/tracking-buildings/data-centres-... "Global data centre electricity demand in 2018 was an estimated 198 TWh, or almost 1% of global final demand for electricity (Masanet et al., 2018)."
coderunner 7y ago
What about in the context of trying to get to an MVP? Is the dev time speedup of using a dynamic programming language and stack significant over using a c++ backed? You wouldn't care much about performance when you're trying to figure out if you'll get traction.
- otabdeveloper4 7y ago
  No, it's not significant. Dev time depends on programmer skill, not the toolset. A good C++ programmer will develop your MVP many times faster than an average Python programmer. Python programmers are much easier to hire, though - you already need a good C++ programmer on the team to hire another one, because HR and corporate management can't into proper hiring process. This last factor is the overarching most important one for BigCorp Enterprise Inc., not development speed or cost.
  - stefano 7y ago
    The same argument can be made for Python (or for most non-trivial jobs, for that matter). You always need a good technical person to hire another one.
    - otabdeveloper4 7y ago
      Not really; hiring Python and Java devs is very amenable to keyword-driven recruitment. Hiring C++ devs in the same manner is a clusterfuck waiting to happen.
  - filleokus 7y ago
    > Dev time depends on programmer skill, not the toolset. This is obviously not strictly true, always. A skilled programmer will use the proper tools for the job. If you for example is tasked with writing a backend service exposing a GraphQL API, I think it would be foolish to do this in C++, and would bet that the average Python programmer would do it quicker than even a top-tier C++ programmer (if the latter would be hellbent on doing it in C++). Especially when working with MVP's (or new projects in general), the ability to leverage already existing tools and frameworks are key to rapid progress. This doesn't necessarily have to be scripting languages, but the Python/Node/Go/etc developer would have a working GraphQL server up and running connected to a database of choice within an afternoon while the skilled C++ developer would have to spend at least a few days implementing a GraphQL server mostly from scratch [0]. [0]: A quick Google show that schema parsers exists for C++, but nothing matching the frameworks/library available for more web-fashionable languages.
sriram_malhar 7y ago
This is precisely the point made by McSherry, Isard and Murray in their lovely paper, "Scalability! But at what COST?" (Usenix HotOS '15). They demonstrate how much performance headroom there is in modern CPU and memory, and show how simple cache-sensitive batch algorithms running on a single core can outperform hundreds of cores running distributed map-reduce style jobs. https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos...
- wiradikusuma 7y ago
  But nobody in BigCo(tm) would do that, because everyone (including the executives) involved want to have Hadoop/BigData(tm) in their resume. /sarcasm
  - tlb 7y ago
    No doubt there are people who do it for cynical reasons. But at least some people do it sincerely thinking it’s the right choice. It’d be more interesting to talk about them, and how they came to make the wrong decision for what they thought were the right reasona.
    - asdfasgasdgasdg 7y ago
      I think it's a bit presumptuous to say that the decision is "wrong". That paper demonstrated that a single server can outperform a small server farm on a toy problem. Nobody, not even google, solves pagerank in production as a batch job. Real problems are often more complex.
      - tluyben2 7y ago
        For many workloads, companies have tech leads and CTOs absolutely overarchitecting their stack. If you can run your entire system off of 4 loadbalanced $200/mo servers, why am I seeing hadoop/kafka/kubernetes/etc running on $10k+/mo premoney, so paid with investor money? Sure there are a lot of cases where this is fine, but I would say (from real life) that there are far more cases where this was a pretty poor choice. Usually proven to be even poorer when the tech lead leaves and no-one seemingly having a clue how it all works together, despite the whole docker/kubernetes/ci stories peddled to management. That is usually when I get asked to take a look. In my experience these problems are definitely not always toy problems while some of them can, easily, be ran on 1 server for the expected lifespan of the company because the company will never get that much users/clients/data even though immensely profitable. Not everyone (almost no one) is a FAANG and I find it offensive to use the company money (which can be investor money) to realise the wet dreams of the cto while the case for that architecture and the cost of it, makes 0 sense to the business and bottomline. Obviously in life, things are nuanced and differ case by case but nah, real world problems are, generally, not more complex. They are more complex in some (rare) cases, like the one you named. But most companies are not doing anything like that, but they do have their cto’s gearing up for that incredibly unlikely future.
anovikov 7y ago
My good friend built whole career doing exactly the same. "Replace a cluster of 10 Elasticsearch servers with 1 running a custom built C app and in-memory database". Of course, it won't work out to replace 1000 Elasticsearch servers - that's where the advantage of a true "big data" tool will show - but none of the clients really have data "that big".
jacobsenscott 7y ago
You can choose a language that optimizes your hardware, or you can choose a language that optimizes your programmers. 99% of the time optimizing the programmers is the right call.
- buzzkillington 7y ago
  If you're using anything Hadoop related you're not optimizing programmers.
- PeCaN 7y ago
  this is the philosophy that brought us software like atom and slack
  - jacobsenscott 7y ago
    Correct. Two very successful products.
- otabdeveloper4 7y ago
  Yes, if by "optimizing programmers" you mean "optimizing the manager's corporate structure footprint and bonus incentives". If the choice is between hiring one good C++ programmer or 15 really dumb Python "backend engineers" (and a team of QA's and sysadmins to support them), what do you think your pointy-hared corporate boss would chose?
  - jacobsenscott 7y ago
    You never choose to hire dumb programmers. The choice is between 15 good cpp programmers and 4 good python programmers.
    - buzzkillington 7y ago
      You have not had to interview the python devs I've had to interview. Last round, for the second best candidate who we hired at 130% the salary we initially offered, after the best candidate was snatched under our nose for what we were told was twice the salary we were offering: >"Last question, I saw that you wrote a 100 line function here, is this because you ran out of time?" >>"No. I don't like to break up my functions. Having many small functions is confusing and bad practice." >What am I doing with my life?
easytiger 7y ago
Been saying this for years. Modern SW trends are horrific
gregdoesit 7y ago
From this thread, I feel a lot of people misunderstand what a scaleable system means for a scale-up startup or BigTechCo. It doesn’t mean cost-efficiency. It means the ability to solve scalability issues at 10x, 100x, 1000x load by throwing money at it (aka buying more instances/machines). Yes, John Carmack is right that a bunch of scaleable systems that see reasonable load, with well-understood traffic requirements could be rewritten more efficient ways. But how long would it take? How much would the extra work cost? And what would happen if traffic goes up, to another 10x, 100x? Can you throw still throw money at that problem, with this new and efficient system already in-place? One of my memorable stories comes from an engineer I worked with when we rewrote our payments system[1]. He told me how at a previous payments company, they had a system that was written in this kind of efficient way, and needed to run on one machine. As the scale went up, the company kept buying bigger hardware, at one point buying the largest commercially available mainframe (we’re talking the cost being in the multi-millions for the hardware). But the growth was faster and they couldn’t keep up. Downtime after downtime followed at peak times, making hundreds of thousands of losses per downtime. They split into two teams. Team #1 kept making performance tweaks on the existing hardware to try to get performance wins, and increase reliability during peak load. Team #2 rewrote their system in a vertically scaleable way. Team #1 struggled and the outages kept getting worse. Team #2 delivered a new system quickly and the company transitioned their system over. What they gained was not cost savings: it was the ability to (finally!) throw money at their growth problems. Now, when traffic went up, they could commission new machines and scale with traffic. And they could eventually throw away their mainframe. Years later, they started to optimise the performance of the system, making millions of $ in savings. The new system - like the old one - cost millions to run. But that was besides the point. Finally, they stopped bleeding tens of millions in lost revenue per year due to their inability to handle sudden, high load. It’s all about trade-offs. Is cost your #1 priority and are you okay spending a lot of development time on optimising your system? Go with C++, Erlang or some other, similarly efficient language. Is product-market fit more important, with the ability to have high uptime, that you can buy? Use the classic, horizontally scaleable distributed systems stack, and worry about optimising later, when you have stable traffic and optimisation is more profitable over product development. [1] https://blog.pragmaticengineer.com/distributed-architecture-concepts-i-have-learned-while-building-payments-systems/ https://blog.pragmaticengineer.com/distributed-architecture-...
useful 7y ago
PCI-e 4.0 and 5/6 enables consuming absolutely insane amounts of data. I can't think of many applications that need that. 128GB/sec? sure! and 256GB/sec is coming. With 10Gb, 40Gb and 100Gb internet links becoming mainstream the roadblock to having cool stuff is developers to make it.
- PedroBatista 7y ago
  Hold my beer, I’m firing up my [ insert dev stack ].
qaq 7y ago
Keep in mind a modern commodity x86 server with 128 physical cores 4 TB RAM and decent amount of SSD storage and dual 100 gbe nics is about 70K. Ability to use something like Rust also changes equation significantly.
- eb0la 7y ago
  I believe the key is able to use something _native_, not interpreted code or bytecode, running on a virtual machine, which runs in a userspace process inside a virtualized server inside a bare metal server. Said that, I believe Rust has a great future in data processing. Just take a look at Apache Arrow Rust bindings and Ballista (https://news.ycombinator.com/item?id=20456273 https://news.ycombinator.com/item?id=20456273)
cryptica 7y ago
Scalability and performance are very different. Opting for a performant but not scalable solution is basically an acknowledgement that: - The project will only succeed up to a certain point. - After the initial launch, no further changes will be made to the project since new additions are likely to cost performance and lower the upper bound on how many users the system can support. Not many projects are willing to accept either of these premises. Nobody wants to set an upper bound on their success. Also, it should be noted that no language is 'much faster' than any other language. Benchmarks which compare the basic operations across multiple languages rarely find more than 50% performance difference. The more significant performance differences are usually caused by implementation differences in more advanced operations and in any given language, different libraries can offer different performance characteristics so it's not fair to say that a language is slow because some of its default advanced operations are slow. Usually, performance problems come down to people not choosing the best abstract data type for the problem. Some kinds of problems would perform better with linked list, others perform better with arrays or maps or binary trees. Time complexity of any given algorithm is way more significant than the baseline performance of the underlying language.
lonelappde 7y ago
A single server cannot be made reliably available, though. At single-node scale, development costs dwarf hardware costs of scaling horizontally.
ungerik 7y ago
This is why Google developed Go. Python like productivity with C++ like performance...
rbanffy 7y ago
OTOH, we can assume one doesn't need to worry about using C++ unless they have many millions of users.

398 points by Impossible 7y ago

I've been programming C++ and assembly for 23 years. Few years ago I became a huge fan of Python. In my opinion Python is amazingly well suited for rapid first revision and can then be swapped out for C++ / asm.

davidcbc 7y ago
This is fine as long as you can convince management to spend the money to rewrite your software. That's usually a hard sell though. In my experience this plan usually ends up with a python monstrosity that everyone hates but is forced to deal with forever.
- j88439h84 7y ago
  Type hints and dataclasses are a game-changer for Python. Much easier to reason about programs that use them.
  - adev_ 7y ago
    > Type hints and dataclasses are a game-changer for Python. Much easier to reason about programs that use them. Type hints made large python codebase switch from "Any change will blow up to my face" to "I can touch it carefully with protection gloves". Through, it is still pretty easy to fool mypy and very far of the compile time guarantees that provide most static typed AOT compiled language. And unfortunately, it also does not bring any advantage in term of performance.
  - insulanian 7y ago
    > Type hints... At that point it makes more sense to use a statically typed language.
- kitsuac 7y ago
  I'm speaking in terms of the reality of what is effective in development. Not in terms of what management at particular companies will approve.
Fazel94 7y ago
You just have to write a tiny part that uses a lot of CPU in C++/asm or anything else. Much of code's performance isn't really reflected on to the scalability since mostly a tiny part of code is really ran a lot of time, and the other parts are just glues or management stuff or rarely used(not used in scale) features.
- kitsuac 7y ago
  It depends. You aren't going to make a very fast modern codec encoder or decoder using Python. The hotspot ends up being the vast majority of the process. That management/glue layer becomes very thin, amounting only to feeding in the bitstream and reading back the raw video frames.

fatbird 7y ago

Has Carmack looked at Rust? I'd be very curious for someone like him to look at it in light of his experience with C/C++ and high performance systems.

Ygg2 7y ago
Pretty sure he's aware of it. And honestly I see the logic, Rust has been top of Tech Empower Benchmarks for a while. That said, I think the benchmarks could use some tweaking. I'm not sure they are representative.
- throwthisaway2 7y ago
  Those benchmarks matter little. If by server this topic is a webserver the backend scripts do little. Its your database, caching, and disk manipulation that slows down performance. Most people write bad queries or have a shallow caching strategy. A router redirecting to a controller and parsing a template is hardly different across languages that use the same queries. If you truly need speed for post processong the off load on a c script like queries are off loaded.
jsty 7y ago
He's had a look: https://www.reddit.com/r/rust/comments/ap2047/john_carmack_writing_rust_code_feels_very/ https://www.reddit.com/r/rust/comments/ap2047/john_carmack_w...
- pizlonator 7y ago
  Yeah but I didn’t see anything about perf there. I think it’s a valid point that the reasons why C++ is good for perf - compiled, statically typed, low-level, manual memory management, high degree of control over memory layout - would also apply to Rust. If they don’t then that would really call into question positive claims being made about Rust.
pizlonator 7y ago
Good question, not sure why you were voted down. And I’m not even a Rust fan.

slx26 7y ago

the title is misleading. sure, it advocates for C++, or Java, or C#, or others above Python, but clarifying the context: "a lot (not all!) of complex “scalable” systems can be done with a simple, single C++ server.", which I take as: "sometimes it's better to write a simple server in a low level language, than a complex server in a higher level language".

dang 7y ago
Submitted title was "John Carmack advocates C++ for server development". We've replaced it with a shortened version of what he said.

kyledrake 7y ago

The analogy here is one of the best ice climbers in the world proposing an ascent of the Matterhorn. Please use testable, easy to prototype with, memory managed languages for production servers unless you are solving a very specific problem and really know what you are doing.

mberning 7y ago
Very apt. I think people are also forgetting “the bad old days” where you worked on a large-ish cpp or java app for a year, nothing worked right, schedules slipped, and then the whole thing was scrapped and teams disbanded to work on other stuff. That was very common. You can’t count on having a team of Carmacks work on your blub app.
- otabdeveloper4 7y ago
  Replace "cpp or java" with "Python or Ruby" and how is it any different today?

BalinKing 7y ago

From the next tweet in the thread: “JAVA [sic] or C# would also be close, and there are good reasons to prefer those over C++ for servers.”

paulddraper 7y ago
Yep. C++ is the nuclear option, when you know you need to keep memory reasonable, or capture the last bit of compute performance. Neither of these are usually the case for web application servers.

fxtentacle 7y ago

Many people don't know about Python's GIL https://wiki.python.org/moin/GlobalInterpreterLock https://wiki.python.org/moin/GlobalInterpreterLock That's the reason why you need to go multi-process if you want to reach a similar level of concurrency in Python as multi-thread in C++. And that surely adds a lot of complexity. As a very practical example of this, TensorFlow has a dedicated page with advice on how to make the Python part that reads the files from disk less slow. Think about that: The bottleneck for training a highly advanced AI with millions of parameters is in the 20 lines of Python code to read in a binary file... https://www.tensorflow.org/guide/data_performance https://www.tensorflow.org/guide/data_performance

lewis1028282 7y ago
Not really IO is always going to be slow compared to the CPU. The Python interpreter will be blocked 90% of that time waiting for IO anyway.
- SaxonRobber 7y ago
  Percentages are misleading, what matters is the actual time it takes. Even small gains will have an effect as you increase the traffic and add additional stages to the system. Small delays accumulate and can have a surprising effect on queue sizes and latency.
- postpawl 7y ago
  Exactly, and I’m surprised none of the other comments mentioned this so far. Often web app endpoints are bottlenecked by IO because they’re spending most of their time talking to a database or cache server. Python is probably not the right tool for a CPU intensive endpoint that needs to serve up hundreds of thousands of requests per minute and can’t be cached. There are a lot of ways to handle the IO intensive scenario in python: * Threading - Works with python libraries written in C, but now you need to add locks to your code to prevent race conditions. Not good for CPU heavy work because it’s switching context every ~100 instructions. * Gevent - Requires minimal code changes, but it usually blocks when running code from python libraries written in C. This uses event loop style concurrency and automatically patches internal python libraries to switch context when IO occurs. This means less time spent unnecessarily switching context, so it can scale better than python’s threading. * multiprocessing - Better for CPU intensive work, but requires more memory than other solutions. You don’t need to worry about race conditions since the processes are separate. * asyncio - Requires code changes and using compatible libraries. It does event loop style concurrency that allows you to specify specifically when to switch context. Like with gevent, this means less time spent unnecessarily switching context. * ...And there are probably a bunch of other ways I’m missing. I’ve heard that this sort of stuff is simpler in Go.

blondin 7y ago

not how i read his tweet. he was lamenting that python is too slow for some server side development use cases. and he gave cpp as an alternative that would be simpler and faster. he even followed up citing java and csharp. totally agree. if all your backend server is mostly complex serialization & de-serialization, and pushing bytes to other sub-systems, i think many other languages have advantages over python. wondering why no mention of go or rust though...

rvz 7y ago

> ...My bias is that a lot (not all!) of complex “scalable” systems can be done with a simple, single C++ server. The second tweet of the discussion: > JAVA or C# would also be close, and there are good reasons to prefer those over C++ for servers. Many other languages would also be up there, the contrast is with Really Slow (but often productive and fun) languages. I'm afraid that Carmack has sided with his own anecdotal experiences of the 1990s to justify the use of C++ for server-side development for the 21st Century. This probably made sense at the time due to the availability of more C++ devs and less language choices, but today in the 2020s? I remain totally unconvinced by his argument. He goes on to suggest Java or C# which still makes sense for many companies for generic server-side development if you are after a more secure backend. Kotlin is pretty much the most sensible for this. But for the sake of Carmack's engineering background however, it is unsurprising why Java/C#/Kotlin are technically unsuitable for high-performance gaming platforms if one was to create one. So what credible languages could be used to compete with C++? I hear Discord is having a great time with using Elixir (Erlang could also be used) and another gaming platform called 'Hadean' is using Rust for their platform.

jariel 7y ago
"But for the sake of Carmack's engineering background however, it is unsurprising why Java/C#/Kotlin are technically unsuitable for high-performance gaming platforms if one was to create one." This really is just not true. Financial, real-time style High Frequency Trading apps are often written in Java - not C++. Much of the JVM is not a VM, it compiles to machine code - in an optimised manner. For starters. Given how difficult it is to develop safely in C++ I can hardly think of a reason to ever use it on the backend.
- SaxonRobber 7y ago
  This and more opinions from the oracle certified java guru.
  - kitd 7y ago
    Which part are opinions?

jandrewrogers 7y ago

Many developers severely underestimate how much workload can be served by a single modern server and high-quality C++ systems code. I've scaled distributed workloads 10x by moving them to a single server and a different software architecture more suited for scale-up, dramatically reducing system complexity as a bonus. The number of compute workloads I see that actually need scale-out is vanishingly small even in industries known for their data intensity. You can often serve millions of requests per second from a single server even when most of your data model resides on disk. We've become so accustomed to extremely inefficient software systems that we've lost all perspective on what is possible.

pleasecalllater 7y ago
Two. The number is two. You always need a backup server :)
- twic 7y ago
  I used to think that. But there are a couple of notable exceptions at either end of the latency spectrum. If your latency requirements are slack, then you can get away with one machine, because you can reboot or reprovision it and carry on processing without meet your requirements. If your latency requirements are tight, you don't have time to fail over anyway, so you might as well run one machine and make sure you can deal with failing to meet your requirements.
  - chucky_z 7y ago
    This is why I ran Redis on a single server with no failover strategy. Our team spent maybe 15 hours workshopping a handful of other things (redis cluster, sentinels + replicas) before realizing we could spend an hour to have everything sit in a degraded state working around it while Redis itself was fixed. Redis only failed once ever and it took all of an hour to fix it, all during non-peak hours.
- dboreham 7y ago
  Three. So you can do maintenance on one while still having HA.

8fingerlouie 7y ago

As someone who has implemented a complex system in C++ in this decade, I’d say he’s not wrong, but you need to carefully weight the pros and cons. In our case latency and real time demands mattered a lot (NASDAQ feed parser), to the point of the (potential) slowdown of a garbage collector kicking in was enough to rule out Java and .NET. It runs entirely in memory and on 64+ cores. We implemented our own reference counting system to keep some of our sanity, and to at least try to avoid the worst memory leaks. This was an edge case, and for almost everything else you’re probably better off implementing it in something that handles memory for you. If performance is an issue, least try it in Go or Rust with a quick PoC before jumping the C++ wagon.

choeger 7y ago
But aren't the edge cases the things that satisfy your pay? The standard cases of today are the introductory examples of tomorrow and will be automated or abstracted away by the end of next week. In JavaScript.
CoolGuySteve 7y ago
If you're parsing market data then you shouldn't really be allocating at all once the initial setup is complete. So there shouldn't be a need for ref counts. This is because the allocator itself can have an unbounded runtime that takes milliseconds, causing you to drop. In the past I've replaced malloc with an implementation that asserts if called on certain threads after init time.
- tom_ 7y ago
  Milliseconds??! What causes that to occur? Though I suppose if they can take that long, avoiding them is an even better idea than it is normally... For constant time allocations, try TLSF: http://www.gii.upv.es/tlsf/ http://www.gii.upv.es/tlsf/
  - cogman10 7y ago
    I'm not the op, but I'm guessing large amounts of fragmented memory is what causes that. It's one benefit of a gced language with compaction, allocations are typically bounded (except when they trigger a gc).

wruza 7y ago

Morals of this story: 1. Always use the most performant language available to you. (What if your program gets a few million users?) 2. Horizontal scalability is too much complexity/work. Just apply the correct amount of optimization when you initially write the code. If you, like me, read this on mobile and did not click [more answers] link under that, then do it. That may save you a minute or two of derealization time.

choeger 7y ago
I really had a hard time to decide whether I agree with that statement or not. If you design a whole service it is obviously not true. But if you develop something like a specialized Backend, say a database, you might want to reconsider the complexity of horizontal scalability.
Aperocky 7y ago
2 Is almost no longer true because there are simple templates you follow that will scale you till infinity with cloud (which also means your cost is going to scale, but hey we saved engineer time from scaling)

api 7y ago

This has been a repeated point since "enterprise" Java (a.k.a. Jabba) became a thing in the late 90s and early 2000s. A ton of enterprise code is comically inefficient and held together with scotch tape and used chewing gum. It ends up boiling down to the fact that compute power is a lot cheaper than developer time and really good developers are more expensive and harder to find than inexperienced ones.

ummonk 7y ago

Yes, I’m always shocked by just how much performance overhead most languages have compared to C and similar lower level languages. It is a price worth paying for better language ergonomics, but I do wonder whether Rust might be able to give us the best of both worlds here.

_bxg1 7y ago
He did say that Java/C# are also up there, and Go is in that family, so it probably remains the best balance for lots of cases. I do think there's also territory to be explored writing hot paths in Rust and interoping from Python/JS.
twic 7y ago
I semi-seriously think the entire modern shape of the cloud is a result of Ruby being really slow. Back when people were writing their backend business apps in C++, COBOL, Java, etc, if there was ever a performance problem, you could usually just get a slightly bigger machine and grow your thread pools a bit. But once the web took off and Ruby exploded onto it, you couldn't do that, because it's an order of magnitude slower, and doesn't really do multithreading. But, as long as you follow twelve-factor discipline, it scales horizontally like a champ. So, we took to horizontal scaling over multiple VMs (and caching things in Redis instead of local memory or Hazelcast or whatever), and that's been the unquestioned way to do scaling ever since.
- celticmusic 7y ago
  I don't necessarily think you're right here, but I do know the number of horror stories that have come out of Heroku over the years certainly validates the opinion. I remember reading about their routing debacle and realizing just how much work went into trying to get ruby to scale.
- ww520 7y ago
  The push for the need of scaling out started with Ruby and Python's lack of performance. The reason being pushed at the time was, "developer time was more expensive than hardware." Well, that didn't count the amortization of developer time over the lifetime of the product once the product was developed.

rossmohax 7y ago

Carmack is moving to AI and inevitably has to deal with a lot of Python, which still bottlenecks process here an there despite all the effort to move computation to C extensions. I have a really high hope that he detours a bit and creates very-very good non-python tooling for ML.

ospider 7y ago
Maybe he could make Python fast once and for all :p
- stefano 7y ago
  Python has some fundamental language semantics that make it really hard, if not impossible, to create an implementation that can match Java. Pypy is probably the best you can do to optimize python, and it shines in tight numerical loops, but it gets less effective as code gets more complex.

zem 7y ago

vibe.d is well worth a look: https://vibed.org/ https://vibed.org/

zerr 7y ago
Isn't D dead practically? https://news.ycombinator.com/item?id=21902953 https://news.ycombinator.com/item?id=21902953
- kal31dic 7y ago
  And yet dicebot continues to work in D, including for me for a while. I'm hiring 25 D programmers, so I suppose it very much depends on what you mean by practically!
  - zerr 7y ago
    What company is it?
- zem 7y ago
  I have watched the ocaml community go through a renaissance when for a while it looked like it was moribund, and the D community looks like it is developing the same undercurrent of momentum. given that D is a great language and the implementation looks solid, I don't think it is in danger of dying.
  - zerr 7y ago
    Ocaml always enjoined institutional support such as from Inria.
- vips7L 7y ago
  I really like a lot of the ideas in D, but I feel like the language suffered from having too many ideas/features.

IndrekR 7y ago

A site for proof. It keeps amusing me on what hardware/software Stack Overflow/Stack Exchange is running on: https://stackexchange.com/performance https://stackexchange.com/performance This is way less in HW than most people in the trade (from web devs to devops) seem to think when asked about it. SO ranks #36 in Alexa right now: https://www.alexa.com/siteinfo/stackoverflow.com https://www.alexa.com/siteinfo/stackoverflow.com

panpanna 7y ago
Note that they use C# and ASP.Net... (SO and DailyWTF are very much into Microsofts ecosystem)
- tyuioyjj 7y ago
  AFAIK they're using ASP.NET Core There's significant difference between ASP.NET and ASP.NET Core https://meta.stackexchange.com/questions/316278/the-road-to-net-core-please-help-stack-exchange-test-ef-core https://meta.stackexchange.com/questions/316278/the-road-to-...
  - Merad 7y ago
    They use Core today, but SO was using .Net long before Core existed.
- FpUser 7y ago
  Anything wrong about that?
  - panpanna 7y ago
    No, I'm just pointing out that they are different from your average SV company.
    - pjmlp 7y ago
      However they are quite average regarding typical European corporate stacks, either Java or .NET.

bsenftner 7y ago

I'm 15 years in writing high performance Internet servers in C++, and I can confirm higher level languages provide an illusion of capability, but once you're talking high performance with high compute requirements and scaling your service, the cost efficiency of C++ is exponential better than any other language. The higher level language ecosystems are bloated beyond repair. I was able to use one 32-core physical server running a C++ http server I wrote providing a rich media web service and replace an AWS server stack that cost my client 120K per month. The client purchased one $8K 32-core server and co-located it behind a firewall at a cost of $125 per month. And the C++ server ran at 30% utilization, plenty of room for user growth. Their AWS stack of a dozen C#, Python, PHP and Node apps were peaking their capacity too. If course, my solution caused existential questioning by the non-geek CEO and the CTO, but they were in crisis and needed to radically revise how they provided their service or close.

willhslade 7y ago
This sounds like an amazing war story that I just want to hear more of. Is there any more? What's your c++ stack like?
- bsenftner 7y ago
  Currently using Restbed (https://github.com/Corvusoft/restbed https://github.com/Corvusoft/restbed) as the server core, wxWidgets as a server side gui, with Boost, Curl, SQLite and Standard Lib. It's not that complex, beyond using lambdas in a few places. It has extremely high performance, and can run on an Intel Compute Stick, but I tend to use an Intel Nuc at minimum, with clients typically using whatever they have, gaining over redundancy and an ability to pair down. The memory management in a C++ application is just another resource one manages with whatever level of algorithm support you feel comfortable. There are ref-counting systems and complete garbage collectors available one can integrate into their business logic, unlike in a high level language that "transparently" manages memory outside application control. Have you ever thought about how much processing a typical 3D video game performs every frame? What if that caliber of optimized algorithm logic were handling a rich media, non-3D game server hosted business application? It would have pretty amazing performance and scale very economically. That's what I do. Before doing this, I wrote 3D video games and their production environments.

twic 7y ago

What's that, Frank McSherry? For horizontally scaled systems, we should ask what the Configuration that Outperforms a Single Thread is? https://blog.acolyer.org/2015/06/05/scalability-but-at-what-cost/ https://blog.acolyer.org/2015/06/05/scalability-but-at-what-...

Silhouette 7y ago
IMO, the linked article is much more insightful than the flippant comment here might suggest. It's a solid argument, backed by real world data, about how easy it is to make bad assumptions equating better scalability with better performance.

amelius 7y ago

The catch is: until you can't.

buboard 7y ago

people will bike to work to "save the environment" but won't use C direct on metal to reduce the carbon footprint of their code. For a guy like Carmack, it may be quite frustrating working with the constraints of pytorch etc. He ll probably end up making his own pytorh frontend in C, which as a bonus people will use to deploy models.

Aperocky 7y ago
I don’t think it’s that big of an language issue. Java isn’t that much slower (logarithmically speaking). But design choices have arbitrarily incorporated huge amount of bloat and inefficiency.
- buboard 7y ago
  yep, logarithmic. Well, the question is what's the motive about those design choices? Today's e.g. web app choices seem to be basically aesthetic with an eye for extreme novelty, and total disregard for how many times a buffer is being copied back and forth before reaching the user.
  - Aperocky 7y ago
    That is compared to python/ruby which would be multiple logs slower or scaling choices that makes the service resemble the microkernels. Usually dedicated, simple engineering can be very fast using either C or java
FpUser 7y ago
Hypocrisy is not such a rare animal. There is however extra factor here: biking to work if possible is your personal choice. Using your favorite tools at work: often not as much
- buboard 7y ago
  probably not so much hypocrisy as lack of consideration. Even though the energy benefit from making apps e.g. twice as fast isn't so big, it's worth giving a try for the time savings.

tyuioyjj 7y ago

What if those "scalable" systems were designed to "just work" and then after it succeed it was changed into pseudo ""scalable""?

fnord77 7y ago

wow, so much speculation, zero empirical evidence.

nbevans 7y ago
His entire comment was based on empirical evidence. Which is why people are disputing it - because there is room to do so.

arcticbull 7y ago

I'm not sure there's any such thing as a simple C++ server ;)

lmm 7y ago

Horizontal scalability carries a lot of overhead. Probably a factor of 10, easily. But the clue is in the name: eventually you'll get to a point where you have to scale. Back in 2010 I worked for a company whose system, in Java, ran on a single web server (with one identical machine for failover). We laughed at our nearest rivals, who were using Ruby, and apparently needed 60(!) machines to run their system, which had about 5x the average request latency of ours. Then traffic doubled, and suddenly we were having to buy six-figure servers and carefully performance-optimize our code, and our rivals with the 120 Ruby servers didn't look so funny any more. And then traffic doubled again.

steev 7y ago
This sounds like an architecture problem, not a language problem. Can you elaborate?
- lmm 7y ago
  Of course it's possible to write a horizontally scaled application in Java or C++. But once you have to deal with horizontal scaling anyway, language performance is much less of an advantage: as Carmack says, the difference between 100 servers and 10 is just accounting.
  - betaby 7y ago
    And the difference between 1000 servers and 100 is not just accounting. In essence, 1 full rack is easy to reason about while N>1 is not, for myriads of non accounting reasons.
  - adrianN 7y ago
    The Internet uses on the order of 10% of the world's electricity. A 10x difference is huge.
    - jsty 7y ago
      Out of curiosity - what's the source on that? Most figures I've seen put datacenter usage an order of magnitude lower: e.g. https://www.iea.org/reports/tracking-buildings/data-centres-and-data-transmission-networks https://www.iea.org/reports/tracking-buildings/data-centres-... "Global data centre electricity demand in 2018 was an estimated 198 TWh, or almost 1% of global final demand for electricity (Masanet et al., 2018)."

coderunner 7y ago

What about in the context of trying to get to an MVP? Is the dev time speedup of using a dynamic programming language and stack significant over using a c++ backed? You wouldn't care much about performance when you're trying to figure out if you'll get traction.

otabdeveloper4 7y ago
No, it's not significant. Dev time depends on programmer skill, not the toolset. A good C++ programmer will develop your MVP many times faster than an average Python programmer. Python programmers are much easier to hire, though - you already need a good C++ programmer on the team to hire another one, because HR and corporate management can't into proper hiring process. This last factor is the overarching most important one for BigCorp Enterprise Inc., not development speed or cost.
- stefano 7y ago
  The same argument can be made for Python (or for most non-trivial jobs, for that matter). You always need a good technical person to hire another one.
  - otabdeveloper4 7y ago
    Not really; hiring Python and Java devs is very amenable to keyword-driven recruitment. Hiring C++ devs in the same manner is a clusterfuck waiting to happen.
- filleokus 7y ago
  > Dev time depends on programmer skill, not the toolset. This is obviously not strictly true, always. A skilled programmer will use the proper tools for the job. If you for example is tasked with writing a backend service exposing a GraphQL API, I think it would be foolish to do this in C++, and would bet that the average Python programmer would do it quicker than even a top-tier C++ programmer (if the latter would be hellbent on doing it in C++). Especially when working with MVP's (or new projects in general), the ability to leverage already existing tools and frameworks are key to rapid progress. This doesn't necessarily have to be scripting languages, but the Python/Node/Go/etc developer would have a working GraphQL server up and running connected to a database of choice within an afternoon while the skilled C++ developer would have to spend at least a few days implementing a GraphQL server mostly from scratch [0]. [0]: A quick Google show that schema parsers exists for C++, but nothing matching the frameworks/library available for more web-fashionable languages.

sriram_malhar 7y ago

This is precisely the point made by McSherry, Isard and Murray in their lovely paper, "Scalability! But at what COST?" (Usenix HotOS '15). They demonstrate how much performance headroom there is in modern CPU and memory, and show how simple cache-sensitive batch algorithms running on a single core can outperform hundreds of cores running distributed map-reduce style jobs. https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos...

wiradikusuma 7y ago
But nobody in BigCo(tm) would do that, because everyone (including the executives) involved want to have Hadoop/BigData(tm) in their resume. /sarcasm
- tlb 7y ago
  No doubt there are people who do it for cynical reasons. But at least some people do it sincerely thinking it’s the right choice. It’d be more interesting to talk about them, and how they came to make the wrong decision for what they thought were the right reasona.
  - asdfasgasdgasdg 7y ago
    I think it's a bit presumptuous to say that the decision is "wrong". That paper demonstrated that a single server can outperform a small server farm on a toy problem. Nobody, not even google, solves pagerank in production as a batch job. Real problems are often more complex.
    - tluyben2 7y ago
      For many workloads, companies have tech leads and CTOs absolutely overarchitecting their stack. If you can run your entire system off of 4 loadbalanced $200/mo servers, why am I seeing hadoop/kafka/kubernetes/etc running on $10k+/mo premoney, so paid with investor money? Sure there are a lot of cases where this is fine, but I would say (from real life) that there are far more cases where this was a pretty poor choice. Usually proven to be even poorer when the tech lead leaves and no-one seemingly having a clue how it all works together, despite the whole docker/kubernetes/ci stories peddled to management. That is usually when I get asked to take a look. In my experience these problems are definitely not always toy problems while some of them can, easily, be ran on 1 server for the expected lifespan of the company because the company will never get that much users/clients/data even though immensely profitable. Not everyone (almost no one) is a FAANG and I find it offensive to use the company money (which can be investor money) to realise the wet dreams of the cto while the case for that architecture and the cost of it, makes 0 sense to the business and bottomline. Obviously in life, things are nuanced and differ case by case but nah, real world problems are, generally, not more complex. They are more complex in some (rare) cases, like the one you named. But most companies are not doing anything like that, but they do have their cto’s gearing up for that incredibly unlikely future.

anovikov 7y ago

My good friend built whole career doing exactly the same. "Replace a cluster of 10 Elasticsearch servers with 1 running a custom built C app and in-memory database". Of course, it won't work out to replace 1000 Elasticsearch servers - that's where the advantage of a true "big data" tool will show - but none of the clients really have data "that big".

jacobsenscott 7y ago

You can choose a language that optimizes your hardware, or you can choose a language that optimizes your programmers. 99% of the time optimizing the programmers is the right call.

buzzkillington 7y ago
If you're using anything Hadoop related you're not optimizing programmers.
PeCaN 7y ago
this is the philosophy that brought us software like atom and slack
- jacobsenscott 7y ago
  Correct. Two very successful products.
otabdeveloper4 7y ago
Yes, if by "optimizing programmers" you mean "optimizing the manager's corporate structure footprint and bonus incentives". If the choice is between hiring one good C++ programmer or 15 really dumb Python "backend engineers" (and a team of QA's and sysadmins to support them), what do you think your pointy-hared corporate boss would chose?
- jacobsenscott 7y ago
  You never choose to hire dumb programmers. The choice is between 15 good cpp programmers and 4 good python programmers.
  - buzzkillington 7y ago
    You have not had to interview the python devs I've had to interview. Last round, for the second best candidate who we hired at 130% the salary we initially offered, after the best candidate was snatched under our nose for what we were told was twice the salary we were offering: >"Last question, I saw that you wrote a 100 line function here, is this because you ran out of time?" >>"No. I don't like to break up my functions. Having many small functions is confusing and bad practice." >What am I doing with my life?

easytiger 7y ago

Been saying this for years. Modern SW trends are horrific

gregdoesit 7y ago

From this thread, I feel a lot of people misunderstand what a scaleable system means for a scale-up startup or BigTechCo. It doesn’t mean cost-efficiency. It means the ability to solve scalability issues at 10x, 100x, 1000x load by throwing money at it (aka buying more instances/machines). Yes, John Carmack is right that a bunch of scaleable systems that see reasonable load, with well-understood traffic requirements could be rewritten more efficient ways. But how long would it take? How much would the extra work cost? And what would happen if traffic goes up, to another 10x, 100x? Can you throw still throw money at that problem, with this new and efficient system already in-place? One of my memorable stories comes from an engineer I worked with when we rewrote our payments system[1]. He told me how at a previous payments company, they had a system that was written in this kind of efficient way, and needed to run on one machine. As the scale went up, the company kept buying bigger hardware, at one point buying the largest commercially available mainframe (we’re talking the cost being in the multi-millions for the hardware). But the growth was faster and they couldn’t keep up. Downtime after downtime followed at peak times, making hundreds of thousands of losses per downtime. They split into two teams. Team #1 kept making performance tweaks on the existing hardware to try to get performance wins, and increase reliability during peak load. Team #2 rewrote their system in a vertically scaleable way. Team #1 struggled and the outages kept getting worse. Team #2 delivered a new system quickly and the company transitioned their system over. What they gained was not cost savings: it was the ability to (finally!) throw money at their growth problems. Now, when traffic went up, they could commission new machines and scale with traffic. And they could eventually throw away their mainframe. Years later, they started to optimise the performance of the system, making millions of $ in savings. The new system - like the old one - cost millions to run. But that was besides the point. Finally, they stopped bleeding tens of millions in lost revenue per year due to their inability to handle sudden, high load. It’s all about trade-offs. Is cost your #1 priority and are you okay spending a lot of development time on optimising your system? Go with C++, Erlang or some other, similarly efficient language. Is product-market fit more important, with the ability to have high uptime, that you can buy? Use the classic, horizontally scaleable distributed systems stack, and worry about optimising later, when you have stable traffic and optimisation is more profitable over product development. [1] https://blog.pragmaticengineer.com/distributed-architecture-concepts-i-have-learned-while-building-payments-systems/ https://blog.pragmaticengineer.com/distributed-architecture-...

useful 7y ago

PCI-e 4.0 and 5/6 enables consuming absolutely insane amounts of data. I can't think of many applications that need that. 128GB/sec? sure! and 256GB/sec is coming. With 10Gb, 40Gb and 100Gb internet links becoming mainstream the roadblock to having cool stuff is developers to make it.

PedroBatista 7y ago
Hold my beer, I’m firing up my [ insert dev stack ].

qaq 7y ago

Keep in mind a modern commodity x86 server with 128 physical cores 4 TB RAM and decent amount of SSD storage and dual 100 gbe nics is about 70K. Ability to use something like Rust also changes equation significantly.

eb0la 7y ago
I believe the key is able to use something _native_, not interpreted code or bytecode, running on a virtual machine, which runs in a userspace process inside a virtualized server inside a bare metal server. Said that, I believe Rust has a great future in data processing. Just take a look at Apache Arrow Rust bindings and Ballista (https://news.ycombinator.com/item?id=20456273 https://news.ycombinator.com/item?id=20456273)

cryptica 7y ago

Scalability and performance are very different. Opting for a performant but not scalable solution is basically an acknowledgement that: - The project will only succeed up to a certain point. - After the initial launch, no further changes will be made to the project since new additions are likely to cost performance and lower the upper bound on how many users the system can support. Not many projects are willing to accept either of these premises. Nobody wants to set an upper bound on their success. Also, it should be noted that no language is 'much faster' than any other language. Benchmarks which compare the basic operations across multiple languages rarely find more than 50% performance difference. The more significant performance differences are usually caused by implementation differences in more advanced operations and in any given language, different libraries can offer different performance characteristics so it's not fair to say that a language is slow because some of its default advanced operations are slow. Usually, performance problems come down to people not choosing the best abstract data type for the problem. Some kinds of problems would perform better with linked list, others perform better with arrays or maps or binary trees. Time complexity of any given algorithm is way more significant than the baseline performance of the underlying language.

lonelappde 7y ago

A single server cannot be made reliably available, though. At single-node scale, development costs dwarf hardware costs of scaling horizontally.

ungerik 7y ago

This is why Google developed Go. Python like productivity with C++ like performance...

rbanffy 7y ago

OTOH, we can assume one doesn't need to worry about using C++ unless they have many millions of users.