14 ms·
For the “Python isn’t fast enough for production backend” crowd from the same company that brought you the largest social network built on PHP + MySQL.
by seanp2k2 3y ago
For the “Python isn’t fast enough for production backend” crowd from the same company that brought you the largest social network built on PHP + MySQL.
- pjmlp 3y agoYou mean the company that had to create their two PHP implementations and a translator to C++, to actually scale?
- disgruntledphd2 3y agoThey ended up compiling PHP to assembly! Facebook is a company that will go to ridiculous lengths to avoid rewrites.
- that_guy_iain 3y agoI thought they tried it multiple times. The problem is they had so many developers writing new code any rewrite attempt would fall behind super quickly? So they created their own language Hack so allow a slow transfer from PHP to Hack. Which is basically the idea behind Kotlin.
- snapetom 3y agoWas there a separate initiative for assembly, or are you thinking about HipHop? HipHop was for C++ and got a lot of attention back when it was released.
- pjmlp 3y agoYes, the 2nd rewrite for a VM, where the JIT compiler turned out to be quite good versus HipHop, and development focus switched to HHVM.
- disgruntledphd2 3y agoI was thinking about HHVM, which I believed was essentially assembly for a VM. Am I wrong on that?
- snapetom 3y agoNo, you're right. HHVM was the 2nd iteration. I was thinking of the first project, which was for C++. Looking at wikipedia, that was called HPHPc.
- jarjoura 3y agoI know you're being hyperbolic, but during my time there, your impact was measured only on measurable things like performance gains but never on lines of code. So engineers would always go in the direction of whatever gets you the biggest win. Sometimes that actually meant rewriting the thing from scratch, and other times that meant adding something else as a middleware to prevent touching the original thing. For the most public success story, React was a complete rewrite of the thing before it, for example.
- disgruntledphd2 3y agoI spent five years there, and made that joke constantly. It's mostly hyperbole but there's definitely a kernel of truth to it. I generally dislike rewrites and I wonder if FBs approach influenced me more than I realized at the time.
- nikanj 3y agoRewriting in hip-tech-du-jour is for startups with lots of funding and little of traction. Why would a massively successful company toss a stick in their own spokes by tearing the product to ground?
- lightbendover 3y agoEvery company should go to ridiculous lengths to avoid rewrites. Rewrites of significant tech (as a proxy, lets say 100+ kloc) spell doom. Doom.
- engcoach 3y agoIt’s definitely not for the timid or mediocre
- Tainnor 3y agoEvery company? What about banks that have old codebases running on Cobol? I'm also very skeptical about big bang rewrites, but there are points where you need to migrate off certain tech. Ideally you can make that transition piece by piece though, but that also introduces its own set of problems (now you have two systems and the new one has to inherit some of the baggage of the old one in order to be compatible).
- jarjoura 3y agoThis is exactly the kind of innovation I see coming out this current AI boom we're in. Take legacy codebases and spit out complete rewritten versions in whatever language that not just translates, but takes it much further and re-interprets it on top of modern architecture designs. It completes it with tests and a basic UI, or whatever makes sense for the project. Those systems are well understood and well tested, so it's not cost effective at the moment to embark on a complete rewrite. Current AI coding systems are also very unreliable, so it's just a matter of time before those two meet on the graph and voila, another boom in moving the legacy world into safe languages.
- jononomo 3y agoI remember when I used to rewrite stuff before fully understanding the drawbacks of what I was rewriting it in.
- DoesntMatter22 3y agoLol. Do you think you are just going to use go and everything is going to scale to FB level size? Facebook was already at a scale larger than 99.99999 percent of sites before they had HHVM
- throwawaymaths 3y agoOf course not. One uses Erlang and scales to Whatsapp scale.
- pjmlp 3y agoRemember when Twitter was written in Ruby?
- jarjoura 3y agoLOL, at least Ruby inspired a new generation of languages that we have today.
- ramesh31 3y ago>Remember when Twitter was written in Ruby? Ruby is the exception to the rule. Friends don't let friends start new Ruby projects in 2023.
- epolanski 3y agoRuby in 2023 is miles ahead it was 10 years ago.
- pdimitar 3y agoYet it's still super behind tech like Elixir or Golang, which are both faster and give you more compile-time guarantees.
- DoesntMatter22 3y agoBehind how? There are no good web frameworks for Golang at all, and relatively few sites being built with it. Elixir? Okay, it's a nice language that is also dynamically typed. It has some advantages but good luck finding anyone with knowledge of how to program in it or has even used a functional language before. I don't personally use Ruby these days but to say it's super behind is just silly. Golang is as similar to Ruby as C++ is.
- rafark 3y agoFacebook had hundreds of millions daily active users before writing a compiler.
- pjmlp 3y agoAnd lots of nodes not able to keep up, rewriting backend services in C++ to make up for it.
- coding123 3y ago98% of the people that initially thought (Python for Threads - OMG performance!) work at a company that will tell you straight up they expect 100M RPS in a year or two, will actually only get to 50k RPS in 5 years before the "Our incredible journey" letter is issued.
- di456 3y agoI use an unscientific rule of thumb that 10-100x scaling is about the most to plan for except for exceptional cases. Anything beyond that ends up with overcomplicated bloat for a "tomorrow" problem. Being able to handle spikes and iterate quickly is probably more important.
- rewmie 3y ago> Facebook had hundreds of millions daily active users before writing a compiler. In other words, Facebook felt compelled to write a compiler when they had hundreds of millions daily active users.
- depereo 3y agoIf you have hundreds of millions of daily active users then needing to write a compiler isn't the end of the world.
- lyu07282 3y agoCompiling PHP into machine code sounds pretty much impossible, are they using some subset of the language? Does the compiled code use a garbage collector? Or reference counting?
- c7DJTLrn 3y agoGuess I'll just write my next project in Bash then, after all, if it's a good idea they will come right? Execution can wait until you've got a few hundred mil.
- ramesh31 3y ago>Guess I'll just write my next project in Bash then, after all, if it's a good idea they will come right? Execution can wait until you've got a few hundred mil. Yea actually. Bash/CGI could handle plenty. If thats your most comfortable language then go for it.
- fullspectrumdev 3y agoBash is seriously underrated as a programming language these days. About 80% of the code I now write is in bash, because usually the problem ones trying to solve has a trivial solution if you just use existing tooling in creative ways.
- pdimitar 3y agoI don't disagree with your premise and I write plenty of bash and zsh scripts but "the creative ways" very often are a rabbit hole that exposes way too many edge cases. At one point I wake up two hours later and realize I could have finished this in 15 minutes with Golang or even Ruby.
- jononomo 3y agoYou're telling me that they chose to build with PHP and had the most successful scaling story of all time?
- porhole 3y agoAny language will be fast enough with enough modern hardware. But it’s not “just Python”. The OS state that needed to be loaded is not Python. The network gear firmware is not just Python. The databases are not Python. Where it becomes just Metas Python is after a whole lot of other necessary work is done by other smarter people (the ones who master the physics of building the machines), not just some dweebs who got pulled the internet onto the host. These articles are just primate rage bait
- xuancanh 3y agoIt may not be because of Python, but Thread is definitely not ready for the current load. Few hours ago no one could access to Zuck's profile page, and he had to take it to private to hide the issue.
- rafark 3y agoTweets load very slowly on my end. So slow I thought there was a js error on my end, you know, like when you get a never ending spinner animation because there was an unhandled exception thrown.
- Solvency 3y agoSurely you mean threads.
- ryantgtg 3y agoI think there have been issues where profile pages say “this account is private and has 0 followers.” I saw that yesterday on every account I clicked on (on web). And then a few minutes later it was back to normal. It might not be that he manually took it private to hide an issue.
- steveBK123 3y agoThey wrote a new app in the language 99.9% of 0-5 year experienced devs know. Absolutely shocking.
- that_guy_iain 3y agoFor me, it does seem really strange since PHP (currently) is faster than Python and PHP wasn't fast enough for Facebook. But it's interesting to see how it'll play out.
- rafark 3y agoIt’s obvious why they went with python. Since Instagram was written in python and Threads is based on instagram, they’re probably using a lot of the libraries they already have for Instagram.
- that_guy_iain 3y agoThat makes some sense.
- DeathArrow 3y agoProbably Python is deemed as more appealing to hipsters than good old PHP.
- anayak 3y agoWhile that is technically correct, they haven't been running vanilla PHP for quite some time - it's jitted into x86 and run natively on the machine. It's also pretty extensively optimized. And the VM that does the jitting is all C++. Their MySQL deployment is also an internal fork that's been heavily modified for scalability (storage/compute separation, sharding, RocksDB based storage engine instead of InnoDB, Raft for synchronous replication, etc.). Lots of really fantastic engineering work has gone into making their backend infra scale. Some worthwhile references: [1] https://engineering.fb.com/2016/08/31/core-data/myrocks-a-space-and-write-optimized-mysql-database/ https://engineering.fb.com/2016/08/31/core-data/myrocks-a-sp... [2] https://research.facebook.com/file/529018501538081/hhvm-jit-a-profile-guided-region-based-compiler-for-php-and-hack.pdf https://research.facebook.com/file/529018501538081/hhvm-jit-... [3] https://research.facebook.com/file/700800348487709/HHVM_ICPE_2023-(1).pdf https://research.facebook.com/file/700800348487709/HHVM_ICPE...
- ddorian43 3y ago> storage/compute separation Have they really done this for OLTP workloads running in mysql? I know they do it for OLAP though.
- grishka 3y agoCuriously, VKontakte also started from PHP+MySQL but went another way. PHP is compiled ahead of time with something called KittenPHP. It's open-source. For databases they switched some parts to their own bespoke solutions in C that talk with PHP over the memcached protocol. These are called KittenDB (or "engines") for the simple single-purpose ones, and there was also a more generic MySQL replacement in development when I left in 2016, MeowDB.
- jonhohle 3y agoI’m not sure why someone of facebook’s scale would want JIT instead of AOT. That’s a lot of servers all JITing the same thing that could be done once at build time.
- 3y ago
- chippiewill 3y agoPHP is actually shockingly fast for a non-JIT'd language these days.
- icelancer 3y agoPHP8 has a JIT now but I get what you mean.
- shortrounddev2 3y agoQuite often, the choice of language is failing to see the forest for the trees. Saving nanoseconds or cycles because you chose C++ over Python pales in comparison to milliseconds spent at network barriers reading from a cache service or database
- pdimitar 3y agoYes and no, I still remember the times when several Ruby on Rails apps I consulted for had 5ms DB queries and Rails' ActiveRecord was taking 120ms. Technically choosing C++ over Python will save you several orders of magnitude more than nanoseconds. Though C++ might be a bad example. I'd replace Python web app with Elixir or Golang.
- yeck 3y agoPerceivable latency isn't the only consideration. Depending on your business, compute for your proprietary workloads might be one of your biggest expenses. You could see an order of magnitude improvement to resource consumption depending on the type of workload and the language it is written in. Ultimately these these choices are all about trade-offs. Maybe python is fine for them, maybe they've built themselves into a corner. Time will tell.
- MengerSponge 3y agoWhen they need a full re-write the specification will be easy: they've already got the pseudocode. (I kid because I love Python)
- throwawaymaths 3y agoAt Facebook scale those nanoseconds are worth millions in compyte, energy, etc.
- shortrounddev2 3y agoBut do the millions you spend a year on compute cost more or less than the millions you would spend a year in labor finding the increasingly rare breed of C++ developers who can optimize things for instruction or cycle count? Such developers usually have over a decade of experience (if not multiple decades). Python developers are a dime a dozen, comparatively Plus, instruction and cycle counting is low hanging fruit compared to memory latency. You can cache-optimize a program in any language so long as the memory representation of some data is relatively transparent.
- nikhizzle 3y agoEarlyish facebook engineer here. Early days FB php was nothing like the php used to template websites. All kinds of specialized libraries to enable a much more sophisticated programming style. Think functional helpers, and asynchronous execution on thousands of cores, spanning trees across data centers using ssh etc. As a tangent a lot of the good stuff I used was written by Evan Priestley, who also did Phabricator and lots of other strong systems.
- iknowSFR 3y agoCan you speak to the onboarding and learning curve in that environment? What was the training like? What kinds of timelines did you experience regarding expectations as a new hire? Asking because I’m curious about these early stage mega-tech companies that are working at a scale I can’t fathom.
- nikhizzle 3y agoSix week boot camp, you are pretty fluent by week 4. Notably though this was before mega tech days, we had under 100 engineers.
- supernikio2 3y agoI'm of course speaking from an outsider's point of view, but do you think PHP was a good option, or was it a "fitting a square peg into a round hole" situation?
- nikhizzle 3y agoIt was not a deliberate choice, just it was what the site was built in, so people like Evan and Marcel (Laverdet) made php beautiful for us.
- troupo 3y ago> but do you think PHP was a good option, or was it a "fitting a square peg into a round hole" situation? Facebook was founded in 2004. Whatever fancy tech you're thinking that would fit the round hole, it didn't exist then. They also never thought it would serve a couple billion of users one day, so they built it in whatever they had and knew at the time. Same for Instagram and Python (they started with Django).
- hello_moto 3y agoYep, Vanilla Python isn't fast enough so they have to do more: "It's running on Instagram's #Cinder fork that includes a JIT, lazy-loaded modules, pre-compiled static modules, and a bunch of other interesting tweaks against vanilla Python 3.10" From the Tweet. Personally, I don't care what language people use. We all know that if you have money, you can just throw in more servers. Until your CFO and CTO decided that the next big thing is to do "Cost Saving" (in the eye of recession). I'm surprised in 2023 people are still debating mainstream programming language "PROD" readiness. It's like a "cool kid" competition.
- ubercore 3y ago> We all know that if you have money, you can just throw in more servers. Definitely not true at Facebook scale. You need to be smart, fast, _and_ throw more servers at it.
- hello_moto 3y agoI'm speaking on the aspect of Python as web app. We both know the issue has always been how you store the data. Be it your main data storage, or the need of multiple storages for different use cases, etc. The app server itself can be implemented in any languages. It's just a data orchestration + transformation logic anyway...
- deleted 3y ago[deleted]
- epistasis 3y agoIt just dawned on me that linking to Twitter is now impossible on the open web. Unless you have an account, a person can not see the linked material. This is a huge loss for the world, and makes me a bit angry. Edit: I had falsely assumed that because GP missed out on the JITing etc. described in the linked tweet, that Twitter was still inaccessible to the open web. However, trying again now, Twitter is again viewable without an account. So Twitter's communication about needing an account to view it was lacking, but so was my diligence.
- DeathArrow 3y agoPython is just fine for all the 100 users Threads will manage to acquire. :)
- latexr 3y ago> all the 100 users That’s off by multiple orders of magnitude, if you believe their reports of 30 million sign ups in under a day. https://www.theverge.com/2023/7/6/23786108/threads-internal-activity-data-exclusive-instagram-meta https://www.theverge.com/2023/7/6/23786108/threads-internal-...
- Twitterboi 3y agoYou mean sign-in's?
- latexr 3y agoI’m using their terminology. The distinction seems irrelevant for the conversation.
- selcuka 3y agoI believe you should create a separate Threads account even though it uses your Instagram account. It's like OAuth2 single sign-on, but technically you still sign up for a different service.
- berkle4455 3y agoExcept PHP actually is fast while Python isn't.
- ldng 3y agoExcept it's not. Bindings to C libraries are.
- berkle4455 3y agoYour semantics are irrelevant. People who write PHP end up with faster performing applications than people who write Python.
- o1y32 3y agoAm I seeing two people debating which language is faster without quoting any numbers?
- berkle4455 3y agoI assumed it was common knowledge at this point. https://www.techempower.com/benchmarks/#section=data-r21&l=zijym7-6bj&test=json https://www.techempower.com/benchmarks/#section=data-r21&l=z...
- arcticbull 3y agoMost of Instagram is Django.
- grumpyprole 3y agoPHP didn't scale for the number of users or the number of developers. Presumably that's why they built their own PHP-like language called Hack with static types and a new runtime.
- albertopv 3y agoThey didn't use vanilla Python, though.
- Klonoar 3y agoIn addition to the Facebook/PHP comments here, an often overlooked one is that Instagram was a big Python/Django shop for years (and probably still is in many parts).
- ineedasername 3y agoPfft. It would be 100x more performant if they'd written it in asm.
- gedy 3y agoWould this pass their System Design interview?
- paultopia 3y agoOTOH threads is kinda slow as shit right now
- preseinger 3y agoi don't think anybody really claims python can't be used to build large software the claim is usually more about how the maintenance costs are just way higher versus a statically typed language
- zackees 3y agoBut is that true if the dynamically typed language has type annotations?
- pdimitar 3y agoThey truly and tangibly help somewhat but people give them too much credit.
- preseinger 3y agoit's a spectrum, of course, but imo yes -- i don't want types to be opt-in at compile time, i want them to be mandatory
- pharmakom 3y agomakes one wonder what the cost of needing bigger / more machines to make up for the slow runtime is though.