28 ms·
Backend of Meta Threads is built with Python 3.10
- ddorian43 3y agoNo tweaks mentioned at all inside. The title is wrong. Kinda weird to start a new project that needs to scale with Python though (I use Python most of the time).
- threeseed 3y ago"JIT, lazy-loaded modules, pre-compiled static modules, and a bunch of other interesting tweaks against vanilla Python 3.10"
- ddorian43 3y agoThat is just CPython fork from instagram https://github.com/facebookincubator/cinder/ https://github.com/facebookincubator/cinder/
- ambivalence 3y agoThe fork is where the tweaks live.
- jackmott 3y ago[dead]
- deleted 3y ago[deleted]
- troupo 3y agoIt seems they are re-purposing Instagram's backend for this. And Instagram's backend is Python
- kleinsch 3y agoIG is built in Python (which is why this is also) and it seems to scale fine.
- super_flanker 3y agoAs far as I can tell from my prior experience of working at Insta is that all the work is done by various services written in C++/hack. Python/Cinder would be just interface layer routing the incoming request/response from/to client.
- threeseed 3y agoWould love to know more details about the rollout. Because it's adding well over a million new users each hour which is pretty staggering.
- zokier 3y agoI would assume its massively overprovisioned at this point to cover first few weeks if not months of projected growth. But still there probably is all sorts of juicy details that would be neat to hear about
- sanderjd 3y agoOne of the "fun" things when I worked at a big company was thinking through how to make the very first launched version of something work at millions-of-users scale. There is no organic growth in traffic from zero when a non-negligible subset of billions of active users will try it out right away. But I think on net it was still an easier problem than scaling a startup product from zero, because the tooling and processes to support huge usage is already so mature at these big companies.
- phyrex 3y agoIt’s just running on the shared meta infrastructure which is built to handle that scale
- pavlov 3y agoMeta has 3.8 billion monthly active users across its products. A few million new people is less than 0.1% of the total. It's entirely within expected variance.
- nomel 3y agoI think they're looking at it from a an individual product perspective. % users of an entire company can be extremely different than % growth of a new product, within that company. My naive assumption is that they have some neat simulated user/load system to help identify any scaling issues that might arise when launching a service, and the extreme % growth. That 3.8 billion was spread over 15 years.
- mlrpfx 3y agoAlways the same people that get their names in the news. I would say that the amount of tweaks that prop up Python is no cause for celebration.
- blitzar 3y agoYet here it is ... powering a global scale app. Too weak for someones hobby project, strong enough for global scale.
- mlrtph100 3y agoSo Google and Dropbox, who switched to Golang, are hobby projects.
- cmrdporcupine 3y agoChoice of language is not what makes things succeed or fail in production. Team best practices, enforcement of coding style and technique, good project management, infrastructure, and team cohesion etc. And weeding out bad engineers who think that "switching to a new [language|framework|religion]" will solve all problems. 10+ years ago Facebook itself did a remarkable job of scaling up, with hardly any outages, a PHP LAMP stack thingy to one of the hugest traffic websites in the world. Meanwhile back then Twitter mucked around with every novel technology they could find or invent, and had constant outages. Also Google didn't "switch to Golang." There's plenty of C++, Java, Python there. I worked there for 10 years and encountered only a handful of Go projects. Lots of cloud services in C++. Would I personally choose Python for a project? No, I don't like it, and I work in Rust full time because I prefer it. But if I worked at Meta / Instagram and had a team of Python engineers and existing libraries & infrastructure, this would be the right approach.
- aeyes 3y agoAlso, what's really different from Instagram? You change the picture for a short text and many other features already existed - comments, likes, subscriptions, recommendations, personal messages... I think this is just an Instagram skin with some feature flags.
- TheAdamist 3y agoIronic that a post about Threads backend is hosted on twitter.
- deleted 3y ago[deleted]
- mihaic 3y agoWhy downvote this? It's a fair point.
- RamblingCTO 3y agoBecause it's not ironic. It's obvious. Twitter is just a medium to post stuff. Threads is as well. I can post this on mastodon even. Also: Europe still can't join in on the fun.
- user_named 3y agoI'm in Europe and just downloaded the apk. All my Chinese friends have joined too.
- Dah00n 3y agoEU != Europe. Biggest country in Europe is Russia. Biggest population is Russians.
- ffhhttt 3y agoJust like America is not just the US. But nobody cares…
- RamblingCTO 3y agoYeah, that's up for debate. It's not as clear as you make it out to be. As a (EU) European, Russia is not. The western most part of it maybe, yeah. Same for Turkey.
- 3y ago
- nunez 3y agoCan we block Twitter links on HN? No point in supporting that user-hostile tire fire. This could have been on Mastodon.
- ambivalence 3y agoIt is on Mastodon as well, actually: https://mastodon.social/@ambv/110665808046065754 https://mastodon.social/@ambv/110665808046065754
- philipov 3y agoCould we get the mastodon link as the main link, please? Can actually view that one.
- aeyes 3y agoIs the Twitter link really not working for you at the moment? I can open it without having an account.
- rsynnott 3y agoYou can't see replies on Twitter links anymore, though that doesn't really matter in this particular case. Of course, who knows what it'll be doing tomorrow. I'd definitely think it's probably time for HN to prefer Mastodon (or Threads, I suppose) links, where a post is available on multiple microblogging things; Twitter has had any one of about five behaviours for non-logged-in linked tweets (work properly, work but without replies, spin forever, redirect to login, generic error page) over the last week, depending on when you try.
- reledi 3y agoThe tweet is currently viewable while logged out. Have you tried?
- timeon 3y agoOnly thing I see is: "Something went wrong. Try reloading." (Mastodon link works.)
- kleton 3y agoReally, all the performance intensive parts are in various c++ aggregator and recommendation type services. But the webserver is Django, yes.
- redsky880 3y agoCorrect me if I'm wrong, but Django is not a web server. It's a framework used in conjunction with an app server e.g. gunicorn and a web server e.g. nginx
- taopai 3y agoThanks for your comment. I'm really interested in this topic. How do you know the web server is Django? I searched but couldn't find this. Why would they use Django? I did some small projects in it but I assumed it wasn't very fast and wouldn't be suitable for a big app like this. I would like to know the pros and cons. Why didn't they build up something in C++ or Rust? Won't python limit the speed of responses despite being build the hard stuff in compiled languages? Sorry for being so naive, I am an amateur.
- leodriesch 3y agoIt is fairly normal to build web servers in a way so they can be scaled horizontally (more instances rather than larger instances). So they can just have more containers run their Django servers and distribute the load between them.
- dporter 3y agoAt least a few years ago, most of Instagram’s server side code was in Python. Python may be slower than C++, but it’s not about raw speed it’s about being “fast enough.”
- tinco 3y agoThey deemed getting to the market fast was more important than hardware costs. If their architecture is good, then they were probably really quick and flexible using Python to glue their architecture together which would have been the development bottle neck to getting the service up and running. All the components of the service can be done by separate teams in whatever language they deem most effective.
- appleaday1 3y agoNever forget https://github.com/twitter/the-algorithm/pull/447 https://github.com/twitter/the-algorithm/pull/447
- aio2 3y agoFinally, a difference between the two.
- lopkeny12ko 3y agoPython is just the web server. All the truly performance-sensitive components--edge proxy, load balancers, backend services, databases, caches, and storage services--are mostly C++.
- citrin_ru 3y agoI would argue a web server which processes all (or most) requests is a performance sensitive part of the system. Straightforward rewriting from Python to Go in many cases gives at least 2x-3x performance improvement which on FAANG scale can mean huge difference in hardware cost.
- brrrrrm 3y agoDeveloper velocity is extremely important at FAANG scale.
- that_guy_iain 3y agoIs it really? Don't they work really slowly compared to most of the industry? I really get the impression that it's slow progress for things at FAANG.
- super_flanker 3y agoNot really, especially Facebook (Meta) moves really fast. They have a slogan "Move fast, break things", okay not "Break things" anymore. But you get the gist, in my experience working at almost all of FAANG, Meta and Netflix move the fastest.
- PartiallyTyped 3y agoI think AWS must be the slowest of the bunch tbh. Too many meetings.
- 3y ago
- seanp2k2 3y agoFor the “Python isn’t fast enough for production backend” crowd from the same company that brought you the largest social network built on PHP + MySQL.
- pjmlp 3y agoYou mean the company that had to create their two PHP implementations and a translator to C++, to actually scale?
- disgruntledphd2 3y agoThey ended up compiling PHP to assembly! Facebook is a company that will go to ridiculous lengths to avoid rewrites.
- that_guy_iain 3y agoI thought they tried it multiple times. The problem is they had so many developers writing new code any rewrite attempt would fall behind super quickly? So they created their own language Hack so allow a slow transfer from PHP to Hack. Which is basically the idea behind Kotlin.
- snapetom 3y agoWas there a separate initiative for assembly, or are you thinking about HipHop? HipHop was for C++ and got a lot of attention back when it was released.
- pjmlp 3y agoYes, the 2nd rewrite for a VM, where the JIT compiler turned out to be quite good versus HipHop, and development focus switched to HHVM.
- disgruntledphd2 3y agoI was thinking about HHVM, which I believed was essentially assembly for a VM. Am I wrong on that?
- manca 3y agoI've been using Threads since late yesterday and it's been very slow... sometimes it wouldn't even load content when you browse specific profiles (especially the active ones with tens of thousands of followers). So I am not surprised the backend is in Python :)
- KolmogorovComp 3y agoI will probably be downvoted for this joke but I cannot resist. I hope they use Threads over async/await.
- deleted 3y ago[deleted]
- beardedman 3y agoSo glad we're condensing entire tech stacks down to a language now.
- zengid 3y agoOk but is the front-end in React Native?
- shaftoe444 3y ago> It's running on Instagram's #Cinder fork that includes a JIT, lazy-loaded modules, pre-compiled static modules, and a bunch of other interesting tweaks against vanilla Python 3.10. So not entirely just python 3.1.
- zitterbewegung 3y agoI have ported code from Cinder to CPython . The fork has some optimizations that can be easily put in CPython and Facebook is open to port features. I’m not sure if Facebook wants to continually have a fork but CPython is open to have those features merged in if they make sense.
- amethyst 3y agoThe Cinder team's longterm goal is to upstream as much as we can, and make the rest available as pip-installable extensions that anyone could theoretically install and use on CPython. The fewer internal changes and patches we need to maintain, the faster we can adopt upstream Python releases and all of the associated performance and tooling wins.
- VagabundoP 3y agoI've been following it with updates from Talk Python To Me and Python Bytes. Great work, much appreciated.
- zitterbewegung 3y agoOkay cool thanks for the clarification
- VectorLock 3y agoThat seems like pretty much the perfect way to go about things. Kudos on that work!
- roflyear 3y agoPython is a spec not an implementation, for what it is worth.
- costco 3y agoThere's definitely some overlap in the backends because https://www.threads.net/t/C https://www.threads.net/t/C is the first Instagram post but on Threads with the exact same amount of likes it has on Instagram yet https://www.threads.net/@kevin https://www.threads.net/@kevin shows no threads.
- czardoz 3y agoI worked on Cinder, and on the web server. Happy to answer technical questions if any :)
- feydaykyn 3y agoHi, Would you recommend using Cinder for stack made of - Django - Cython - Numpy Thanks!
- czardoz 3y agoIf this stack is heavily skewed towards numerical computations, Cinder might require a lot of tuning to be effective.
- vmsp 3y agoCan you expand on the web server? Does it implement WSGI?
- czardoz 3y agoIt has some elements of WSGI, but has expanded into its own thing which supports asyncio (this happened before ASGI was a thing).
- ntonozzi 3y agoWhat are the most impactful and likely features that you'd like to see merged into CPython?
- czardoz 3y agoI'd love to see lazy imports become part of the upstream at some point.
- whalesalad 3y agoWho would benefit from Cinder? What kinds of workloads? The README is kinda lacking on this.
- mattfrommars 3y agoMakes me wonder why did they not opt for Java or C# as their backend choice. I better stop learning C# and move on.
- vips7L 3y agoUse what you know! That is the only reason they chose Python. Instagram is a Python/Django shop.
- andrewstuart 3y agoIf you know you’ve got gigantic scale why would you not use a compiled language? Surely it makes massive difference to hardware utilization? I like Python and javascript but there’s perfectly fine tools around now for building systems that allow one server to do a lot more. They say you should not prematurely optimize. Equally you should not be prematurely unoptimised.
- paulddraper 3y agoUsing a compiled languages for web stuff is like using a Ferrari for a commute. It's better, but not the limiting factor.
- andrewstuart 3y agoThreads is intended to be massive scale. I’d use the analogy of a Japanese bullet train as the compiled language and a diesel train as Python. Python is my primary language along with JS but if I was building a new service for Meta I wouldn’t use either at the back end.
- o1y32 3y ago...and apparently Meta won't care a tiny little bit about your opinion, because they have a product that is running quite well.
- callamdelaney 3y agoBecause Zuck probably told his team to make this a few weeks ago to annoy Musk. Weird ego thing going on. Python is likely faster to push out an app like this than say rust or even go. Also why spend time optimising something that will be dead in a few months?
- smrtinsert 3y ago!remindme 2 months. After 30 million signups in a day I'm pretty sure we just saw the birth of a new Meta social network.
- Solvency 3y agoWhy has everyone on here been comparing Threads to Twitter when threads.com literally says its a Slack alternative for makers? The product has absolutely nothing to do with a Twitter style social network.
- PartiallyTyped 3y agoThreads.com is not Threads.net. The latter owned by Instagram/Meta.
- Solvency 3y agoJesus Christ this is confusing.
- nologic01 3y agoNo matter what the true combination of reasons, this is a massive endorsement of the python ecosystem. Its an interesting dynamic: popularity draws attention, attention helps adress pain points, this makes the platform more attractive and the popularity increases...
- mrcwinn 3y agoI’m building a tiny little product for trail runners and cyclists in Python 3.10 (Flask) at https://www.outandback.com https://www.outandback.com. Python has been a lot of fun to pick up (I rely heavily on pandas and some ML libraries so it felt like the right stack).
- justsocrateasin 3y agoCould you describe your tech stack more? I'm a data engineer trying to teach myself React so I can build a full stack app using some APIs and it's certainly a good exercise, but I could also certainly build that in Python. Your app looks great and I'd like to explore that more! P.S - fellow runner and cyclist and out and I'm checking out the app more now.
- caeril 3y agoAside: I see a ton of hype about threads.net all over HN and elsewhere, but it's still useless thus far. It loads a javascript-driven animation of what appears to be a galaxy, any not much else. There is a QR Code at the bottom right corner that appears to be invalid. Not sure what I'm missing. 10MM+ signups through what mechanism? There is no signup form as far as I can tell. Do they restrict full content access for linux-hosted browsers? edit: after saving and reversing saturation, I've resolved the QR Code to https://www.threads.net/download/redirect https://www.threads.net/download/redirect, which only offers binary download options, and only for mobile platforms. Where is the web interface?
- wilg 3y agoThere is no web interface, but they will almost certainly add one. MVP baby!
- caeril 3y agoGot it, thanks. But that's really odd because most mobile MVPs are either just PWAs or some React Native shit compiled to binaries. Why the hell wouldn't they just release it on the web?
- heywire 3y agoMost people access social media using mobile apps, not websites. I don’t think I’ve used the Facebook, Instagram, or Twitter websites in years.
- wilg 3y agoFocusing on native mobile apps over web for launch is a completely sound strategy. 80% of Twitter use is from mobile, and probably near 100% for Instagram since they barely have a web version. But I’d expect them to do a web version relatively quickly to appeal to Twitter refugees.
- anuraaga 3y agoThreads is a repurposing of Instagram's backends / APIs. Instagram has been based on Python from the start. Therefore Threads happens to use Python. It doesn't mean that Python is the language that one should pick when building this sort of app given alternatives that don't require an in-house fork. The full post makes this reasonably easy to grok but I'm sure the hacker news "title-only" UI is misrepresenting.
- jononomo 3y agoHonestly, the only kind of re-writes I respect are when someone decides to rewrite something written in the hot new tech and uses 2005-era PHP instead. I recommend solid tools that have been proven to scale to billions of users.
- jononomo 3y agoPython is the fastest language in the world if you count in human brain cycles. You should only worry about CPU cycles after you've made your second million.
- jononomo 3y agoDjango is so boring, stable, and well-understood, and there are so many solid code examples, that ChatGPT is basically a super-powered coding assistant when you're working in Django.