5 ms·

> Instagram uses Python (Django) for its backend. By using the same backend for Threads, we could leverage a lot of the existing tech stack for Threads and reus

> Instagram uses Python (Django) for its backend. By using the same backend for Threads, we could leverage a lot of the existing tech stack for Threads and reuse most of our existing data models, business logic, security features, and server infrastructure. This also meant users could sign in to the app with their existing Instagram account, making it super simple to onboard and set up your Threads app. Looks like Python and Django is extremely scalable at Facebook scale, even after using Threads I didn't realise any downtime at all. It would be possible for Mastodon to learn from this so that they too can scale to 200 million users.

organsnyder 3y ago
Mastodon has such a different system architecture—at a business level—that I think many of the scaling challenges are entirely different. While my experience with Mastodon is as a (very satisfied) user, in general scaling challenges are as much figuring out how to scale the interactions between systems as they are scaling the individual systems themselves. That gets incredibly more difficult when those individual systems are spread out among various organizations with different priorities, philosophies, technical abilities, and resources.
- YouSuck2 3y ago
  [flagged]
weego 3y ago
The only lesson to learn from this is that given exceptional resources and a long list of talent any stack can be made to scale. That doesn't help most projects in the world.
- tjrgergw 3y ago
  [flagged]
- roflyear 3y ago
  Python apps using popular frameworks (not all frameworks) are generally (not always) easy to scale. The app itself is mostly just "spawn another instance somewhere" - storage and sessions and all that can be complex, but that's not really impacted by the app framework/language.
- tivert 3y ago
  > The only lesson to learn from this is that given exceptional resources and a long list of talent any stack can be made to scale. Exactly. IIRC, Facebook itself is/was a giant ball of PHP, and they had to do all kinds of crazy things to make it scale, up to and including writing their own compilers and VMs. That doesn't say anything about the suitability of the scalability of stock PHP.
  - ozzydave 3y ago
    I think the lesson is, avoid rewrites and don’t break apart your monorepo; have teams improve the platform while your product engineers go full-steam ahead. Compare with Twitter which spent 2 years in the woods rewriting to Java while FB kept growing, giant ball of PHP and all.
    - highwaylights 3y ago
      You’re getting downvoted for this, but you’re not wrong. Just keep moving forward unless the sky is falling is 100% the lesson to be learned here.
      - esafak 3y ago
        In the worst case, you'll fail to scale due to technical shortcomings like Friendster and become forgotten while later entrants like Facebook eat your lunch. There was a time when "being web scale" was a real concern. That's one reason companies like Google make such a big deal about it in their technical interviews. Now most of the hard work has been done and exposed as services so today even an untrained teenager can stand up a web-scale service.
willio58 3y ago
I don't use Python or Django anymore but I'm always happy to hear about it and kinda surprised to learn it's still being chosen today for massive-scale social media sites. To me, Django was my go-to for cranking out a usable MVP in the shortest time possible. It made it so easy, I could define a DB schema and it would generate these forms with validation built in. I kinda miss those days.
- roflyear 3y ago
  What you're saying is correct, but oh boy have I worked on Django apps that have had awful security issues because of this. "look, all my endpoints are there, user A must only be able to reset their own password... right? ;)" Also you don't have to miss those days! You can still use Django or a similar library? I still use Flask, it is great.
  - willio58 3y ago
    I'm pretty deep in the Typescript world these days but hey, maybe one day I'll make my way back into Python!
- colesantiago 3y ago
  I sort of wish that Go, Rust, Zig has these kind of frameworks like Django, Laravel and Rails. I believe there is an opportunity here to make the next Django in these languages.
  - olah_1 3y ago
    Each of those absolutely do have them. Here is a list for just Golang: https://github.com/avelino/awesome-go#web-frameworks https://github.com/avelino/awesome-go#web-frameworks
ajhai 3y ago
There are a lot of things that goes on in production to scale any service to handle that level of requests. But Django as a web framework is good at what it does. It comes in with most things that one needs to put web apps in production. We recently open-sourced an LLM apps platform (https://github.com/trypromptly/LLMStack https://github.com/trypromptly/LLMStack) that is entirely built with django as backend (drf for APIs, channels for websockets and reactjs for frontend).
Mxbonn 3y ago
This is largely due to Meta maintaining an internal performance-oriented production version of CPython: https://github.com/facebookincubator/cinder https://github.com/facebookincubator/cinder .
thewataccount 3y ago
(I use django in production myself, nothing against it) It's probably worth noting Meta made their own JIT'd version of CPython just for Instagram/Threads. Maybe mastodon devs can use their interpreter, but Meta's human/compute resources are magnitudes more than mastodon, the lessons from Meta might not be very applicable. https://github.com/facebookincubator/cinder https://github.com/facebookincubator/cinder
- ushakov 3y ago
  Isn’t Mastodon written in Ruby?
  - thewataccount 3y ago
    Yes, I wasn't sure if parent was suggesting they switch to django, or simply "using what facebook learned" - I didn't want to imply that there's no way Mastodon devs couldn't use Cinder to their advantage. I do however highly doubt Cinder would be of use to Mastodon at the current moment.
- parthokr 3y ago
  recently they made cinder public… this is a JIT’d version of cpython
smfugit 3y ago
These systems should not scale. Giving everyone free broadcast capability is a sign of mentally bankruptcy. Its like giving every neuron in your head broadcast capability. What happens when they all fire at the same time? Or like giving everyone in the world a radio transmitter that can broadcast on any frequency. What happens when they all transmit on the same frequency at the same time? These idiots are scaling things because its become possible technically. There is no other logic to whats going on. And its producing/enabling massive amounts of useless noise. The UN report on the Attention Economy says 0.5% of content on the net is actually consumed by anyone.
- tivert 3y ago
  > These systems should not scale. Giving everyone free broadcast capability is a sign of mentally bankruptcy. ... These idiots are scaling things because its become possible technically. There is no other logic to whats going on. I agree, but I don't think it's a sign of "mentally bankruptcy," but an indication of the flaws and limitations of contemporary ideology. They're not just doing it because it's technically possible, they wouldn't even try and no one would sign up if our ideology didn't say it was a good thing to do.
deleted 3y ago
[deleted]
freedomben 3y ago
Python/Django (and most other frameworks) isn't the problem with scaling. It's following the 12-factor app rules.
bee_rider 3y ago
Python is known to be pretty wasteful in terms of compute performance, but is the compute grunt of a given node a big deal in these kinds of networks? I assumed it is a big communication problem. Python shouldn’t hurt there, right? It isn’t like a Python-generated packet goes through the network more slowly.
KaiserPro 3y ago
> t would be possible for Mastodon to learn from this so that they too can scale to 200 million users. I mean possibly, if it was designed to be a centralised bohemoth with a staff of 600 to look after it. Its decentralised by design, much like the web used to be.
interactivecode 3y ago
So did they setup a separate backend for threads or did they just extend the current instagram backend to support text posts?

organsnyder 3y ago

Mastodon has such a different system architecture—at a business level—that I think many of the scaling challenges are entirely different. While my experience with Mastodon is as a (very satisfied) user, in general scaling challenges are as much figuring out how to scale the interactions between systems as they are scaling the individual systems themselves. That gets incredibly more difficult when those individual systems are spread out among various organizations with different priorities, philosophies, technical abilities, and resources.

YouSuck2 3y ago
[flagged]

weego 3y ago

The only lesson to learn from this is that given exceptional resources and a long list of talent any stack can be made to scale. That doesn't help most projects in the world.

tjrgergw 3y ago
[flagged]
roflyear 3y ago
Python apps using popular frameworks (not all frameworks) are generally (not always) easy to scale. The app itself is mostly just "spawn another instance somewhere" - storage and sessions and all that can be complex, but that's not really impacted by the app framework/language.
tivert 3y ago
> The only lesson to learn from this is that given exceptional resources and a long list of talent any stack can be made to scale. Exactly. IIRC, Facebook itself is/was a giant ball of PHP, and they had to do all kinds of crazy things to make it scale, up to and including writing their own compilers and VMs. That doesn't say anything about the suitability of the scalability of stock PHP.
- ozzydave 3y ago
  I think the lesson is, avoid rewrites and don’t break apart your monorepo; have teams improve the platform while your product engineers go full-steam ahead. Compare with Twitter which spent 2 years in the woods rewriting to Java while FB kept growing, giant ball of PHP and all.
  - highwaylights 3y ago
    You’re getting downvoted for this, but you’re not wrong. Just keep moving forward unless the sky is falling is 100% the lesson to be learned here.
    - esafak 3y ago
      In the worst case, you'll fail to scale due to technical shortcomings like Friendster and become forgotten while later entrants like Facebook eat your lunch. There was a time when "being web scale" was a real concern. That's one reason companies like Google make such a big deal about it in their technical interviews. Now most of the hard work has been done and exposed as services so today even an untrained teenager can stand up a web-scale service.

willio58 3y ago

I don't use Python or Django anymore but I'm always happy to hear about it and kinda surprised to learn it's still being chosen today for massive-scale social media sites. To me, Django was my go-to for cranking out a usable MVP in the shortest time possible. It made it so easy, I could define a DB schema and it would generate these forms with validation built in. I kinda miss those days.

roflyear 3y ago
What you're saying is correct, but oh boy have I worked on Django apps that have had awful security issues because of this. "look, all my endpoints are there, user A must only be able to reset their own password... right? ;)" Also you don't have to miss those days! You can still use Django or a similar library? I still use Flask, it is great.
- willio58 3y ago
  I'm pretty deep in the Typescript world these days but hey, maybe one day I'll make my way back into Python!
colesantiago 3y ago
I sort of wish that Go, Rust, Zig has these kind of frameworks like Django, Laravel and Rails. I believe there is an opportunity here to make the next Django in these languages.
- olah_1 3y ago
  Each of those absolutely do have them. Here is a list for just Golang: https://github.com/avelino/awesome-go#web-frameworks https://github.com/avelino/awesome-go#web-frameworks

ajhai 3y ago

There are a lot of things that goes on in production to scale any service to handle that level of requests. But Django as a web framework is good at what it does. It comes in with most things that one needs to put web apps in production. We recently open-sourced an LLM apps platform (https://github.com/trypromptly/LLMStack https://github.com/trypromptly/LLMStack) that is entirely built with django as backend (drf for APIs, channels for websockets and reactjs for frontend).

Mxbonn 3y ago

This is largely due to Meta maintaining an internal performance-oriented production version of CPython: https://github.com/facebookincubator/cinder https://github.com/facebookincubator/cinder .

thewataccount 3y ago

(I use django in production myself, nothing against it) It's probably worth noting Meta made their own JIT'd version of CPython just for Instagram/Threads. Maybe mastodon devs can use their interpreter, but Meta's human/compute resources are magnitudes more than mastodon, the lessons from Meta might not be very applicable. https://github.com/facebookincubator/cinder https://github.com/facebookincubator/cinder

ushakov 3y ago
Isn’t Mastodon written in Ruby?
- thewataccount 3y ago
  Yes, I wasn't sure if parent was suggesting they switch to django, or simply "using what facebook learned" - I didn't want to imply that there's no way Mastodon devs couldn't use Cinder to their advantage. I do however highly doubt Cinder would be of use to Mastodon at the current moment.
parthokr 3y ago
recently they made cinder public… this is a JIT’d version of cpython

smfugit 3y ago

These systems should not scale. Giving everyone free broadcast capability is a sign of mentally bankruptcy. Its like giving every neuron in your head broadcast capability. What happens when they all fire at the same time? Or like giving everyone in the world a radio transmitter that can broadcast on any frequency. What happens when they all transmit on the same frequency at the same time? These idiots are scaling things because its become possible technically. There is no other logic to whats going on. And its producing/enabling massive amounts of useless noise. The UN report on the Attention Economy says 0.5% of content on the net is actually consumed by anyone.

tivert 3y ago
> These systems should not scale. Giving everyone free broadcast capability is a sign of mentally bankruptcy. ... These idiots are scaling things because its become possible technically. There is no other logic to whats going on. I agree, but I don't think it's a sign of "mentally bankruptcy," but an indication of the flaws and limitations of contemporary ideology. They're not just doing it because it's technically possible, they wouldn't even try and no one would sign up if our ideology didn't say it was a good thing to do.

deleted 3y ago

[deleted]

freedomben 3y ago

Python/Django (and most other frameworks) isn't the problem with scaling. It's following the 12-factor app rules.

bee_rider 3y ago

Python is known to be pretty wasteful in terms of compute performance, but is the compute grunt of a given node a big deal in these kinds of networks? I assumed it is a big communication problem. Python shouldn’t hurt there, right? It isn’t like a Python-generated packet goes through the network more slowly.

KaiserPro 3y ago

> t would be possible for Mastodon to learn from this so that they too can scale to 200 million users. I mean possibly, if it was designed to be a centralised bohemoth with a staff of 600 to look after it. Its decentralised by design, much like the web used to be.

interactivecode 3y ago

So did they setup a separate backend for threads or did they just extend the current instagram backend to support text posts?