7 ms·

Web Service Efficiency at Instagram with Python

195 points by ksashikumar 10y ago

throwaway_45 10y ago
Do you real need all this engineering for a basically a CRUD application? What code are all these people deploying.
- erubin 10y ago
  Anyone can bake a cake. It's a lot harder to bake a billion cakes.
  - ben_jones 10y ago
    But baking cakes scales horizontally, just buy 10 million bakeries and hire 20 million bakers. You don't need a baking engineer to reinvent the oven, especially if that'll potentially end up costing you more in the long run. Throwing away the metaphor, IMO horizontal scaling is sometimes cheaper then vertical scaling.
    - teej 10y ago
      Yes, but if you have 10 million bakeries, you can afford a team of 10 software engineers to increase your efficiency to produce the same output with 7 million bakeries and 10 million bakers.
- optimuspaul 10y ago
  I'm sure it's far more than Crud. Access control, analytics, filtering, probably some pseudo-science, etc.
  - kirkdouglas 10y ago
    Also they run ads
pm90 10y ago
Wish they would share some of that profiling code...
fovc 10y ago
I wonder if they looked into using existing APM solutions before building out the profiler and monitor
lifeisstillgood 10y ago
I am digging into a gnarly bug in thousands of lines in python 2.6 right now and, well, a custom cProfile, C types support and auto-reporting sounds ... Lovely :-) But I know that they will have their rough edge cases and developers banging their heads on the desk making this custom code for their needs. But the grass does seem greener today :-)
lifeisstillgood 10y ago
I think on reflection however we need to remember, with all our sprints and agile iterations, that just in Instagram billions of web requests are being served by just dozens of developers based on code written over 25 years now, following good practises and hard work - kudos to the Python Dev team on 25 years.
- dalke 10y ago
  To be fair, that's more like "started 25 years ago". Very little of the original Python 1.0 code exists (especially given the great renaming when the "Py"/"PY"-prefix occurred.) I like to think of the thousands of developers involved. ;) For example, assuming Instagram uses gcc to build Python or any of the other components, gcc started in 1987. Parts of glibc have copyrights dating from the 1980s, like netinet/tcp.h Copyright (c) 1982, The Regents of the University of California .
  - lifeisstillgood 10y ago
    Of course ...
sorenjan 10y ago
If engineering time wasn't an issue, how much would a company of Instagram's scale save by switching from Python to a lower level language like C++? Pretend you could wave a magic wand and overnight have the entire code base transformed from Python to well written C++, how would that affect the hardware requirements, and future development? I guess what I'm really asking is what pros and cons are there with using dynamic and relatively slow languages compared to faster languages like C++, Java, or C#? My uninformed reaction is that it's easy for startups to get started with these easy languages but that it leads to problems when it's time to scale up. Dropbox is writing their own Python implementation, and Facebook have their PHP to C++ compiler.
- nemothekid 10y ago
  >how much would a company of Instagram's scale switching from Python to a lower level language like C++ I guess one thing you have to consider is 1. How many engineers do you have to hire to make Python "fast enough" vs 2. What does it cost to deploy a full C++ team? Is it easier for new programmers to learn & write safe C++ code vs. Python. I wonder what is cheaper?
  - SEJeff 10y ago
    s/C\+\+/golang/g
- kirkdouglas 10y ago
  From the users perspective they're doing fine (haven't had any problems with Instagram performance).
- draw_down 10y ago
  Major con: Good luck finding people who want to write a web app in C++.
  - weberc2 10y ago
    This is probably an extension of the "if developer time wasn't an issue"; at least that's a major factor for me (not being able to develop quickly).
acd 10y ago
Is Pypy used? http://speed.pypy.org/ http://speed.pypy.org/
piquadrat 10y ago
Instagram is a great story for Django, it's the best argument to bring against doubters of Django's scalability. But I wonder how much Instagram contributes to Django itself. I follow Django's development relatively closely, and I'm not aware of any major work contributed by Instagram engineers. They aren't listed on Django's fundraising page, neither. https://www.djangoproject.com/fundraising/ https://www.djangoproject.com/fundraising/
- kirkdouglas 10y ago
  As far as I know, Disqus (https://disqus.com/ https://disqus.com/) and Eventbrite (https://www.eventbrite.com/ https://www.eventbrite.com/) are also major users of Django.
  - piquadrat 10y ago
    Some of the most prolific Django developers work at Eventbrite. I can't say how much, if any, of their contributions to the Django ecosystem are on Eventbrite's dime, though.
    - andreagrandi 10y ago
      Well... Andrew Godwin ( https://twitter.com/andrewgodwin https://twitter.com/andrewgodwin ) works at Eventbrite. I think he doesn't need any introduction :) For lazy people: he wrote South, Django migrations and he's currently working on Channels.
- fefifofu 10y ago
  The fundraising page is depressing. They're only asking for US$200,000 and they only have 19% at half way through the year. In other news... Instagram hits half a billion monthly active users... but not on Django's fundraising page. I'm not knowledgeable about open-source issues, pros/cons, etc., but man... only 19% of $200K. Doesn't seem right given Python is one of the major languages and I believe Django is the most popular web framework in Python.
  - 10y ago
meteorfox 10y ago
This part I'm skeptical: "Compared to CPU time, CPU instruction is a better metric, as it reports the same numbers regardless of CPU models and CPU loads for the same request." CPU instructions will be more stable than CPU time for sure, and it does show that their metric is stable based on their chart but a single CPU instruction can take multiple cycles, especially if there are stalls in the pipeline or other processes are "polluting" the cache. Depending on the CPU model, the number of uops that can be issued concurrently varies, and the latency of these instructions will also vary on the size, and access pattern to their memory hierarchy. Also, what about a change in compiler version? That can also vary the number of instructions. Unless they are referring to a Python bytecode instruction as a CPU instruction. Would measuring CPI be a better indicative of their efficiency? They could also track both, no need to settle for one.
- zerohp 10y ago
  I agree with your skepticism. They will not detect a performance regression when there is a change in instruction mix or cache miss ratio.
wheaties 10y ago
Wow, CPU is the limiting factor? I've never been CPU bound in a Web app, only I/O bound. Am I doing something wrong? Granted, I don't use python anymore for Web dev. I'm using languages that can handle 10x the traffic per box that Django used to be able to handle. This isn't a knick against python. I love python. Just such a different perspective.
- hardwaresofton 10y ago
  This also immediately popped out to me as the wrong assumption to make (and start off with, for that matter)... But I could see it being true if: - You are essentially maxing out incoming and outgoing network IO - You have no database to speak to (or your DB is in-memory) In that case, I guess it is just a race to do some computation and fetch the right data from RAM, then the limiting factors would be RAM access and the number of the computations you could do to try and keep up with inbound requests?
- dchuk 10y ago
  What languages/frameworks are you using now?
- pbreit 10y ago
  I'm guessing that you're working on something that is fractionally as successful as Instagram. Getting to the point where you need to scale is hard, scaling is easy.
instakill 10y ago
If any Instagram people are reading this, please for the love of all that is sane, please I beg you add some kind of progress indicator for your API. 3 weeks in, I find myself loading my app's API client page to see if it's still stuck in a greyed out "go live" mode. I've got an "Instagram API" search saved for twitter and StackOverflow to check checking if anyone else gets to slip through but others struggle just as much. Just a little transparency and predictability is all that we ask for.
spullara 10y ago
They should really show what they are doing with their CPU. How many requests per second and their characteristics? Generally Python (Ruby, etc) is terribly inefficient relative to the next tier of languages like Java and Go. At Twitter we were able to reduce number of servers by 10x and decrease latency by 10x by moving the web application from Ruby to Scala/Java with no architectural changes, essentially byte for byte (and bug for bug) compatible. My guess is they have done this math and would rather pay for more servers and take the latency hit rather than rewrite it.
- deno 10y ago
  Did you at the time compare it with a JIT implementation like PyPy (for Python, not sure what would be the Ruby equivalent)?
  - pritambaral 10y ago
    A JITing implementation of Ruby would by JRuby. Depending on JRuby's state at that time, Twitter could have used JRuby and gradually rewritten parts of it in Scala. Perhaps.

195 points by ksashikumar 10y ago

throwaway_45 10y ago

Do you real need all this engineering for a basically a CRUD application? What code are all these people deploying.

erubin 10y ago
Anyone can bake a cake. It's a lot harder to bake a billion cakes.
- ben_jones 10y ago
  But baking cakes scales horizontally, just buy 10 million bakeries and hire 20 million bakers. You don't need a baking engineer to reinvent the oven, especially if that'll potentially end up costing you more in the long run. Throwing away the metaphor, IMO horizontal scaling is sometimes cheaper then vertical scaling.
  - teej 10y ago
    Yes, but if you have 10 million bakeries, you can afford a team of 10 software engineers to increase your efficiency to produce the same output with 7 million bakeries and 10 million bakers.
optimuspaul 10y ago
I'm sure it's far more than Crud. Access control, analytics, filtering, probably some pseudo-science, etc.
- kirkdouglas 10y ago
  Also they run ads

pm90 10y ago

Wish they would share some of that profiling code...

fovc 10y ago

I wonder if they looked into using existing APM solutions before building out the profiler and monitor

lifeisstillgood 10y ago

I am digging into a gnarly bug in thousands of lines in python 2.6 right now and, well, a custom cProfile, C types support and auto-reporting sounds ... Lovely :-) But I know that they will have their rough edge cases and developers banging their heads on the desk making this custom code for their needs. But the grass does seem greener today :-)

lifeisstillgood 10y ago

I think on reflection however we need to remember, with all our sprints and agile iterations, that just in Instagram billions of web requests are being served by just dozens of developers based on code written over 25 years now, following good practises and hard work - kudos to the Python Dev team on 25 years.

dalke 10y ago
To be fair, that's more like "started 25 years ago". Very little of the original Python 1.0 code exists (especially given the great renaming when the "Py"/"PY"-prefix occurred.) I like to think of the thousands of developers involved. ;) For example, assuming Instagram uses gcc to build Python or any of the other components, gcc started in 1987. Parts of glibc have copyrights dating from the 1980s, like netinet/tcp.h Copyright (c) 1982, The Regents of the University of California .
- lifeisstillgood 10y ago
  Of course ...

sorenjan 10y ago

If engineering time wasn't an issue, how much would a company of Instagram's scale save by switching from Python to a lower level language like C++? Pretend you could wave a magic wand and overnight have the entire code base transformed from Python to well written C++, how would that affect the hardware requirements, and future development? I guess what I'm really asking is what pros and cons are there with using dynamic and relatively slow languages compared to faster languages like C++, Java, or C#? My uninformed reaction is that it's easy for startups to get started with these easy languages but that it leads to problems when it's time to scale up. Dropbox is writing their own Python implementation, and Facebook have their PHP to C++ compiler.

nemothekid 10y ago
>how much would a company of Instagram's scale switching from Python to a lower level language like C++ I guess one thing you have to consider is 1. How many engineers do you have to hire to make Python "fast enough" vs 2. What does it cost to deploy a full C++ team? Is it easier for new programmers to learn & write safe C++ code vs. Python. I wonder what is cheaper?
- SEJeff 10y ago
  s/C\+\+/golang/g
kirkdouglas 10y ago
From the users perspective they're doing fine (haven't had any problems with Instagram performance).
draw_down 10y ago
Major con: Good luck finding people who want to write a web app in C++.
- weberc2 10y ago
  This is probably an extension of the "if developer time wasn't an issue"; at least that's a major factor for me (not being able to develop quickly).

acd 10y ago

Is Pypy used? http://speed.pypy.org/ http://speed.pypy.org/

piquadrat 10y ago

Instagram is a great story for Django, it's the best argument to bring against doubters of Django's scalability. But I wonder how much Instagram contributes to Django itself. I follow Django's development relatively closely, and I'm not aware of any major work contributed by Instagram engineers. They aren't listed on Django's fundraising page, neither. https://www.djangoproject.com/fundraising/ https://www.djangoproject.com/fundraising/

kirkdouglas 10y ago
As far as I know, Disqus (https://disqus.com/ https://disqus.com/) and Eventbrite (https://www.eventbrite.com/ https://www.eventbrite.com/) are also major users of Django.
- piquadrat 10y ago
  Some of the most prolific Django developers work at Eventbrite. I can't say how much, if any, of their contributions to the Django ecosystem are on Eventbrite's dime, though.
  - andreagrandi 10y ago
    Well... Andrew Godwin ( https://twitter.com/andrewgodwin https://twitter.com/andrewgodwin ) works at Eventbrite. I think he doesn't need any introduction :) For lazy people: he wrote South, Django migrations and he's currently working on Channels.
fefifofu 10y ago
The fundraising page is depressing. They're only asking for US$200,000 and they only have 19% at half way through the year. In other news... Instagram hits half a billion monthly active users... but not on Django's fundraising page. I'm not knowledgeable about open-source issues, pros/cons, etc., but man... only 19% of $200K. Doesn't seem right given Python is one of the major languages and I believe Django is the most popular web framework in Python.
- 10y ago

meteorfox 10y ago

This part I'm skeptical: "Compared to CPU time, CPU instruction is a better metric, as it reports the same numbers regardless of CPU models and CPU loads for the same request." CPU instructions will be more stable than CPU time for sure, and it does show that their metric is stable based on their chart but a single CPU instruction can take multiple cycles, especially if there are stalls in the pipeline or other processes are "polluting" the cache. Depending on the CPU model, the number of uops that can be issued concurrently varies, and the latency of these instructions will also vary on the size, and access pattern to their memory hierarchy. Also, what about a change in compiler version? That can also vary the number of instructions. Unless they are referring to a Python bytecode instruction as a CPU instruction. Would measuring CPI be a better indicative of their efficiency? They could also track both, no need to settle for one.

zerohp 10y ago
I agree with your skepticism. They will not detect a performance regression when there is a change in instruction mix or cache miss ratio.

wheaties 10y ago

Wow, CPU is the limiting factor? I've never been CPU bound in a Web app, only I/O bound. Am I doing something wrong? Granted, I don't use python anymore for Web dev. I'm using languages that can handle 10x the traffic per box that Django used to be able to handle. This isn't a knick against python. I love python. Just such a different perspective.

hardwaresofton 10y ago
This also immediately popped out to me as the wrong assumption to make (and start off with, for that matter)... But I could see it being true if: - You are essentially maxing out incoming and outgoing network IO - You have no database to speak to (or your DB is in-memory) In that case, I guess it is just a race to do some computation and fetch the right data from RAM, then the limiting factors would be RAM access and the number of the computations you could do to try and keep up with inbound requests?
dchuk 10y ago
What languages/frameworks are you using now?
pbreit 10y ago
I'm guessing that you're working on something that is fractionally as successful as Instagram. Getting to the point where you need to scale is hard, scaling is easy.

instakill 10y ago

If any Instagram people are reading this, please for the love of all that is sane, please I beg you add some kind of progress indicator for your API. 3 weeks in, I find myself loading my app's API client page to see if it's still stuck in a greyed out "go live" mode. I've got an "Instagram API" search saved for twitter and StackOverflow to check checking if anyone else gets to slip through but others struggle just as much. Just a little transparency and predictability is all that we ask for.

spullara 10y ago

They should really show what they are doing with their CPU. How many requests per second and their characteristics? Generally Python (Ruby, etc) is terribly inefficient relative to the next tier of languages like Java and Go. At Twitter we were able to reduce number of servers by 10x and decrease latency by 10x by moving the web application from Ruby to Scala/Java with no architectural changes, essentially byte for byte (and bug for bug) compatible. My guess is they have done this math and would rather pay for more servers and take the latency hit rather than rewrite it.

deno 10y ago
Did you at the time compare it with a JIT implementation like PyPy (for Python, not sure what would be the Ruby equivalent)?
- pritambaral 10y ago
  A JITing implementation of Ruby would by JRuby. Depending on JRuby's state at that time, Twitter could have used JRuby and gradually rewritten parts of it in Scala. Perhaps.