13 ms·

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that y

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are looking for ways to make the maintenance easier? And before I get down-voted to hell - I completely understand using Python for many things. It a good technical choice for many different problems, but navigating a million lines of Python seems just daunting to me (although maybe I'm just not experienced enough with Python).

hacker_9 9y ago
The evolution is fairly simple: python is very easy to setup, especially with django for working on the web, and for writing scripts quickly it work really well when everything is just an 'import' away. Then as the product gets bigger, you'll hire python developers to keep up with the workload - and the best ones will be the ones who have committed their lives to Python. So you'll now end up with more and more python code. Before you know it, you aren't writing small scripts anymore, but now you are writing quite large features, that take weeks and that require intimate knowledge of the code base so you don't keep backtracking and repeating yourself. But Python doesn't help you at this point, you traded static types for flexibility and now you have to pay the price. At this point you're screwed, too many man hours spent on the codebase to redo it, so what to do? Well if you have the man power, build your own static type checker of course! I mean after all, if you have 100s of engineers who cares? You just throw more people at the problem until it goes away. Then wrap it up in a nice little package, and slap yourself on the back while you ride the instagram bubble.
- oblio 9y ago
  Rewriting a whole app would be at least as big of an investment, in my opinion...
  - sidlls 9y ago
    But not as useful for the resumes of the devs involved.
    - scrollaway 9y ago
      Did it occur to you that rewriting an internal app has no benefit to the community at large, whereas publishing a tool is a clear improvement on the community's tools? Did it occur to you that instagram published a valuable and useful tool that now just exists, and this is now a non-issue for anyone else in their situation? Like, why are you complaining about resume boasting?! It's like you want people to do useless work that has no positive effect on the open source community. Are you just upset people are using Python or what?
nawgszy 9y ago
>Is this done by purpose, or did they end up with a million lines of Python code and are looking for ways to make the maintenance easier? Definitely the latter. I've seen this discussion a few times before, and it's always the same. Your initial developers are not looking down the road to the million lines of code milestone, they're just trying to make a product that might actually make some money here and now. I'm sure Instagram was exactly that. They needed to handle images and some guy knew how to do it in Python. They wrote Python code, and then people liked Instagram. They eventually became a billion dollar company with millions of lines of code and no where along the road was there time to say "hey we need to refactor this whole thing". Or if that was said, management laughed and said "we need this feature". So here is where you end up. The developers need to clean things up but they don't have time to clean it up by using a language, realistically, they probably don't know as well as the Python they wrote the millions of lines of code in. Re: your last comment, navigating a million lines of any codebase is daunting, and especially more so if you aren't a developer in that language. I'm not sure what exactly "Python" has to do with that, besides that you're not a Python dev.
- rusk 9y ago
  I'm not sure what exactly "Python" has to do with that Well as he said, a statically typed language is better in that kind of situation because it enables a better class of tooling and the typing system enforces certain style constraints, that enables better quality of code analysis en mass. Python specifically is very lightweight in this regards with little in the way of naming constraints (vs for instance Ruby having different formatting rules for different types) So yeah, it’s not “just like any other language” - horses for courses
- imiric 9y ago
  To add to this, note that type hinting is quite a new feature in Python (introduced in v3.5, released in 2015), and this functionality simply wasn't available before. So any company heavily invested in Python today obviously wants to improve their runtime reliability, without having to rewrite parts of their stack. Stricter typing goes a long way to achieve this, and gradual typing allows you to upgrade the code base at your own pace, which is great. Consider this study[0] about TypeScript and Flow, which use the same approach for JavaScript, which found both able to detect ~15% of runtime bugs. So no wonder companies with large Python code bases would be the first to invest in this space. Personally I feel this is a great addition to the language, and hope type checking becomes a first class citizen too, instead of being delegated to external tools like mypy[1] or pytype[2]. [0]: http://ttendency.cs.ucl.ac.uk/projects/type_study/ http://ttendency.cs.ucl.ac.uk/projects/type_study/ [1]: http://mypy-lang.org/ http://mypy-lang.org/ [2]: https://github.com/google/pytype https://github.com/google/pytype
scrollaway 9y ago
I really don't get people. Instagram and Dropbox, through typing annotations in Python, are gradually improving a language that has codebases running globally, from YouTube to NASA. Clearly something is right with the situation when the incentives are aligned for a tech company to contribute back to the open source community in such fundamental ways. So why look for the mole and think "They should have done it differently", when doing it differently has a high likelihood to mean not being as successful as they are today, and not having the occasion to contribute back? It's like telling a successful charity "You should just take everyone's money and spend it on lamborghinis instead of wasting time building wells in africa".
- kod 9y ago
  No, it's like asking someone who spent a lot of time building an octagonal wheel and is now trying to shave down the corners... why didn't you use a circle to begin with.
  - ageofwant 9y ago
    They did not spend a lot of time building an octagonal wheel, they built a billion dollar company using Python. Now, when the code base has been proven, and the business rules solidified, they retrofit what they believe will make the code base easier to maintain. Python is an excellent enabler of this kind of dynamic system evolution.
    - angelsl 9y ago
      You've totally missed the analogy. See pfranz's comment.
      - depressedpanda 9y ago
        No, I the analogy just wasn't very apt. Python allowed them to build a successful company. Now, when their stack is mature and maintenance is more important than rapid prototyping, Python allows them to add type hinting. Because they are engineers, they built a tool (in Python) that allows them to do it in an automated manner. And all of this is great! They are evolving their code to fit their needs; it's nothing like making a octagonal wheel and wishing you'd have gone for a round one in the beginning.
bobwaycott 9y ago
You appear to be ignoring who this coming from. Instagram is huge, and they built their service with Python. It isn’t coming from someone who is arguing this is how it should be done from the beginning. Hell, given the typical lifecycle of a startup, this situation occurs exactly because a dynamic language is chosen to speed getting a product to market and try to build a business around it.
vasilipupkin 9y ago
I literally just had this exact thought independently of you. I mean if you need to do something like this, doesn't that mean you should be writing code in a statically typed language?
- nerfhammer 9y ago
  there's a difference between should be and should have been maybe it's easier to do this half-measure than rewrite your entire code base
  - vasilipupkin 9y ago
    I get that part, but for new projects, this perhaps is a hint to do things in Java?
    - newfoundglory 9y ago
      80% of startups are going to die before their codebase reaches this size. They should probably not be making tech stack choices based on "what if we succeed beyond all likelihood".
      - vasilipupkin 9y ago
        two things come to mind: maybe those startups would die at a slower rate if their code was comprehensible to begin with ? but leaving that aside, for companies where survival is not an issue then, doesn't this indicate that using a dynamically typed language is not great?
        panopticon 9y ago
        I would be interested in seeing the number of startups that fail due to technical debt. My instinct is that most startups fail for business reasons (no clear need, not enough/right sales, poor management, bad pitch, solving the wrong problem, etc).
        9y ago
Kalium 9y ago
It's rarely easy to re-implement large or complex codebases in a new language. That kind of major effort requires significant signoffs from leadership and a large effort. This tool? This tool takes an existing set of codebases and makes them safer. No major boiling of oceans required.
- deleted 9y ago
  [deleted]
azhenley 9y ago
Didn't Facebook do this for PHP (Hack) and Microsoft with JavaScript (TypeScript)? From a technical aspect, I do find these projects cool. I wonder if its more efficient for large companies to initially develop using dynamic languages then transition them with these optionally typed languages.
- vorotato 9y ago
  Small projects scream dynamic types, but not every baby never grows up. Eventually the cognitive load becomes crippling and you're crying for static types.
- xzel 9y ago
  Facebook did even more with PHP with HHVM and other tooling. I'm not sure if JS was exactly the same for Microsoft as I saw it more as a cool internal project that got big vs something that eventually became necessary.
  - WorldMaker 9y ago
    From what I read, several big teams at Microsoft were dogfooding Typescript immediately during development, even before it was ever externally released.
- jermaustin1 9y ago
  Dynamic languages definitely help with the rapid development of a small application or prototype, but when you have a large team trying to maintain the application, static typing allows for better compile time error catching. Weird hybrid approaches makes me wary.
jpmoyn 9y ago
This tool seems to be built for teams that have already created large codebases in non-typed languages. This is a pretty common thing today. They are addressing a very real problem that a lot of people face which is "oh we fucked up, what's the most painless way we can fix it". We just recently moved our entire codebase from JS to TypeScript which was pretty hard but super worth it.
realbarack 9y ago
I agree with you but I think it's pretty straightforward how this sort of thing happens: 1. Startup builds thing fast in dynamic language because they need to optimize for development speed and iteration, not maintainability or scalability. 2. Startup grows and continues to hire for expertise in the tech stack they are mostly already using. 3. Repeat for some years and some hundreds of engineers and you arrive at this exact scenario.
- fiatjaf 9y ago
  Isn't it faster to write everything in Go, for example, that has a compiler guiding you all the way and you rarely get runtime errors? I feel my developer time very much "optimized" when writing backends in Go than when I wrote then in Python (I also tried Node, which was a disaster).
  - xapata 9y ago
    Empirically? Doesn't seem like it. But it's hard to run a good controlled experiment. The confounding effects of the team members, the project goal, and the vagaries of business are too noisy.
  - jimnotgym 9y ago
    Go wasn't a serious option for Instagram though. According to Wikipedia Instagram launched in 2010, Go launched in 2009. That would have made them very early adopters, was the library support there back then like it was for Python? Just because there may be better tools now, should they scrap their working code that earned their fortune?
    - fiatjaf 9y ago
      I'm not talking about Instagram, I'm answering the parent comment, which talked about developer productivity in general.
    - lmm 9y ago
      Well, OCaml, Haskell or maybe Scala would've been mature options in 2010, combining safety with language-level productivity that's comparable to Python, though the extent to which web frameworks available in those languages at that time were Django-equivalent is arguable. (Personally I would - and did - happily choose Scala with Wicket over Django in 2010) In terms of what Instagram should do now, I'd say they should do what Facebook did: introduce thrift or similar, gradually move business logic into backend services written in more suitable languages, leaving the Python to eventually become just a thin web frontend. Retrofitting types onto code involves a lot of the same effort as rewriting it into a better language, and the rewards for the latter are higher, IME.
walrus1066 9y ago
Think you trade maintainability for initial productivity with a dynamic language. You can churn out a greenfield project faster as you don't have to spend upfront time mapping out interfaces, DTO's etc. It's also easier to 'hack'. Of course the above makes the code harder to maintain and reason with, unless written by very disciplined engineers. So these languages are a natural fit for startups (and things like prototyping and scripting). Instagram would have been in this category, and it's worked out well for them.
SatvikBeri 9y ago
My job is mostly machine learning and statistics. I would love to use something like Haskell or even Java, but the NumPy/SciPy/Scikit-learn/Pandas etc. ecosystem is just so far ahead of everything else that it's not worth it.
lambda 9y ago
> Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are looking for ways to make the maintenance easier? Being in a similar (though much smaller scale) situation ourselves, I suspect your latter suggestion is the answer; they wound up with a large amount of Python code due to expediency (Python is good at quickly getting things done), and are now finding the code base quite hard to maintain. Something to remember is that as of 10 years ago, statically typed languages consisted of C (for some definition of "statically typed"), C++ (complicated and cumbersome, and didn't yet have widespread availability of "modern" C++ features), Java (cumbersome, heavyweight, lots of missing opportunities for abstraction back then), C# (heavily tied into the Microsoft ecosystem, not yet open source), and a bunch of weird academic languages that required tutorials about burritos to learn (ML, Haskell, etc). While statically typed languages like C, C++, and Java were the mainstream languages of the 90s, C was too limited for a lot of people, C++ and Java too cumbersome to use, and so lightweight dynamically typed languages like Perl, Python, PHP, Ruby, and JavaScript picked up a lot of steam due to how much easier to pick up and more productive many programmers found themselves in those languages. But now we have large, fairly un-maintainable code bases in these languages, and people are realizing the value of static typing, in part due to the maintenance hassle and in part due to newer, more expressive and accessible statically typed languages being available (Elm, Rust, TypeScript, Scala, Go, Kotlin, as well as improvements to C++, Java, and C#). But that leaves all of these old codebases, that are hard to maintain. Rather than doing a complete rewrite, adding static typing capabilities that can be applied to existing codebases is a way to make them more maintainable without spending all of the time of a complete rewrite and having everyone have to spend all of the time learning the new language while still maintaining the old codebase.
- ageofwant 9y ago
  Also, these old code bases can now be fairly easily and optionally retrofitted using modern Python's support for type hints and tools like mypy and now MonkeyType. So people can continue using their existing code base and the vast Python ecosystem, with the added benefit of type checking where appropriate.
nikhilalmeida 9y ago
reasons (advantages) for using python (or any other dynamic lang.) when in startup mode is more than just typing. Similarly the advantages of continuing to use python outweigh the costs of doing this little "type-dance".
emmelaich 9y ago
There's lots of good and bad things about Python. The worst thing -- to me -- is dynamic typing. So why not fix it?
user5994461 9y ago
>>> but navigating a million lines of Python seems just daunting to me (although maybe I'm just not experienced enough with Python). It's not that bad. The first thing you learn when you're in a million lines codebase is that you will only work within your project of maybe a hundred files. Once in a while, there is a guy who is asking for help on his project or there is an old weird bug to fix and you dive in other stuff. Otherwise, it's like it's not there.
- AlexCoventry 9y ago
  Sounds like the projects you worked on had excellent encapsulation.
  - user5994461 9y ago
    I wouldn't say excellent encapsulation. It's just normal to use sub directories to split things. The project was over ten millions lines of python.
    - AlexCoventry 9y ago
      Subdirectory structure is sort of orthogonal to the issue of encapsulation. It's more about providing API's and clean abstractions which prevent incidental interactions between implementation details across module boundaries.
Yxven 9y ago
Personally, I would love to ditch python for a language with strong types and type inference, but what is the replacement for django? Where do I find a well-designed, well-documented, battle-tested framework that I can easily hire developers for? Currently, I think the nearest competitor is node with typescript, and I'd rather stick with python. Please tell me if I'm wrong.
- weavie 9y ago
  I dream of a time when OCaml becomes the obvious answer to this question.
- conceptme 9y ago
  PHP 7
- bennofs 9y ago
  Java? But that's another kind of hell
sidlls 9y ago
Rewriting over a million lines of code in a statically typed language coming from one likely as riddled with type errors as this codebase is is unlikely to be productive. They're making the best of a terrible situation. I also think that if they want to use types the correct approach is to apply a tool like this as a stop-gap but write new components going forward and bug fixes/feature re-writes in a language that supports types "properly" (i.e. in the way they seem to want, that is static types checked at compile time). I think tools like this are great for companies in situations like this. I don't think they're good to use from the outset: the team should just use an actually statically typed language instead.
nickm12 9y ago
It's important to realize that these tools just didn't exist 7-10 years ago when companies like Instagram and Dropbox were getting started. Though type inference has been around in so-called "academic" languages for decades, it hit a tipping point in the last 10 years, to the point that every major dynamic language has static type checkers or dialects (like Typescript) that support static checking. Meanwhile, even traditional statically typed languages are growing stronger type checkers.

hacker_9 9y ago

The evolution is fairly simple: python is very easy to setup, especially with django for working on the web, and for writing scripts quickly it work really well when everything is just an 'import' away. Then as the product gets bigger, you'll hire python developers to keep up with the workload - and the best ones will be the ones who have committed their lives to Python. So you'll now end up with more and more python code. Before you know it, you aren't writing small scripts anymore, but now you are writing quite large features, that take weeks and that require intimate knowledge of the code base so you don't keep backtracking and repeating yourself. But Python doesn't help you at this point, you traded static types for flexibility and now you have to pay the price. At this point you're screwed, too many man hours spent on the codebase to redo it, so what to do? Well if you have the man power, build your own static type checker of course! I mean after all, if you have 100s of engineers who cares? You just throw more people at the problem until it goes away. Then wrap it up in a nice little package, and slap yourself on the back while you ride the instagram bubble.

oblio 9y ago
Rewriting a whole app would be at least as big of an investment, in my opinion...
- sidlls 9y ago
  But not as useful for the resumes of the devs involved.
  - scrollaway 9y ago
    Did it occur to you that rewriting an internal app has no benefit to the community at large, whereas publishing a tool is a clear improvement on the community's tools? Did it occur to you that instagram published a valuable and useful tool that now just exists, and this is now a non-issue for anyone else in their situation? Like, why are you complaining about resume boasting?! It's like you want people to do useless work that has no positive effect on the open source community. Are you just upset people are using Python or what?

nawgszy 9y ago

>Is this done by purpose, or did they end up with a million lines of Python code and are looking for ways to make the maintenance easier? Definitely the latter. I've seen this discussion a few times before, and it's always the same. Your initial developers are not looking down the road to the million lines of code milestone, they're just trying to make a product that might actually make some money here and now. I'm sure Instagram was exactly that. They needed to handle images and some guy knew how to do it in Python. They wrote Python code, and then people liked Instagram. They eventually became a billion dollar company with millions of lines of code and no where along the road was there time to say "hey we need to refactor this whole thing". Or if that was said, management laughed and said "we need this feature". So here is where you end up. The developers need to clean things up but they don't have time to clean it up by using a language, realistically, they probably don't know as well as the Python they wrote the millions of lines of code in. Re: your last comment, navigating a million lines of any codebase is daunting, and especially more so if you aren't a developer in that language. I'm not sure what exactly "Python" has to do with that, besides that you're not a Python dev.

rusk 9y ago
I'm not sure what exactly "Python" has to do with that Well as he said, a statically typed language is better in that kind of situation because it enables a better class of tooling and the typing system enforces certain style constraints, that enables better quality of code analysis en mass. Python specifically is very lightweight in this regards with little in the way of naming constraints (vs for instance Ruby having different formatting rules for different types) So yeah, it’s not “just like any other language” - horses for courses
imiric 9y ago
To add to this, note that type hinting is quite a new feature in Python (introduced in v3.5, released in 2015), and this functionality simply wasn't available before. So any company heavily invested in Python today obviously wants to improve their runtime reliability, without having to rewrite parts of their stack. Stricter typing goes a long way to achieve this, and gradual typing allows you to upgrade the code base at your own pace, which is great. Consider this study[0] about TypeScript and Flow, which use the same approach for JavaScript, which found both able to detect ~15% of runtime bugs. So no wonder companies with large Python code bases would be the first to invest in this space. Personally I feel this is a great addition to the language, and hope type checking becomes a first class citizen too, instead of being delegated to external tools like mypy[1] or pytype[2]. [0]: http://ttendency.cs.ucl.ac.uk/projects/type_study/ http://ttendency.cs.ucl.ac.uk/projects/type_study/ [1]: http://mypy-lang.org/ http://mypy-lang.org/ [2]: https://github.com/google/pytype https://github.com/google/pytype

scrollaway 9y ago

I really don't get people. Instagram and Dropbox, through typing annotations in Python, are gradually improving a language that has codebases running globally, from YouTube to NASA. Clearly something is right with the situation when the incentives are aligned for a tech company to contribute back to the open source community in such fundamental ways. So why look for the mole and think "They should have done it differently", when doing it differently has a high likelihood to mean not being as successful as they are today, and not having the occasion to contribute back? It's like telling a successful charity "You should just take everyone's money and spend it on lamborghinis instead of wasting time building wells in africa".

kod 9y ago
No, it's like asking someone who spent a lot of time building an octagonal wheel and is now trying to shave down the corners... why didn't you use a circle to begin with.
- ageofwant 9y ago
  They did not spend a lot of time building an octagonal wheel, they built a billion dollar company using Python. Now, when the code base has been proven, and the business rules solidified, they retrofit what they believe will make the code base easier to maintain. Python is an excellent enabler of this kind of dynamic system evolution.
  - angelsl 9y ago
    You've totally missed the analogy. See pfranz's comment.
    - depressedpanda 9y ago
      No, I the analogy just wasn't very apt. Python allowed them to build a successful company. Now, when their stack is mature and maintenance is more important than rapid prototyping, Python allows them to add type hinting. Because they are engineers, they built a tool (in Python) that allows them to do it in an automated manner. And all of this is great! They are evolving their code to fit their needs; it's nothing like making a octagonal wheel and wishing you'd have gone for a round one in the beginning.

bobwaycott 9y ago

You appear to be ignoring who this coming from. Instagram is huge, and they built their service with Python. It isn’t coming from someone who is arguing this is how it should be done from the beginning. Hell, given the typical lifecycle of a startup, this situation occurs exactly because a dynamic language is chosen to speed getting a product to market and try to build a business around it.

vasilipupkin 9y ago

I literally just had this exact thought independently of you. I mean if you need to do something like this, doesn't that mean you should be writing code in a statically typed language?

nerfhammer 9y ago
there's a difference between should be and should have been maybe it's easier to do this half-measure than rewrite your entire code base
- vasilipupkin 9y ago
  I get that part, but for new projects, this perhaps is a hint to do things in Java?
  - newfoundglory 9y ago
    80% of startups are going to die before their codebase reaches this size. They should probably not be making tech stack choices based on "what if we succeed beyond all likelihood".
    - vasilipupkin 9y ago
      two things come to mind: maybe those startups would die at a slower rate if their code was comprehensible to begin with ? but leaving that aside, for companies where survival is not an issue then, doesn't this indicate that using a dynamically typed language is not great?
      - panopticon 9y ago
        I would be interested in seeing the number of startups that fail due to technical debt. My instinct is that most startups fail for business reasons (no clear need, not enough/right sales, poor management, bad pitch, solving the wrong problem, etc).
      - 9y ago

Kalium 9y ago

It's rarely easy to re-implement large or complex codebases in a new language. That kind of major effort requires significant signoffs from leadership and a large effort. This tool? This tool takes an existing set of codebases and makes them safer. No major boiling of oceans required.

deleted 9y ago
[deleted]

azhenley 9y ago

Didn't Facebook do this for PHP (Hack) and Microsoft with JavaScript (TypeScript)? From a technical aspect, I do find these projects cool. I wonder if its more efficient for large companies to initially develop using dynamic languages then transition them with these optionally typed languages.

vorotato 9y ago
Small projects scream dynamic types, but not every baby never grows up. Eventually the cognitive load becomes crippling and you're crying for static types.
xzel 9y ago
Facebook did even more with PHP with HHVM and other tooling. I'm not sure if JS was exactly the same for Microsoft as I saw it more as a cool internal project that got big vs something that eventually became necessary.
- WorldMaker 9y ago
  From what I read, several big teams at Microsoft were dogfooding Typescript immediately during development, even before it was ever externally released.
jermaustin1 9y ago
Dynamic languages definitely help with the rapid development of a small application or prototype, but when you have a large team trying to maintain the application, static typing allows for better compile time error catching. Weird hybrid approaches makes me wary.

jpmoyn 9y ago

This tool seems to be built for teams that have already created large codebases in non-typed languages. This is a pretty common thing today. They are addressing a very real problem that a lot of people face which is "oh we fucked up, what's the most painless way we can fix it". We just recently moved our entire codebase from JS to TypeScript which was pretty hard but super worth it.

realbarack 9y ago

I agree with you but I think it's pretty straightforward how this sort of thing happens: 1. Startup builds thing fast in dynamic language because they need to optimize for development speed and iteration, not maintainability or scalability. 2. Startup grows and continues to hire for expertise in the tech stack they are mostly already using. 3. Repeat for some years and some hundreds of engineers and you arrive at this exact scenario.

fiatjaf 9y ago
Isn't it faster to write everything in Go, for example, that has a compiler guiding you all the way and you rarely get runtime errors? I feel my developer time very much "optimized" when writing backends in Go than when I wrote then in Python (I also tried Node, which was a disaster).
- xapata 9y ago
  Empirically? Doesn't seem like it. But it's hard to run a good controlled experiment. The confounding effects of the team members, the project goal, and the vagaries of business are too noisy.
- jimnotgym 9y ago
  Go wasn't a serious option for Instagram though. According to Wikipedia Instagram launched in 2010, Go launched in 2009. That would have made them very early adopters, was the library support there back then like it was for Python? Just because there may be better tools now, should they scrap their working code that earned their fortune?
  - fiatjaf 9y ago
    I'm not talking about Instagram, I'm answering the parent comment, which talked about developer productivity in general.
  - lmm 9y ago
    Well, OCaml, Haskell or maybe Scala would've been mature options in 2010, combining safety with language-level productivity that's comparable to Python, though the extent to which web frameworks available in those languages at that time were Django-equivalent is arguable. (Personally I would - and did - happily choose Scala with Wicket over Django in 2010) In terms of what Instagram should do now, I'd say they should do what Facebook did: introduce thrift or similar, gradually move business logic into backend services written in more suitable languages, leaving the Python to eventually become just a thin web frontend. Retrofitting types onto code involves a lot of the same effort as rewriting it into a better language, and the rewards for the latter are higher, IME.

walrus1066 9y ago

Think you trade maintainability for initial productivity with a dynamic language. You can churn out a greenfield project faster as you don't have to spend upfront time mapping out interfaces, DTO's etc. It's also easier to 'hack'. Of course the above makes the code harder to maintain and reason with, unless written by very disciplined engineers. So these languages are a natural fit for startups (and things like prototyping and scripting). Instagram would have been in this category, and it's worked out well for them.

SatvikBeri 9y ago

My job is mostly machine learning and statistics. I would love to use something like Haskell or even Java, but the NumPy/SciPy/Scikit-learn/Pandas etc. ecosystem is just so far ahead of everything else that it's not worth it.

lambda 9y ago

> Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are looking for ways to make the maintenance easier? Being in a similar (though much smaller scale) situation ourselves, I suspect your latter suggestion is the answer; they wound up with a large amount of Python code due to expediency (Python is good at quickly getting things done), and are now finding the code base quite hard to maintain. Something to remember is that as of 10 years ago, statically typed languages consisted of C (for some definition of "statically typed"), C++ (complicated and cumbersome, and didn't yet have widespread availability of "modern" C++ features), Java (cumbersome, heavyweight, lots of missing opportunities for abstraction back then), C# (heavily tied into the Microsoft ecosystem, not yet open source), and a bunch of weird academic languages that required tutorials about burritos to learn (ML, Haskell, etc). While statically typed languages like C, C++, and Java were the mainstream languages of the 90s, C was too limited for a lot of people, C++ and Java too cumbersome to use, and so lightweight dynamically typed languages like Perl, Python, PHP, Ruby, and JavaScript picked up a lot of steam due to how much easier to pick up and more productive many programmers found themselves in those languages. But now we have large, fairly un-maintainable code bases in these languages, and people are realizing the value of static typing, in part due to the maintenance hassle and in part due to newer, more expressive and accessible statically typed languages being available (Elm, Rust, TypeScript, Scala, Go, Kotlin, as well as improvements to C++, Java, and C#). But that leaves all of these old codebases, that are hard to maintain. Rather than doing a complete rewrite, adding static typing capabilities that can be applied to existing codebases is a way to make them more maintainable without spending all of the time of a complete rewrite and having everyone have to spend all of the time learning the new language while still maintaining the old codebase.

ageofwant 9y ago
Also, these old code bases can now be fairly easily and optionally retrofitted using modern Python's support for type hints and tools like mypy and now MonkeyType. So people can continue using their existing code base and the vast Python ecosystem, with the added benefit of type checking where appropriate.

nikhilalmeida 9y ago

reasons (advantages) for using python (or any other dynamic lang.) when in startup mode is more than just typing. Similarly the advantages of continuing to use python outweigh the costs of doing this little "type-dance".

emmelaich 9y ago

There's lots of good and bad things about Python. The worst thing -- to me -- is dynamic typing. So why not fix it?

user5994461 9y ago

>>> but navigating a million lines of Python seems just daunting to me (although maybe I'm just not experienced enough with Python). It's not that bad. The first thing you learn when you're in a million lines codebase is that you will only work within your project of maybe a hundred files. Once in a while, there is a guy who is asking for help on his project or there is an old weird bug to fix and you dive in other stuff. Otherwise, it's like it's not there.

AlexCoventry 9y ago
Sounds like the projects you worked on had excellent encapsulation.
- user5994461 9y ago
  I wouldn't say excellent encapsulation. It's just normal to use sub directories to split things. The project was over ten millions lines of python.
  - AlexCoventry 9y ago
    Subdirectory structure is sort of orthogonal to the issue of encapsulation. It's more about providing API's and clean abstractions which prevent incidental interactions between implementation details across module boundaries.

Yxven 9y ago

Personally, I would love to ditch python for a language with strong types and type inference, but what is the replacement for django? Where do I find a well-designed, well-documented, battle-tested framework that I can easily hire developers for? Currently, I think the nearest competitor is node with typescript, and I'd rather stick with python. Please tell me if I'm wrong.

weavie 9y ago
I dream of a time when OCaml becomes the obvious answer to this question.
conceptme 9y ago
PHP 7
bennofs 9y ago
Java? But that's another kind of hell

sidlls 9y ago

Rewriting over a million lines of code in a statically typed language coming from one likely as riddled with type errors as this codebase is is unlikely to be productive. They're making the best of a terrible situation. I also think that if they want to use types the correct approach is to apply a tool like this as a stop-gap but write new components going forward and bug fixes/feature re-writes in a language that supports types "properly" (i.e. in the way they seem to want, that is static types checked at compile time). I think tools like this are great for companies in situations like this. I don't think they're good to use from the outset: the team should just use an actually statically typed language instead.

nickm12 9y ago

It's important to realize that these tools just didn't exist 7-10 years ago when companies like Instagram and Dropbox were getting started. Though type inference has been around in so-called "academic" languages for decades, it hit a tipping point in the last 10 years, to the point that every major dynamic language has static type checkers or dialects (like Typescript) that support static checking. Meanwhile, even traditional statically typed languages are growing stronger type checkers.