21 ms·

Open-sourcing MonkeyType – Let your Python code type-hint itself

427 points by ambivalence 9y ago

ValleyOfTheMtns 9y ago
FYI, it requires Python 3.6+. It mentions it in the article towards the end, but if you're like me and prefer to jump straight into trying something out you may not have seen it. I wasted a bit of time trying to figure out what the ContextManager is in the Python typing module and why it couldn't find it.
JosephRedfern 9y ago
This sounds similar to Dropbox's PyAnnotate -- Guido van Rossum writes about it here: http://mypy-lang.blogspot.co.uk/2017/11/dropbox-releases-pyannotate-auto.html http://mypy-lang.blogspot.co.uk/2017/11/dropbox-releases-pya... Would be interesting to see how MonkeyType and PyAnnotate compare.
- sbuccini 9y ago
  I'd also like to hear more about this -- both the feature set and the development process. It's interesting that two large engineering organizations responsible for some of the most popular applications on the planet spent hundreds of engineering hours building almost the same exact tool at around the exact same time. I'd also like to know how much quicker or better it could have been completed if it had been done out in the open.
  - maxxxxx 9y ago
    "I'd also like to know how much quicker or better it could have been completed if it had been done out in the open. " I assume they scratched their own itch so it was probably faster to do it on their own addressing their specific needs.
  - ambivalence 9y ago
    The idea to gather types at runtime is as old as PEP 484. The Dropbox and Facebook teams working on Python type checking know each other. We both worked on our implementations independently since we wanted to first test internally whether the idea holds water. For example, I personally thought it wouldn't be as useful in practice as it turned out to be! We knew we're going to open-source each others' implementations, esp. that Instagram's is focusing solely on Python 3 which isn't useful for Dropbox at the moment. It just took a while to get through the process of open sourcing what we had (cleaning up the early implementation with limited documentation, decoupling from internal data stores, etc.). Would it be cheaper if this started out in the open? Probably, but I don't think by quite the margin as you expect.
    - Traveler42 9y ago
      Even though MonkeyType is focused on Python 3, can the .pyi files be used with Python 2?
- ambivalence 9y ago
  MonkeyType is Python 3-centric, PyAnnotate is Python 2-centric. PyAnnotate couples re-applying the types with the tool, and uses type comments for this. MonkeyType generates .pyi files that you can either use directly or re-apply them to your code as proper type annotations. Other than that, MonkeyType is used on a daily basis internally and solves a bunch of common annoyances of systems like these, like duplicates in unions, applying better types to stuff that was already hand-annotated with Any, etc.
muizelaar 9y ago
I wonder how this compares to PyAnnotate: http://mypy-lang.blogspot.ca/2017/11/dropbox-releases-pyannotate-auto.html http://mypy-lang.blogspot.ca/2017/11/dropbox-releases-pyanno...
hacker_9 9y ago
"At Instagram we have hundreds of engineers working on well over a million lines of Python 3." It always amazes me that some of the most popular products around are built with the worst technology choices. And now they had to build their own static type checker, which slows down random samples of real users, just to shore up the language's weaknesses? Outstanding.
- grandmczeb 9y ago
  If lots of successful projects use what you consider the “worst” technology, perhaps the problem is with your perception rather than the technology?
  - devoply 9y ago
    No the person has a point. Often the weakness of the tech stack are not known ahead of time. Or what is considered a weakness is not a weakness when the choices are initially made. Most developers if they had to do stuff over again would make different choices in one way or another. Also later in the game if you ever get there, when you have capital then certain problems that were caused by the choices made can be resolved a different way... as is the case here.
  - alsadi 9y ago
    Those people believe that the right choice is assembly unless you can write directly in hex /s
    - AnimalMuppet 9y ago
      Nonsense. Octal, not hex. (Not totally a joke. My mother wrote in octal before she got an assembler.)
    - thechao 9y ago
      I was taught assembly at the job by an old assembly guru. His first statement to me was something along the lines of: "we'll use an assembler to start with, but it doesn't generate very good code; so, once you're comfortable, we'll hand assemble our machine code".
troels 9y ago
Interestingly, I made a similar tool for php some time ago. It was recently revived and is now in active development to bring it up to speed with recent developments in the language. https://github.com/troelskn/phpweaver https://github.com/troelskn/phpweaver
- muglug 9y ago
  That is fascinating. I’m the author of a static analysis tool for PHP that can generate types, but clearly not at the same level as runtime analysis. I’ll use it on my company’s codebase and report back.
  - troels 9y ago
    It would be great if you can give any feedback. There are quite a few rough edges currently.
zarita_existe 9y ago
Yet another hacky solution to address one of the biggest Python’s flaws. Why not switch to Java instead?
- sixo 9y ago
  Because the product would never have been released given the same constraints on developer time x count x qualifications
  - anon11082016 9y ago
    Maybe for the good of us all, maybe in that case they SHOULD have used Java ;)
- alsadi 9y ago
  One man's flaw is another's feature. Life is too short to strict typing
- scrollaway 9y ago
  I was going to address the "why" by explaining actually why (Refactoring time required, impact difference, other negative tradeoffs, active devs on the projects / hiring requirements, benefit to the global python community, ...). But it looks like this is a troll account, so flag & move on.
  - deleted 9y ago
    [deleted]
maltalex 9y ago
Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are looking for ways to make the maintenance easier? And before I get down-voted to hell - I completely understand using Python for many things. It a good technical choice for many different problems, but navigating a million lines of Python seems just daunting to me (although maybe I'm just not experienced enough with Python).
- hacker_9 9y ago
  The evolution is fairly simple: python is very easy to setup, especially with django for working on the web, and for writing scripts quickly it work really well when everything is just an 'import' away. Then as the product gets bigger, you'll hire python developers to keep up with the workload - and the best ones will be the ones who have committed their lives to Python. So you'll now end up with more and more python code. Before you know it, you aren't writing small scripts anymore, but now you are writing quite large features, that take weeks and that require intimate knowledge of the code base so you don't keep backtracking and repeating yourself. But Python doesn't help you at this point, you traded static types for flexibility and now you have to pay the price. At this point you're screwed, too many man hours spent on the codebase to redo it, so what to do? Well if you have the man power, build your own static type checker of course! I mean after all, if you have 100s of engineers who cares? You just throw more people at the problem until it goes away. Then wrap it up in a nice little package, and slap yourself on the back while you ride the instagram bubble.
  - oblio 9y ago
    Rewriting a whole app would be at least as big of an investment, in my opinion...
    - sidlls 9y ago
didibus 9y ago
Its interesting, this is a similar learning Clojure came around with, that the types weren't really useful unless everything is typed. Though Typed Racket's solution was to promote types to runtime validation at those borders between things with types and things without. I do find it intriguing though, that adding back types manually is so hard and slow. Is it slower when done retroactively? Or is it just as slow when done at the same time, but we don't realize its overhead?
- lmm 9y ago
  It's a lot slower to do retroactively. You basically have to tell the computer why you believe something is correct - e.g. if you're moving to having a distinct type for non-empty lists because some functions are only valid for non-empty lists, you have to explain why you believe a list you're passing to such a function is non-empty. That's a lot easier to do at the same time you're doing it (even in Python you'd probably still ask yourself whether you knew the list was non-empty as you were writing it) than to come back months or years later and remember why.
- mratzloff 9y ago
  It's slower when done retroactively. A developer writing typed code has the problem domain and software design unmarshaled into his brain while writing his small section. Type testing occurs concurrently with feature testing. Types added retroactively need to be applied to a large codebase and undergo separate testing.
Cieplak 9y ago
Given that people have asked why not use a statically typed language, seems appropriate to mention that it's possible to write pythonic-looking C++: http://preshing.com/20141202/cpp-has-become-more-pythonic/ http://preshing.com/20141202/cpp-has-become-more-pythonic/ I have been using C++ a lot lately but really wish there were more tools for reflection at compile time, e.g., ability to iterate over all the members of a class. Other than that, I'm really loving C++17's auto template parameters and type deduction capabilities, plus code that's 200x faster at runtime than most interpreted languages. I've found autocompletion in CLion to be slightly better than autocompletion PyCharm, but not quite as good as IPython or IntelliJ with Java.
- aub3bhat 9y ago
  Please show me C++ equivalent of sorted([(k.weight, k.name) for k in somelist], reverse=True)
  - nickelbox 9y ago
    You could implement the following with a decent bit of work (declaring k and reversed to be variables of special, hand-written/macro-generated types with overloaded operator, and operator=). sorted( (k.weight, k.name).for_(k).in(somelist), reverse=true)); You would never be able to get that past code review, however.
    - joshuamorton 9y ago
      Sure, but at that point you've just reimplemented python in macros ;) The other thing to note is that sorted([(k.weight, k.name) for k in somelist], reverse=True) is essentially already typechecked: def biggest_ks(ks: K): return sorted([(k.weight, k.name) for k in ks], reverse=True) The above code now has all the same type guarantees as your c++, actually maybe more since the macros you use are going to be...uhhh, mysterious.
      - nickelbox 9y ago
        The macro would only be used to generate k for convenience. It could be all implemented in straight, macro-less C++98 in O(minutes). My point was just that you can implement almost whatever you want (even without macros, they'll just expand the design space).
domenukk 9y ago
Similarly, I'm amazed by Pycharm (IDE), which supports pretty good inferred type hints from debugging and static code anlysis, btw. Makes writing code a lot easier. Looking forward to trying MonkeyType, it seems awesome for lager projects.
- pvg 9y ago
  Pycharm relies on pre-generated annotations as well, they're just built-in - it's not as magically trace-y and infer-y as it might seem at a casual glance.
CGamesPlay 9y ago
Why didn't you make the Python 3 type checking advisory instead? Like what Facebook did with Flow and Hack, why not make write a product that lets you statically analyze the types and will never itself cause runtime errors, and transition to types that way? What advantage does building this tool have? I understand that the thing I'm describing involves modifying the way Python 3 handles type annotations, but it doesn't seem like more work than building out all of the instrumentation you've done here.
- ambivalence 9y ago
  I'm not quite sure what you mean. Python has an external static checker for types, it's called mypy. Python's type annotations are in fact very similar to Flow and Hack in the sense that they provide gradual typing. The specification (see: PEP 484) describes that only annotated functions are type checked. Calls to non-annotated code are treated as accepting any type in arguments and returning the Any type (a special type which effectively silences the type checker). This generates a chicken and egg problem: if you don't have enough functions annotated, the type checker won't be able to provide meaningful output to you. So convincing people to annotate their code is harder: they don't see the benefit right away. Worse yet, you already have tens of thousands of functions in your code that you know work in production but were written before type annotations were introduced. It's not really feasible to come back and fill this information manually. MonkeyType is a tool that gathers types at runtime and enables putting them back in your code as annotations. The goal is for mypy to have more information to work with, making it way more useful.
  - CGamesPlay 9y ago
    Sorry, I guess I jumped to the conclusion that you profiled in production because you had a mandatory type checker which would cause runtime errors if invalid types were passed. Let me try another question. Do you think that a project like MonkeyType would substantially aid adding flow annotations to a JavaScript project? If so, why is the value to the Python ecosystem higher than the value to the Flow/JavaScript ecosystem (i.e. why doesn't it exist for JS)?
alexchamberlain 9y ago
Fantastic contribution back to the community; I look forward to trying it out. I must say this is the first time I've been disappointed with the quality of discussion on HN. For a community that promotes using the right tool for the job at the time, I would have thought people would be more open to the choices the early engineers made. I'm sure that Instagram are using a variety of tech across their stack.
- blub 9y ago
  Most people would expect software to crash, hang, be slow or somehow leak their personal information. That's normal behaviour for the products of the software industry. For a long time there have been efforts to ensure at least a degree of quality and robustness through processes, practices and verification tools. One such tool is a type system which allows encoding requirements and expectations that will be automatically verified with the help of a checker. This is exactly what Instagram is attempting in an effort to increase quality and ease maintenance. It seems many are wondering why they haven't done this from the beginning. A typical (and probably correct) answer is development speed and flexibility: early stage companies need to be nimble and compromise on quality if they want to survive. Fair enough, I understand that. We can applaud Instagram's success on the markets, but that doesn't have to mean that they're a role model of technical excellence for building a million line Python code base. This tool is the proof that Python has significant problems at scale, which is something the Python community has denied for a long time. They're still doing it in this thread, but the lesson looks pretty clear to me: if you plan on building large scale, don't use Python. Or PHP while we're at it (see Facebook). Definitely not JavaScript (see FirefoxOS). The founders of a start-up can continue to do whatever they want in the name of success. Instagram at their beginnings was basically a different company from today's Instagram, no one would have used them as a technical role-model. Now they're at best a role model for large Python code bases, but many here seem to be drawing the wrong conclusion, namely that it's a good idea to do large-scale Python in the first place.
  - kbd 9y ago
    > This tool is the proof that Python has significant problems at scale... This does not prove your point. Annotating a large dynamically-typed codebase with type information is a large amount of work, regardless of the language. This tool makes that easier.
jimnotgym 9y ago
Reading all of the comments from engineers who seem to either posess a time machine to send current tech back in time, or are criticising the technical choices that made the founders $squillions, is making me a bit mad. As a diversion perhaps some of them could list a few billion dollar startups that made perfect choices at the start and never had any cause to refactor or reimplement code as they grew?
- mbid 9y ago
  It's not obvious (and IMO somewhat doubtful) that it was the technical choices of the founders that made them successful. Their choices could well have been bad, just not bad enough to make their business fail.
  - deleted 9y ago
    [deleted]
- swsieber 9y ago
  Whatsap? Just kidding, they probably had to refactor code too. But their technical story seems a lot better than most startups
true_religion 9y ago
How is this different from Dropbox's PyAnnotate[1] which was released a few weeks ago? [1] https://github.com/dropbox/pyannotate https://github.com/dropbox/pyannotate
dilap 9y ago
> At Instagram we have hundreds of engineers working on well over a million lines of Python 3. Man, that's crazy. At the time they were acquired by Facebook, they had 13 employees.
- kilpikaarna 9y ago
  This was the part that stood out to me also. They must have a ton of new stuff in the pipeline, or their "display photos, insert some ads in between" loop is way more complex than it seems. Or that's total number of engineers and way fewer actually twiddle the Python...
sitkack 9y ago
As both a dynamic and static type enthusiast, back typing dynamic code is extremely problematic. Fluent use of a dynamic language will use and create constructs that are nearly un-typeable. If you want to make typed code, start typed. If you code with implicit types, use a good type inferred language (ML, F#, etc). If you want to use type checking in Python, use the annotations and MyPy from the beginning. That said, I am not saying this tool is bad. It could very well help a lot of codebases, but I would warn against using it as part of the operational workflow.
- viraptor 9y ago
  > Fluent use of a dynamic language will use and create constructs that are nearly un-typeable. Do you mean cases where you accept "anything iterable", an issue with callbacks / template types, or something else?
- joshuamorton 9y ago
  >Fluent use of a dynamic language will use and create constructs that are nearly un-typeable. Can you give examples? In practice, I haven't noticed this being an issue.
- joshuamorton 9y ago
  >Fluent use of a dynamic language will use and create constructs that are nearly un-typeable. Can you give examples? In practice, I haven't noticed this being an issue. A lot of python code already had/has docstrings explaining the types, these annotations just formalize that a bit more.
Lxr 9y ago
Is their goal to annotate everything or just the non-obvious things? Also how would a tool like this handle cases where the “correct” type is a generic base class but at runtime it only sees a certain subclass? To be pythonic, a function that accepts a tuple should usually also accept a list for example, but at runtime that may never happen.
- ambivalence 9y ago
  Since this is how gradual typing works, the goal is to annotate every last function. Good question about abstract base classes! Paraphrasing a well known cliché: types in functions should be forgiving in arguments (what the function accepts) and strict in return values (what the function emits). In our case, the human reviewer needs to decide if the argument types collected by MonkeyType should be generalized. In fact, the collected types might not even work in all cases and the type checker might complain. It's because annotations describe "what should be" whereas MonkeyType finds "what is". This is why a system like MonkeyType shouldn't even attempt to use abstract base classes in place of concrete types that it collected.
BucketSort 9y ago
I come from a statically typed background (C++), but have been doing a lot of analytics in python in the past two years. It is frustrating not to have compile time guarantees when dealing with mathematical programs, because some things have to be a particular type (i.e. matrices of compatible dimensions). The result is a copious use of asserts, but it feels bad when you know that if you did this in a functional language,let's say, you could prove implementations are correct by the nature of the type system. In short, I'd love to see more strong type support in python.
- joshuamorton 9y ago
  >(i.e. matrices of compatible dimensions) What language do you use where you can get these kinds of guarantees? As far as I know very few languages provide those kinds of dependent types statically.
  - OskarS 9y ago
    In graphics programming, you have specific types for lots of small vectors and matrices (for vectors, there are separate types for every sizes 4 and below, and matrices usually comes in a variety of sizes as well, at the very least 3x3, 3x4 and 4x4). Typechecking is very useful here: if you try to transform a point in space represented by a Vector3 by a general 4x4 matrix, it fails compilation because you have to convert the point to homogeneous coordinates first. Very useful information from the type system.
    - joshuamorton 9y ago
      I'm aware of that, I was more thinking in the general case, as comes up in machine learning for example, where you have sizes like 128, 192, and odd shapes like 12x3x100x100 4-tensors, etc. That is, generalized matrix types, not simply rotation matrix types or whatnot for special cases.
  - antoinealb 9y ago
    Eigen (a popular matrix library) does it in c++.
tandav 9y ago
Why not use cython? It has types
z3t4 9y ago
> With MonkeyType’s help, we’ve already annotated over a third of the functions in our codebase, and we’re already seeing type-checking catch many bugs that would have otherwise likely shipped to production Are most of their code not yet in production ? Or why do they produce more bugs now with static types, then before with dynamic types !? This sounds a lot like homeopathy, that can both detect and cure diseases with placebo.
mkolodny 9y ago
This looks wonderful :) I'd love to try it out at some point. One thing that I think could really improve the documentation is a few examples! One of my favorite things about the Python docs and the community is the wealth of examples. From looking at the docs, I couldn't find the main thing I wanted to see - what would MoneyType's annotations look like if I used it?

427 points by ambivalence 9y ago

FYI, it requires Python 3.6+. It mentions it in the article towards the end, but if you're like me and prefer to jump straight into trying something out you may not have seen it. I wasted a bit of time trying to figure out what the ContextManager is in the Python typing module and why it couldn't find it.

JosephRedfern 9y ago

This sounds similar to Dropbox's PyAnnotate -- Guido van Rossum writes about it here: http://mypy-lang.blogspot.co.uk/2017/11/dropbox-releases-pyannotate-auto.html http://mypy-lang.blogspot.co.uk/2017/11/dropbox-releases-pya... Would be interesting to see how MonkeyType and PyAnnotate compare.

sbuccini 9y ago
I'd also like to hear more about this -- both the feature set and the development process. It's interesting that two large engineering organizations responsible for some of the most popular applications on the planet spent hundreds of engineering hours building almost the same exact tool at around the exact same time. I'd also like to know how much quicker or better it could have been completed if it had been done out in the open.
- maxxxxx 9y ago
  "I'd also like to know how much quicker or better it could have been completed if it had been done out in the open. " I assume they scratched their own itch so it was probably faster to do it on their own addressing their specific needs.
- ambivalence 9y ago
  The idea to gather types at runtime is as old as PEP 484. The Dropbox and Facebook teams working on Python type checking know each other. We both worked on our implementations independently since we wanted to first test internally whether the idea holds water. For example, I personally thought it wouldn't be as useful in practice as it turned out to be! We knew we're going to open-source each others' implementations, esp. that Instagram's is focusing solely on Python 3 which isn't useful for Dropbox at the moment. It just took a while to get through the process of open sourcing what we had (cleaning up the early implementation with limited documentation, decoupling from internal data stores, etc.). Would it be cheaper if this started out in the open? Probably, but I don't think by quite the margin as you expect.
  - Traveler42 9y ago
    Even though MonkeyType is focused on Python 3, can the .pyi files be used with Python 2?
ambivalence 9y ago
MonkeyType is Python 3-centric, PyAnnotate is Python 2-centric. PyAnnotate couples re-applying the types with the tool, and uses type comments for this. MonkeyType generates .pyi files that you can either use directly or re-apply them to your code as proper type annotations. Other than that, MonkeyType is used on a daily basis internally and solves a bunch of common annoyances of systems like these, like duplicates in unions, applying better types to stuff that was already hand-annotated with Any, etc.

muizelaar 9y ago

I wonder how this compares to PyAnnotate: http://mypy-lang.blogspot.ca/2017/11/dropbox-releases-pyannotate-auto.html http://mypy-lang.blogspot.ca/2017/11/dropbox-releases-pyanno...

hacker_9 9y ago

"At Instagram we have hundreds of engineers working on well over a million lines of Python 3." It always amazes me that some of the most popular products around are built with the worst technology choices. And now they had to build their own static type checker, which slows down random samples of real users, just to shore up the language's weaknesses? Outstanding.

grandmczeb 9y ago
If lots of successful projects use what you consider the “worst” technology, perhaps the problem is with your perception rather than the technology?
- devoply 9y ago
  No the person has a point. Often the weakness of the tech stack are not known ahead of time. Or what is considered a weakness is not a weakness when the choices are initially made. Most developers if they had to do stuff over again would make different choices in one way or another. Also later in the game if you ever get there, when you have capital then certain problems that were caused by the choices made can be resolved a different way... as is the case here.
- alsadi 9y ago
  Those people believe that the right choice is assembly unless you can write directly in hex /s
  - AnimalMuppet 9y ago
    Nonsense. Octal, not hex. (Not totally a joke. My mother wrote in octal before she got an assembler.)
  - thechao 9y ago
    I was taught assembly at the job by an old assembly guru. His first statement to me was something along the lines of: "we'll use an assembler to start with, but it doesn't generate very good code; so, once you're comfortable, we'll hand assemble our machine code".

troels 9y ago

Interestingly, I made a similar tool for php some time ago. It was recently revived and is now in active development to bring it up to speed with recent developments in the language. https://github.com/troelskn/phpweaver https://github.com/troelskn/phpweaver

muglug 9y ago
That is fascinating. I’m the author of a static analysis tool for PHP that can generate types, but clearly not at the same level as runtime analysis. I’ll use it on my company’s codebase and report back.
- troels 9y ago
  It would be great if you can give any feedback. There are quite a few rough edges currently.

zarita_existe 9y ago

Yet another hacky solution to address one of the biggest Python’s flaws. Why not switch to Java instead?

sixo 9y ago
Because the product would never have been released given the same constraints on developer time x count x qualifications
- anon11082016 9y ago
  Maybe for the good of us all, maybe in that case they SHOULD have used Java ;)
alsadi 9y ago
One man's flaw is another's feature. Life is too short to strict typing
scrollaway 9y ago
I was going to address the "why" by explaining actually why (Refactoring time required, impact difference, other negative tradeoffs, active devs on the projects / hiring requirements, benefit to the global python community, ...). But it looks like this is a troll account, so flag & move on.
- deleted 9y ago
  [deleted]

maltalex 9y ago

Reading this as someone who writes mostly in statically typed languages, the whole exercise seems odd. Having so much dynamically typed code to maintain that you need to run production code using a separate tool just to figure out the types sounds just wrong. Why not use a statically typed language for such a large code-base? Is this done by purpose, or did they end up with a million lines of Python code and are looking for ways to make the maintenance easier? And before I get down-voted to hell - I completely understand using Python for many things. It a good technical choice for many different problems, but navigating a million lines of Python seems just daunting to me (although maybe I'm just not experienced enough with Python).

hacker_9 9y ago
The evolution is fairly simple: python is very easy to setup, especially with django for working on the web, and for writing scripts quickly it work really well when everything is just an 'import' away. Then as the product gets bigger, you'll hire python developers to keep up with the workload - and the best ones will be the ones who have committed their lives to Python. So you'll now end up with more and more python code. Before you know it, you aren't writing small scripts anymore, but now you are writing quite large features, that take weeks and that require intimate knowledge of the code base so you don't keep backtracking and repeating yourself. But Python doesn't help you at this point, you traded static types for flexibility and now you have to pay the price. At this point you're screwed, too many man hours spent on the codebase to redo it, so what to do? Well if you have the man power, build your own static type checker of course! I mean after all, if you have 100s of engineers who cares? You just throw more people at the problem until it goes away. Then wrap it up in a nice little package, and slap yourself on the back while you ride the instagram bubble.
- oblio 9y ago
  Rewriting a whole app would be at least as big of an investment, in my opinion...
  - sidlls 9y ago

didibus 9y ago

Its interesting, this is a similar learning Clojure came around with, that the types weren't really useful unless everything is typed. Though Typed Racket's solution was to promote types to runtime validation at those borders between things with types and things without. I do find it intriguing though, that adding back types manually is so hard and slow. Is it slower when done retroactively? Or is it just as slow when done at the same time, but we don't realize its overhead?

lmm 9y ago
It's a lot slower to do retroactively. You basically have to tell the computer why you believe something is correct - e.g. if you're moving to having a distinct type for non-empty lists because some functions are only valid for non-empty lists, you have to explain why you believe a list you're passing to such a function is non-empty. That's a lot easier to do at the same time you're doing it (even in Python you'd probably still ask yourself whether you knew the list was non-empty as you were writing it) than to come back months or years later and remember why.
mratzloff 9y ago
It's slower when done retroactively. A developer writing typed code has the problem domain and software design unmarshaled into his brain while writing his small section. Type testing occurs concurrently with feature testing. Types added retroactively need to be applied to a large codebase and undergo separate testing.

Cieplak 9y ago

Given that people have asked why not use a statically typed language, seems appropriate to mention that it's possible to write pythonic-looking C++: http://preshing.com/20141202/cpp-has-become-more-pythonic/ http://preshing.com/20141202/cpp-has-become-more-pythonic/ I have been using C++ a lot lately but really wish there were more tools for reflection at compile time, e.g., ability to iterate over all the members of a class. Other than that, I'm really loving C++17's auto template parameters and type deduction capabilities, plus code that's 200x faster at runtime than most interpreted languages. I've found autocompletion in CLion to be slightly better than autocompletion PyCharm, but not quite as good as IPython or IntelliJ with Java.

aub3bhat 9y ago
Please show me C++ equivalent of sorted([(k.weight, k.name) for k in somelist], reverse=True)
- nickelbox 9y ago
  You could implement the following with a decent bit of work (declaring k and reversed to be variables of special, hand-written/macro-generated types with overloaded operator, and operator=). sorted( (k.weight, k.name).for_(k).in(somelist), reverse=true)); You would never be able to get that past code review, however.
  - joshuamorton 9y ago
    Sure, but at that point you've just reimplemented python in macros ;) The other thing to note is that sorted([(k.weight, k.name) for k in somelist], reverse=True) is essentially already typechecked: def biggest_ks(ks: K): return sorted([(k.weight, k.name) for k in ks], reverse=True) The above code now has all the same type guarantees as your c++, actually maybe more since the macros you use are going to be...uhhh, mysterious.
    - nickelbox 9y ago
      The macro would only be used to generate k for convenience. It could be all implemented in straight, macro-less C++98 in O(minutes). My point was just that you can implement almost whatever you want (even without macros, they'll just expand the design space).

domenukk 9y ago

Similarly, I'm amazed by Pycharm (IDE), which supports pretty good inferred type hints from debugging and static code anlysis, btw. Makes writing code a lot easier. Looking forward to trying MonkeyType, it seems awesome for lager projects.

pvg 9y ago
Pycharm relies on pre-generated annotations as well, they're just built-in - it's not as magically trace-y and infer-y as it might seem at a casual glance.

CGamesPlay 9y ago

Why didn't you make the Python 3 type checking advisory instead? Like what Facebook did with Flow and Hack, why not make write a product that lets you statically analyze the types and will never itself cause runtime errors, and transition to types that way? What advantage does building this tool have? I understand that the thing I'm describing involves modifying the way Python 3 handles type annotations, but it doesn't seem like more work than building out all of the instrumentation you've done here.

ambivalence 9y ago
I'm not quite sure what you mean. Python has an external static checker for types, it's called mypy. Python's type annotations are in fact very similar to Flow and Hack in the sense that they provide gradual typing. The specification (see: PEP 484) describes that only annotated functions are type checked. Calls to non-annotated code are treated as accepting any type in arguments and returning the Any type (a special type which effectively silences the type checker). This generates a chicken and egg problem: if you don't have enough functions annotated, the type checker won't be able to provide meaningful output to you. So convincing people to annotate their code is harder: they don't see the benefit right away. Worse yet, you already have tens of thousands of functions in your code that you know work in production but were written before type annotations were introduced. It's not really feasible to come back and fill this information manually. MonkeyType is a tool that gathers types at runtime and enables putting them back in your code as annotations. The goal is for mypy to have more information to work with, making it way more useful.
- CGamesPlay 9y ago
  Sorry, I guess I jumped to the conclusion that you profiled in production because you had a mandatory type checker which would cause runtime errors if invalid types were passed. Let me try another question. Do you think that a project like MonkeyType would substantially aid adding flow annotations to a JavaScript project? If so, why is the value to the Python ecosystem higher than the value to the Flow/JavaScript ecosystem (i.e. why doesn't it exist for JS)?

alexchamberlain 9y ago

Fantastic contribution back to the community; I look forward to trying it out. I must say this is the first time I've been disappointed with the quality of discussion on HN. For a community that promotes using the right tool for the job at the time, I would have thought people would be more open to the choices the early engineers made. I'm sure that Instagram are using a variety of tech across their stack.

blub 9y ago
Most people would expect software to crash, hang, be slow or somehow leak their personal information. That's normal behaviour for the products of the software industry. For a long time there have been efforts to ensure at least a degree of quality and robustness through processes, practices and verification tools. One such tool is a type system which allows encoding requirements and expectations that will be automatically verified with the help of a checker. This is exactly what Instagram is attempting in an effort to increase quality and ease maintenance. It seems many are wondering why they haven't done this from the beginning. A typical (and probably correct) answer is development speed and flexibility: early stage companies need to be nimble and compromise on quality if they want to survive. Fair enough, I understand that. We can applaud Instagram's success on the markets, but that doesn't have to mean that they're a role model of technical excellence for building a million line Python code base. This tool is the proof that Python has significant problems at scale, which is something the Python community has denied for a long time. They're still doing it in this thread, but the lesson looks pretty clear to me: if you plan on building large scale, don't use Python. Or PHP while we're at it (see Facebook). Definitely not JavaScript (see FirefoxOS). The founders of a start-up can continue to do whatever they want in the name of success. Instagram at their beginnings was basically a different company from today's Instagram, no one would have used them as a technical role-model. Now they're at best a role model for large Python code bases, but many here seem to be drawing the wrong conclusion, namely that it's a good idea to do large-scale Python in the first place.
- kbd 9y ago
  > This tool is the proof that Python has significant problems at scale... This does not prove your point. Annotating a large dynamically-typed codebase with type information is a large amount of work, regardless of the language. This tool makes that easier.

jimnotgym 9y ago

Reading all of the comments from engineers who seem to either posess a time machine to send current tech back in time, or are criticising the technical choices that made the founders $squillions, is making me a bit mad. As a diversion perhaps some of them could list a few billion dollar startups that made perfect choices at the start and never had any cause to refactor or reimplement code as they grew?

mbid 9y ago
It's not obvious (and IMO somewhat doubtful) that it was the technical choices of the founders that made them successful. Their choices could well have been bad, just not bad enough to make their business fail.
- deleted 9y ago
  [deleted]
swsieber 9y ago
Whatsap? Just kidding, they probably had to refactor code too. But their technical story seems a lot better than most startups

true_religion 9y ago

How is this different from Dropbox's PyAnnotate[1] which was released a few weeks ago? [1] https://github.com/dropbox/pyannotate https://github.com/dropbox/pyannotate

dilap 9y ago

> At Instagram we have hundreds of engineers working on well over a million lines of Python 3. Man, that's crazy. At the time they were acquired by Facebook, they had 13 employees.

kilpikaarna 9y ago
This was the part that stood out to me also. They must have a ton of new stuff in the pipeline, or their "display photos, insert some ads in between" loop is way more complex than it seems. Or that's total number of engineers and way fewer actually twiddle the Python...

sitkack 9y ago

As both a dynamic and static type enthusiast, back typing dynamic code is extremely problematic. Fluent use of a dynamic language will use and create constructs that are nearly un-typeable. If you want to make typed code, start typed. If you code with implicit types, use a good type inferred language (ML, F#, etc). If you want to use type checking in Python, use the annotations and MyPy from the beginning. That said, I am not saying this tool is bad. It could very well help a lot of codebases, but I would warn against using it as part of the operational workflow.

viraptor 9y ago
> Fluent use of a dynamic language will use and create constructs that are nearly un-typeable. Do you mean cases where you accept "anything iterable", an issue with callbacks / template types, or something else?
joshuamorton 9y ago
>Fluent use of a dynamic language will use and create constructs that are nearly un-typeable. Can you give examples? In practice, I haven't noticed this being an issue.
joshuamorton 9y ago
>Fluent use of a dynamic language will use and create constructs that are nearly un-typeable. Can you give examples? In practice, I haven't noticed this being an issue. A lot of python code already had/has docstrings explaining the types, these annotations just formalize that a bit more.

Lxr 9y ago

Is their goal to annotate everything or just the non-obvious things? Also how would a tool like this handle cases where the “correct” type is a generic base class but at runtime it only sees a certain subclass? To be pythonic, a function that accepts a tuple should usually also accept a list for example, but at runtime that may never happen.

ambivalence 9y ago
Since this is how gradual typing works, the goal is to annotate every last function. Good question about abstract base classes! Paraphrasing a well known cliché: types in functions should be forgiving in arguments (what the function accepts) and strict in return values (what the function emits). In our case, the human reviewer needs to decide if the argument types collected by MonkeyType should be generalized. In fact, the collected types might not even work in all cases and the type checker might complain. It's because annotations describe "what should be" whereas MonkeyType finds "what is". This is why a system like MonkeyType shouldn't even attempt to use abstract base classes in place of concrete types that it collected.

BucketSort 9y ago

I come from a statically typed background (C++), but have been doing a lot of analytics in python in the past two years. It is frustrating not to have compile time guarantees when dealing with mathematical programs, because some things have to be a particular type (i.e. matrices of compatible dimensions). The result is a copious use of asserts, but it feels bad when you know that if you did this in a functional language,let's say, you could prove implementations are correct by the nature of the type system. In short, I'd love to see more strong type support in python.

joshuamorton 9y ago
>(i.e. matrices of compatible dimensions) What language do you use where you can get these kinds of guarantees? As far as I know very few languages provide those kinds of dependent types statically.
- OskarS 9y ago
  In graphics programming, you have specific types for lots of small vectors and matrices (for vectors, there are separate types for every sizes 4 and below, and matrices usually comes in a variety of sizes as well, at the very least 3x3, 3x4 and 4x4). Typechecking is very useful here: if you try to transform a point in space represented by a Vector3 by a general 4x4 matrix, it fails compilation because you have to convert the point to homogeneous coordinates first. Very useful information from the type system.
  - joshuamorton 9y ago
    I'm aware of that, I was more thinking in the general case, as comes up in machine learning for example, where you have sizes like 128, 192, and odd shapes like 12x3x100x100 4-tensors, etc. That is, generalized matrix types, not simply rotation matrix types or whatnot for special cases.
- antoinealb 9y ago
  Eigen (a popular matrix library) does it in c++.

tandav 9y ago

Why not use cython? It has types

z3t4 9y ago

> With MonkeyType’s help, we’ve already annotated over a third of the functions in our codebase, and we’re already seeing type-checking catch many bugs that would have otherwise likely shipped to production Are most of their code not yet in production ? Or why do they produce more bugs now with static types, then before with dynamic types !? This sounds a lot like homeopathy, that can both detect and cure diseases with placebo.

mkolodny 9y ago

This looks wonderful :) I'd love to try it out at some point. One thing that I think could really improve the documentation is a few examples! One of my favorite things about the Python docs and the community is the wealth of examples. From looking at the docs, I couldn't find the main thing I wanted to see - what would MoneyType's annotations look like if I used it?