9 ms·
Our journey to type checking 4M lines of Python
- linux2647 7y agoGenuinely curious: how does a project like Dropbox grow to four million lines? In my (albeit naïve/inexperienced) mind, I could see how a file syncing app could grow to several thousand lines (even tens of thousands), but what all is in those four million lines? Is it a lot of UI code? Backwards compatibility for older data models? Building components themselves instead of just `pip install`ing?
- jdm2212 7y agoPer the blog post, it's both the client code and server code. The server code is >75% of the annotated Python. Which makes sense, given the amount of stuff you have to do to run one of the world's larger distributed file system with tight SLAs (and auth, and your own data centers, etc).
- paggle 7y agoClient support for every random configuration you can imagine (Windows XP in Arabic for example) and a ton of server side performance optimization.
- ampersandy 7y agoLet's just dig into one aspect of a file syncing app: storing the files at Dropbox scale. Without spending too long thinking about it, I'd say we need: * An API to request files/upload files * Authentication/encryption (how do we ensure DB employees aren't arbitrarily reading data on servers?) * Service that shards data and handles multi-region replication * That entails multiple datacenters (how much code does it take to keep a DC running?) * Automated backups to cold storage * Automated restoration + testing of cold storage volumes * Soft + hard deletion (deleting hot/warm/cold storage volumes reliably) * Error handling (retries, host errors, network failures, filesystem corruption, finding data when a host dies) * Fail-safes like blocking requests when hosts can't handle them, shedding load, etc. I'd also add that at this scale, very few libraries start handling the types of problems you have. For example, there might be a memcache library that makes writing queries easy. However, when you have thousands of memcache servers globally you can't just hardcode IP addresses anymore. You know have to write your own custom 'service-service' that lets you look up what host you should talk to for something.
- abernard1 7y agoI could imagine a list 10X that size that still does not need even 400K lines of Python code, much less 4M. I agree that people generally underestimate the lines of code needed to operate systems at scale, but having used dropbox, I'd still say that's excessive. Note, this is just the Python code, not the UI stuff or the golang or Rust. I've been in multiple 1M-10M line codebases at scale and I just cannot fathom with how with their product simplicity (not engineering simplicity) they could be at that size with a language as expressive as Python. My guess is this is generously counting a lot of forked libraries. Ungenerously, it makes me think there's a lot of NIH syndrome.
- jdm2212 7y agoThe worst, buggiest, least maintainable code I've ever dealt with was a 500ish line (heavily tested!) untyped Python file at my last job. Caused more trouble than all the Java and C++ and Go combined. My personal experience with compile time type checking is that the benefits kick in for a single developer project within a week, and for a multi-developer project immediately.
- tus88 7y agoHow are you sure typing would have made a difference to that piece of Python? Are you sure there is no troublesome code written anywhere in Java or C++?
- swiley 7y agoIMO type checking is to python what bounds checking is to C.
- speedplane 7y ago> IMO type checking is to python what bounds checking is to C. Interesting analogy, but in my experience it doesn't hold up. If you have a Python typing bug, it will crash your program almost immediately and you can quickly catch it and fix it. In contrast, C boundary checking issues often come up in relatively rare corner cases, and when they do occur, they often do not result in an exception / error especially if the memory is already paged and set to zero. In production code, boundary checking errors in C occur all the time, but Python type bugs are pretty rare.
- justwalt 7y agoI think he's talking specifically about the Java and C++ code that accompanied that Python file. I'm sure he's seen bad Java and C++ :)
- jdm2212 7y agoI've seen some disgusting Java and C++, but that company had pretty high engineering standards so the Java and C++ was always good. The Python was _supposed_ to be good, too, but 20+ devs making changes to this one config generator script meant people were constantly making their own (and breaking each other's) assumptions about how the code should behave.
- Waterluvian 7y agoHow is python's added typing compared to TypeScript? Is it expressive enough to completely supplant traditional Python or is it just something you add in sometimes to help out a little?
- justwalt 7y agoIf I'm not mistaken, all of the types are akin to comments: completely ignored by the interpreter.
- jdm2212 7y agoSo there's the same situation as with TS where you can't do runtime reflection to get type annotations? That seems frustrating. The one really rough edge I've seen in TS is type guards. Not the end of the world, obviously, but still kinda gross coming from Java.
- j88439h84 7y agoYou can get annotations for functions and classes with `typing.get_type_hints()`.
- michael_j_ward 7y agoYou can get type annotations at runtime. And you can use this for validation with something you roll yourself or with 3rd party packages. I've used pydantic [1] before and found it quite nice. [1] https://pydantic-docs.helpmanual.io https://pydantic-docs.helpmanual.io
- deleted 7y ago[deleted]
- j88439h84 7y agoPretty much all of my Python is typed with attrs and mypy. It's great.
- 7y ago
- stmw 7y agoThis is impressive. But IMHO, anyone starting today should just use Rust from the beginning, and avoid having to endure all of the problems for years and then taking on a big HN-worthy post about 4M lines.
- jdm2212 7y agoHaving never used Rust: how's the ecosystem around it like for maintaining it on production servers? Are there equivalents of Java Mission Control and `jcmd`?
- stmw 7y agoThe ecosystem is good and growing, but not as extensive as java, of course. We use generic cloud monitoring/SRE tools (think statsd, not jcmd, for example). From prior experience, I also think JVMs somehow tend to require more of this kind of tooling, compared to Rust - or, for that matter, other languages & runtimes.
- jdm2212 7y agoI'm curious how much Java needs the tooling in an all else equal situation (totally possible 'cuz Java is _old_) vs how much Java has all this cool tooling that enables/encourages people to do things that aren't possible in Go but which require the tooling. I've mostly used the Java, C++ and Go in my career, and it's pretty amazing to me that I can write write safe, readable code (like Go) on top of high-performance libraries (like C++) and step through code on a prod machine with a remote debugger, get live performance metrics with a single CLI command, etc.
- viraptor 7y agoNot in the same way. Since there's no significant runtime behind rust, there's no specific way to query it in an abstract way. On the other hand, native code means that you can use any common way to query the system. Anything you'd use for C or C++ will work as well. Profilers, stack querying, monitoring the process itself, user-level probes, etc. will work just fine. There's lots of tools that will work for this.
- j88439h84 7y agoUsing mypy with http://attrs.org http://attrs.org for all my classes has been a revolution in my Python codebases. I can't emphasize enough how big a difference it makes in readability and correctness. @attr.dataclass class Person: name: str age: int
- meowface 7y agoWhat advantage is there to using attrs instead of the new dataclasses module in the standard library?
- j88439h84 7y agoFrom "Why attrs"[1]: PEP 557 added Data Classes to Python 3.7 that resemble attrs in many ways. They are the result of the Python community’s wish to have an easier way to write classes in the standard library that doesn’t carry the problems of namedtuples. To that end, attrs and its developers were involved in the PEP process and while we may disagree with some minor decisions that have been made, it’s a fine library and if it stops you from abusing namedtuples, they are a huge win. Nevertheless, there are still reasons to prefer attrs over Data Classes whose relevancy depends on your circumstances: attrs supports all mainstream Python versions, including CPython 2.7 and PyPy. Data Classes are intentionally less powerful than attrs. There is a long list of features that were sacrificed for the sake of simplicity and while the most obvious ones are validators, converters, and __slots__, it permeates throughout all APIs. On the other hand, Data Classes currently do not offer any significant feature that attrs doesn’t already have. attrs can and will move faster. We are not bound to any release schedules and we have a clear deprecation policy. One of the reasons to not vendor attrs in the standard library was to not impede attrs’s future developement. [1] http://www.attrs.org/en/stable/why.html#data-classes http://www.attrs.org/en/stable/why.html#data-classes
- arugulum 7y agoMy own experience with dataclasses vs attrs: `dataclass` primarily just provides a faster syntax for defining a simple struct-like class. For that alone I've found it tremendously useful. At the same time, sometimes you do need more functionality around a "struct-like" class, that that's where attrs comes in (e.g. I use dataclasses/attrs to wrap configuration objects, which mostly just hold values but occasionally need a little more logic). Even then, I often end up extending the basic dataclass with some of my own common methods (e.g. serialization).
- j88439h84 7y agoThe article mentions PyPy wasn't fast enough on their code, so they wrote a whole "new language". I wonder if they tried contributing some relevant performance improvements to PyPy. I ask because in my experience PyPy can make a massive difference if I'm using it on CPU-intensive code, and it'd benefit everyone to have a faster PyPy.
- zawerf 7y agoDropbox's implementation of python was pyston: https://github.com/dropbox/pyston https://github.com/dropbox/pyston But I think it's dead now: https://blog.pyston.org/ https://blog.pyston.org/
- j88439h84 7y ago> We thought about running it automatically on every test build and/or collecting types from a small fraction of live network requests, but decided against it as either approach is too risky. I wonder what's too risky about it? Did you try it and something broke? Instagram's MonkeyType seems to think it works ok in production. https://monkeytype.readthedocs.io/en/stable/stores.html https://monkeytype.readthedocs.io/en/stable/stores.html
- boulos 7y agoTreating “all the types we happened to see at runtime” as “all the types allowed” automatically is I think what they meant as “risky”. MonkeyType seems to record the answers and then let you decide (via apply) if you want to agree. I’m a little surprised more people didn’t use this approach, but it probably makes for fairly clumsy type sets that the human is better off saying “Eh, I’ll just document it correctly”.
- epage 7y agoI feel mypy has been a big help in the correctness and approachability in my code I write. My big issue is that it is too easy for `Any` to (implicitly) show up, making it so parts of my code are (silently) not checked. I wish there was a good way to spot this; maybe I need to dig more into the docs.
- j88439h84 7y agomypy --help Disallow the use of the dynamic 'Any' type under certain conditions. --disallow-any-unimported Disallow Any types resulting from unfollowed imports --disallow-subclassing-any Disallow subclassing values of type 'Any' when defining classes (inverse: --allow-subclassing- any) --disallow-any-expr Disallow all expressions that have type Any --disallow-any-decorated Disallow functions that have Any in their signature after decorator transformation --disallow-any-explicit Disallow explicit Any in type positions --disallow-any-generics Disallow usage of generic types that do not specify explicit type parameters (inverse: --allow-any-generics)
- azhenley 7y agoThis is an incredible story! Love hearing how academic projects can evolve into success stories like this. I'm looking forward to an update a few years from now.
- hannofcart 7y agoOn a near daily basis, I thank you guys for writing mypy. I have a personal project spanning 2 years or so that I used mypy on 3 months back, and it has been a pleasure to work with ever since. Just a few days back, I had had to refactor one of the core classes used everywhere and mypy told me exactly what to fix. At that point I remember wondering what I'd have done without mypy. Thanks a ton.
- gitgud 7y agoThe article show's several graphs of Lines of Code over Time. Isn't this an outdated way to measure anything in software? "Measuring software productivity by lines of code is like measuring progress on an airplane by how much it weighs." - Bill Gates
- thundergolfer 7y agoIt’s quite a decent way to show how much you use a language in an organisation. With this much Python, Dropbox is in a position where integrating MyPy with Bazel is worth the trouble.
- stagger87 7y agoBut they aren't measuring software productivity by lines of code...
- chrisseaton 7y agoWhere did you get the idea that they’re measuring productivity by lines of code? I think you’ve imagined that. Nothing in the article suggests they’re measuring anything but the scale of the problem.
- speedplane 7y ago> The article show's several graphs of Lines of Code over Time. Isn't this an outdated way to measure anything in software? "Measuring software productivity by lines of code is like measuring progress on an airplane by how much it weighs." - Bill Gates Lines of code is not a good measure of productivity, but it's a relatively decent measure of complexity. Bad developers can write many lines of code that adds little productive functionality, but still increases the overall code-base's complexity. A 100k LOC code base is probably not 100X more functional than a 1k LOC code base, but it is very roughly 100X more complex.
- ajxs 7y agoThe only shocking thing here is that no one stepped in earlier. It's not like Java/C# are radical new technologies in the online space. Obviously statically typed languages are not a new concept either. I'm just aghast at this culture. It's actually quite shocking that this debate is a real one taking up a large amount of airtime in the industry.
- flyinglizard 7y agoIt’s even more shocking that the solution is to shoehorn some half assed typing to Python instead of dropping it altogether in favor of something suitable.
- mbar84 7y ago> I was trying to find ways to make it possible to use the > same programming language for projects ranging from tiny > scripts to multi-million line sprawling codebases, > without compromising too much at any point in the > continuum. An important part of this was the idea of > gradual growth from an untyped prototype to a battle- > tested, statically typed product.
- ledauphin 7y agoMyPy has worked very well for our small team of backend developers. It's already production-ready in my opinion, but it still lacks some modern type-system features that you may be used to from other languages. Ironically, for instance, before Protocols were introduced (and they're still not part of Python core), there was no way to express structural subtyping in MyPy (for a duck-typed language!). And I'd argue they've made the same mistake (and a worse one in some ways) with TypedDict, which is a strict/nominal set rather than an a structural one, making it impossible to use on many real-world collections that have key-value pairs that you want to forgo typing for whatever reason. Still, Python with MyPy is a huge step up from Python without, and mypyc looks interesting as well.
- Barrin92 7y agothe article mentions that the desktop client app also uses python, is dropbox still using qt? the reason I'm asking is I remember a blog post from earlier this summer announcing the new desktop app which looked very webtech-ish.
- siempreb 7y agoAbout 4 million lines of Python code is just a bad decision from the start. Now they are trying to fix it with types, but IMHO the fault is not in the language. They just should have started out with a statically typed language that scales better.
- ledauphin 7y agoYeah, I'm sure Dropbox's founders remember and regret that fateful day when they said to each other, let's write 4 million lines of Python to get this fledgling business idea off the ground!