30 ms·
How we rolled out one of the largest Python 3 migrations
- thijsvandien 8y agoInteresting write-up, though it leaves me terrified how a relatively small, more or less single purpose application like the Dropbox client has over 1 million lines of code in Python alone.
- larrik 8y agoDropbox doesn't feel that small or "single purpose" to me anymore.
- thijsvandien 8y agoWhat does the Dropbox client do for me other than syncing files and exposing a bit of the online functionality such as generating share links? (Serious question...)
- iscrewyou 8y agoWe can simply start by asking what does “syncing files” include? Watching files. Keeping backup of files. Keeping conflicts resolved. Watching Selective Sync files and folders. Watching Smart Sync files and folders. Notifications for synced files. Etc. etc. There’s way more the client does than what I mention.
- thijsvandien 8y agoI agree that syncing is far from trivial, but that doesn't change it's a single purpose.
- kiriakasis 8y agoNot to build a straw man but by this logic also a browser is a single purpose application
- paulddraper 8y agoA rocket ship is also "single purpose."
- PurpleRamen 8y agoI don't think there is much code for conflict-resolving in dropbox. Usually in case of conflicts it renames one of the involved files and add a message about conflict and the date to the name and moves on.
- varenc 8y agoPerhaps there's lots of code to minimize the number of times a real conflict occurs?
- paulddraper 8y ago> small > more or less single purpose application > Dropbox client Heh.
- fermienrico 8y agoI found the Dropbox lecture [1] at Stanford one of the most riveting things ever. There is just so much technology behind Dropbox, it is staggering. There is a reason why it is so much better than iCloud sync, Google Drive, Box or OneDrive. [1] https://www.youtube.com/watch?v=PE4gwstWhmc https://www.youtube.com/watch?v=PE4gwstWhmc
- travbrack 8y agoThis led me to find another loosely related but very entertaining piece of dropbox history. The original "Show HN" post: [1]. It's funny to see so much skepticism knowing now what the company became. [1] https://news.ycombinator.com/item?id=8863 https://news.ycombinator.com/item?id=8863
- billforsternz 8y agoYes, this is one of the classics - right up there with the "less space than a Nomad, no wireless, lame" comment (which wasn't on HN I don't think - but we all know it could have been :) Edit: I see the motherlode is in place earlier in the thread "Especially when you could build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem"
- mixmastamyk 8y agoThe iPod comment was from slashdot, if memory serves.
- wiredfool 8y agoYes. CmdrTaco, when posting to the home page: https://slashdot.org/story/01/10/23/1816257/Apple-releases-iPod https://slashdot.org/story/01/10/23/1816257/Apple-releases-i... (that post is old enough to vote in this year's election).
- anentropic 8y agoha, that's a nice counterpart to the 'trivial' thread above https://news.ycombinator.com/item?id=18071820 https://news.ycombinator.com/item?id=18071820
- panic 8y agoThe size of a codebase is less about its function and more about the number of people adding code to it.
- thijsvandien 8y agoWhat would you consider the main cause(s) of that?
- deleted 8y ago[deleted]
- panic 8y agoThe job of a programmer is to write code, so that’s what they do. I doubt there are very many people at Dropbox whose job it is to remove code!
- maltalex 8y agoThat’s an extremely narrow definition of a programmer’s job. I think that in terms of lines if code, my contribution to many projects was net negative.
- deleted 8y ago[deleted]
- MisterOctober 8y agoI dunno - refactoring often involves removal of a fair bit of code
- icebraining 8y agoThe classic story: https://www.folklore.org/StoryView.py?story=Negative_2000_Lines_Of_Code.txt https://www.folklore.org/StoryView.py?story=Negative_2000_Li...
- flukus 8y agoIn turn this becomes the number of people required to maintain a codebase.
- chubot 8y agoYou might only use a small part of Dropbox, but I bet there is a lot of functionality that you don't care about but which is critcial to Dropbox as a business/product for others. The fact that you think it's small means they're probably doing something right! Relevant: https://danluu.com/sounds-easy/ https://danluu.com/sounds-easy/ (FWIW I don't use Dropbox myself, but I definitely had people ask me why Google needed 3,000 employees back in the day. Apparently it now has nearly 90K employees.)
- carlmr 8y agoFrom the article >There's also a wide body of research that's found that decreasing latency has a roughly linear effect on revenue over a pretty wide range of latencies for some businesses. Increasing performance also has the benefit of reducing costs. I wish he cited some of that research, because Google doesn't show much except for this amazon study with the 100ms. I'm especially interested if there's any research on engineering tools and their latency (long build times) etc., which are chronically under addressed in quite a few large corporations. I'm just wondering if there's some studies that would make the case for me if I were to present this to management.
- PascLeRasc 8y agoEspecially when you could build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem!
- orf 8y agoSarcasm doesn't translate well on the internet, so I'm really not sure if suggesting using CVS (of all things) over a mounted FTP share as a replacement for Dropbox is a joke!
- _ZeD_ 8y agoYou have no idea how web development is different from desktop one
- Marazan 8y agoIt might be single purpose but that single purpose is a reallllllly difficult problem.
- rawoke083600 8y agoI would have thought that Go would be perfect for this sort of system-level-like programming ?
- kenhwang 8y agoLooks like they already have Rust/C++/obj-c in use, which all fill that purpose a bit better than Go would.
- basic1 8y agoGo wasn't a thing in 2007.
- mrgordon 8y agoGo wasn't created until 2009 and Dropbox already had tons of Python code by then as well as extremely accomplished Python programmers (they hired the creator of Python only a few years later). I also don't think Go would have allowed the tight integration with, for example, OS X where Dropbox actually superimposes their icons onto your Finder icons. As far as I understand, it was only because of smart programming and use of OS X's Python / Ruby bridge functionality that they were able to do it.
- Longhanks 8y agoThere’s an official API since 2014 (Yosemite): https://developer.apple.com/library/archive/documentation/General/Conceptual/ExtensibilityPG/Finder.html https://developer.apple.com/library/archive/documentation/Ge...
- NegativeLatency 8y agoDropbox predates that api though. They used to hack into Finder.app's api IIRC.
- Longhanks 8y agoWhich is part of why that extension point was introduced :) https://twitter.com/nickjshearer/status/608833134902140928 https://twitter.com/nickjshearer/status/608833134902140928
- amanzi 8y agoThey forgot step 1 - hire the creator of Python... But seriously, that's an interesting post.
- amelius 8y agoGuido van Rossem recently stepped down from leading the Python project. https://news.ycombinator.com/item?id=17515492 https://news.ycombinator.com/item?id=17515492
- romed 8y ago
- chrisseaton 8y agoWhat happened to their new implementation of Python they were building? I know it was cancelled, but not really why.
- phonon 8y agoProbably because it was 2.7 only, and only 10% faster on Dropbox's code. https://blog.pyston.org/2017/01/31/pyston-0-6-1-released-and-future-plans/ https://blog.pyston.org/2017/01/31/pyston-0-6-1-released-and...
- chrisseaton 8y agoI'm surprised 10% wasn't enough - 10% bottom-line improvement in programming language implementation is normally massive. Twitter is singing from the roof-tops about 10% improvement in Java performance from the new Graal JIT compiler.
- ummonk 8y agoI would imagine that for code that is performance sensitive enough that a 10% improvement matters, they would be porting to a language with saner performance instead?
- phonon 8y agoPyPy is already way better than that, and has a Python 3.5 implementation.
- pmart123 8y agoI'm by no means an expert on the subject, but isn't there an advantage to having a JIT compiler use LLVM? On another note, I believe one thing that has been problematic for pypy adoption is that it does not automatically work with C extensions or Cython, and generally if someone already had performance issues with CPython, they would have written some C/Cython extensions?
- dwheeler 8y agoIf you can't get it directly (I can't), there is an archive available at https://web.archive.org/web/20180925210033/https://blogs.dropbox.com/tech/2018/09/how-we-rolled-out-one-of-the-largest-python-3-migrations-ever/ https://web.archive.org/web/20180925210033/https://blogs.dro...
- yieFiseh 8y agoBut their Debian packages still depend on Python 2.
- mr_toad 8y ago> However, rather than use the native toolchains directly, such as Xcode for macOS, we delegated the creation of platform-compliant binaries to py2exe for Windows, py2app for macOS, and bbfreeze for Linux. I wish the authors of more Python tools would deploy standalone applications. I do not like having to maintain various sets of Python installers/package managers (because every Python tool seems to use a different installer). Especially on cloud servers that often lack a whole set of dependencies that Python developers just seem to take for granted. I’m not a Python developer. I don’t have the time or the inclination to repackage various tools and untangle dependencies. Given the choice between trying to figure out how to get multiple Python tools to behave together, or using another tool, I’ll almost always choose an alternative.
- cjhanks 8y agoThis is possible and actually pretty trivial. You install all 3rd party dependencies into some directory. The command line entry point is then a simple BASH script which sets the PYTHONPATH to the appropriate installation location and then does the appropriate exec call. You then have a functionally portable python installation.
- rtpg 8y agoin the ideal, those making the tools would provide this packaging, instead of asking all users to have `pip` properly set up. Similar issues exist w/r/t npm in my opinion.
- orf 8y agoIsn't this just a hacky virtualenv? 'python3 -mvenv dir && ./dir/bin/pip install requirements'?
- jofer 8y agoThere are a lot of minor differences, but the biggest difference is that you're able to be completely independent of the system python install. You bundle a complete python interpreter, all libraries needed, etc. The user doesn't need to have python installed at all, and if they have 2.x instead of 3.x or 3.3 when you're expecting features that are only present in >= 3.5, it's no issue. This may sound trivial, but it's a _huge_ deal, particularly when you need to deploy something that runs on multiple different OSes and versions of OSes. Other than that, the "directory full of libs, binaries, and code" approach is a lot easier to package into something that will work well with the native package manager (e.g. an .msi for windows, etc).
- vemv 8y agoI hope this kills (or helps killing) the 100+ thread count I have always seen in macOS. It surpasses any other thread count from far more important/sophisticated processes. I'd say that's a waste (if not abuse) of the system's resources and scheduling system.
- Born_Again 8y agoGreat write-up and those two graphs are interesting. It's cool to learn how different companies treat their beta users. I wish this article touched upon more of the technically difficulties with switching from Python 2 to 3 too. A̵l̵s̵o̵ ̵a̵ ̵s̵m̵a̵l̵l̵ ̵h̵e̵a̵d̵s̵-̵u̵p̵:̵ ̵a̵t̵ ̵t̵h̵e̵ ̵e̵n̵d̵ ̵o̵f̵ ̵t̵h̵e̵ ̵f̵i̵r̵s̵t̵ ̵p̵a̵r̵a̵g̵r̵a̵p̵h̵,̵ ̵"̵v̵e̵n̵e̵r̵a̵b̵l̵e̵"̵ ̵s̵h̵o̵u̵l̵d̵ ̵b̵e̵ ̵"̵v̵u̵l̵n̵e̵r̵a̵b̵l̵e̵"̵ ̵(̵u̵n̵l̵e̵s̵s̵ ̵y̵o̵u̵ ̵m̵e̵a̵n̵ ̵w̵e̵ ̵s̵h̵o̵u̵l̵d̵ ̵l̵o̵o̵k̵ ̵b̵a̵c̵k̵ ̵a̵t̵ ̵t̵h̵e̵ ̵s̵a̵c̵r̵e̵d̵ ̵p̵y̵w̵i̵n̵3̵2̵ ̵l̵i̵b̵r̵a̵r̵y̵ ̵w̵i̵t̵h̵ ̵h̵o̵n̵o̵r̵)̵.̵
- pzone 8y agoVenerable is correct here.
- Born_Again 8y agoYou're right, I haven't seen it used outside of a religious context before.
- makecheck 8y agoIt’s funny, after Mojave was released recently I thought it might finally have a python3 installed, even if it’s not the default. Nope, still Python 2.7. This is good for me since I deployed a Python-dependent app under the assumption that the system Python would be stable and reliable. It allows relatively complex things to be achieved with a tiny download package. I’ve been prepared to adopt Python 3 for awhile but it just isn’t necessary when using system defaults.
- Longhanks 8y agoIt'll be very interesting to see what happens with the next macOS when Python 2 will be EOL (which is roughly 3 months after its release). The upgrade to Python 3 is long overdue.
- CJefferson 8y agoPython being EOL sounds scary but actually won't matter to Apple. They already apply custom patches, they can carry on running python 2.7 forever, with minor bug fixes where really required.
- guitarbill 8y ago> they can carry on running python 2.7 forever would they do this? does anything macOS internal depend on python 2.7?
- varenc 8y agothe `xattr` terminal command relies on the system installed Python, though this seems to be the only example. If you look at the source in `/usr/bin/xattr` it does some work to deal with different versions of Python. All the work ultimately gets handled by the xattr module preinstalled with the system Python. This module has Apple's copyright in it and is different than the `xattr` module on pypi. Wonder how this Python one-off in macOS came to be.
- madrox 8y agoAs a python engineer still living in 2.7, it's great to see major codebases making the move. I know I have a similar experience coming in my future, and I appreciate hearing what seems like more or less a success story come from it.
- sandGorgon 8y agoIs there any sample code that illustrates this kind of python embedding versus one of the freezer scripts ? We used to build desktop software using pyqt and freeze it. I wonder how that entire toolchain looks like in this new way.
- jnwatson 8y agoSee https://docs.python.org/3.7/extending/embedding.html https://docs.python.org/3.7/extending/embedding.html. The idea behind embedding is you might have a Python shell in a larger app. But you can also use it to tightly control the execution of the interpreter.
- wiz21c 8y agoFTA : >>> On the surface, the application would more closely resemble what the platform expects, while behind various libraries, teams would have more flexibility to use their choice of programming language or tooling. I'm always fascinated by how the implementation of the core principles of an application is dictated by factors alien to it, such as OS, company organisation, etc. Therefore, the job of coding is often a small part compared to the amounts of trivialities, project management decisions, customer's ideas, corporate policies, etc. Although my soul is a coder's one, I always realize how much coding is just a small part of what I call application development.
- drej 8y agoHere's a nice overview of how Facebook migrated their codebase to Python 3. While it's different in nature (server side vs. client side), it's rather interesting. https://www.youtube.com/watch?v=H4SS9yVWJYA https://www.youtube.com/watch?v=H4SS9yVWJYA
- d__k 8y agoA guide to migrating to Python 3 (with pleasure :) https://github.com/arogozhnikov/python3_with_pleasure https://github.com/arogozhnikov/python3_with_pleasure
- peter_retief 8y agoI hope the people who told me that python wasn't good for large rollouts read this
- j7ake 8y agoAnyone curious to estimate the cost for Dropbox to migrate to Python 3?
- xchaotic 8y agoand also calculating the cost savings if any. On a technical level the whole exercise seems to lead to a slightly more elegant, consistent code base, but it's still a long way from earning or saving any actual money.