25 ms·

Go + Services = One Goliath Project

418 points by yawn 7y ago

Possiblyheroin 7y ago
I was under the impression that Go was more performant than Kotlin. TMYK :)
- tus88 7y ago
  Go was written for servers, Kotlin for mobile apps. Seems an obvious choice really.
  - endorphone 7y ago
    Kotlin is just a less verbose, more modern language targeting the JVM. It has nothing to do with mobile apps beyond that Google, much later, decided to support it for Android development. Go is an efficient platform, however in these switches it usually goes something like "we took something hugely overbuilt, with layers and layers and layers and abstractions and abstractions, and rebuilt it with minimalist Go and now it's faster", which, of course it is.
  - hn_throwaway_99 7y ago
    > Go was written for servers, Kotlin for mobile apps. Not really. Kotlin was just a better JVM language developed by the JetBrains folks before Google adopted it as a first class Android language, but I don't believe it was specifically developed for mobile initially.
    - dehrmann 7y ago
      The open question is if Java will start borrowing the best features from Kotlin and it will become less relevant (like Scala). To Kotlin's credit, it has better IDE support, it feels simpler than Java (whereas Scala feels more complex), and it cleans up a lot of fundamental language issues that Java can't get away from.
  - sosodev 7y ago
    I wouldn’t say that Kotlin was written for mobile apps. JetBrains, the company that created Kotlin, doesn’t even have mobile apps AFAIK. It’s just meant to be a more modern JVM language and that means it does anything Java does just better. Servers included.
  - xwowsersx 7y ago
    I don't think that's true. You're confusing the fact that Kotlin has become the officially supported language for Android development with the idea that it was written for mobile apps.
  - lloydde 7y ago
    You may be confusing the adoption of Kotlin for Android. That happened more recently Kotlin has been open source since 2012 and designed as a direct (compatible) replacement for Java.
  - joshbaptiste 7y ago
    The only thing I see where Golang wins over the JVM is overall memory usage for daemons.. As Khan is non-profit they want to save at any avenue.
- mrspatula 7y ago
  JVM is pretty nippy
- defertoreptar 7y ago
  I have found that there can be drastic differences in Go performance in the particular way you structure the program. Writing Go code in a Python-like way is going to be less performant than if you run escape analysis every compile and make deliberate effort to stay on the stack.
  - pcwalton 7y ago
    > Writing Go code in a Python-like way is going to be less performant than if you run escape analysis every compile and make deliberate effort to stay on the stack. The fact that people are doing this tedious optimization is strong evidence of the fact that Go needs a generational garbage collector with bump allocation in the nursery. The JVM has a fast generational GC, and as a result you don't have to do this kind of optimization to get good allocation performance.
- coldtea 7y ago
  Why would Go be more performant than Kotlin? The JVM has been fine-tuned for 3 decades (JIT and GC) whereas Go is pretty primitive in a lot of its backend.
  - Thaxll 7y ago
    See for yourself: https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/go.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
    - coldtea 7y ago
      See what? This is an unofficial benchmark game - and based on implementations people bothered to provide. Not some scientific test, and nothing that guarantees these are the best implementations. And even if they were they're not representative of server/long running program behavior (where JIT quality and especially GC implementation) matters. Even so, the page linked starts with "Back in April 2010, Russ Cox charitably suggested that only fannkuch-redux, fasta, k-nucleotide, mandlebrot, nbody, reverse-complement and spectral-norm were close to fair comparisons". Since Russ Cox is one of the Go co-developers, lets see the ones he accepts are fair. Of those 5 are present in the page, on 3 of which Java wins with decent margins (fannkuch-redux: 18%, k-nucleotide: 21%, reverse-complement: 17.5%) and on 2 of which Go barely comes ahead (fasta: 6.3%, spectral-norm: 6%).
      - Thaxll 7y ago
        I used both Java and Go in production and yes Go can be faster or Java it depends but Go usually use between 3 and 10x less memory. https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ ( real world benchmark with networking / serialization ect ... ) Get over it yes Go can be faster than Java it's not a secret.
        apta 7y ago
        > real world benchmark Highly debatable. Even then, you see that golang ranks in the 102nd place for the plaintext benchmark, almost 6x slower than Netty, which is very established in the JVM world. Same with the JSON benchmark, golang is in the 126th place, ~3.5x slower than Netty and Vertx. > but Go usually use between 3 and 10x less memory. The JVM by default will use whatever memory is assigned to it. This makes sense from an efficiency and utilization point of view, as it generally aims for maintaining good throughput (whereas golang is only tuned for latency). The JVM now ships with new low latency GCs (ZGC and Shenandoah) which are currently available in experimental phase.
- Thaxll 7y ago
  Overall Go is.
atmosx 7y ago
How does Khan academy make money to sustain the website? Is it all donations?
- sosodev 7y ago
  From what I understand it’s mostly donations but I think they receive some grants too.
- bhaumik 7y ago
  Non-profit, with several $1m+ donations: https://www.khanacademy.org/about/our-supporters https://www.khanacademy.org/about/our-supporters
  - patneedham 7y ago
    To be more specific (for anyone else who hasn't checked the link yet), the "Lifetime giving" section has: 9 donations >10m, 4 donations between 5-10m, 20 donations between 1-5m So looks like there has been at least 120m in donations!
    - atmosx 7y ago
      Thanks
bodka 7y ago
GO gorever!
purple-again 7y ago
We turned our monolith into a bunch of micro services almost 6 years ago to the day. For a long time I was very happy with the new pattern but over the years the weight of keeping everything updated along with the inevitable corners that fall behind and have...questionable..security due to how long they sit neglected has really left me wondering if I am happy with it after all. I would love hear some thoughts from others that made the move, especially anyone that decided to move back to a monolith repo.
- tuckerconnelly 7y ago
  I moved back to monolith and am very happy. I think of the monolith now as a collection of modules. The rule is now, one should be able to drag any of the modules to the top-level of our monorepo and create a new microservice pretty easily when the time comes. I think the microservices book (that came from that Uber engineer...?) suggests a rule of 5 engineers per service.
  - virmundi 7y ago
    How are you preventing transaction couplings? For example, module A and B are called by C. C starts a transaction that wraps A and B. If you move B up as a network feature, you lose the transactionalty.
    - m0zg 7y ago
      Usually the answer to this is "we pray to god it doesn't fail".
    - deleted 7y ago
      [deleted]
    - tuckerconnelly 7y ago
      Good question, and this rule is meant to be bent in those scenarios. I try to avoid these dependencies if at all possible, but if not possible, the "writes" for those modules all belong to a single service, and any other service, depending on how "pure" I need to be in the project, will make network calls to the other service, or just grab that data directly from the database. For a more concrete example, I recently built a service ("scraper") that scraped data and upserted a large tree of structured data to postgres in a transaction. Writes were only allowed from scraper, but "api" could SELECT data for reporting to the frontend "web" as much as it wanted. In the future, "api" might make be refactored to make internal HTTP request to "scraper," so they could have totally separate databases.
peterwwillis 7y ago
So, some potential pitfalls: - The decision seems to be primarily a software architecture one, without much mention of all the other architects whose input will shape how the finished product is run and supported. In a modern software development environment, all the other parts of the org should be consulted on greenfield work to "Shift Left" anything that may need to change down the pike. Design in a silo leads to ineffective products. - They're going from "hmm we need to upgrade from Python 2 to Python 3", to "we need to redesign everything in a new language with a radically different software architecture". This is definitely the second system effect. It's going to take years to make this thing reliable and sunset the old product. - They're porting over the logic? Even if this is actually the right move, wouldn't a clean-room implementation potentially give better outcomes? - Why are they continuing to use App Engine if the writing's on the wall for 2024?
- batter 7y ago
  Go + AppEngine is the most unstable combination i have ever seen. While we tried to deliver project during 1 year, it was almost fully rewritten couple times because of new Go or AppEngine API. Having NodeJS with far less problems. And AppEngine has huge price tag.
  - pm90 7y ago
    GAE v2 let’s you use docker containers and ime it has been pretty stable and fantastic.
- dangoor 7y ago
  I don't disagree with your pitfalls, but I do think we're working to avoid them. To your first point, "The decision seems to be primarily a software architecture one...", this project has had involvement of the whole engineering team since the beginning. The whole org is on board with this change. It's definitely not happening in a silo. > This is definitely the second system effect. It's going to take years to make this thing reliable and sunset the old product. I hope not, but obviously we're not done yet, so I can't say how long it will end up taking to completely decommission the Python 2 app. What I can say is this: there are aspects to this project that are _simplifying_ our system and, for what's left moving from Python to Go, our intention is to port the business logic as close to a straight up port as we can get. > - They're porting over the logic? Even if this is actually the right move, wouldn't a clean-room implementation potentially give better outcomes? _That's_ second system effect, to me. We can't change everything and fix every problem now, so we're focusing on the changes that will help us move from Python to Go faster. > - Why are they continuing to use App Engine if the writing's on the wall for 2024? I don't think Google Cloud is disappearing in 2024, for one. Beyond that, again, we're not changing everything about our architecture. The way our data is stored is staying the same.
benatkin 7y ago
The article says this: "Moving from Python 2 to 3 is not an easy task." I disagree with this. It's a Python project's dependencies that make it hard to move from 2 to 3, and most libraries have been updated. Of course, you could argue that it isn't easy to migrate a codebase from one major version of a language (or framework, or database) to another, but when you eliminate easy from your vocabulary it becomes harder to describe different levels of difficulty.
- foolfoolz 7y ago
  migrating from python 2 to 3 is such a large task that migrating to any other language is a comparable effort. this is not just a library problem the language itself changed significantly source: no python services at my company are going to be migrated to python 3; it’s all moving to a JVM
  - was_boring 7y ago
    What? I've done it on some sufficiently large code bases, and small ones, and it was done way faster then a rewrite. With tools like 2to3 you can assign it to an intern and have it done pretty quickly.
    - edoceo 7y ago
      It's not quite that easy.
  - zo1 7y ago
    Migrating to 2-3 can be a large task in some very rare cases possibly, but for the most part it is practically effortless if you don't have to support both simultaneously. The biggest hurdle might be the "fear" of the unicode change, but that can be dealt with. Source: All python services at my current workplace are in the process of being migrated to 3.*, and I'm doing one of the main ones at the moment and it's a breeze, including compiled c-extensions. For curiosity, a good list of actual python3 syntax changes: https://docs.python.org/release/3.0.1/whatsnew/3.0.html#overview-of-syntax-changes https://docs.python.org/release/3.0.1/whatsnew/3.0.html#over...
timwaagh 7y ago
Seems like such a waste. Is switching to python 3 really that hard? Is hardware that expensive? If this is indeed the right call it doesn't bode well for traditional scripting languages as the web scales to fewer high traffic apps. We might start to see more jvm, go (apparently) or even rust and c(++), rather than speed of development languages like Python or Ruby. Trend seems to be the reverse though, with python the second and most rapidly growing language.
- neurobashing 7y ago
  Everyone’s project/code base is different but in my experience there’s been a critical mass of libraries for a few years. I presume the “it’s hard to move to 3” is dev teams wanting a new toy as much as “the rewrite is too complex”. Library use, size of code base etc are all big factors but at the end of the day, I think team motivation is really the deciding factor.
  - Consultant32452 7y ago
    This is my experience too. Someone or a couple someones on the team decide they want to try out some new tech or expand their resume. Then it becomes a quest to justify the switch rather than a quest to make the best business decision.
  - freyr 7y ago
    That mirrors my experience as well. Someone with influence is bored or wants to level up, so they'll drag the entire company into a long, expensive quagmire. Unless the existing codebase is mired in technical debt and completely unsalvageable or cannot scale further, this seems like a very radical move.
    - ksec 7y ago
      Which is precisely why most of the Web Dev should not be called Engineers.
    - lonelappde 7y ago
      Replacing an old unholy mess with a new unholy mess is usually a bad plan. It's called the Second System Effect.
kucing 7y ago
Looking at the case where khanacademy is migrating their server only after about 10 years. I realize more that I don't have to worry that much about being locked into certain technologies (unless it's clearly untransferable, e.g. storing part of customer data in 3rd party server), because after all, we might keep it only for about 10-20 years, and the thing I'm working at almost certainly will only last < 2-3 years.
kissgyorgy 7y ago
Huge mistake. Reminds me of https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...
thatswrong0 7y ago
Ok this is going to sound ignorant, as my only experiences in backend services have been Go and Python. I don't like either. Is there something I'm missing? For simple CRUD apps, both are sufficient. But (in my limited experience), the moment I've wanted to create more complex business logic with stricter constraints, neither has been quite up to the task. Go doesn't make things easy. It asks you to repeat yourself. I don't like the lack of basic functions like a generic map / filter function.. I know Rob Pike just says "use for loops", but it feels so unnecessarily unexpressive. When I see map, I know what's going on almost immediately. For loops take more reading to understand. Nil pointers shouldn't be, yet still are, a thing (developers aren't perfect - why can't the type system help?). It feels like a straight downgrade from Python from a code clarity perspective. And it's typed, sure, but the type system doesn't let me express constraints that other languages allow me to do that would prevent entire classes of bugs. It doesn't feel worth it compared to Python. Yes, the core language ends up being comparatively "simple", but simple building blocks doesn't guarantee a simple overall system. And my company is very diligent from an architectural perspective. But then when I look at Python, I'd rather just use Javascript with Lodash, esp. when it comes to the treatment of functions as first class objects[0]. Throw Typescript in there, and you get, in my opinion, a better type system than Go, so unless language performance is a major constraint (which it hasn't been for my company, our DB usage patterns it the biggest thing instead), why would I want to use either of these rather than Typescript? [0] Edit: Dumb and wrong, I meant its treatment of anonymous functions. I don’t like lambdas.
- catalogia 7y ago
  Go seems to be optimized for onboarding new developers (particularly straight out of school) quickly, rather than for the long term comfort of developers using it. There are Rob Pike quotes that speak to the first part of that at least. If I was a business owner, I'd love Go. But what I can't really figure out is why so many devs love it. It's far from the worst thing in the world, I don't hate it, but I just can't get excited over it either.
  - adtac 7y ago
    Defers, channels, inexpensive goroutines, a mostly consistent standard library, very good performance, garbage collection, being similar to C, strongly typed, and readability are the reasons why I've enjoyed writing Go exclusively for the past couple of years. Almost none of these is exclusive to Go, but the intersection of all these certainly is.
aazaa 7y ago
> Now, in 2019, Python 3 versions are dominant and the Python Software Foundation has said that Python 2 reaches its official end-of-life on January 1, 2020 , so that they can focus their limited time fully on the future. Undoubtedly, there are still millions of lines of Python 2 out there, but the truth is undeniable: Python 2 is on its way out. The Python 2/3 split is by far the most annoying thing about Python. I don't develop software in Python but about half the time I've had to use a Python library or program the problem of 2/3 incompatibility has cropped up. Some projects don't make it clear whether one or the other is required, leading to further confusion. If anything the Python 2 EOL could make a bad situation worse. Like Khan Academy, each Python 2 package maintainer will be forced to make a decision: move to Python 3 or abandon and maybe move to an entirely new language. It think many will choose to abandon, leaving these packages to rot. Second on the list are the multiple package managers (or things looking like package managers). Third on the list of annoyances are native extensions, driven by the poor performance of Python itself. These extensions make it difficult to use certain libraries across operating systems. So as a non-Python developer I don't look forward to the occasions when I must use a Python-based piece of software.
- zo1 7y ago
  If you're installing a package via the package manager, it will very quickly tell you if you can install it on your specific version of python. Unless you're downloading some rather obscure and un-loved library where the author didn't explicitly state which versions of python they support. Multiple package managers: There has only been 2 big ones from my long-term general usage of python. Easy-install and pip, the former of which is falling in favor but still semi-supported. Pretty much everything runs off of pip. What may be confusing you, and does confuse me at times as well, is their naming and the installation instructions as provided by library authors. E.g. some say python setup.py -install others just tell you to "pip install" it. Some would say use "setuptools", etc. Other would tell you to use things such as "conda" or "anaconda", pipx, and to create virtualenvs. All secondary, but things that should not ideally distract you from just plain using pip. 3. This has also been getting a whole lot better in the last 5 or so years. Microsoft has been funding dev-time to make the ecosystem for python (including extension compilation) much more pleasant in the Windows space. Also, the package managers and library authors are doing a whole lot better in that binary distributions are much more prominent so the compilation of the extensions never has to happen on your machine.
Thaxll 7y ago
Kotlin looks nice now but on the long term it's a bad choice since they're stuck on Java 8 they will always lag behind the real JVM and won't be able to catch up since they need heavy modifications.
- m0shen 7y ago
  If you mean the java 8 bytecode version, that hasn't been true since April: https://blog.jetbrains.com/kotlin/2019/04/kotlin-1-3-30-released/#more-6991 https://blog.jetbrains.com/kotlin/2019/04/kotlin-1-3-30-rele...
  - Thaxll 7y ago
    But they don't benefit from new runtime features right ? Loom, GC ect ... are no go for Kotlin.
    - m0shen 7y ago
      I don't see why it wouldn't benefit from the GC changes. AFAIK Loom has no scheduled release date. They seem like they're committed to supporting new features, but you're probably right they will lag on some things.
buboard 7y ago
If they were going for performance, and web-related, how come they didn't consider PHP?
ngngngng 7y ago
Unpopular opinion, writing Go is faster than Python. With the compiler, strong typing, and no versioning hell, I'm much more productive in Go. Whenever I use python I run into problems with versions and dependencies. And the whole community just tells me to use pyenv or virtualenv and it will "fix all my issues". Only it doesn't.
- milofeynman 7y ago
  I doubt that's unpopular. It's the same reason I significantly prefer Go over Ruby.
- kart23 7y ago
  For larger stuff, yes. For small stuff, no. If you just need to get something small done quick and dirty, python will be easier and faster.
  - jerf 7y ago
    Yes. My current estimate of the cutover, for myself, is about three weeks of solid, 40hour/week development. After that, my Python (or other dynamic language) starts the process of seizing up, where instead of rewriting some module I just put a little hack in there to make it backwards compatible with other code, since I haven't got a great way of being quite sure what's calling this code, so I use a __setitem__ or have a function that takes "a thing or an array of that thing", and I find myself increasingly reluctant to refactor the Python. YMMV on the exact number, but that's been my experience several times now. I know it can be done; I've seen it done, I've done it myself. But refactoring without even the rudimentary static type system Go has just becomes an increasing nightmare at scale. And I use unit testing in Python, etc. But, flipside, yes, Go isn't a great language for just bashing a script together in. Maybe not the worst, with a bit of library work, but not a great language.
- watermelon0 7y ago
  Just as a counterpoint, I'd say that using Python can IMHO be similarly productive than Go. Regarding the dependencies, you have tools on top of virtualenv, such as pipenv/poentry, which handle dependencies, and are easy to use. Biggest issue that I've encountered would probably be when two or more dependencies require the same package, with no intersect between supported versions. I don't think Go handle this any better, thought. Type hints (and mypy for static type checking) are a must, and coupled with a good IDE, they really improve the productivity. I'd say that mypy's type system is more advanced than Go's, but it strongly lacks in type safety (due to the fact that majority of the libraries are not taking advantage of it yet, and that Python is still a dynamic language by its nature, and there is no runtime type checking).
bootlooped 7y ago
The article mentions Go's superior compile time, when compared to Kotlin. I have done a lot more Java development than Kotlin, but my recollection is that both of them compiled fairly fast. Is Go really significantly faster to compile for similarly sized projects?
- tbrock 7y ago
  I’ve written some things in both languages and compiling go feels an order of magnitude faster.
  - abraxas 7y ago
    It's mostly because most setups will use maven or gradle which do much more than just compile code. They will often pull down dependencies, check for bugs and style inconsistencies, run unit tests etc. If you run plain javac against a bunch of files it will compile just as fast as Go. In any case it will be in a blink of an eye.
- sheeshkebab 7y ago
  Yes, it’s much faster than these java and jvm based codebases (including all jvm based languages like kotlin, scala etc). Go as a language is just so much simpler. Although compiling overhead of large JavaScript codebases, that make up most of modern websites, kills any kind of benefit in improving build performance of some backend service.
- fizx 7y ago
  It's rare that I notice the compile time for either. Incremental compilers are great!
- tapirl 7y ago
  The article mentions why choosing Go over Kotlin is memory consumption.
tybit 7y ago
As much as I personally don’t enjoy writing Go I really can’t fault them. I still find it interesting that for a relatively obvious feature set of fast compiles, fast startup and fast runtime there really isn’t anything mainstream out there to compete with Go. I really hope something like Kotlin, Swift, ReasonML or even AOT JVM/.NET brings something to the table soon. Or perhaps I’ll just have to wait for WASM to really take off server side.
- genuine_smiles 7y ago
  > fast compiles, fast startup and fast runtime All I can think of is D, but it’s not quite mainstream. Are there any other less popular languages that meet all 3 conditions?
  - amedvednikov 7y ago
    https://vlang.io https://vlang.io
    - sagichmal 7y ago
      Still snake oil.
      - fouc 7y ago
        Site looks pretty legit, can you explain why it's snake oil?
        doteka 7y ago
        Because it literally uses string manipulation to generate c code from source. There is no concept of an AST or anything - just string bashing.
    - iteratorloopmap 7y ago
      LoL
    - 7y ago
sibeliuss 7y ago
Isn't there a quote somewhere along the lines of "full rewrites are suicidal?" Seems pretty risky to me.
- stoicShell 7y ago
  The saying only applies when your product is software itself, not in most cases where software is just the means to deliver your product — here a website providing education in video form etc. Besides, the reality is that most business software out there gets rewritten every 3-15 years (really depends on use-case and conditions, but on average 4-5-6 years is a good bet). After some time it's just not worth it to keep refactoring, you'd rather start anew with hopefully better tech and certainly with better knowledge of your problem — they say you should write everything 3 times to make sure you really nailed it. In many businesses, these rewrites would constitute a new major version, more comparable to the feeling we always got in the waterfall era — new version = big changes, new UI, new stuff. That's when it's possibly lethal, if you really break the thing, and that thing is your product, not a means to it.
  - guitarbill 7y ago
    the number of failed migrations, modernizations, and "tech transformations" in non-software industries, as well as the number of consulting outfits and their profits doesn't seem to back this up. [0] IT disasters now part of modern life - https://www.afr.com/technology/it-disasters-now-part-of-modern-life-20160628-gptyw6 https://www.afr.com/technology/it-disasters-now-part-of-mode... [1] Number of IT failures at banks and other firms is unacceptable, say MPs - https://www.theguardian.com/business/2019/oct/28/number-of-it-failures-at-banks-and-other-firms-is-unacceptable-say-mps https://www.theguardian.com/business/2019/oct/28/number-of-i... [2] The Biggest IT Failures of 2018 - https://spectrum.ieee.org/riskfactor/computing/it/it-failures-2018-all-the-old-familiar-faces https://spectrum.ieee.org/riskfactor/computing/it/it-failure... we can argue whether these are "rewrites", but big changes can be rewarding but are inherently risky. balancing this is hard, and rewrites uncover and introduce the unknown unknowns.
    - 7y ago
Solar19 7y ago
Interesting read. It always surprises me that companies go many years running major apps and infrastructure with interpreter-based languages like Python and Ruby to begin with. It's an incredible waste of energy, compute, and for web apps sometimes, users' time. Developers need to be a lot more disciplined about performance and efficiency. I'm glad Khan went to Go, but man all those years wasted.
tony 7y ago
You can start porting your live codebase to python 3 right now - no downtime at all. You can have a 2+3 compatible codebase live in a week or two, it'll get you about 98% the way there [1] My projects (e.g. https://tmuxp.git-pull.com https://tmuxp.git-pull.com, https://unihan-etl.git-pull.com https://unihan-etl.git-pull.com) are both python 2/3 compatible. I learned by reading through other projects like Sphinx, Flask, Werkzeug, and SQLAlchemy: - Flask: https://github.com/pallets/flask https://github.com/pallets/flask - Werkzeug: https://github.com/pallets/werkzeug https://github.com/pallets/werkzeug - Sphinx https://github.com/sphinx-doc/sphinx/tree/v1.8.5 https://github.com/sphinx-doc/sphinx/tree/v1.8.5 (see compat: https://github.com/sphinx-doc/sphinx/blob/v1.8.5/sphinx/util/compat.py https://github.com/sphinx-doc/sphinx/blob/v1.8.5/sphinx/util...) - https://github.com/sqlalchemy/sqlalchemy https://github.com/sqlalchemy/sqlalchemy Helpful blog posts: http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/ http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python..., http://lucumr.pocoo.org/2013/5/21/porting-to-python-3-redux/ http://lucumr.pocoo.org/2013/5/21/porting-to-python-3-redux/ As for automation tools, for 2/3 compatibility, I used futurize and huge codebase to great success: https://python-future.org/futurize.html https://python-future.org/futurize.html. I started with this: futurize --write --stage1 --unicode-literals --nobackups <files> and https://docs.python.org/2/library/2to3.html https://docs.python.org/2/library/2to3.html 2to3-2.7 -w -n <files> Really helped. Also, wire Travis / your CI system to have your tests run on python 2 and 3. If you want to test your python 3 codebase live on a subdomain / same SQL/NoSQL DB, be careful about jobs/tasks! Pickle version mismatches and stuff. Use a separate redis/whatever DB for the deployments. After you're finally on python 3, you can use https://github.com/asottile/pyupgrade https://github.com/asottile/pyupgrade to modern your code. [1] For the other 2%, use conditionals in your requirements file: enum34==1.0.4;python_version<"3" ushlex==0.99.1;python_version<"3"
- dangoor 7y ago
  I appreciate all of the links and the fact that, for a lot of folks, this is the state of Python 2 to 3 migration today. What you're suggesting is, imho, the best path for most (and I appreciate you mentioning the pickle incompatibility because that is a thing which would trip people up when trying to make the move). If we would have been able to have a 2&3 compatible codebase live in a week or two, we absolutely would have done that. Incompatible changes in some libraries we use, App Engine first gen to second gen changes (which are for the better, but still a big deal), the choice of storing some pickles permanently, plus a need to really verify unicode handling all over the place (especially in a 10 year old codebase), and other factors that aren't coming to mind at the moment mean that this is not a couple week thing for us. Moving to Go is more work than moving to Python 3, but in our particular case it's not as much more work as people might expect.
RandyRanderson 7y ago
Why do they call them "micro services" and not distributed systems? Oh right, it's because distributed systems are obviously really hard to create correctly and no sane person would ever agree to pay for that. Nice: re-branding. I can't wait for the, maybe "consolidated computing" manifesto (aka turning micro services back into monoliths).
- thethimble 7y ago
  What if the services you are writing are independent in that they solve separate business problems, are built by separate teams, have little to no data coupling (e.g. Only basic auth), have different scalability profiles, etc? Separate services are really effective for these cases. Neither micro services nor monoliths are silver bullets. Instead it's possible for each approach to be the best approach in a particular business context.
  - RandyRanderson 7y ago
    In several decades of it experience , I've not known or heard of a nontrivial system like the one you describe in the first part of your message. In the latter part, that must be a disingenuous dichotomy. You don't really believe that just because an avenue exists we should include it in an evaluation?
  - pjmlp 7y ago
    What about writing modular libraries then?
- olingern 7y ago
  Why are you getting hung up on nomenclature? The point of the article is clear. If you feel that using 'microservices' is too trite or "buzzword-y" then that's about you and not the article.
  - RandyRanderson 7y ago
    I guess you're agreeing with me sarcastically? Very funny.
wonjohnchoi 7y ago
It is crazy to see they are still using python 2. Seeing how slow the conversions to python 3 have been, was creating python 3 a good decision for python community? Can it be argued that developing python 2 further in a backward compatible way would have been better for the community? I know that evaluating this kind of thing is hard as metrics are bound to be subjective and speculative. But I am curious if there was any serious attempt to figure it out.
- ddxxdd 7y ago
  I remember a long time ago, I used the "six" library (Python (2 * 3) ) to code in Python 2 with an API based on Python 3. I'm not entirely sure how well the line of code: "import six" would fix any compatibility issues nowadays.
- speedplane 7y ago
  > was creating python 3 a good decision for python community? ... I know that evaluating this kind of thing is hard I don't think there is any question here: Python 3 is a complete disaster. Years and years of engineering effort wasted on changing string libraries. Sadly, the Python leadership refuses to acknowledge the failing, perhaps because such an acknowledgement would challenge their omnipotence ... it would, and it should.
bhntr3 7y ago
> If we moved from Python to a language that is an order of magnitude faster, we can both improve how responsive our site is and decrease our server costs dramatically. I see people say things like this a lot but my experience is that while other languages are 10x or more faster than python in some benchmarks it's very rare that computation time dominates server latency or that servers are running at 60%+ cpu across all cores. If 90% of your service latency is not directly on the cpu and/or you haven't profiled to see that the performance bottleneck is evenly distributed across all tasks, then it's super dangerous to migrate to a new language thinking that will fix it. I hope people inside Khan Academy know this and it's just a clickbait blog. If they really think "go is 10x faster than python so we'll only need 1 server for every 10 when we migrate" then I think they'll be disappointed.
- alfalfasprout 7y ago
  Well Brad, that's very dependent on what exactly the service is. Moreover oftentimes low CPU utilization is actually a limitation of the implementation on a slow language (eg; Python technically does have async webservers but adds a lot of idle overhead). Indeed, there are many benefits these more performant languages have over Python aside from raw single-core performance. For starters, more efficient concurrency and parallelism can help reduce average latency when combined with a quality async webserver. Then there's gains due to shared memory across threads. So in many cases-- absolutely, you can only need 1 server vs. 10 when you migrate. It's thus not fair to say that these gains are "very rare".
- pm90 7y ago
  * It’s not just that Go is faster to run but also faster to iterate on. If python can be neither, its offering little benefit. * they moved from a monolith to a microservices architecture; concern that any of the services in the request path could add latency just because of the overall runtime speed is slow is a legitimate one. * their primary deployment method is Google App Engine where you are billed by CPU used. Any change that consumed less CPUs has a tangible effect on their costs
  - andy_ppp 7y ago
apta 7y ago
> Go, however, used a lot less memory, which means that it can scale down to smaller instances. I could be mistaken, but this sounds like they went ahead with the default JVM settings, where it tends to use as much memory it is allowed to (which makes sense from a utilization and efficiency perspective). If memory usage is a concern, the JVM can be tuned for such.
- jacques_chester 7y ago
  The JVM hasn't yet become fully container-friendly because it bases its calculation on the host OS figures, not the container figures. You can use a calculator to get precise, proven settings for any supported JVM: https://github.com/cloudfoundry/java-buildpack-memory-calculator/ https://github.com/cloudfoundry/java-buildpack-memory-calcul...
  - apta 7y ago
    I thought this was a solved issue since JDK 10: https://www.docker.com/blog/improved-docker-container-integration-with-java-10/ https://www.docker.com/blog/improved-docker-container-integr...
    - jacques_chester 7y ago
      In my understanding, no. It's improved but not fully "fixed". Unfortunately that's as far as my understanding extends.
deleted 7y ago
[deleted]
tmpfile 7y ago
Yikes! It's a lot of effort to reduce memory use. They might be better off creating a new Go entrypoint/server that can call into CPython to reuse all their existing/tested modules (treat their Python as a microservice called by Go). They could then use Go to create/call new microservices or replace various routes on a selective basis.
remote_phone 7y ago
What a mistake. Not only going from a monolith to microservices but also changing the language? This is a mistake that rookies make. This will be one of those post-Mortems where they will sheepishly admit they bit more than they could chew, and it wasted years of productivity. There’s no reason to move to Go. Stick with Python for now. Migrate safely to python 3. Once everything is stable, start breaking things up into thrift or protobuf services. They don’t even need to be microservices but you need the contract. Once that is stable migrate to whatever language you want. But at this point you will have the well-defined api and test cases. Trying to do too much all at once is a no-brained disaster.
- pjmlp 7y ago
  This kind of decisions is usually always taken when the monetary cost is not measured. Apparently not mapping developer hours to real money keeps not being a thing outside consulting shops.
  - deleted 7y ago
    [deleted]
todd3834 7y ago
> We’ll only generate web pages via React server side rendering, eliminating the Jinja server-side templating we’ve been using I’ve been down this road. Deep down this road. Let me just give you a heads up on something I didn’t consider at the time: Most template languages do not parse every single node, one by one. In a sense they are just doing string concatenation. Not so with server side rendering and React. I’m not saying it can’t be done but just realize it is going to take a lot more compute power. Caching is great of course but won’t help you if you plan to customize user content during the server side rendering as well. My recommendation is that you don’t do any user authenticated stuff during SSR. Also consider how you are going to handle cookies if you do plan to make authenticated requests to server side rendering. Also solvable but for some reason people had the hardest time understanding why we had to forward cookies to the domains we controlled in an API request and definitely not to any other servers. I’m not sure I would pick React for an SEO driven website. It is hard to get a competitive “time to first byte”. Unless of course you can pre warm a cache of every one of your pages. Lastly, you’re going to need Node for the SSR. I’m sure you know this but that might take you out of app engine and into cloud compute. Not a big deal but thought I’d mention. Good luck! It is doable. If you ever want to chat about how we solved some of these problems I’d love to save you some time if I can. Hit me up in my profile email.
- dangoor 7y ago
  Thanks for the suggestions and the offer to chat! We've been doing SSR for quite a while now and are improving our CDN use as we go along. We already took steps to ensure that there's no user-specific information showing up in our server-side react rendering which would damage cacheability. Our frontend infrastructure team essentially owns the React render server. I'll let them know you offered to chat.
plinkplonk 7y ago
just trying to understand - you guys think moving a Python2 monolith to Python 3 is too painful, and so you are going to port all the code from Python2 to a completely new language (Go), change the architecture (monolith -> microservices) and move the HTTP API to React + GraphQL, all in one year? 2020 is going to be in an interesting year at Khan Academy ;-)
- CGamesPlay 7y ago
  The move from Python 2 to 3 would likely have also involved changing the architecture so they could migrate components incrementally. Since they were going to do that regardless, and if they already wanted to change the interfaces from HTTP to GraphQL, this is a natural time to do it. Though, this migration has nothing to do with React--they were already and will continue to be using it.
  - deleted 7y ago
    [deleted]
- conradfr 7y ago
  At least the article makes it seem it's not a decision taken on a whim and they did some kind of POC and planned the transition. Of course the obvious thing missing in the article is how they expect to deliver new business features while recoding everything in a new language. It's fun to read this and the Etsy thread currently on the frontpage as well.
  - dangoor 7y ago
    > Of course the obvious thing missing in the article is how they expect to deliver new business features while recoding everything in a new language. For new features, the new parts of the GraphQL schema for those features will be written in Go as part of the new services. Our frontend is already in large part a single page app in React which requests data via GraphQL, so the frontend for the features will look just the same as it would on our monolith.
- dangoor 7y ago
deleted 7y ago
[deleted]
kureikain 7y ago
Go is a very weird language :-(. It's very limited when you started to do complex thing. Example, let's say you are building websocket. You will have a hard time to write type safe websocket handler to process the payload from client for all the events... I started to do Rust/Crystal and both of them are better than Go(performance, type system). Yet, whenever I build something for work, I come back to Go :-(. I told myself to use Rust or Crystal. Then I realized that Go is a practical language. It compiled fast so it makes testing easier. The cross compiler just make it so easy to build binary run on everything thing. And the limitation of Go makes it very consistent on how you do thing. This makes working with Go become faster event by the fact that it slows you down on other parts. So I think Go is a language that people easier to fall into because it has the speed of interpreter language like Ruby/Python(or even faster) during development and have a better performance/type safe story.
- apta 7y ago
  Java and C# provide the benefits you mentioned, while being more expressive languages and having better runtimes compared to golang.
  - hu3 7y ago
    Hardly. OP mentioned fast compilation and limited ways to code the same solution. C# and Java are slower to compile and offer way more options to do the same thing. Here's Uncle Bob take on testing with Go: https://m.youtube.com/watch?v=2dKZ-dWaCiU&t=36m40s https://m.youtube.com/watch?v=2dKZ-dWaCiU&t=36m40s
    - apta 7y ago
      > C# and Java are slower to compile Not for any meaningful work in my experience. As a matter of fact, I found the change/compile/run loop in golang to be slower on projects I've been working on due to the fact that it doesn't support incremental compilation, so any change I make ends up recompiling the entire program and writing out a 100+MB binary anyway. Compared to a Scala project I worked on before (and Scala is notorious for slow compiles), after the first compilation, all modifications happen very quickly as only the respective classes are re-compiled. > Here's Uncle Bob take on testing with Go Again, this doesn't apply for any non-trivial/large project. On a project I'm working on, it literally takes 7-8 minutes to do a clean build + run all unit tests in golang.
sethammons 7y ago
I've been in a similar boat. We've been splitting up or converting large Python 2.6/2.7 applications into Go services (and doing the same to large Perl applications) for a long time now. Go has consistently been 10-20x performant (allowing for dramatically reduced hardware needs), easier to maintain, and more productive to produce code in than our previous Python (Twisted) and Perl (AnyEvent). Hopefully KhanAcademy has solid telemetry data in both legacy and new code so they can quantify benefits. They will also have a learning curve for managing multiple micro services vs monoliths. Accessing shared data will be a problem they will likely have to solve. We've opted for each service controlling its own data - no reaching into another service's data behind its back. Everything through APIs. This gives the microservice the ability to alter its datastore as it needs to and not be blocked by other teams' need to update how they access the data. Debugging a distributed solution is much harder than a single service. Distributed tracing, consistent structured logging with log aggregators that let you do fancy searches (like Splunk), and application telemetry and metrics will be even more important than before.
- dangoor 7y ago
  Does sound similar! We have established the rule that each service owns its own data. We've already got Stackdriver set up to give us distributed tracing and have set up standards around logging.
mjpuser 7y ago
I think the real problem is that they didn't properly maintain their code. Rewriting it in Go won't prevent them from dealing with this in a few years for when this Go version reaches end of life. I would have liked to see an article on "introducing process" side of programming.
- dangoor 7y ago
  Thanks for the comment! A couple of things about this... 1. the Go team is working very hard to ensure that there are no such compatibility issues. Code written for Go 1.0 should still compile with Go 1.14 beta today. 2. It's possible there's more we could have done along the way, and I tend to think that statically typed languages make it easier to safely refactor more ruthlessly. But I do think we've actually done quite a bit of change incrementally along the way. Our move to React on the frontend and GraphQL on the backend have been good examples of that. Plus, we did a huge refactoring a couple of years ago to draw better boundaries in our monolith, and that has made a move to services possible.
harel 7y ago
Migration from python 2 to 3 is easy and fast. I've migrated multiple large apps and it took about a day each. Most libraries that matter have been migrated. Some don't even support python 2 anymore. It's practically 2020. This should not even be a consideration. After 2 to 3 is done they should consider again If they want to redo the stack but first I'd focus on this small maintenance task.
- AnonymousPlanet 7y ago
  Hahaha, sorry but this is a very cute thing to say, in my view. At our company we just barely finished migrating our software with nearly a million lines of legacy Python 2 code to Python 3. This took over a year of nearly exclusive migration effort, just making our code work with both. The entire migration project started way before I joined the company several years ago. So, no, things are not as simple if you're not dealing with toy projects. And no, you can't assume that it's the same for everyone if you're not in their shoes. Your comment is pretty much the equivalent of "I don't see a bug. Works for me."
  - StavrosK 7y ago
    It very much depends on how good the codebase is. I also spent a year on and off porting a large codebase from 2 to 3, and it would have gone an order of magnitude faster if the codebase were in better shape.
    - AnonymousPlanet 7y ago
      I agree that code quality is a big factor. But what is good code quality in Python? In our case, the oldest code is the most "pythonic" and is at the same time the worst to maintain. The better code mitigates the drawbacks of dynamic typing and by that moves away from the pythonic standard you see in many libraries. But even if you nail the types to the board (e.g. assert isinstance(...)), use (the somewhat weak) Mypy wherever you can, and have good test coverage, you still have to grep your code base for usage of, e.g., .keys(), eyeball hundreds of modules for subtle Unicode madness or hunt for the odd division, replace every sort() that doesn't use key= yet, etc. The todos add up and someone has to go into the code and change those lines.
d_burfoot 7y ago
I heard about this project from a friend who works at KA. I am concerned about the strategy, and I think the following approach would yield better results: 1. Write in Go an exact reimplementation of the current Python codebase. Use the same database schema, front-end HTML/JS, test suite, and so on. To whatever extent possible, use the same names for classes and functions. Check the reimplementation correctness by using a comparison tool that calls both the Python and Go version of a page/function/search and making sure that they produce the same results. 2. Change the production code over to the Go version, perhaps using a ramping strategy where X% of servers are running the Go code, and you gradually increase X, while monitoring vital statistics like server load and response time. 3. Now that the production site is running Go, incrementally split off components into their own services. This approach leads you to the same destination, but with a lot less risk. It is very unhealthy to have a situation where the production site is running one codebase but all the developers are working on another codebase. Note that you will realize the benefits of Go (performance, type safety) after step 2, which is much sooner than OP's plan. Joel Spolsky's classic essay about how you should never do full codebase rewrites is worth reviewing: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...
- jordanbeiber 7y ago
  Is moving directly to #3 not an option? I’m thinking that defining parts that can be moved to separate services, and start consuming these could be a way to organically transition to a new architecture.
- krenel 7y ago
  100% agree. We've just finished to roll out our implementation in Go migrating a subsystem from PHP and receiving around 150req/second and demultiplexing those request to 1500-2000req/second to legacy backends. The key to the success of the project was that the API was an exact match, and we could compare both implementations for exact requests. The deploy strategy of the new version: - Reply the real traffic to the new Go service comparing the results with the old one - Then implement a toggle feature than enabled different traffic sources to use one backend or the other - Keep changing backends to the new system and ensure that metrics were unaffected Having e2e and integration tests for the Golang project was of a huge help, since we could fix all differences using TDD. Although we changed some of the implementations to take advantage of Go constructs, just a 1-to-1 replacement would have had a huge performance impact.
zmmmmm 7y ago
I think some of the misunderstanding in these comments comes from not fully appreciating the perspective of not-for-profit organisations. While I can't speak for Khan Academy, I know that in every NFP organisation I have worked for there is an acute awareness that funding could dry up one day and the prime directive is to ensure that in a scenario like that, the work of the organisation can continue. In this case, it leads to a higher concern about minimising the cost of the operational services than you might have in a for-profit organisation. In all the strategic planning I have been involved in with NFP, we always have the "what if worst case scenario arises" plan and in that plan the ability to scale down to bare minimum operational cost is key. It may not be conscious but I suspect that may be part of the reason the performance savings from moving to Go are so attractive in this case, where most profit-making companies just ask the question of whether they can afford to pay for the servers with their current margin or not and if they can they have more important things to worry about.

418 points by yawn 7y ago

Possiblyheroin 7y ago

I was under the impression that Go was more performant than Kotlin. TMYK :)

tus88 7y ago
Go was written for servers, Kotlin for mobile apps. Seems an obvious choice really.
- endorphone 7y ago
  Kotlin is just a less verbose, more modern language targeting the JVM. It has nothing to do with mobile apps beyond that Google, much later, decided to support it for Android development. Go is an efficient platform, however in these switches it usually goes something like "we took something hugely overbuilt, with layers and layers and layers and abstractions and abstractions, and rebuilt it with minimalist Go and now it's faster", which, of course it is.
- hn_throwaway_99 7y ago
  > Go was written for servers, Kotlin for mobile apps. Not really. Kotlin was just a better JVM language developed by the JetBrains folks before Google adopted it as a first class Android language, but I don't believe it was specifically developed for mobile initially.
  - dehrmann 7y ago
    The open question is if Java will start borrowing the best features from Kotlin and it will become less relevant (like Scala). To Kotlin's credit, it has better IDE support, it feels simpler than Java (whereas Scala feels more complex), and it cleans up a lot of fundamental language issues that Java can't get away from.
- sosodev 7y ago
  I wouldn’t say that Kotlin was written for mobile apps. JetBrains, the company that created Kotlin, doesn’t even have mobile apps AFAIK. It’s just meant to be a more modern JVM language and that means it does anything Java does just better. Servers included.
- xwowsersx 7y ago
  I don't think that's true. You're confusing the fact that Kotlin has become the officially supported language for Android development with the idea that it was written for mobile apps.
- lloydde 7y ago
  You may be confusing the adoption of Kotlin for Android. That happened more recently Kotlin has been open source since 2012 and designed as a direct (compatible) replacement for Java.
- joshbaptiste 7y ago
  The only thing I see where Golang wins over the JVM is overall memory usage for daemons.. As Khan is non-profit they want to save at any avenue.
mrspatula 7y ago
JVM is pretty nippy
defertoreptar 7y ago
I have found that there can be drastic differences in Go performance in the particular way you structure the program. Writing Go code in a Python-like way is going to be less performant than if you run escape analysis every compile and make deliberate effort to stay on the stack.
- pcwalton 7y ago
  > Writing Go code in a Python-like way is going to be less performant than if you run escape analysis every compile and make deliberate effort to stay on the stack. The fact that people are doing this tedious optimization is strong evidence of the fact that Go needs a generational garbage collector with bump allocation in the nursery. The JVM has a fast generational GC, and as a result you don't have to do this kind of optimization to get good allocation performance.
coldtea 7y ago
Why would Go be more performant than Kotlin? The JVM has been fine-tuned for 3 decades (JIT and GC) whereas Go is pretty primitive in a lot of its backend.
- Thaxll 7y ago
  See for yourself: https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/go.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
  - coldtea 7y ago
    See what? This is an unofficial benchmark game - and based on implementations people bothered to provide. Not some scientific test, and nothing that guarantees these are the best implementations. And even if they were they're not representative of server/long running program behavior (where JIT quality and especially GC implementation) matters. Even so, the page linked starts with "Back in April 2010, Russ Cox charitably suggested that only fannkuch-redux, fasta, k-nucleotide, mandlebrot, nbody, reverse-complement and spectral-norm were close to fair comparisons". Since Russ Cox is one of the Go co-developers, lets see the ones he accepts are fair. Of those 5 are present in the page, on 3 of which Java wins with decent margins (fannkuch-redux: 18%, k-nucleotide: 21%, reverse-complement: 17.5%) and on 2 of which Go barely comes ahead (fasta: 6.3%, spectral-norm: 6%).
    - Thaxll 7y ago
      I used both Java and Go in production and yes Go can be faster or Java it depends but Go usually use between 3 and 10x less memory. https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ ( real world benchmark with networking / serialization ect ... ) Get over it yes Go can be faster than Java it's not a secret.
      - apta 7y ago
        > real world benchmark Highly debatable. Even then, you see that golang ranks in the 102nd place for the plaintext benchmark, almost 6x slower than Netty, which is very established in the JVM world. Same with the JSON benchmark, golang is in the 126th place, ~3.5x slower than Netty and Vertx. > but Go usually use between 3 and 10x less memory. The JVM by default will use whatever memory is assigned to it. This makes sense from an efficiency and utilization point of view, as it generally aims for maintaining good throughput (whereas golang is only tuned for latency). The JVM now ships with new low latency GCs (ZGC and Shenandoah) which are currently available in experimental phase.
Thaxll 7y ago
Overall Go is.

atmosx 7y ago

How does Khan academy make money to sustain the website? Is it all donations?

sosodev 7y ago
From what I understand it’s mostly donations but I think they receive some grants too.
bhaumik 7y ago
Non-profit, with several $1m+ donations: https://www.khanacademy.org/about/our-supporters https://www.khanacademy.org/about/our-supporters
- patneedham 7y ago
  To be more specific (for anyone else who hasn't checked the link yet), the "Lifetime giving" section has: 9 donations >10m, 4 donations between 5-10m, 20 donations between 1-5m So looks like there has been at least 120m in donations!
  - atmosx 7y ago
    Thanks

bodka 7y ago

GO gorever!

purple-again 7y ago

We turned our monolith into a bunch of micro services almost 6 years ago to the day. For a long time I was very happy with the new pattern but over the years the weight of keeping everything updated along with the inevitable corners that fall behind and have...questionable..security due to how long they sit neglected has really left me wondering if I am happy with it after all. I would love hear some thoughts from others that made the move, especially anyone that decided to move back to a monolith repo.

tuckerconnelly 7y ago
I moved back to monolith and am very happy. I think of the monolith now as a collection of modules. The rule is now, one should be able to drag any of the modules to the top-level of our monorepo and create a new microservice pretty easily when the time comes. I think the microservices book (that came from that Uber engineer...?) suggests a rule of 5 engineers per service.
- virmundi 7y ago
  How are you preventing transaction couplings? For example, module A and B are called by C. C starts a transaction that wraps A and B. If you move B up as a network feature, you lose the transactionalty.
  - m0zg 7y ago
    Usually the answer to this is "we pray to god it doesn't fail".
  - deleted 7y ago
    [deleted]
  - tuckerconnelly 7y ago
    Good question, and this rule is meant to be bent in those scenarios. I try to avoid these dependencies if at all possible, but if not possible, the "writes" for those modules all belong to a single service, and any other service, depending on how "pure" I need to be in the project, will make network calls to the other service, or just grab that data directly from the database. For a more concrete example, I recently built a service ("scraper") that scraped data and upserted a large tree of structured data to postgres in a transaction. Writes were only allowed from scraper, but "api" could SELECT data for reporting to the frontend "web" as much as it wanted. In the future, "api" might make be refactored to make internal HTTP request to "scraper," so they could have totally separate databases.

peterwwillis 7y ago

So, some potential pitfalls: - The decision seems to be primarily a software architecture one, without much mention of all the other architects whose input will shape how the finished product is run and supported. In a modern software development environment, all the other parts of the org should be consulted on greenfield work to "Shift Left" anything that may need to change down the pike. Design in a silo leads to ineffective products. - They're going from "hmm we need to upgrade from Python 2 to Python 3", to "we need to redesign everything in a new language with a radically different software architecture". This is definitely the second system effect. It's going to take years to make this thing reliable and sunset the old product. - They're porting over the logic? Even if this is actually the right move, wouldn't a clean-room implementation potentially give better outcomes? - Why are they continuing to use App Engine if the writing's on the wall for 2024?

batter 7y ago
Go + AppEngine is the most unstable combination i have ever seen. While we tried to deliver project during 1 year, it was almost fully rewritten couple times because of new Go or AppEngine API. Having NodeJS with far less problems. And AppEngine has huge price tag.
- pm90 7y ago
  GAE v2 let’s you use docker containers and ime it has been pretty stable and fantastic.
dangoor 7y ago
I don't disagree with your pitfalls, but I do think we're working to avoid them. To your first point, "The decision seems to be primarily a software architecture one...", this project has had involvement of the whole engineering team since the beginning. The whole org is on board with this change. It's definitely not happening in a silo. > This is definitely the second system effect. It's going to take years to make this thing reliable and sunset the old product. I hope not, but obviously we're not done yet, so I can't say how long it will end up taking to completely decommission the Python 2 app. What I can say is this: there are aspects to this project that are _simplifying_ our system and, for what's left moving from Python to Go, our intention is to port the business logic as close to a straight up port as we can get. > - They're porting over the logic? Even if this is actually the right move, wouldn't a clean-room implementation potentially give better outcomes? _That's_ second system effect, to me. We can't change everything and fix every problem now, so we're focusing on the changes that will help us move from Python to Go faster. > - Why are they continuing to use App Engine if the writing's on the wall for 2024? I don't think Google Cloud is disappearing in 2024, for one. Beyond that, again, we're not changing everything about our architecture. The way our data is stored is staying the same.

benatkin 7y ago

The article says this: "Moving from Python 2 to 3 is not an easy task." I disagree with this. It's a Python project's dependencies that make it hard to move from 2 to 3, and most libraries have been updated. Of course, you could argue that it isn't easy to migrate a codebase from one major version of a language (or framework, or database) to another, but when you eliminate easy from your vocabulary it becomes harder to describe different levels of difficulty.

foolfoolz 7y ago
migrating from python 2 to 3 is such a large task that migrating to any other language is a comparable effort. this is not just a library problem the language itself changed significantly source: no python services at my company are going to be migrated to python 3; it’s all moving to a JVM
- was_boring 7y ago
  What? I've done it on some sufficiently large code bases, and small ones, and it was done way faster then a rewrite. With tools like 2to3 you can assign it to an intern and have it done pretty quickly.
  - edoceo 7y ago
    It's not quite that easy.
- zo1 7y ago
  Migrating to 2-3 can be a large task in some very rare cases possibly, but for the most part it is practically effortless if you don't have to support both simultaneously. The biggest hurdle might be the "fear" of the unicode change, but that can be dealt with. Source: All python services at my current workplace are in the process of being migrated to 3.*, and I'm doing one of the main ones at the moment and it's a breeze, including compiled c-extensions. For curiosity, a good list of actual python3 syntax changes: https://docs.python.org/release/3.0.1/whatsnew/3.0.html#overview-of-syntax-changes https://docs.python.org/release/3.0.1/whatsnew/3.0.html#over...

timwaagh 7y ago

Seems like such a waste. Is switching to python 3 really that hard? Is hardware that expensive? If this is indeed the right call it doesn't bode well for traditional scripting languages as the web scales to fewer high traffic apps. We might start to see more jvm, go (apparently) or even rust and c(++), rather than speed of development languages like Python or Ruby. Trend seems to be the reverse though, with python the second and most rapidly growing language.

neurobashing 7y ago
Everyone’s project/code base is different but in my experience there’s been a critical mass of libraries for a few years. I presume the “it’s hard to move to 3” is dev teams wanting a new toy as much as “the rewrite is too complex”. Library use, size of code base etc are all big factors but at the end of the day, I think team motivation is really the deciding factor.
- Consultant32452 7y ago
  This is my experience too. Someone or a couple someones on the team decide they want to try out some new tech or expand their resume. Then it becomes a quest to justify the switch rather than a quest to make the best business decision.
- freyr 7y ago
  That mirrors my experience as well. Someone with influence is bored or wants to level up, so they'll drag the entire company into a long, expensive quagmire. Unless the existing codebase is mired in technical debt and completely unsalvageable or cannot scale further, this seems like a very radical move.
  - ksec 7y ago
    Which is precisely why most of the Web Dev should not be called Engineers.
  - lonelappde 7y ago
    Replacing an old unholy mess with a new unholy mess is usually a bad plan. It's called the Second System Effect.

kucing 7y ago

Looking at the case where khanacademy is migrating their server only after about 10 years. I realize more that I don't have to worry that much about being locked into certain technologies (unless it's clearly untransferable, e.g. storing part of customer data in 3rd party server), because after all, we might keep it only for about 10-20 years, and the thing I'm working at almost certainly will only last < 2-3 years.

kissgyorgy 7y ago

Huge mistake. Reminds me of https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...

thatswrong0 7y ago

Ok this is going to sound ignorant, as my only experiences in backend services have been Go and Python. I don't like either. Is there something I'm missing? For simple CRUD apps, both are sufficient. But (in my limited experience), the moment I've wanted to create more complex business logic with stricter constraints, neither has been quite up to the task. Go doesn't make things easy. It asks you to repeat yourself. I don't like the lack of basic functions like a generic map / filter function.. I know Rob Pike just says "use for loops", but it feels so unnecessarily unexpressive. When I see map, I know what's going on almost immediately. For loops take more reading to understand. Nil pointers shouldn't be, yet still are, a thing (developers aren't perfect - why can't the type system help?). It feels like a straight downgrade from Python from a code clarity perspective. And it's typed, sure, but the type system doesn't let me express constraints that other languages allow me to do that would prevent entire classes of bugs. It doesn't feel worth it compared to Python. Yes, the core language ends up being comparatively "simple", but simple building blocks doesn't guarantee a simple overall system. And my company is very diligent from an architectural perspective. But then when I look at Python, I'd rather just use Javascript with Lodash, esp. when it comes to the treatment of functions as first class objects[0]. Throw Typescript in there, and you get, in my opinion, a better type system than Go, so unless language performance is a major constraint (which it hasn't been for my company, our DB usage patterns it the biggest thing instead), why would I want to use either of these rather than Typescript? [0] Edit: Dumb and wrong, I meant its treatment of anonymous functions. I don’t like lambdas.

catalogia 7y ago
Go seems to be optimized for onboarding new developers (particularly straight out of school) quickly, rather than for the long term comfort of developers using it. There are Rob Pike quotes that speak to the first part of that at least. If I was a business owner, I'd love Go. But what I can't really figure out is why so many devs love it. It's far from the worst thing in the world, I don't hate it, but I just can't get excited over it either.
- adtac 7y ago
  Defers, channels, inexpensive goroutines, a mostly consistent standard library, very good performance, garbage collection, being similar to C, strongly typed, and readability are the reasons why I've enjoyed writing Go exclusively for the past couple of years. Almost none of these is exclusive to Go, but the intersection of all these certainly is.

aazaa 7y ago

> Now, in 2019, Python 3 versions are dominant and the Python Software Foundation has said that Python 2 reaches its official end-of-life on January 1, 2020 , so that they can focus their limited time fully on the future. Undoubtedly, there are still millions of lines of Python 2 out there, but the truth is undeniable: Python 2 is on its way out. The Python 2/3 split is by far the most annoying thing about Python. I don't develop software in Python but about half the time I've had to use a Python library or program the problem of 2/3 incompatibility has cropped up. Some projects don't make it clear whether one or the other is required, leading to further confusion. If anything the Python 2 EOL could make a bad situation worse. Like Khan Academy, each Python 2 package maintainer will be forced to make a decision: move to Python 3 or abandon and maybe move to an entirely new language. It think many will choose to abandon, leaving these packages to rot. Second on the list are the multiple package managers (or things looking like package managers). Third on the list of annoyances are native extensions, driven by the poor performance of Python itself. These extensions make it difficult to use certain libraries across operating systems. So as a non-Python developer I don't look forward to the occasions when I must use a Python-based piece of software.

zo1 7y ago
If you're installing a package via the package manager, it will very quickly tell you if you can install it on your specific version of python. Unless you're downloading some rather obscure and un-loved library where the author didn't explicitly state which versions of python they support. Multiple package managers: There has only been 2 big ones from my long-term general usage of python. Easy-install and pip, the former of which is falling in favor but still semi-supported. Pretty much everything runs off of pip. What may be confusing you, and does confuse me at times as well, is their naming and the installation instructions as provided by library authors. E.g. some say python setup.py -install others just tell you to "pip install" it. Some would say use "setuptools", etc. Other would tell you to use things such as "conda" or "anaconda", pipx, and to create virtualenvs. All secondary, but things that should not ideally distract you from just plain using pip. 3. This has also been getting a whole lot better in the last 5 or so years. Microsoft has been funding dev-time to make the ecosystem for python (including extension compilation) much more pleasant in the Windows space. Also, the package managers and library authors are doing a whole lot better in that binary distributions are much more prominent so the compilation of the extensions never has to happen on your machine.

Thaxll 7y ago

Kotlin looks nice now but on the long term it's a bad choice since they're stuck on Java 8 they will always lag behind the real JVM and won't be able to catch up since they need heavy modifications.

m0shen 7y ago
If you mean the java 8 bytecode version, that hasn't been true since April: https://blog.jetbrains.com/kotlin/2019/04/kotlin-1-3-30-released/#more-6991 https://blog.jetbrains.com/kotlin/2019/04/kotlin-1-3-30-rele...
- Thaxll 7y ago
  But they don't benefit from new runtime features right ? Loom, GC ect ... are no go for Kotlin.
  - m0shen 7y ago
    I don't see why it wouldn't benefit from the GC changes. AFAIK Loom has no scheduled release date. They seem like they're committed to supporting new features, but you're probably right they will lag on some things.

buboard 7y ago

If they were going for performance, and web-related, how come they didn't consider PHP?

ngngngng 7y ago

Unpopular opinion, writing Go is faster than Python. With the compiler, strong typing, and no versioning hell, I'm much more productive in Go. Whenever I use python I run into problems with versions and dependencies. And the whole community just tells me to use pyenv or virtualenv and it will "fix all my issues". Only it doesn't.

milofeynman 7y ago
I doubt that's unpopular. It's the same reason I significantly prefer Go over Ruby.
kart23 7y ago
For larger stuff, yes. For small stuff, no. If you just need to get something small done quick and dirty, python will be easier and faster.
- jerf 7y ago
  Yes. My current estimate of the cutover, for myself, is about three weeks of solid, 40hour/week development. After that, my Python (or other dynamic language) starts the process of seizing up, where instead of rewriting some module I just put a little hack in there to make it backwards compatible with other code, since I haven't got a great way of being quite sure what's calling this code, so I use a __setitem__ or have a function that takes "a thing or an array of that thing", and I find myself increasingly reluctant to refactor the Python. YMMV on the exact number, but that's been my experience several times now. I know it can be done; I've seen it done, I've done it myself. But refactoring without even the rudimentary static type system Go has just becomes an increasing nightmare at scale. And I use unit testing in Python, etc. But, flipside, yes, Go isn't a great language for just bashing a script together in. Maybe not the worst, with a bit of library work, but not a great language.
watermelon0 7y ago
Just as a counterpoint, I'd say that using Python can IMHO be similarly productive than Go. Regarding the dependencies, you have tools on top of virtualenv, such as pipenv/poentry, which handle dependencies, and are easy to use. Biggest issue that I've encountered would probably be when two or more dependencies require the same package, with no intersect between supported versions. I don't think Go handle this any better, thought. Type hints (and mypy for static type checking) are a must, and coupled with a good IDE, they really improve the productivity. I'd say that mypy's type system is more advanced than Go's, but it strongly lacks in type safety (due to the fact that majority of the libraries are not taking advantage of it yet, and that Python is still a dynamic language by its nature, and there is no runtime type checking).

bootlooped 7y ago

The article mentions Go's superior compile time, when compared to Kotlin. I have done a lot more Java development than Kotlin, but my recollection is that both of them compiled fairly fast. Is Go really significantly faster to compile for similarly sized projects?

tbrock 7y ago
I’ve written some things in both languages and compiling go feels an order of magnitude faster.
- abraxas 7y ago
  It's mostly because most setups will use maven or gradle which do much more than just compile code. They will often pull down dependencies, check for bugs and style inconsistencies, run unit tests etc. If you run plain javac against a bunch of files it will compile just as fast as Go. In any case it will be in a blink of an eye.
sheeshkebab 7y ago
Yes, it’s much faster than these java and jvm based codebases (including all jvm based languages like kotlin, scala etc). Go as a language is just so much simpler. Although compiling overhead of large JavaScript codebases, that make up most of modern websites, kills any kind of benefit in improving build performance of some backend service.
fizx 7y ago
It's rare that I notice the compile time for either. Incremental compilers are great!
tapirl 7y ago
The article mentions why choosing Go over Kotlin is memory consumption.

tybit 7y ago

As much as I personally don’t enjoy writing Go I really can’t fault them. I still find it interesting that for a relatively obvious feature set of fast compiles, fast startup and fast runtime there really isn’t anything mainstream out there to compete with Go. I really hope something like Kotlin, Swift, ReasonML or even AOT JVM/.NET brings something to the table soon. Or perhaps I’ll just have to wait for WASM to really take off server side.

genuine_smiles 7y ago
> fast compiles, fast startup and fast runtime All I can think of is D, but it’s not quite mainstream. Are there any other less popular languages that meet all 3 conditions?
- amedvednikov 7y ago
  https://vlang.io https://vlang.io
  - sagichmal 7y ago
    Still snake oil.
    - fouc 7y ago
      Site looks pretty legit, can you explain why it's snake oil?
      - doteka 7y ago
        Because it literally uses string manipulation to generate c code from source. There is no concept of an AST or anything - just string bashing.
  - iteratorloopmap 7y ago
    LoL
  - 7y ago

sibeliuss 7y ago

Isn't there a quote somewhere along the lines of "full rewrites are suicidal?" Seems pretty risky to me.

stoicShell 7y ago
The saying only applies when your product is software itself, not in most cases where software is just the means to deliver your product — here a website providing education in video form etc. Besides, the reality is that most business software out there gets rewritten every 3-15 years (really depends on use-case and conditions, but on average 4-5-6 years is a good bet). After some time it's just not worth it to keep refactoring, you'd rather start anew with hopefully better tech and certainly with better knowledge of your problem — they say you should write everything 3 times to make sure you really nailed it. In many businesses, these rewrites would constitute a new major version, more comparable to the feeling we always got in the waterfall era — new version = big changes, new UI, new stuff. That's when it's possibly lethal, if you really break the thing, and that thing is your product, not a means to it.
- guitarbill 7y ago
  the number of failed migrations, modernizations, and "tech transformations" in non-software industries, as well as the number of consulting outfits and their profits doesn't seem to back this up. [0] IT disasters now part of modern life - https://www.afr.com/technology/it-disasters-now-part-of-modern-life-20160628-gptyw6 https://www.afr.com/technology/it-disasters-now-part-of-mode... [1] Number of IT failures at banks and other firms is unacceptable, say MPs - https://www.theguardian.com/business/2019/oct/28/number-of-it-failures-at-banks-and-other-firms-is-unacceptable-say-mps https://www.theguardian.com/business/2019/oct/28/number-of-i... [2] The Biggest IT Failures of 2018 - https://spectrum.ieee.org/riskfactor/computing/it/it-failures-2018-all-the-old-familiar-faces https://spectrum.ieee.org/riskfactor/computing/it/it-failure... we can argue whether these are "rewrites", but big changes can be rewarding but are inherently risky. balancing this is hard, and rewrites uncover and introduce the unknown unknowns.
  - 7y ago

Solar19 7y ago

Interesting read. It always surprises me that companies go many years running major apps and infrastructure with interpreter-based languages like Python and Ruby to begin with. It's an incredible waste of energy, compute, and for web apps sometimes, users' time. Developers need to be a lot more disciplined about performance and efficiency. I'm glad Khan went to Go, but man all those years wasted.

tony 7y ago

You can start porting your live codebase to python 3 right now - no downtime at all. You can have a 2+3 compatible codebase live in a week or two, it'll get you about 98% the way there [1] My projects (e.g. https://tmuxp.git-pull.com https://tmuxp.git-pull.com, https://unihan-etl.git-pull.com https://unihan-etl.git-pull.com) are both python 2/3 compatible. I learned by reading through other projects like Sphinx, Flask, Werkzeug, and SQLAlchemy: - Flask: https://github.com/pallets/flask https://github.com/pallets/flask - Werkzeug: https://github.com/pallets/werkzeug https://github.com/pallets/werkzeug - Sphinx https://github.com/sphinx-doc/sphinx/tree/v1.8.5 https://github.com/sphinx-doc/sphinx/tree/v1.8.5 (see compat: https://github.com/sphinx-doc/sphinx/blob/v1.8.5/sphinx/util/compat.py https://github.com/sphinx-doc/sphinx/blob/v1.8.5/sphinx/util...) - https://github.com/sqlalchemy/sqlalchemy https://github.com/sqlalchemy/sqlalchemy Helpful blog posts: http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/ http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python..., http://lucumr.pocoo.org/2013/5/21/porting-to-python-3-redux/ http://lucumr.pocoo.org/2013/5/21/porting-to-python-3-redux/ As for automation tools, for 2/3 compatibility, I used futurize and huge codebase to great success: https://python-future.org/futurize.html https://python-future.org/futurize.html. I started with this: futurize --write --stage1 --unicode-literals --nobackups <files> and https://docs.python.org/2/library/2to3.html https://docs.python.org/2/library/2to3.html 2to3-2.7 -w -n <files> Really helped. Also, wire Travis / your CI system to have your tests run on python 2 and 3. If you want to test your python 3 codebase live on a subdomain / same SQL/NoSQL DB, be careful about jobs/tasks! Pickle version mismatches and stuff. Use a separate redis/whatever DB for the deployments. After you're finally on python 3, you can use https://github.com/asottile/pyupgrade https://github.com/asottile/pyupgrade to modern your code. [1] For the other 2%, use conditionals in your requirements file: enum34==1.0.4;python_version<"3" ushlex==0.99.1;python_version<"3"

dangoor 7y ago
I appreciate all of the links and the fact that, for a lot of folks, this is the state of Python 2 to 3 migration today. What you're suggesting is, imho, the best path for most (and I appreciate you mentioning the pickle incompatibility because that is a thing which would trip people up when trying to make the move). If we would have been able to have a 2&3 compatible codebase live in a week or two, we absolutely would have done that. Incompatible changes in some libraries we use, App Engine first gen to second gen changes (which are for the better, but still a big deal), the choice of storing some pickles permanently, plus a need to really verify unicode handling all over the place (especially in a 10 year old codebase), and other factors that aren't coming to mind at the moment mean that this is not a couple week thing for us. Moving to Go is more work than moving to Python 3, but in our particular case it's not as much more work as people might expect.

RandyRanderson 7y ago

Why do they call them "micro services" and not distributed systems? Oh right, it's because distributed systems are obviously really hard to create correctly and no sane person would ever agree to pay for that. Nice: re-branding. I can't wait for the, maybe "consolidated computing" manifesto (aka turning micro services back into monoliths).

thethimble 7y ago
What if the services you are writing are independent in that they solve separate business problems, are built by separate teams, have little to no data coupling (e.g. Only basic auth), have different scalability profiles, etc? Separate services are really effective for these cases. Neither micro services nor monoliths are silver bullets. Instead it's possible for each approach to be the best approach in a particular business context.
- RandyRanderson 7y ago
  In several decades of it experience , I've not known or heard of a nontrivial system like the one you describe in the first part of your message. In the latter part, that must be a disingenuous dichotomy. You don't really believe that just because an avenue exists we should include it in an evaluation?
- pjmlp 7y ago
  What about writing modular libraries then?
olingern 7y ago
Why are you getting hung up on nomenclature? The point of the article is clear. If you feel that using 'microservices' is too trite or "buzzword-y" then that's about you and not the article.
- RandyRanderson 7y ago
  I guess you're agreeing with me sarcastically? Very funny.

wonjohnchoi 7y ago

It is crazy to see they are still using python 2. Seeing how slow the conversions to python 3 have been, was creating python 3 a good decision for python community? Can it be argued that developing python 2 further in a backward compatible way would have been better for the community? I know that evaluating this kind of thing is hard as metrics are bound to be subjective and speculative. But I am curious if there was any serious attempt to figure it out.

ddxxdd 7y ago
I remember a long time ago, I used the "six" library (Python (2 * 3) ) to code in Python 2 with an API based on Python 3. I'm not entirely sure how well the line of code: "import six" would fix any compatibility issues nowadays.
speedplane 7y ago
> was creating python 3 a good decision for python community? ... I know that evaluating this kind of thing is hard I don't think there is any question here: Python 3 is a complete disaster. Years and years of engineering effort wasted on changing string libraries. Sadly, the Python leadership refuses to acknowledge the failing, perhaps because such an acknowledgement would challenge their omnipotence ... it would, and it should.

bhntr3 7y ago

> If we moved from Python to a language that is an order of magnitude faster, we can both improve how responsive our site is and decrease our server costs dramatically. I see people say things like this a lot but my experience is that while other languages are 10x or more faster than python in some benchmarks it's very rare that computation time dominates server latency or that servers are running at 60%+ cpu across all cores. If 90% of your service latency is not directly on the cpu and/or you haven't profiled to see that the performance bottleneck is evenly distributed across all tasks, then it's super dangerous to migrate to a new language thinking that will fix it. I hope people inside Khan Academy know this and it's just a clickbait blog. If they really think "go is 10x faster than python so we'll only need 1 server for every 10 when we migrate" then I think they'll be disappointed.

alfalfasprout 7y ago
Well Brad, that's very dependent on what exactly the service is. Moreover oftentimes low CPU utilization is actually a limitation of the implementation on a slow language (eg; Python technically does have async webservers but adds a lot of idle overhead). Indeed, there are many benefits these more performant languages have over Python aside from raw single-core performance. For starters, more efficient concurrency and parallelism can help reduce average latency when combined with a quality async webserver. Then there's gains due to shared memory across threads. So in many cases-- absolutely, you can only need 1 server vs. 10 when you migrate. It's thus not fair to say that these gains are "very rare".
pm90 7y ago
* It’s not just that Go is faster to run but also faster to iterate on. If python can be neither, its offering little benefit. * they moved from a monolith to a microservices architecture; concern that any of the services in the request path could add latency just because of the overall runtime speed is slow is a legitimate one. * their primary deployment method is Google App Engine where you are billed by CPU used. Any change that consumed less CPUs has a tangible effect on their costs
- andy_ppp 7y ago

apta 7y ago

> Go, however, used a lot less memory, which means that it can scale down to smaller instances. I could be mistaken, but this sounds like they went ahead with the default JVM settings, where it tends to use as much memory it is allowed to (which makes sense from a utilization and efficiency perspective). If memory usage is a concern, the JVM can be tuned for such.

jacques_chester 7y ago
The JVM hasn't yet become fully container-friendly because it bases its calculation on the host OS figures, not the container figures. You can use a calculator to get precise, proven settings for any supported JVM: https://github.com/cloudfoundry/java-buildpack-memory-calculator/ https://github.com/cloudfoundry/java-buildpack-memory-calcul...
- apta 7y ago
  I thought this was a solved issue since JDK 10: https://www.docker.com/blog/improved-docker-container-integration-with-java-10/ https://www.docker.com/blog/improved-docker-container-integr...
  - jacques_chester 7y ago
    In my understanding, no. It's improved but not fully "fixed". Unfortunately that's as far as my understanding extends.

deleted 7y ago

[deleted]

tmpfile 7y ago

Yikes! It's a lot of effort to reduce memory use. They might be better off creating a new Go entrypoint/server that can call into CPython to reuse all their existing/tested modules (treat their Python as a microservice called by Go). They could then use Go to create/call new microservices or replace various routes on a selective basis.

remote_phone 7y ago

What a mistake. Not only going from a monolith to microservices but also changing the language? This is a mistake that rookies make. This will be one of those post-Mortems where they will sheepishly admit they bit more than they could chew, and it wasted years of productivity. There’s no reason to move to Go. Stick with Python for now. Migrate safely to python 3. Once everything is stable, start breaking things up into thrift or protobuf services. They don’t even need to be microservices but you need the contract. Once that is stable migrate to whatever language you want. But at this point you will have the well-defined api and test cases. Trying to do too much all at once is a no-brained disaster.

pjmlp 7y ago
This kind of decisions is usually always taken when the monetary cost is not measured. Apparently not mapping developer hours to real money keeps not being a thing outside consulting shops.
- deleted 7y ago
  [deleted]

todd3834 7y ago

> We’ll only generate web pages via React server side rendering, eliminating the Jinja server-side templating we’ve been using I’ve been down this road. Deep down this road. Let me just give you a heads up on something I didn’t consider at the time: Most template languages do not parse every single node, one by one. In a sense they are just doing string concatenation. Not so with server side rendering and React. I’m not saying it can’t be done but just realize it is going to take a lot more compute power. Caching is great of course but won’t help you if you plan to customize user content during the server side rendering as well. My recommendation is that you don’t do any user authenticated stuff during SSR. Also consider how you are going to handle cookies if you do plan to make authenticated requests to server side rendering. Also solvable but for some reason people had the hardest time understanding why we had to forward cookies to the domains we controlled in an API request and definitely not to any other servers. I’m not sure I would pick React for an SEO driven website. It is hard to get a competitive “time to first byte”. Unless of course you can pre warm a cache of every one of your pages. Lastly, you’re going to need Node for the SSR. I’m sure you know this but that might take you out of app engine and into cloud compute. Not a big deal but thought I’d mention. Good luck! It is doable. If you ever want to chat about how we solved some of these problems I’d love to save you some time if I can. Hit me up in my profile email.

dangoor 7y ago
Thanks for the suggestions and the offer to chat! We've been doing SSR for quite a while now and are improving our CDN use as we go along. We already took steps to ensure that there's no user-specific information showing up in our server-side react rendering which would damage cacheability. Our frontend infrastructure team essentially owns the React render server. I'll let them know you offered to chat.

plinkplonk 7y ago

just trying to understand - you guys think moving a Python2 monolith to Python 3 is too painful, and so you are going to port all the code from Python2 to a completely new language (Go), change the architecture (monolith -> microservices) and move the HTTP API to React + GraphQL, all in one year? 2020 is going to be in an interesting year at Khan Academy ;-)

CGamesPlay 7y ago
The move from Python 2 to 3 would likely have also involved changing the architecture so they could migrate components incrementally. Since they were going to do that regardless, and if they already wanted to change the interfaces from HTTP to GraphQL, this is a natural time to do it. Though, this migration has nothing to do with React--they were already and will continue to be using it.
- deleted 7y ago
  [deleted]
conradfr 7y ago
At least the article makes it seem it's not a decision taken on a whim and they did some kind of POC and planned the transition. Of course the obvious thing missing in the article is how they expect to deliver new business features while recoding everything in a new language. It's fun to read this and the Etsy thread currently on the frontpage as well.
- dangoor 7y ago
  > Of course the obvious thing missing in the article is how they expect to deliver new business features while recoding everything in a new language. For new features, the new parts of the GraphQL schema for those features will be written in Go as part of the new services. Our frontend is already in large part a single page app in React which requests data via GraphQL, so the frontend for the features will look just the same as it would on our monolith.
dangoor 7y ago

deleted 7y ago

[deleted]

kureikain 7y ago

Go is a very weird language :-(. It's very limited when you started to do complex thing. Example, let's say you are building websocket. You will have a hard time to write type safe websocket handler to process the payload from client for all the events... I started to do Rust/Crystal and both of them are better than Go(performance, type system). Yet, whenever I build something for work, I come back to Go :-(. I told myself to use Rust or Crystal. Then I realized that Go is a practical language. It compiled fast so it makes testing easier. The cross compiler just make it so easy to build binary run on everything thing. And the limitation of Go makes it very consistent on how you do thing. This makes working with Go become faster event by the fact that it slows you down on other parts. So I think Go is a language that people easier to fall into because it has the speed of interpreter language like Ruby/Python(or even faster) during development and have a better performance/type safe story.

apta 7y ago
Java and C# provide the benefits you mentioned, while being more expressive languages and having better runtimes compared to golang.
- hu3 7y ago
  Hardly. OP mentioned fast compilation and limited ways to code the same solution. C# and Java are slower to compile and offer way more options to do the same thing. Here's Uncle Bob take on testing with Go: https://m.youtube.com/watch?v=2dKZ-dWaCiU&t=36m40s https://m.youtube.com/watch?v=2dKZ-dWaCiU&t=36m40s
  - apta 7y ago
    > C# and Java are slower to compile Not for any meaningful work in my experience. As a matter of fact, I found the change/compile/run loop in golang to be slower on projects I've been working on due to the fact that it doesn't support incremental compilation, so any change I make ends up recompiling the entire program and writing out a 100+MB binary anyway. Compared to a Scala project I worked on before (and Scala is notorious for slow compiles), after the first compilation, all modifications happen very quickly as only the respective classes are re-compiled. > Here's Uncle Bob take on testing with Go Again, this doesn't apply for any non-trivial/large project. On a project I'm working on, it literally takes 7-8 minutes to do a clean build + run all unit tests in golang.

sethammons 7y ago

I've been in a similar boat. We've been splitting up or converting large Python 2.6/2.7 applications into Go services (and doing the same to large Perl applications) for a long time now. Go has consistently been 10-20x performant (allowing for dramatically reduced hardware needs), easier to maintain, and more productive to produce code in than our previous Python (Twisted) and Perl (AnyEvent). Hopefully KhanAcademy has solid telemetry data in both legacy and new code so they can quantify benefits. They will also have a learning curve for managing multiple micro services vs monoliths. Accessing shared data will be a problem they will likely have to solve. We've opted for each service controlling its own data - no reaching into another service's data behind its back. Everything through APIs. This gives the microservice the ability to alter its datastore as it needs to and not be blocked by other teams' need to update how they access the data. Debugging a distributed solution is much harder than a single service. Distributed tracing, consistent structured logging with log aggregators that let you do fancy searches (like Splunk), and application telemetry and metrics will be even more important than before.

dangoor 7y ago
Does sound similar! We have established the rule that each service owns its own data. We've already got Stackdriver set up to give us distributed tracing and have set up standards around logging.

mjpuser 7y ago

I think the real problem is that they didn't properly maintain their code. Rewriting it in Go won't prevent them from dealing with this in a few years for when this Go version reaches end of life. I would have liked to see an article on "introducing process" side of programming.

dangoor 7y ago
Thanks for the comment! A couple of things about this... 1. the Go team is working very hard to ensure that there are no such compatibility issues. Code written for Go 1.0 should still compile with Go 1.14 beta today. 2. It's possible there's more we could have done along the way, and I tend to think that statically typed languages make it easier to safely refactor more ruthlessly. But I do think we've actually done quite a bit of change incrementally along the way. Our move to React on the frontend and GraphQL on the backend have been good examples of that. Plus, we did a huge refactoring a couple of years ago to draw better boundaries in our monolith, and that has made a move to services possible.

harel 7y ago

Migration from python 2 to 3 is easy and fast. I've migrated multiple large apps and it took about a day each. Most libraries that matter have been migrated. Some don't even support python 2 anymore. It's practically 2020. This should not even be a consideration. After 2 to 3 is done they should consider again If they want to redo the stack but first I'd focus on this small maintenance task.

AnonymousPlanet 7y ago
Hahaha, sorry but this is a very cute thing to say, in my view. At our company we just barely finished migrating our software with nearly a million lines of legacy Python 2 code to Python 3. This took over a year of nearly exclusive migration effort, just making our code work with both. The entire migration project started way before I joined the company several years ago. So, no, things are not as simple if you're not dealing with toy projects. And no, you can't assume that it's the same for everyone if you're not in their shoes. Your comment is pretty much the equivalent of "I don't see a bug. Works for me."
- StavrosK 7y ago
  It very much depends on how good the codebase is. I also spent a year on and off porting a large codebase from 2 to 3, and it would have gone an order of magnitude faster if the codebase were in better shape.
  - AnonymousPlanet 7y ago
    I agree that code quality is a big factor. But what is good code quality in Python? In our case, the oldest code is the most "pythonic" and is at the same time the worst to maintain. The better code mitigates the drawbacks of dynamic typing and by that moves away from the pythonic standard you see in many libraries. But even if you nail the types to the board (e.g. assert isinstance(...)), use (the somewhat weak) Mypy wherever you can, and have good test coverage, you still have to grep your code base for usage of, e.g., .keys(), eyeball hundreds of modules for subtle Unicode madness or hunt for the odd division, replace every sort() that doesn't use key= yet, etc. The todos add up and someone has to go into the code and change those lines.

d_burfoot 7y ago

I heard about this project from a friend who works at KA. I am concerned about the strategy, and I think the following approach would yield better results: 1. Write in Go an exact reimplementation of the current Python codebase. Use the same database schema, front-end HTML/JS, test suite, and so on. To whatever extent possible, use the same names for classes and functions. Check the reimplementation correctness by using a comparison tool that calls both the Python and Go version of a page/function/search and making sure that they produce the same results. 2. Change the production code over to the Go version, perhaps using a ramping strategy where X% of servers are running the Go code, and you gradually increase X, while monitoring vital statistics like server load and response time. 3. Now that the production site is running Go, incrementally split off components into their own services. This approach leads you to the same destination, but with a lot less risk. It is very unhealthy to have a situation where the production site is running one codebase but all the developers are working on another codebase. Note that you will realize the benefits of Go (performance, type safety) after step 2, which is much sooner than OP's plan. Joel Spolsky's classic essay about how you should never do full codebase rewrites is worth reviewing: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...

jordanbeiber 7y ago
Is moving directly to #3 not an option? I’m thinking that defining parts that can be moved to separate services, and start consuming these could be a way to organically transition to a new architecture.
krenel 7y ago
100% agree. We've just finished to roll out our implementation in Go migrating a subsystem from PHP and receiving around 150req/second and demultiplexing those request to 1500-2000req/second to legacy backends. The key to the success of the project was that the API was an exact match, and we could compare both implementations for exact requests. The deploy strategy of the new version: - Reply the real traffic to the new Go service comparing the results with the old one - Then implement a toggle feature than enabled different traffic sources to use one backend or the other - Keep changing backends to the new system and ensure that metrics were unaffected Having e2e and integration tests for the Golang project was of a huge help, since we could fix all differences using TDD. Although we changed some of the implementations to take advantage of Go constructs, just a 1-to-1 replacement would have had a huge performance impact.

zmmmmm 7y ago

I think some of the misunderstanding in these comments comes from not fully appreciating the perspective of not-for-profit organisations. While I can't speak for Khan Academy, I know that in every NFP organisation I have worked for there is an acute awareness that funding could dry up one day and the prime directive is to ensure that in a scenario like that, the work of the organisation can continue. In this case, it leads to a higher concern about minimising the cost of the operational services than you might have in a for-profit organisation. In all the strategic planning I have been involved in with NFP, we always have the "what if worst case scenario arises" plan and in that plan the ability to scale down to bare minimum operational cost is key. It may not be conscious but I suspect that may be part of the reason the performance savings from moving to Go are so attractive in this case, where most profit-making companies just ask the question of whether they can afford to pay for the servers with their current margin or not and if they can they have more important things to worry about.