12 ms·

Seems like such a waste. Is switching to python 3 really that hard? Is hardware that expensive? If this is indeed the right call it doesn't bode well for tradit

Seems like such a waste. Is switching to python 3 really that hard? Is hardware that expensive? If this is indeed the right call it doesn't bode well for traditional scripting languages as the web scales to fewer high traffic apps. We might start to see more jvm, go (apparently) or even rust and c(++), rather than speed of development languages like Python or Ruby. Trend seems to be the reverse though, with python the second and most rapidly growing language.

neurobashing 7y ago
Everyone’s project/code base is different but in my experience there’s been a critical mass of libraries for a few years. I presume the “it’s hard to move to 3” is dev teams wanting a new toy as much as “the rewrite is too complex”. Library use, size of code base etc are all big factors but at the end of the day, I think team motivation is really the deciding factor.
- Consultant32452 7y ago
  This is my experience too. Someone or a couple someones on the team decide they want to try out some new tech or expand their resume. Then it becomes a quest to justify the switch rather than a quest to make the best business decision.
- freyr 7y ago
  That mirrors my experience as well. Someone with influence is bored or wants to level up, so they'll drag the entire company into a long, expensive quagmire. Unless the existing codebase is mired in technical debt and completely unsalvageable or cannot scale further, this seems like a very radical move.
  - ksec 7y ago
    Which is precisely why most of the Web Dev should not be called Engineers.
  - lonelappde 7y ago
    Replacing an old unholy mess with a new unholy mess is usually a bad plan. It's called the Second System Effect.
- Alex3917 7y ago
  I worked on a fairly large codebase that needed to be rewritten from scratch when migrating from 2 to 3, primarily because all the tests were written using a test framework that was no longer maintained. So given that you might need to start over anyway, I think it's reasonable to consider other options. That said, yeah it's difficult to understand how KA's web server costs aren't already basically zero, and how their endpoints aren't already basically instantaneous.
  - dangoor 7y ago
    > That said, yeah it's difficult to understand how KA's web server costs aren't already basically zero, and how their endpoints aren't already basically instantaneous. I find that this is the outsider's view of a great many products. Things always seem a lot simpler on the outside. In Khan Academy's case, I think a lot of folks just think of our site as being a collection of more-or-less static pages with videos on them. There's a lot more going on than that, though. We've got a CMS that supports articles with math and interactive elements, in addition to the videos... and many, many exercises with hints. All translated into dozens of languages. We have to remember every exercise people have done so that we know which ones to present to them next, and we need to display that progress when they look at topic pages. Oh yeah, and if they're in a classroom, we need to present that progress to teachers (or coaches/parents, outside of the classroom). Teachers can also assign content. Now, we're also offering features for school districts: https://www.khanacademy.org/district https://www.khanacademy.org/district Plus, there's the official SAT prep, which connects to the College Board directly to provide personalized guidance about what to work on... and that's only one of the test preparation areas of our site. And, as you can imagine, there are a bunch of other features and aspects of the features above that I'm not mentioning. It adds up.
    - Alex3917 7y ago
      Fair, but what percentage of spend is actually on web servers as opposed to database, data transfer, static asset storage, caches, CDN, etc? The features you listed are kind of what I expected, but I still wouldn't expect the web servers to be more than 15% or so of your hosting costs. I know you guys get a ton of traffic, but on most web sites at least 90% of traffic is logged out and doesn't even need to hit the web servers in the first place.
WilliamEdward 7y ago
I don't think upgrading to python 3 will be as hard as rewriting the entire thing, but rewriting does come with the benefit that you are not stuck with the problems that come with dynamic typing in a huge codebase, and i'm assuming this is why sticking with python is hard. They also said a faster language will improve their server.
- zo1 7y ago
  This is perhaps a very big misconception with python usage. Just because it has "dynamic" or "duck" typing, doesn't mean that you have to consider chaotic and unpredictable data running through your code paths. It's just not the case. In a typical codebase, it's probably 98% very specific and known data types linked to the variables in your code. With the remaining 2% being things that are just "easier" to solve with dynamic typing rather than coming up with complicated interface/inheritance hierarchies that you typically find in compiled languages. And with type-hinting now being there in python, you have a very good way of "codifying" that dynamic or duck-typing. Such that you can expect almost 100% knowledge of all the data types coming in/out of your classes/functions. At this point, I'd argue it's got one of the most robust "type" systems out there, if one can call it that at all. Just don't use text editor + MyPy, or VSCode for your python development, and you'll be in good hands. I.e. Use PyCharm.
  - makapuf 7y ago
    Re: your last sentence, what is text editor + my so bad ? How does pycharm warrant this definitive advices ?
    - zo1 7y ago
      Honestly, at this point mypy is inferior to the built-in PyCharm when it comes to speed, integration and in some cases the type-inference as well. I've also compared it to the VS python language server and it too doesn't stack up. The other metric would be what a person coming from established, full-fledged IDEs such as VS would expect. With PyCharm, you get intellisense almost on-par with what you get from VS for C#/VB, assuming you use type-hints that is. Not trying to be difficult, but I'm being honest about them providing a really good python experience that is for the most part free. There is no need to putz-around with VScode, json configs, plugins, mypy, etc and still end up getting a relatively inferior experience. Doubly so for new-developers.
tptacek 7y ago
What's really the ergonomic difference between "traditional scripting languages" and Go? I came up writing professional C code, and spent most of the last 15 years writing "traditional scripting language" code, and Go feels a lot closer to scripting than to C to me, despite compiling down to machine code. Go has static types, and that distinguishes it from Python, Ruby, and Perl. But the trend now seems to be for languages to move towards static typing anyways; 2005-era Python was wrong about that.
- zozbot234 7y ago
  > Go has static types, and that distinguishes it from Python, Ruby, and Perl. It also has interface{} and that's a non-trivial commonality with Python, Ruby and Perl.
- freyr 7y ago
  > 2005-era Python was wrong about that. Given Python's incredible rise since 2005, and even in the past few years, I'm not sure we can say they were "wrong." It's serves a purpose.
  - doteka 7y ago
    That rise has been entirely accounted for by machine learning and data science. Python as a language for actual software engineering has been slowly dying for a while now.
- rtpg 7y ago
  It’s usually the DSL argument. In Go you can establish calling patterns for errors, but you’re really limited in terms of providing libraries with nice APIs that prevent you from making mistakes. In Python you have a lot of ways to make sure someone does a thing. Exceptions are good for making sure an error is handled. Context managers make sure a resource is cleaned up properly. I might be wrong but I feel like writing something like jquery (with its fluent API) would be really tough in Go.
orev 7y ago
For any web-based app, it really makes sense to take the approach of using a rapid development language first, then as you need to scale, convert to something that’s compiled and focuses on speed. It’s not one or the other kind of thing — they both have a role (at least until we reach the holy grail where fast to develop is also fast to run).
- zozbot234 7y ago
  What's a "rapid development" language? I know about RAD, but that was a buzzword that was only ever used to push terrible languages like Visual Basic, and somewhat less terrible ones like Embarcadero Delphi.
  - cle 7y ago
    Dynamic languages (so you can hack together stuff and quickly bypass any roadblocks), with REPLs (quick feedback and avoiding writing tests), and low cognitive overhead (so new folks can ramp up quickly). Some example languages that come to mind here: Python, Ruby, JavaScript, Clojure, Groovy. Go actually comes close here, even though it doesn't have a REPL and isn't very dynamic, because of its focus on minimizing cognitive overhead and getting the job done with minimal fuss. Other languages carry a lot of community baggage. Java is one of the worst IMO...if you try to hire Java programmers, it's going to take a lot of effort and risk to find and reject applicants who've read too many design pattern books, are architecture astronauts, or come from an enterprise-y background. The signal-to-noise ratio is just really poor.
    - AtlasBarfed 7y ago
      Rubys devs said the same thing about Java when Rails first came out. Now look at Ruby, its stack is gigantic and cumbersome. I guess Go will try to resist that, but over time Go will become burdened with a huge stack of outmoded software.
- scarface74 7y ago
  Like one of the former engineers at Twitter said about their early issues with stability and when someone blamed Ruby for the performance issues - short version it was a bad architecture not the language. Stateless web servers are one of those things which are ridiculously and easily parallelizable, you can scale a web server horizontally easily. The ROI of rewriting everything in another language as opposed to just adding more web servers would probably take years unless you’re running at a ridiculously large scale. For the cost of one developer’s fully allocated salary, you can throw a lot of hardware at performance issues with web apps. I prefer statically typed languages, but performance isn’t one of them. Besides, how much processing is a typical web app doing?
stinky613 7y ago
> Is switching to python 3 really that hard? I think it's more that given their specific codebase it's similarly difficult to switch to python 3 as it is to switch to a number of other, entirely different programming languages. Once you recognize that rough equivalency, then it's worth considering the stability, compile times, necessary production resources, etc of those other programming languages. If your project's specific dependencies are so intrinsically stuck on a python 2.x implementation you might be caught between having to redesign that dependency in-house or switching to a language where you wouldn't need to do that in-house work.
- freyr 7y ago
  > it's similarly difficult to switch to python 3 But that's almost certainly not the case. Even if their existing codebase relies very heavily on the small subset of Python2 features that require manual porting, switching to Python3 will be much less work than rewriting everything in a new language.
  - dangoor 7y ago
    The Python 2 to 3 conversion was only part of it. The libraries were a huge difference for us. I will agree that porting to Python 3 would be less work than porting to Go, but we believe that the difference is less than people would suspect.
    - bb88 7y ago
      Never announce a plan to do something. Announce the results after doing it. If you announce a plan, you're making a bet (sometimes with your reputation) that's it's going to be successful, and telling the world about it. If however, you didn't announce it, you could still get the benefit of success, or make it looks like learning the lessons of a failed experiment.
      - dangoor 7y ago
        We have the good fortune of not having to be super secretive about our tech. We _want_ to talk about this project as we go along and share what we learn. We've already got more interesting stuff to talk about, and it'd be a lot less interesting, I think, without this context for the overall project. As mentioned in the post, a small piece of our GraphQL schema is already in Go running in production. This blog post isn't just "we're thinking of doing this thing". It's "we've already done a bunch of research, thinking, _and_ built some of it."
ljm 7y ago
I've worked with or known about too many places that have jumped on the microservices bandwagon and the only thing I've encountered is a lack of maturity (mine included) and a knee-jerk reaction against monolithic code, when the problem isn't the size of the codebase but its organic growth over time. Go is an excellent language for it but distributing your business logic and functionality over a network fundamentally changes the behaviour of your app and how you have to think about it; you can't just tear out bits of the monolith and make it an API. I've come to some fairly comfortable conclusions: 1. If your team is small; don't do it. The mental overhead of that architecture will bring your team's ability to deliver down to it's knees. 2. If you've got a long-lived app with lots of legacy code; don't do it. You will have to maintain and add new features to the old codebase because it's easier, which means you have more things to rewrite. 3. If you're small-ish/mid scale. Don't do it. Kubernetes and similar are tools for people handling Facebook/Google/Netflix kind of loads. 4. If your organisational structure doesn't fit (i.e. you don't have enough people to split into smaller teams); don't do it. Scaling is considered a good problem to have, right? That means that your current, ugly monolith is actually successful, and somehow the first thought is to replace it all with a complete—but trendy—unknown? If the engineering team is so unhappy about the architecture then Go is the wrong choice. Maybe they could consider service oriented architecture and some refactoring/tech debt time so they feel happier about that codebase. Pull some of the code out into modules and then start figuring out where bits of the codebase would actually belong, while still being one deployable. And then after that, if you really want to, distribute it over the network, and then start thinking about porting it. Otherwise, throw away your first successful prototype at the first instance and go all in on distributed architecture and microservices. At least then you have the luxury of figuring it out from scratch.
- goostavos 7y ago
  >knee-jerk reaction against monolithic code It's almost religious. The reaction some people have when you suggest monoliths is completely baffling. It's apparently just "known" that it is the correct approach to all problems, so, y'know, it's embarrassing for you to have even suggested otherwise.
  - 7y ago
ggregoire 7y ago
Python is growing only because of tensorflow, pytorch, numpy, pandas and friends. Web dev is moving away from dynamic typing and that’s a good thing.
zik 7y ago
In terms of general compute speed Go is many times faster than Python - approximately on a par with Java speed-wise but with much less memory use. That means your server costs are many times cheaper and your page latencies are much faster than Python. It's significant.
- PretzelFisch 7y ago
  In the past companies would just compile python down to c to get the memory and perf they need. Probably would be the right answer here too, but that would not look as cool on the resume.
  - WoodTree 7y ago
    By compiling you mean rewriting it in C or Cython. Which is only a good idea for certain applications.
    - PretzelFisch 7y ago
      yeah, I was thinking of Cython getting used. Anly particular reason it would not work for this kind of use case?
  - _ph_ 7y ago
    Moving compute intensive tasks to C would speed up many programs. That is the reason that many dynamic languages have overall good performance - they rely on C libraries for the computative heavy lifting. This has one big backdraw however. Not only do you need to write in two different languages, the rewriting in C requires a lot of care, as the language protects you much less than the high level language you implement for. One big attraction of Go is, that it is high level and productive enough, to be the main implementation language, and for time critical stuff very efficient. So you don't have to cross language bareers to implement speed critical code and you get the full type and memory safety in the whole stack. Interestingly, one can write Python extensions in Go, so in most cases that would be my choice these days for speeding up critical code paths in Python.
- 7y ago
marcosdumay 7y ago
> Is hardware that expensive? Hardware is cheap. Hardware is on the "accessible to a 3rd world middle class person" level of cheap. But with enough scale, it adds up, while the costs of a rewrite don't. And the difference between a language like Go and one like Python is on the hundreds of times.
- dangoor 7y ago
  FWIW, our tests showed Go as realistically being about 10x faster than Python for our tasks.
ram_rar 7y ago
Switching to python3 is hard, especially if you have a massive python codebase interacting with other systems. Its not as easy as importing unicode_literals. Unicode breaks in very subtle ways.
- guitarbill 7y ago
  and rewriting it in another language will magically fix this and not introduce new bugs? besides, in my experience Python 3's clear separation between bytes and str makes these breakages much less subtle than it silently going wrong in Python 2. i wish them all the best, but would've been much more impressed if they'd done it, not simply announced to do it.
  - cwyers 7y ago
    The question isn't, is porting Py2 to Py3 easier than porting Py2 to Go. The question is, if you spend the same effort on porting Py2 to Go that you would have spent porting to Py3, and that only gets you 60% of the way there, but the ROI on that work is much higher, are you better off?
  - speedplane 7y ago
    > in my experience Python 3's clear separation between bytes and str makes these breakages much less subtle than it silently going wrong in Python 2. No question Python 3 is better than 2, but it's not better enough to justify the move. People will only move when they absolutely have to. That isn't progress, it's inefficiency.
dangoor 7y ago
I _like_ Python. Even released a Python web framework. I think there are many projects it's a good fit for, which is why it's continuing to be quite successful, despite the pain of Python 3. But that doesn't mean it's a great fit for all projects. Personally, I've come to find that code in statically typed languages is easier to maintain over time, especially from a big team. I guess a lot of Python folks agree, which is why Python 3 allows static typing as well. At a certain point, server costs _do_ add up to real money and some applications are not purely database-bound. Go's tooling makes it almost as fast to work with as a scripting language, but with much better performance. The language itself is certainly not as succinct as Python, but I think it has made reasonable tradeoffs. Also: there's already a lot of JVM on the web. Finally, I'll just note that _not all_ Python 3 migrations are that hard. It depends on a lot on the libraries used.
novok 7y ago
In my experience it's more annoying than you think, especially if you want to prevent small sneaking regressions. Python not being statically typed also means all of the breaking type changes they made with string now being byte or similar means crashes are not revealed until possibly production if your unit tests didn't get that oneeeee edge case right. If I'm going to do a new project in the future, I will demand it will be a statically typed language. I'm sick of dynamic languages.

neurobashing 7y ago

Everyone’s project/code base is different but in my experience there’s been a critical mass of libraries for a few years. I presume the “it’s hard to move to 3” is dev teams wanting a new toy as much as “the rewrite is too complex”. Library use, size of code base etc are all big factors but at the end of the day, I think team motivation is really the deciding factor.

Consultant32452 7y ago
This is my experience too. Someone or a couple someones on the team decide they want to try out some new tech or expand their resume. Then it becomes a quest to justify the switch rather than a quest to make the best business decision.
freyr 7y ago
That mirrors my experience as well. Someone with influence is bored or wants to level up, so they'll drag the entire company into a long, expensive quagmire. Unless the existing codebase is mired in technical debt and completely unsalvageable or cannot scale further, this seems like a very radical move.
- ksec 7y ago
  Which is precisely why most of the Web Dev should not be called Engineers.
- lonelappde 7y ago
  Replacing an old unholy mess with a new unholy mess is usually a bad plan. It's called the Second System Effect.
Alex3917 7y ago
I worked on a fairly large codebase that needed to be rewritten from scratch when migrating from 2 to 3, primarily because all the tests were written using a test framework that was no longer maintained. So given that you might need to start over anyway, I think it's reasonable to consider other options. That said, yeah it's difficult to understand how KA's web server costs aren't already basically zero, and how their endpoints aren't already basically instantaneous.
- dangoor 7y ago
  > That said, yeah it's difficult to understand how KA's web server costs aren't already basically zero, and how their endpoints aren't already basically instantaneous. I find that this is the outsider's view of a great many products. Things always seem a lot simpler on the outside. In Khan Academy's case, I think a lot of folks just think of our site as being a collection of more-or-less static pages with videos on them. There's a lot more going on than that, though. We've got a CMS that supports articles with math and interactive elements, in addition to the videos... and many, many exercises with hints. All translated into dozens of languages. We have to remember every exercise people have done so that we know which ones to present to them next, and we need to display that progress when they look at topic pages. Oh yeah, and if they're in a classroom, we need to present that progress to teachers (or coaches/parents, outside of the classroom). Teachers can also assign content. Now, we're also offering features for school districts: https://www.khanacademy.org/district https://www.khanacademy.org/district Plus, there's the official SAT prep, which connects to the College Board directly to provide personalized guidance about what to work on... and that's only one of the test preparation areas of our site. And, as you can imagine, there are a bunch of other features and aspects of the features above that I'm not mentioning. It adds up.
  - Alex3917 7y ago
    Fair, but what percentage of spend is actually on web servers as opposed to database, data transfer, static asset storage, caches, CDN, etc? The features you listed are kind of what I expected, but I still wouldn't expect the web servers to be more than 15% or so of your hosting costs. I know you guys get a ton of traffic, but on most web sites at least 90% of traffic is logged out and doesn't even need to hit the web servers in the first place.

WilliamEdward 7y ago

I don't think upgrading to python 3 will be as hard as rewriting the entire thing, but rewriting does come with the benefit that you are not stuck with the problems that come with dynamic typing in a huge codebase, and i'm assuming this is why sticking with python is hard. They also said a faster language will improve their server.

zo1 7y ago
This is perhaps a very big misconception with python usage. Just because it has "dynamic" or "duck" typing, doesn't mean that you have to consider chaotic and unpredictable data running through your code paths. It's just not the case. In a typical codebase, it's probably 98% very specific and known data types linked to the variables in your code. With the remaining 2% being things that are just "easier" to solve with dynamic typing rather than coming up with complicated interface/inheritance hierarchies that you typically find in compiled languages. And with type-hinting now being there in python, you have a very good way of "codifying" that dynamic or duck-typing. Such that you can expect almost 100% knowledge of all the data types coming in/out of your classes/functions. At this point, I'd argue it's got one of the most robust "type" systems out there, if one can call it that at all. Just don't use text editor + MyPy, or VSCode for your python development, and you'll be in good hands. I.e. Use PyCharm.
- makapuf 7y ago
  Re: your last sentence, what is text editor + my so bad ? How does pycharm warrant this definitive advices ?
  - zo1 7y ago
    Honestly, at this point mypy is inferior to the built-in PyCharm when it comes to speed, integration and in some cases the type-inference as well. I've also compared it to the VS python language server and it too doesn't stack up. The other metric would be what a person coming from established, full-fledged IDEs such as VS would expect. With PyCharm, you get intellisense almost on-par with what you get from VS for C#/VB, assuming you use type-hints that is. Not trying to be difficult, but I'm being honest about them providing a really good python experience that is for the most part free. There is no need to putz-around with VScode, json configs, plugins, mypy, etc and still end up getting a relatively inferior experience. Doubly so for new-developers.

tptacek 7y ago

What's really the ergonomic difference between "traditional scripting languages" and Go? I came up writing professional C code, and spent most of the last 15 years writing "traditional scripting language" code, and Go feels a lot closer to scripting than to C to me, despite compiling down to machine code. Go has static types, and that distinguishes it from Python, Ruby, and Perl. But the trend now seems to be for languages to move towards static typing anyways; 2005-era Python was wrong about that.

zozbot234 7y ago
> Go has static types, and that distinguishes it from Python, Ruby, and Perl. It also has interface{} and that's a non-trivial commonality with Python, Ruby and Perl.
freyr 7y ago
> 2005-era Python was wrong about that. Given Python's incredible rise since 2005, and even in the past few years, I'm not sure we can say they were "wrong." It's serves a purpose.
- doteka 7y ago
  That rise has been entirely accounted for by machine learning and data science. Python as a language for actual software engineering has been slowly dying for a while now.
rtpg 7y ago
It’s usually the DSL argument. In Go you can establish calling patterns for errors, but you’re really limited in terms of providing libraries with nice APIs that prevent you from making mistakes. In Python you have a lot of ways to make sure someone does a thing. Exceptions are good for making sure an error is handled. Context managers make sure a resource is cleaned up properly. I might be wrong but I feel like writing something like jquery (with its fluent API) would be really tough in Go.

orev 7y ago

For any web-based app, it really makes sense to take the approach of using a rapid development language first, then as you need to scale, convert to something that’s compiled and focuses on speed. It’s not one or the other kind of thing — they both have a role (at least until we reach the holy grail where fast to develop is also fast to run).

zozbot234 7y ago
What's a "rapid development" language? I know about RAD, but that was a buzzword that was only ever used to push terrible languages like Visual Basic, and somewhat less terrible ones like Embarcadero Delphi.
- cle 7y ago
  Dynamic languages (so you can hack together stuff and quickly bypass any roadblocks), with REPLs (quick feedback and avoiding writing tests), and low cognitive overhead (so new folks can ramp up quickly). Some example languages that come to mind here: Python, Ruby, JavaScript, Clojure, Groovy. Go actually comes close here, even though it doesn't have a REPL and isn't very dynamic, because of its focus on minimizing cognitive overhead and getting the job done with minimal fuss. Other languages carry a lot of community baggage. Java is one of the worst IMO...if you try to hire Java programmers, it's going to take a lot of effort and risk to find and reject applicants who've read too many design pattern books, are architecture astronauts, or come from an enterprise-y background. The signal-to-noise ratio is just really poor.
  - AtlasBarfed 7y ago
    Rubys devs said the same thing about Java when Rails first came out. Now look at Ruby, its stack is gigantic and cumbersome. I guess Go will try to resist that, but over time Go will become burdened with a huge stack of outmoded software.
scarface74 7y ago
Like one of the former engineers at Twitter said about their early issues with stability and when someone blamed Ruby for the performance issues - short version it was a bad architecture not the language. Stateless web servers are one of those things which are ridiculously and easily parallelizable, you can scale a web server horizontally easily. The ROI of rewriting everything in another language as opposed to just adding more web servers would probably take years unless you’re running at a ridiculously large scale. For the cost of one developer’s fully allocated salary, you can throw a lot of hardware at performance issues with web apps. I prefer statically typed languages, but performance isn’t one of them. Besides, how much processing is a typical web app doing?

stinky613 7y ago

> Is switching to python 3 really that hard? I think it's more that given their specific codebase it's similarly difficult to switch to python 3 as it is to switch to a number of other, entirely different programming languages. Once you recognize that rough equivalency, then it's worth considering the stability, compile times, necessary production resources, etc of those other programming languages. If your project's specific dependencies are so intrinsically stuck on a python 2.x implementation you might be caught between having to redesign that dependency in-house or switching to a language where you wouldn't need to do that in-house work.

freyr 7y ago
> it's similarly difficult to switch to python 3 But that's almost certainly not the case. Even if their existing codebase relies very heavily on the small subset of Python2 features that require manual porting, switching to Python3 will be much less work than rewriting everything in a new language.
- dangoor 7y ago
  The Python 2 to 3 conversion was only part of it. The libraries were a huge difference for us. I will agree that porting to Python 3 would be less work than porting to Go, but we believe that the difference is less than people would suspect.
  - bb88 7y ago
    Never announce a plan to do something. Announce the results after doing it. If you announce a plan, you're making a bet (sometimes with your reputation) that's it's going to be successful, and telling the world about it. If however, you didn't announce it, you could still get the benefit of success, or make it looks like learning the lessons of a failed experiment.
    - dangoor 7y ago
      We have the good fortune of not having to be super secretive about our tech. We _want_ to talk about this project as we go along and share what we learn. We've already got more interesting stuff to talk about, and it'd be a lot less interesting, I think, without this context for the overall project. As mentioned in the post, a small piece of our GraphQL schema is already in Go running in production. This blog post isn't just "we're thinking of doing this thing". It's "we've already done a bunch of research, thinking, _and_ built some of it."

ljm 7y ago

I've worked with or known about too many places that have jumped on the microservices bandwagon and the only thing I've encountered is a lack of maturity (mine included) and a knee-jerk reaction against monolithic code, when the problem isn't the size of the codebase but its organic growth over time. Go is an excellent language for it but distributing your business logic and functionality over a network fundamentally changes the behaviour of your app and how you have to think about it; you can't just tear out bits of the monolith and make it an API. I've come to some fairly comfortable conclusions: 1. If your team is small; don't do it. The mental overhead of that architecture will bring your team's ability to deliver down to it's knees. 2. If you've got a long-lived app with lots of legacy code; don't do it. You will have to maintain and add new features to the old codebase because it's easier, which means you have more things to rewrite. 3. If you're small-ish/mid scale. Don't do it. Kubernetes and similar are tools for people handling Facebook/Google/Netflix kind of loads. 4. If your organisational structure doesn't fit (i.e. you don't have enough people to split into smaller teams); don't do it. Scaling is considered a good problem to have, right? That means that your current, ugly monolith is actually successful, and somehow the first thought is to replace it all with a complete—but trendy—unknown? If the engineering team is so unhappy about the architecture then Go is the wrong choice. Maybe they could consider service oriented architecture and some refactoring/tech debt time so they feel happier about that codebase. Pull some of the code out into modules and then start figuring out where bits of the codebase would actually belong, while still being one deployable. And then after that, if you really want to, distribute it over the network, and then start thinking about porting it. Otherwise, throw away your first successful prototype at the first instance and go all in on distributed architecture and microservices. At least then you have the luxury of figuring it out from scratch.

goostavos 7y ago
>knee-jerk reaction against monolithic code It's almost religious. The reaction some people have when you suggest monoliths is completely baffling. It's apparently just "known" that it is the correct approach to all problems, so, y'know, it's embarrassing for you to have even suggested otherwise.
- 7y ago

ggregoire 7y ago

Python is growing only because of tensorflow, pytorch, numpy, pandas and friends. Web dev is moving away from dynamic typing and that’s a good thing.

zik 7y ago

In terms of general compute speed Go is many times faster than Python - approximately on a par with Java speed-wise but with much less memory use. That means your server costs are many times cheaper and your page latencies are much faster than Python. It's significant.

PretzelFisch 7y ago
In the past companies would just compile python down to c to get the memory and perf they need. Probably would be the right answer here too, but that would not look as cool on the resume.
- WoodTree 7y ago
  By compiling you mean rewriting it in C or Cython. Which is only a good idea for certain applications.
  - PretzelFisch 7y ago
    yeah, I was thinking of Cython getting used. Anly particular reason it would not work for this kind of use case?
- _ph_ 7y ago
  Moving compute intensive tasks to C would speed up many programs. That is the reason that many dynamic languages have overall good performance - they rely on C libraries for the computative heavy lifting. This has one big backdraw however. Not only do you need to write in two different languages, the rewriting in C requires a lot of care, as the language protects you much less than the high level language you implement for. One big attraction of Go is, that it is high level and productive enough, to be the main implementation language, and for time critical stuff very efficient. So you don't have to cross language bareers to implement speed critical code and you get the full type and memory safety in the whole stack. Interestingly, one can write Python extensions in Go, so in most cases that would be my choice these days for speeding up critical code paths in Python.
7y ago

marcosdumay 7y ago

> Is hardware that expensive? Hardware is cheap. Hardware is on the "accessible to a 3rd world middle class person" level of cheap. But with enough scale, it adds up, while the costs of a rewrite don't. And the difference between a language like Go and one like Python is on the hundreds of times.

dangoor 7y ago
FWIW, our tests showed Go as realistically being about 10x faster than Python for our tasks.

ram_rar 7y ago

Switching to python3 is hard, especially if you have a massive python codebase interacting with other systems. Its not as easy as importing unicode_literals. Unicode breaks in very subtle ways.

guitarbill 7y ago
and rewriting it in another language will magically fix this and not introduce new bugs? besides, in my experience Python 3's clear separation between bytes and str makes these breakages much less subtle than it silently going wrong in Python 2. i wish them all the best, but would've been much more impressed if they'd done it, not simply announced to do it.
- cwyers 7y ago
  The question isn't, is porting Py2 to Py3 easier than porting Py2 to Go. The question is, if you spend the same effort on porting Py2 to Go that you would have spent porting to Py3, and that only gets you 60% of the way there, but the ROI on that work is much higher, are you better off?
- speedplane 7y ago
  > in my experience Python 3's clear separation between bytes and str makes these breakages much less subtle than it silently going wrong in Python 2. No question Python 3 is better than 2, but it's not better enough to justify the move. People will only move when they absolutely have to. That isn't progress, it's inefficiency.

dangoor 7y ago

I _like_ Python. Even released a Python web framework. I think there are many projects it's a good fit for, which is why it's continuing to be quite successful, despite the pain of Python 3. But that doesn't mean it's a great fit for all projects. Personally, I've come to find that code in statically typed languages is easier to maintain over time, especially from a big team. I guess a lot of Python folks agree, which is why Python 3 allows static typing as well. At a certain point, server costs _do_ add up to real money and some applications are not purely database-bound. Go's tooling makes it almost as fast to work with as a scripting language, but with much better performance. The language itself is certainly not as succinct as Python, but I think it has made reasonable tradeoffs. Also: there's already a lot of JVM on the web. Finally, I'll just note that _not all_ Python 3 migrations are that hard. It depends on a lot on the libraries used.

novok 7y ago

In my experience it's more annoying than you think, especially if you want to prevent small sneaking regressions. Python not being statically typed also means all of the breaking type changes they made with string now being byte or similar means crashes are not revealed until possibly production if your unit tests didn't get that oneeeee edge case right. If I'm going to do a new project in the future, I will demand it will be a statically typed language. I'm sick of dynamic languages.