4 ms·

I hoped to have debunked, though not thoroughly, the myth that dynamic languages cannot scale to power large projects in my recent talk at Pycon CA. Just take a

I hoped to have debunked, though not thoroughly, the myth that dynamic languages cannot scale to power large projects in my recent talk at Pycon CA. Just take a look at Openstack. It's powering clouds in CERN and Yahoo! Japan where it provides the infrastructure for delivering earthquake notifications to the public -- a valuable and critical service! And it's written largely in Python. It does rely on systems written in static languages and is itself written in one... but dynamic languages are very useful for a class of problem where deferring decisions about specifics until run-time is a huge advantage. Normally this is reserved for exploratory programming but in practice it is also useful for dynamically configured systems that can introspect themselves at run-time and change that configuration or adapt to unstructured input. Update: I'm also one of the unwashed masses that, gasp, didn't go to university and study PLT. I still have a dog-eared copy of SICP, The Dragon Book, Concrete Mathematics, et al... and started by making terrible pong clones and text adventures in BASIC on an Amiga 500.

coldtea 11y ago
>I hoped to have debunked, though not thoroughly, the myth that dynamic languages cannot scale to power large projects in my recent talk at Pycon CA. Just take a look at Openstack. It's powering clouds in CERN and Yahoo! Japan where it provides the infrastructure for delivering earthquake notifications to the public -- a valuable and critical service! And it's written largely in Python. That doesn't say much in itself though. Such infrastructure for "earthquake notification" could just be a few thousand lines or even mostly "glue" code. As you say "It does rely on systems written in static languages and is itself written in one" which kind of refutes the whole argument, doesn't it? >It does rely on systems written in static languages and is itself written in one... but dynamic languages are very useful for a class of problem where deferring decisions about specifics until run-time is a huge advantage. For those cases couldn't one just use a DSL or at worse an sandboxed isolated dynamic language from inside a STL talking care for the rest of the system? There are several "business rules" systems for STL.
- mugsie 11y ago
  >That doesn't say much in itself though. Such infrastructure for "earthquake notification" could just be a few thousand lines or even mostly "glue" code. https://github.com/openstack/neutron https://github.com/openstack/neutron I assure you that OpenStack is a little more than "glue" code. Not sure what part of OpenStack is written in a STL, but the vast majority of its hundreds of thousands of lines of code is python.
- agentultra 11y ago
  > That doesn't say much in itself though. Such infrastructure for "earthquake notification" could just be a few thousand lines or even mostly "glue" code. In total Openstack encompasses around +4M SLOC written in Python. How much is deployed in a given cloud depends on the configuration. However the community must still shepherd and manage all of that code and make regular, stable releases on a six-month cadence. It's not "just" glue code. > As you say "It does rely on systems written in static languages and is itself written in one" which kind of refutes the whole argument, doesn't it? Not necessarily. Awk was necessarily written because querying unstructured input is a huge pain in plain C. Kernighan gave a recent example of this in one of his recent talks on programming language design. The caveat is there is a limit at which adding more lines to an awk program becomes counter-productive. It's essentially a good DSL but a terrible language. I think Python defies that -- with the millions of lines of code managed by the community of thousands of developers a large system written in a dynamic language isn't impossible or necessarily hindered by its lack of static type analysis. > For those cases couldn't one just use a DSL or at worse an sandboxed isolated dynamic language from inside a STL talking care for the rest of the system? In some sense this is how I view Python and languages of its kind... but I think in practice it's more of a philosophical viewpoint. In practice dynamic languages are general purpose and much suited to certain kinds of programming that static languages are not. Common Lisp and Smalltalk are great examples of dynamic languages kicking butt. Starting in the debugger and developing self-introspecting systems has a peculiar effect on the programmer. One tends to develop a domain language for solving a class of problems in order to make them tractable. One thing I dislike however is how divorced it can be from the machine. Some Common Lisp compilers can produce really awesome code but it does take a little more burden from the programmer to gain such efficiency from the compiler (compared to the tactical omissions of defined behavior from languages like C which allow almost trivial optimizations... with the trade off that certain errors are quite likely on my part if I'm not careful). I think there is room for both in the world. I really enjoy OCaml, C, Python, and Common Lisp... and I would seriously consider the problem before the language. Our job after all is to transform data and not spend time writing pretty, elegant, or clever code for the code's sake. update: grammar.
rwmj 11y ago
OpenStack is the perfect example of how Python doesn't work well for larger projects. It's only because of an absolutely massive investment in testing that OpenStack works at all. I think it's more accurate to say that OpenStack works despite an unsuitable choice for implementation language.
- agentultra 11y ago
  I assure you that even if you wrote it in Haskell you'd have a few unit tests and integration tests in there.
  - ldrndll 11y ago
    I don't think he claimed otherwise. There's a difference between a code base that makes judicious use of tests to validate business logic and one that relies on the test suite just to keep it from falling apart. I've worked on the latter, so I know ;) I'm not making any representations as to whether or not this is true of OpenStack. I honestly wouldn't know.
    - agentultra 11y ago
      I can assure you, as an active technical contributor, that the testing infrastructure in Openstack is verifying business logic, ensuring consistent coding standards, running unit tests and code coverage, and all of the usual activities associated with delivering reliable software. The difference for many projects is the scale at which it is automated thanks to the Openstack Infra and QA teams. Any suggestion that Python code cannot possibly execute without it is an exaggeration.
- mugsie 11y ago
  the level of testing OpenStack does is the only way you can really test something like OpenStack no matter what the language.

coldtea 11y ago

>I hoped to have debunked, though not thoroughly, the myth that dynamic languages cannot scale to power large projects in my recent talk at Pycon CA. Just take a look at Openstack. It's powering clouds in CERN and Yahoo! Japan where it provides the infrastructure for delivering earthquake notifications to the public -- a valuable and critical service! And it's written largely in Python. That doesn't say much in itself though. Such infrastructure for "earthquake notification" could just be a few thousand lines or even mostly "glue" code. As you say "It does rely on systems written in static languages and is itself written in one" which kind of refutes the whole argument, doesn't it? >It does rely on systems written in static languages and is itself written in one... but dynamic languages are very useful for a class of problem where deferring decisions about specifics until run-time is a huge advantage. For those cases couldn't one just use a DSL or at worse an sandboxed isolated dynamic language from inside a STL talking care for the rest of the system? There are several "business rules" systems for STL.

mugsie 11y ago
>That doesn't say much in itself though. Such infrastructure for "earthquake notification" could just be a few thousand lines or even mostly "glue" code. https://github.com/openstack/neutron https://github.com/openstack/neutron I assure you that OpenStack is a little more than "glue" code. Not sure what part of OpenStack is written in a STL, but the vast majority of its hundreds of thousands of lines of code is python.
agentultra 11y ago
> That doesn't say much in itself though. Such infrastructure for "earthquake notification" could just be a few thousand lines or even mostly "glue" code. In total Openstack encompasses around +4M SLOC written in Python. How much is deployed in a given cloud depends on the configuration. However the community must still shepherd and manage all of that code and make regular, stable releases on a six-month cadence. It's not "just" glue code. > As you say "It does rely on systems written in static languages and is itself written in one" which kind of refutes the whole argument, doesn't it? Not necessarily. Awk was necessarily written because querying unstructured input is a huge pain in plain C. Kernighan gave a recent example of this in one of his recent talks on programming language design. The caveat is there is a limit at which adding more lines to an awk program becomes counter-productive. It's essentially a good DSL but a terrible language. I think Python defies that -- with the millions of lines of code managed by the community of thousands of developers a large system written in a dynamic language isn't impossible or necessarily hindered by its lack of static type analysis. > For those cases couldn't one just use a DSL or at worse an sandboxed isolated dynamic language from inside a STL talking care for the rest of the system? In some sense this is how I view Python and languages of its kind... but I think in practice it's more of a philosophical viewpoint. In practice dynamic languages are general purpose and much suited to certain kinds of programming that static languages are not. Common Lisp and Smalltalk are great examples of dynamic languages kicking butt. Starting in the debugger and developing self-introspecting systems has a peculiar effect on the programmer. One tends to develop a domain language for solving a class of problems in order to make them tractable. One thing I dislike however is how divorced it can be from the machine. Some Common Lisp compilers can produce really awesome code but it does take a little more burden from the programmer to gain such efficiency from the compiler (compared to the tactical omissions of defined behavior from languages like C which allow almost trivial optimizations... with the trade off that certain errors are quite likely on my part if I'm not careful). I think there is room for both in the world. I really enjoy OCaml, C, Python, and Common Lisp... and I would seriously consider the problem before the language. Our job after all is to transform data and not spend time writing pretty, elegant, or clever code for the code's sake. update: grammar.

rwmj 11y ago

OpenStack is the perfect example of how Python doesn't work well for larger projects. It's only because of an absolutely massive investment in testing that OpenStack works at all. I think it's more accurate to say that OpenStack works despite an unsuitable choice for implementation language.

agentultra 11y ago
I assure you that even if you wrote it in Haskell you'd have a few unit tests and integration tests in there.
- ldrndll 11y ago
  I don't think he claimed otherwise. There's a difference between a code base that makes judicious use of tests to validate business logic and one that relies on the test suite just to keep it from falling apart. I've worked on the latter, so I know ;) I'm not making any representations as to whether or not this is true of OpenStack. I honestly wouldn't know.
  - agentultra 11y ago
    I can assure you, as an active technical contributor, that the testing infrastructure in Openstack is verifying business logic, ensuring consistent coding standards, running unit tests and code coverage, and all of the usual activities associated with delivering reliable software. The difference for many projects is the scale at which it is automated thanks to the Openstack Infra and QA teams. Any suggestion that Python code cannot possibly execute without it is an exaggeration.
mugsie 11y ago
the level of testing OpenStack does is the only way you can really test something like OpenStack no matter what the language.