6 ms·

Hi, I'm the Founder at Distelli and I wrote the original version of the Distelli agent in Python that Brian rebuilt using Lua. The main reason I picked Python 2

by kt9 11y ago

Hi, I'm the Founder at Distelli and I wrote the original version of the Distelli agent in Python that Brian rebuilt using Lua. The main reason I picked Python 2.4 is because we have large enterprise customers that are / were running Python 2.4 on servers and I had to support Python 2.4 to serve their use case.

bb88 11y ago
In that case, Lua makes a lot of sense, frankly. I can write 100 lines of code in perl, but trying to get that deployed on mac, windows and linux can be a nightmare, especially if its using a library external to perl. (EDIT: LUA->Lua)
- glibgil 11y ago
  LUA? Still?
  - deleted 11y ago
    [deleted]
  - deleted 11y ago
    [deleted]
    - talloaktrees 11y ago
      I think he referring to LUA vs Lua
  - etiene 11y ago
    Lua used to be LUA, you'll find LUA written in old source code comments :) Don't be so mad about the Lua Uppercase Accident :)
batbomb 11y ago
They were probably running RHEL5, but even in 2014 they would have been in the Production 3 phase which is really should only be used for legacy purposes, and that's definitely not the sort of system you push new software to. Red Hat even pushes those customers towards virtualization because a) hardware rarely lasts 7 years and b) they won't bother backporting drivers for new hardware to the old kernel at that point. I still use a few RHEL5 machines but nobody would ever dream of pushing new software to them.
- teh_klev 11y ago
  In the real world...if you have a large fleet of RHEL5 boxes/VM's, this a fact of life. We're slowly phasing out our CentOS 5 boxes, but you still need to be able to run new software on them.
- nine_k 11y ago
  How about running a (newer) local version of Python, without of course touching the system Python? To my mind, it's usually a good idea not to use system Python, both packages and the interpreter, if you're not also controlling them. This makes building your deployables more involved, but makes the actual deployment more painless.
- hueving 11y ago
  >not the sort of system you push new software to. Telling customers that they are doing things wrong isn't really a great way to win customers.
  - takeda 11y ago
    This is something I don't understand. If someone is using Java, they'll install the version of Java they need. If they use Ruby, they'll install the version of Ruby they need. If they need Lua, they'll do the same thing. Why everyone is relying on python that's installed there for RedHat's needs (e.g. yum). Unlike other languages, Python is probably the best one that can have multiple versions coexist together.
    - iofj 11y ago
      As a consultant I have very different experiences. Even in Java development you find shops that refuse to upgrade their JVM for various reasons. RHEL 5 is really common.
      - ghshephard 11y ago
        A lot (all that I'm familiar with) of the enterprise software packages that I've worked with ship with their own, very specific (read: regression tested against) version of java. Enterprise software usually requires a ton of servers, so it's not like anything else on those servers will be using the jvm.
Pharaoh2 11y ago
"I had never seriously used Python before joining Distelli, so there was a bit of a learning curve" That was your problem, you just made a huge mistake and depending on how big your python code base was, you have now alienated most of your developers by moving to a language they are not already comfortable with. Do your customers already have lua installed on their servers or are you bundling it now? You could have just a easily bundled a newer python interpreter. Lua is a nice scripting language, if you want to embed it into another app or to create a plugin system. It's not a nice general purpose scripting language, mostly because of the vast pypi repo.
- bigtunacan 11y ago
  How is this alienating anyone? This is a command line tool. The end users are suddenly using a different language as a result. Their install process was streamlined and system dependencies were removed by including it in the install package. The same thing could have been achieved within Python, but from an end user perspective I'm guessing this is an improvement for most of their customers.
  - Pharaoh2 11y ago
    It's alienating Distelli developers. The reason cited for the move is totally unacceptable in terms of the new skillset that their developers will need to acquire to continue development of the tool. The cost of effort their developers will have to put in to be as effective as a LUA developer as they were as a python developer is really not worth this move. There are more to moving to a different platform/language than just the end users.
    - Sanddancer 11y ago
      The end users are the ones that keep the lights on though. It sounds a lot like that Python was more and more of an anchor around the team, and the costs of migration were smaller than the costs of the other options. Yes, some developers were probably upset that they couldn't use their preferred language anymore. However, it sounds like because they're now using a language designed to be embedded, they can focus a lot more on solving problems than fighting with a language that has a lot more platform-specific behavior.
orf 11y ago
That's fair enough, but it sounds like you've constrained yourselves and your code for the sake of a few enterprise customers, and now you've had to clear your technical debt with a complete rewrite. Even if batbomb were right I would argue that any solution is better than supporting 2.4. The very minimum you would want is 2.5 at a stretch.
techie128 11y ago
Couldn't you have run two versions of Python at the same time on their systems? Would it have been possible to ship a Python version with your software?
- collyw 11y ago
  easily
brownbat 11y ago
> I had to support Python 2.4... large enterprise customers that are / were running Python 2.4 on servers... How did the switch to Lua not break this in the same way? Sorry for the naive question, I'm just not following how it can be possible to change languages, but impossible to change versions. At first glance, the latter seems like a special case of the former.
- iofj 11y ago
  Lua and Luajit are small embeddable languages. They link into a single executable, including if you want the program they'll run.
- kt9 11y ago
  Shipping it as a single download with zero dependencies meant that we would not have to worry in the future about what else was installed on the machine and the versions of system packaged the customers were running.
  - oblio 11y ago
    Couldn't you just bundle Python instead?
christianbryant 11y ago
I'm a Python devotee, but I appreciate the anecdote. I'm curious whether the switch 1) cost any features/algorithms and 2) whether the anticipated performance improvement really panned out since the switch, or if the gain wound up being simply architecture/deployment-related. Stepping back, did the whole re-write really balance out with all the work that had to be done, or could you have stuck with Python and simply solved the problems at hand?
- brimworks 11y ago
  1) All features/algorithms were ported, and we only added new functionality that never existed in the old Python code base. A few of the features that got added include: dispatching builds, getting live logs, re-encrypt keys, deploy encrypted artifacts, fix many bugs, support running multiple agents on a single computer, etc. I should also mention that we previously had two separate downloads: a CLI used for uploading releases, and the agent used for dispatching deployments. The functionality of both of these code bases where merged into our new Lua based "client" so customers only have to install a single binary that is less than 10MB (the binary size depends on your platform). 2) Deployments with the new agent were noticeably faster, (like 1 second deployments that used to take 20+ seconds). However, this speed up was largely due to the Python codes design which had a "continuation" loop which was polling based, but the Lua code used coroutines and simply continued the threads when the steps were complete. Overall, I think the choice to use Lua (or more specifically luvi) was a great decision. Note that it did take a few iterations in order to come up with the optimal way of using Lua + libuv. Originally I was taking the Node.JS approach of using callbacks, but that approach has two problems: [a] It is difficult to properly implement "back pressure" so that all queues between producer and consumer are bounded. [b] It is difficult to handle errors properly. Lua coroutines allowed me to write a simple "green thread" library that encapsulated these challenges.

by kt9 11y ago

bb88 11y ago

In that case, Lua makes a lot of sense, frankly. I can write 100 lines of code in perl, but trying to get that deployed on mac, windows and linux can be a nightmare, especially if its using a library external to perl. (EDIT: LUA->Lua)

glibgil 11y ago
LUA? Still?
- deleted 11y ago
  [deleted]
- deleted 11y ago
  [deleted]
  - talloaktrees 11y ago
    I think he referring to LUA vs Lua
- etiene 11y ago
  Lua used to be LUA, you'll find LUA written in old source code comments :) Don't be so mad about the Lua Uppercase Accident :)

batbomb 11y ago

They were probably running RHEL5, but even in 2014 they would have been in the Production 3 phase which is really should only be used for legacy purposes, and that's definitely not the sort of system you push new software to. Red Hat even pushes those customers towards virtualization because a) hardware rarely lasts 7 years and b) they won't bother backporting drivers for new hardware to the old kernel at that point. I still use a few RHEL5 machines but nobody would ever dream of pushing new software to them.

teh_klev 11y ago
In the real world...if you have a large fleet of RHEL5 boxes/VM's, this a fact of life. We're slowly phasing out our CentOS 5 boxes, but you still need to be able to run new software on them.
nine_k 11y ago
How about running a (newer) local version of Python, without of course touching the system Python? To my mind, it's usually a good idea not to use system Python, both packages and the interpreter, if you're not also controlling them. This makes building your deployables more involved, but makes the actual deployment more painless.
hueving 11y ago
>not the sort of system you push new software to. Telling customers that they are doing things wrong isn't really a great way to win customers.
- takeda 11y ago
  This is something I don't understand. If someone is using Java, they'll install the version of Java they need. If they use Ruby, they'll install the version of Ruby they need. If they need Lua, they'll do the same thing. Why everyone is relying on python that's installed there for RedHat's needs (e.g. yum). Unlike other languages, Python is probably the best one that can have multiple versions coexist together.
  - iofj 11y ago
    As a consultant I have very different experiences. Even in Java development you find shops that refuse to upgrade their JVM for various reasons. RHEL 5 is really common.
    - ghshephard 11y ago
      A lot (all that I'm familiar with) of the enterprise software packages that I've worked with ship with their own, very specific (read: regression tested against) version of java. Enterprise software usually requires a ton of servers, so it's not like anything else on those servers will be using the jvm.

Pharaoh2 11y ago

"I had never seriously used Python before joining Distelli, so there was a bit of a learning curve" That was your problem, you just made a huge mistake and depending on how big your python code base was, you have now alienated most of your developers by moving to a language they are not already comfortable with. Do your customers already have lua installed on their servers or are you bundling it now? You could have just a easily bundled a newer python interpreter. Lua is a nice scripting language, if you want to embed it into another app or to create a plugin system. It's not a nice general purpose scripting language, mostly because of the vast pypi repo.

bigtunacan 11y ago
How is this alienating anyone? This is a command line tool. The end users are suddenly using a different language as a result. Their install process was streamlined and system dependencies were removed by including it in the install package. The same thing could have been achieved within Python, but from an end user perspective I'm guessing this is an improvement for most of their customers.
- Pharaoh2 11y ago
  It's alienating Distelli developers. The reason cited for the move is totally unacceptable in terms of the new skillset that their developers will need to acquire to continue development of the tool. The cost of effort their developers will have to put in to be as effective as a LUA developer as they were as a python developer is really not worth this move. There are more to moving to a different platform/language than just the end users.
  - Sanddancer 11y ago
    The end users are the ones that keep the lights on though. It sounds a lot like that Python was more and more of an anchor around the team, and the costs of migration were smaller than the costs of the other options. Yes, some developers were probably upset that they couldn't use their preferred language anymore. However, it sounds like because they're now using a language designed to be embedded, they can focus a lot more on solving problems than fighting with a language that has a lot more platform-specific behavior.

orf 11y ago

That's fair enough, but it sounds like you've constrained yourselves and your code for the sake of a few enterprise customers, and now you've had to clear your technical debt with a complete rewrite. Even if batbomb were right I would argue that any solution is better than supporting 2.4. The very minimum you would want is 2.5 at a stretch.

techie128 11y ago

Couldn't you have run two versions of Python at the same time on their systems? Would it have been possible to ship a Python version with your software?

collyw 11y ago
easily

brownbat 11y ago

> I had to support Python 2.4... large enterprise customers that are / were running Python 2.4 on servers... How did the switch to Lua not break this in the same way? Sorry for the naive question, I'm just not following how it can be possible to change languages, but impossible to change versions. At first glance, the latter seems like a special case of the former.

iofj 11y ago
Lua and Luajit are small embeddable languages. They link into a single executable, including if you want the program they'll run.
kt9 11y ago
Shipping it as a single download with zero dependencies meant that we would not have to worry in the future about what else was installed on the machine and the versions of system packaged the customers were running.
- oblio 11y ago
  Couldn't you just bundle Python instead?

christianbryant 11y ago

I'm a Python devotee, but I appreciate the anecdote. I'm curious whether the switch 1) cost any features/algorithms and 2) whether the anticipated performance improvement really panned out since the switch, or if the gain wound up being simply architecture/deployment-related. Stepping back, did the whole re-write really balance out with all the work that had to be done, or could you have stuck with Python and simply solved the problems at hand?

brimworks 11y ago
1) All features/algorithms were ported, and we only added new functionality that never existed in the old Python code base. A few of the features that got added include: dispatching builds, getting live logs, re-encrypt keys, deploy encrypted artifacts, fix many bugs, support running multiple agents on a single computer, etc. I should also mention that we previously had two separate downloads: a CLI used for uploading releases, and the agent used for dispatching deployments. The functionality of both of these code bases where merged into our new Lua based "client" so customers only have to install a single binary that is less than 10MB (the binary size depends on your platform). 2) Deployments with the new agent were noticeably faster, (like 1 second deployments that used to take 20+ seconds). However, this speed up was largely due to the Python codes design which had a "continuation" loop which was polling based, but the Lua code used coroutines and simply continued the threads when the steps were complete. Overall, I think the choice to use Lua (or more specifically luvi) was a great decision. Note that it did take a few iterations in order to come up with the optimal way of using Lua + libuv. Originally I was taking the Node.JS approach of using callbacks, but that approach has two problems: [a] It is difficult to properly implement "back pressure" so that all queues between producer and consumer are bounded. [b] It is difficult to handle errors properly. Lua coroutines allowed me to write a simple "green thread" library that encapsulated these challenges.