6 ms·

I'm really impressed you used Julia in production. Despite its immaturity and all the reasons you mentioned, building your product on it is a testament to the l

I'm really impressed you used Julia in production. Despite its immaturity and all the reasons you mentioned, building your product on it is a testament to the language, its potential and the core group working on it. I wonder what the performance differences in your product are going from Julia to python? I've had my eye on it for the last two years and really look forward to its 1.0 release. There is no doubt there is an audience for it!

philip1209 11y ago
Our Julia library did shift generation and shift assignment in one step. We split these tasks into two separate microservices as we transitioned away from Julia. So, it is tough to compare benchmarks. We are using some compiled libraries in Python that make these calculations parallel. I was playing with Legos recently and thought of a new way to approach the shift decomposition problem, and I'm amid a prototype in Go that is promising.
- idunning 11y ago
  Are you using column generation/Dantzig–Wolfe decomposition?
  - philip1209 11y ago
    We're not using column generation in the current algorithms (but Leanform is just launching with column generation on top of CBC - https://angel.co/leanframe https://angel.co/leanframe ). We're not using Dantzig-Wolfe decomposition, but our original scheduling algorithm used a dynamic programming approach that was quite similar. We're currently amid rewriting the Chomp decomposition algorithm to be a more specialized branch and bound variant.
    - sebastienroy 11y ago
      Yes do do have a column generation algorithm that leverage my expertise I gained working at ad-opt (airline pilot rostering). We also migrated from julia to python. Another vote for the MIT teams that worked on the project! It really helped us at the beginning! We are also playing with the google glop solver that gives pretty good results so far. The main drawback of python for column generation (and as far as I understand, the same flaw exists in juliaopt/JuMP) is twofold. 1) most libraries are "math formulation oriented" which makes implementation hard to maintain/debug 2) biggest issue: the wrappers around the underlying solver (cbc, glpk, etc) keeps all references (to variables and coefficients) and do not offer a clean add/remove interface (only add/set bounds). Even playing with coefficients isn't always straigthforward. So we had to build our own wrapper on top of the LP wrappers out there (Pulp, pyomo, cylp etc). When solving large scale problems with column generation, you generate hundreds of thousands of different variables even if only a fraction of them are "active" at a time leading to memory and runtime leaks in the usual wrappers. We had cases where the LP library was spending about 85% of the time just writing and reading files for CBC. The or-tools library wraps the C library itself via SWIG and is our best horse as of writing this. For sure we will think of rolling our own binder too. Last year I was hoping to do it is scheme :) We might get "stuck" with python for the time being though.
dnautics 11y ago
I'm currently using julia to write automated continuous testing for a computer chip. My only beef is that it isn't zero indexed. I probably could have done it with python, but IMO the language just is cleaner and easier to read. The macro system helps, a lot. The typesystem also gives me confidence in the correctness of some of the stranger code I'm emitting. It's a really great prototyping language for mathematical concepts.
- philip1209 11y ago
  The indexing change while rewriting some of our tests from Julia to Python was the most painful part.
tomrod 11y ago
Data scientist in the financial industry here. I speak for more than myself when I say I look forward to the day when Julia is 1.0.
- eggy 11y ago
  Yep, waiting for 1.0. I really like Julia. Zero-indexing would be preferable, or at least a switch like APL to choose what your default indexing is. It's easy to switch if you use somebody's else's code that is 1-based.

philip1209 11y ago

Our Julia library did shift generation and shift assignment in one step. We split these tasks into two separate microservices as we transitioned away from Julia. So, it is tough to compare benchmarks. We are using some compiled libraries in Python that make these calculations parallel. I was playing with Legos recently and thought of a new way to approach the shift decomposition problem, and I'm amid a prototype in Go that is promising.

idunning 11y ago
Are you using column generation/Dantzig–Wolfe decomposition?
- philip1209 11y ago
  We're not using column generation in the current algorithms (but Leanform is just launching with column generation on top of CBC - https://angel.co/leanframe https://angel.co/leanframe ). We're not using Dantzig-Wolfe decomposition, but our original scheduling algorithm used a dynamic programming approach that was quite similar. We're currently amid rewriting the Chomp decomposition algorithm to be a more specialized branch and bound variant.
  - sebastienroy 11y ago
    Yes do do have a column generation algorithm that leverage my expertise I gained working at ad-opt (airline pilot rostering). We also migrated from julia to python. Another vote for the MIT teams that worked on the project! It really helped us at the beginning! We are also playing with the google glop solver that gives pretty good results so far. The main drawback of python for column generation (and as far as I understand, the same flaw exists in juliaopt/JuMP) is twofold. 1) most libraries are "math formulation oriented" which makes implementation hard to maintain/debug 2) biggest issue: the wrappers around the underlying solver (cbc, glpk, etc) keeps all references (to variables and coefficients) and do not offer a clean add/remove interface (only add/set bounds). Even playing with coefficients isn't always straigthforward. So we had to build our own wrapper on top of the LP wrappers out there (Pulp, pyomo, cylp etc). When solving large scale problems with column generation, you generate hundreds of thousands of different variables even if only a fraction of them are "active" at a time leading to memory and runtime leaks in the usual wrappers. We had cases where the LP library was spending about 85% of the time just writing and reading files for CBC. The or-tools library wraps the C library itself via SWIG and is our best horse as of writing this. For sure we will think of rolling our own binder too. Last year I was hoping to do it is scheme :) We might get "stuck" with python for the time being though.

dnautics 11y ago

I'm currently using julia to write automated continuous testing for a computer chip. My only beef is that it isn't zero indexed. I probably could have done it with python, but IMO the language just is cleaner and easier to read. The macro system helps, a lot. The typesystem also gives me confidence in the correctness of some of the stranger code I'm emitting. It's a really great prototyping language for mathematical concepts.

philip1209 11y ago
The indexing change while rewriting some of our tests from Julia to Python was the most painful part.

tomrod 11y ago

Data scientist in the financial industry here. I speak for more than myself when I say I look forward to the day when Julia is 1.0.

eggy 11y ago
Yep, waiting for 1.0. I really like Julia. Zero-indexing would be preferable, or at least a switch like APL to choose what your default indexing is. It's easy to switch if you use somebody's else's code that is 1-based.