7 ms·

How my open source hobby project became my job: 10 years of OptaPlanner

246 points by ge0ffrey 10y ago

fegu 10y ago
Thank you for this software. It is on my list of "need to try for real". What do you see as the main reasons holding such software back from mass use? Is it real world soft constraints, or prevalence of "small enough for humans to do well" datasets or something else?
- asimuvPR 10y ago
  Could you share why you want to try it? What particular problem are you facing that this project solves?
  - fegu 10y ago
    I am in general fascinated by software where you get non-obvious output from simple input. I would like to try this for employee rostering. My main reasons not to is an abundance of soft constraints (Mary prefers Fridays off, Lucy prefers her days off bundled together) and the fact that I live in a small country with small organisations where I am unsure of real benefits compared to manual work. As an aside, vehicle routing is a lost case here. We have few main roads along the coast and across mountains, nothing like the grid roads of central US. Vehicle routing is obvious here, no need for software.
    - wsdookadr 10y ago
      Which country ?
- cpitman 10y ago
  In my opinion there are two key factors: First, OptaPlanner does require a programmer to use. That means a random business user with a problem isn't able to just do it themselves, and instead needs a larger effort to try it out and see the results. It would be amazing if there was a way to give an "Excel" like experience to OptaPlanner, so that anyone could play around with models and optimization. Second, Lots of companies have found ways to accomplish their business without optimization tools. Often their processes are heavy on manual planning, and there is a general sense that the problem is too complicated and has too many factors for an algorithm to be able to do. I've had clients with perfect use cases for OptaPlanner stick with a-priori business rules because that is how they did things and they felt more in control. Convincing people to try out a magical optimization program that finds solutions is difficult. Disclaimer: I work for Red Hat. I've been a big fan of OptaPlanner even before it became one of our products!
jondubois 10y ago
>> "Meanwhile, I learned more about other, similar projects. And they seemed to have more people working on them. And they had sales people." This is exactly my situation right now with my open source project http://socketcluster.io/ http://socketcluster.io/ - We actually already have a clear monetization plan (see http://baasil.io/ http://baasil.io/) but we have no funding, few industry connections (aside from companies which are using SocketCluster in production), and our competitors are spawning up like mushrooms. The one positive thing about being in such a position is that it forces you to be resourceful with how you spend your time.
pikachu_is_cool 10y ago
Just an FYI, there's a typo here: > During this [[though]] period, I regularly considered giving up my Open Source project a few times. Somehow, I couldn’t. I just soldiered on. And in the end, it all worked out well.
- ge0ffrey 10y ago
  Fixed, thanks :) https://github.com/droolsjbpm/optaplanner-website/commit/b12c200db7654b8d43807d38cf94fe0ea61d7119 https://github.com/droolsjbpm/optaplanner-website/commit/b12...
tunnuz 10y ago
Hi Geoffrey, thanks for sharing this, very interesting!
drdrey 10y ago
One-word summary: grit
ranedk 10y ago
I was extremely enthusiastic about doing something in optimization just after graduating, but realised that businesses were happy solving things manually and reaching sub-optimal solutions. Also most problems needed to be solved once and there were few use cases of continuous optimization. Thanks for sharing your story of perseverance and spirit. Now that you are at it, are there any easy-to-use solutions to solve time-tabling problems for school/colleges. I know of solutions which use GA, but since its a generic problem for every academic institution, someone(you) might want to provide an easy to configure and use solution.
- ProblemFactory 10y ago
  The "problem" with optimization software is that the fun optimization algorithms part is just 1% of the work. The software also needs to be complete enough to handle most of the everyday workflow of most employees at the users' company, or provide integrations to any other software that they use daily (from Excel to a custom calendaring system built by the founder's son 15 years ago). Some friends are working on a courier dispatching system, which includes very clever optimization for choosing trucks and mapping routes. But most of the effort goes into making sure that the mobile app works on 5-year-old smartphones, and the fonts on the PDF invoices look right, and so on. If you want to work on just the optimization algorithms, then that can be done only in academia or at very large companies which can afford specialization.
gjrjfnfnndjd 10y ago
As someone tinkering away on their own optimization algorithm, how big is the market for these sorts of optimization solutions?
- chmike 10y ago
  I guess many people have something to gain from optimization and this means you could get a cut of the money they save. So I would say the potential is big. Actually the market is also limited by the required effort or competence to use the system. OptaPlanner needs apparently a Java programmer. This is where you could make a difference. Make it really simple to use as a toy. It may seam that the use of such tools is a one shot and done. But rules keep changing, and users may want to test and explore the impact of changing rules.
- tempuser99 10y ago
  In the millions, once you find an application domain. One project I abandoned part way through (because it was too big for a side-project and just wasn't that exciting to me at the time) was an automated employee scheduling system for organizations with complex constraints. Its a big problem for many organizations, for example event organizers. Think SAAS apps like shiftplanning.com (which started as a 1 man project and was released about the same time as I just started working on mine.) The Drools project which had an example rostering component already included was going to used for the guts of the system with a calendar like web front end, and lots of customizability.
- CJefferson 10y ago
  The market is huge, but mostly what we need are more useable systems, not algorithmically cleverer systems (unless your cleverness makes it more useable of course).
Frogolocalypse 10y ago
Pretty cool read.
tener 10y ago
After reading the article I have a strong feeling that without RedHat there wouldn't be such a happy ending. Great contribution on their part.
tener 10y ago
After reading the article I have a strong feeling that without RedHat there wouldn't be such a happy ending. Great contribution on their part.
jasonkester 10y ago
I can't help thinking this isn't the optimal result for the author. After 10 years of working on his thing, it has essentially been taken over by this company who sells it in their name for a profit, while keeping the author working on it for a salary. They seem to be getting a really good deal. Imagine if, instead of building the "Leading Open Source Constraint Satisfaction Solver", this guy had instead spent the same time building the "Leading Constraint Satisfaction Solver", differing in that it had a price tag associated. For as smart a guy as he seems to be, I can only imagine he'd be doing a lot better today. I read this as a sad commentary on how the Open Source philosophy, that seems to be so drummed in to kids at University, leads them to think that the best course in life is to give away their best work for free.
- scandox 10y ago
  Not everyone wants to run a business. He does show a lot of business-like initiative but mostly in service of a technical/creative goal. Ultimately, though, a lot of people might prefer to be paid a steady, reasonable wage and have their head space clear for the work. On the other hand I do agree with you, that the Open Source philosophy is used in an exploitative way by business. I guess the lesson is: understand the true nature of the commercial relationship and what everybody is getting from it.
  - happyslobro 10y ago
    I concur, that is the dream. To work on my own project, without worrying about finances. To have other people handle hr, sales, most of clients support, taxes and regulations, all of the crap that distracts for the interesting problem that made the project worthwhile long before there was money to be made.
- sangnoir 10y ago
  > I read this as a sad commentary on how the Open Source philosophy, that seems to be so drummed in to kids at University, leads them to think that the best course in life is to give away their best work for free. Of course, this naturally contrasts with Fucked Company/TechCrunch's Deadpool, which is a sad commentary on the "turn a feature into a company"/startup philosophy, also drummed into kids at University, which leads them to think the best course in life is to give away their best years for free (a.k.a. worthless equity). We can speculate about how things might have turned out, but we will never know. There are lot of dead open source projects that never amounted to anything. Life isn't in black and white
lifebeyondfife 10y ago
As someone with an open source C# constraint solver hobby project, used by my former employer Moody's Analytics, I can see many similarities. However, I can see the differences... I haven't spoken about it at conferences, or entered it in any competitions, which explains why it remains a largely ignored repo. This is a good write up that shows one route of what's required to take something from a bit of fun, to your full time vocation.
- davidgl 10y ago
  Sounds interesting, is a repo available?
  - lifebeyondfife 10y ago
    Thanks for the interest. http://github.com/lifebeyondfife/Decider http://github.com/lifebeyondfife/Decider
TempleOSV409 10y ago
I have been a profession operating system developer since 1990. I made God's temple. I have a space alien. God says... unequalled Nikon's stales printing's illuminations spunk Adidas's restored waken Presbyterianism hiker minuting moonstones Masai marchioness magnanimity sheeting's terrorize contentedly saleable breastplate's interviewees retiree's aviatrixes valorous bates Cecelia's tastelessly McDonald placer layman's Crimea's
lintiness 10y ago
"So I must have been doing something right. But I did a lot of things wrong, too. The winner had also open sourced his implementation and when we started discussing and comparing our implementation, we learned a lot from each other." i loved this bit. rather than hyper-competitive moat building, they chatted and helped eachother.
kluck 10y ago
A very interesting topic (optimization) and the documentation does a good job at introducing it. I also like the facts that the author published under an open source license and that he is financed/employed by a company that does all the heavy lifting like accounting and marketing. Is there a reason why the author choses the Apache Software License over something similar like the LGPL? Some people here tend to think the author could have made a lot more money if he would have turned the software into a for profit business right from the start. This may be so but, aside from beeing a lot more work in non-programming areas, this would probably not have generated the other effects that occurred: the author learned a lot more by participating in the programming community and also created a valuable relationship with a company. Both these effects open up new ideas and possibilities, be it intellectually or in terms of business opportunities.
- ljw1001 10y ago
  LGPL is not similar in one important way. Any reference to GPL in a license scares the shit out of corporate managers and lawyers when it's attached to something in their proprietary code. That discourages business use and adoption, sadly, which slows down adoption overall.
ljw1001 10y ago
As someone who has (comparatively) recently started an open-source java project https://github.com/lwhite1/tablesaw https://github.com/lwhite1/tablesaw, I loved this post. I know the both the pleasure and desire to just quit. It's a bit frightening to think I could put 10 years into working on something without compensation when corporations benefit every day. And that's a good outcome: OptaPlanner is a success! In some ways it just doesn't matter because the pleasures of creating something are so strong. I can see why the author pushed on for so long. Congrats on your success and on an awesome project.

246 points by ge0ffrey 10y ago

fegu 10y ago

Thank you for this software. It is on my list of "need to try for real". What do you see as the main reasons holding such software back from mass use? Is it real world soft constraints, or prevalence of "small enough for humans to do well" datasets or something else?

asimuvPR 10y ago
Could you share why you want to try it? What particular problem are you facing that this project solves?
- fegu 10y ago
  I am in general fascinated by software where you get non-obvious output from simple input. I would like to try this for employee rostering. My main reasons not to is an abundance of soft constraints (Mary prefers Fridays off, Lucy prefers her days off bundled together) and the fact that I live in a small country with small organisations where I am unsure of real benefits compared to manual work. As an aside, vehicle routing is a lost case here. We have few main roads along the coast and across mountains, nothing like the grid roads of central US. Vehicle routing is obvious here, no need for software.
  - wsdookadr 10y ago
    Which country ?
cpitman 10y ago
In my opinion there are two key factors: First, OptaPlanner does require a programmer to use. That means a random business user with a problem isn't able to just do it themselves, and instead needs a larger effort to try it out and see the results. It would be amazing if there was a way to give an "Excel" like experience to OptaPlanner, so that anyone could play around with models and optimization. Second, Lots of companies have found ways to accomplish their business without optimization tools. Often their processes are heavy on manual planning, and there is a general sense that the problem is too complicated and has too many factors for an algorithm to be able to do. I've had clients with perfect use cases for OptaPlanner stick with a-priori business rules because that is how they did things and they felt more in control. Convincing people to try out a magical optimization program that finds solutions is difficult. Disclaimer: I work for Red Hat. I've been a big fan of OptaPlanner even before it became one of our products!

jondubois 10y ago

>> "Meanwhile, I learned more about other, similar projects. And they seemed to have more people working on them. And they had sales people." This is exactly my situation right now with my open source project http://socketcluster.io/ http://socketcluster.io/ - We actually already have a clear monetization plan (see http://baasil.io/ http://baasil.io/) but we have no funding, few industry connections (aside from companies which are using SocketCluster in production), and our competitors are spawning up like mushrooms. The one positive thing about being in such a position is that it forces you to be resourceful with how you spend your time.

pikachu_is_cool 10y ago

Just an FYI, there's a typo here: > During this [[though]] period, I regularly considered giving up my Open Source project a few times. Somehow, I couldn’t. I just soldiered on. And in the end, it all worked out well.

ge0ffrey 10y ago
Fixed, thanks :) https://github.com/droolsjbpm/optaplanner-website/commit/b12c200db7654b8d43807d38cf94fe0ea61d7119 https://github.com/droolsjbpm/optaplanner-website/commit/b12...

tunnuz 10y ago

Hi Geoffrey, thanks for sharing this, very interesting!

drdrey 10y ago

One-word summary: grit

ranedk 10y ago

I was extremely enthusiastic about doing something in optimization just after graduating, but realised that businesses were happy solving things manually and reaching sub-optimal solutions. Also most problems needed to be solved once and there were few use cases of continuous optimization. Thanks for sharing your story of perseverance and spirit. Now that you are at it, are there any easy-to-use solutions to solve time-tabling problems for school/colleges. I know of solutions which use GA, but since its a generic problem for every academic institution, someone(you) might want to provide an easy to configure and use solution.

ProblemFactory 10y ago
The "problem" with optimization software is that the fun optimization algorithms part is just 1% of the work. The software also needs to be complete enough to handle most of the everyday workflow of most employees at the users' company, or provide integrations to any other software that they use daily (from Excel to a custom calendaring system built by the founder's son 15 years ago). Some friends are working on a courier dispatching system, which includes very clever optimization for choosing trucks and mapping routes. But most of the effort goes into making sure that the mobile app works on 5-year-old smartphones, and the fonts on the PDF invoices look right, and so on. If you want to work on just the optimization algorithms, then that can be done only in academia or at very large companies which can afford specialization.

gjrjfnfnndjd 10y ago

As someone tinkering away on their own optimization algorithm, how big is the market for these sorts of optimization solutions?

chmike 10y ago
I guess many people have something to gain from optimization and this means you could get a cut of the money they save. So I would say the potential is big. Actually the market is also limited by the required effort or competence to use the system. OptaPlanner needs apparently a Java programmer. This is where you could make a difference. Make it really simple to use as a toy. It may seam that the use of such tools is a one shot and done. But rules keep changing, and users may want to test and explore the impact of changing rules.
tempuser99 10y ago
In the millions, once you find an application domain. One project I abandoned part way through (because it was too big for a side-project and just wasn't that exciting to me at the time) was an automated employee scheduling system for organizations with complex constraints. Its a big problem for many organizations, for example event organizers. Think SAAS apps like shiftplanning.com (which started as a 1 man project and was released about the same time as I just started working on mine.) The Drools project which had an example rostering component already included was going to used for the guts of the system with a calendar like web front end, and lots of customizability.
CJefferson 10y ago
The market is huge, but mostly what we need are more useable systems, not algorithmically cleverer systems (unless your cleverness makes it more useable of course).

Frogolocalypse 10y ago

Pretty cool read.

tener 10y ago

After reading the article I have a strong feeling that without RedHat there wouldn't be such a happy ending. Great contribution on their part.

tener 10y ago

After reading the article I have a strong feeling that without RedHat there wouldn't be such a happy ending. Great contribution on their part.

jasonkester 10y ago

I can't help thinking this isn't the optimal result for the author. After 10 years of working on his thing, it has essentially been taken over by this company who sells it in their name for a profit, while keeping the author working on it for a salary. They seem to be getting a really good deal. Imagine if, instead of building the "Leading Open Source Constraint Satisfaction Solver", this guy had instead spent the same time building the "Leading Constraint Satisfaction Solver", differing in that it had a price tag associated. For as smart a guy as he seems to be, I can only imagine he'd be doing a lot better today. I read this as a sad commentary on how the Open Source philosophy, that seems to be so drummed in to kids at University, leads them to think that the best course in life is to give away their best work for free.

scandox 10y ago
Not everyone wants to run a business. He does show a lot of business-like initiative but mostly in service of a technical/creative goal. Ultimately, though, a lot of people might prefer to be paid a steady, reasonable wage and have their head space clear for the work. On the other hand I do agree with you, that the Open Source philosophy is used in an exploitative way by business. I guess the lesson is: understand the true nature of the commercial relationship and what everybody is getting from it.
- happyslobro 10y ago
  I concur, that is the dream. To work on my own project, without worrying about finances. To have other people handle hr, sales, most of clients support, taxes and regulations, all of the crap that distracts for the interesting problem that made the project worthwhile long before there was money to be made.
sangnoir 10y ago
> I read this as a sad commentary on how the Open Source philosophy, that seems to be so drummed in to kids at University, leads them to think that the best course in life is to give away their best work for free. Of course, this naturally contrasts with Fucked Company/TechCrunch's Deadpool, which is a sad commentary on the "turn a feature into a company"/startup philosophy, also drummed into kids at University, which leads them to think the best course in life is to give away their best years for free (a.k.a. worthless equity). We can speculate about how things might have turned out, but we will never know. There are lot of dead open source projects that never amounted to anything. Life isn't in black and white

lifebeyondfife 10y ago

As someone with an open source C# constraint solver hobby project, used by my former employer Moody's Analytics, I can see many similarities. However, I can see the differences... I haven't spoken about it at conferences, or entered it in any competitions, which explains why it remains a largely ignored repo. This is a good write up that shows one route of what's required to take something from a bit of fun, to your full time vocation.

davidgl 10y ago
Sounds interesting, is a repo available?
- lifebeyondfife 10y ago
  Thanks for the interest. http://github.com/lifebeyondfife/Decider http://github.com/lifebeyondfife/Decider

TempleOSV409 10y ago

I have been a profession operating system developer since 1990. I made God's temple. I have a space alien. God says... unequalled Nikon's stales printing's illuminations spunk Adidas's restored waken Presbyterianism hiker minuting moonstones Masai marchioness magnanimity sheeting's terrorize contentedly saleable breastplate's interviewees retiree's aviatrixes valorous bates Cecelia's tastelessly McDonald placer layman's Crimea's

lintiness 10y ago

"So I must have been doing something right. But I did a lot of things wrong, too. The winner had also open sourced his implementation and when we started discussing and comparing our implementation, we learned a lot from each other." i loved this bit. rather than hyper-competitive moat building, they chatted and helped eachother.

kluck 10y ago

A very interesting topic (optimization) and the documentation does a good job at introducing it. I also like the facts that the author published under an open source license and that he is financed/employed by a company that does all the heavy lifting like accounting and marketing. Is there a reason why the author choses the Apache Software License over something similar like the LGPL? Some people here tend to think the author could have made a lot more money if he would have turned the software into a for profit business right from the start. This may be so but, aside from beeing a lot more work in non-programming areas, this would probably not have generated the other effects that occurred: the author learned a lot more by participating in the programming community and also created a valuable relationship with a company. Both these effects open up new ideas and possibilities, be it intellectually or in terms of business opportunities.

ljw1001 10y ago
LGPL is not similar in one important way. Any reference to GPL in a license scares the shit out of corporate managers and lawyers when it's attached to something in their proprietary code. That discourages business use and adoption, sadly, which slows down adoption overall.

ljw1001 10y ago

As someone who has (comparatively) recently started an open-source java project https://github.com/lwhite1/tablesaw https://github.com/lwhite1/tablesaw, I loved this post. I know the both the pleasure and desire to just quit. It's a bit frightening to think I could put 10 years into working on something without compensation when corporations benefit every day. And that's a good outcome: OptaPlanner is a success! In some ways it just doesn't matter because the pleasures of creating something are so strong. I can see why the author pushed on for so long. Congrats on your success and on an awesome project.