5 ms·

Open Sourcing super expensive proprietary software

8 points by mqsiuser 12y ago

mqsiuser 12y ago
So I have been open sourcing the heart of a super expensive IBM software product, which IBM currently/today sells regularly for 100 tsd (Dollar or Euro, doesn't matter much) per CPU core (discounts of 40% to 95% usually apply). This product is IBM WebSphere Message Broker or now IBM Integration Bus (IIB) and was thought to be IBM's "SOA" offering. Let's say mildly that SOA failed and unfortunately the remaining parts are not "a lot" (also not worth that much). I was in fact able to understand them and it's been possible for me to open source the very most important parts in a POC. So the very most important part is the message processing and transformation language "ESQL". IBM implemented it in c and I implemented it in Java. I called the project "use-the-tree" and open sourced it on github. A deployed version is online at www.use-the-tree.com. Actually it turned out that I added a couple of improvements which makes my implementation superior to the one IBM has. There are open source ESB's and Broker's, but I did not find (and I think it is unlikely that there are) comparable proper implementations. I managed to work in this SOA trouble and Integration world for about 8 years and I am the contributor of important components to IIB which I open sourced as "GenericESQLUtilities". My contributions are confirmed (by IBM Hursley) and by the community (on www.mqseries.net) and are actually used. Basically I proved that xslt is a performance disaster and on the other hand I showed how to properly do message transformation and processing. I also just implemented stuff that IBM didn't know how to (implement), since they bought the product from another company (NEON) and certainly had some brain drain during that process. Main areas of future work are (also documented in the git hub wiki): - Add support for JSON - Add all quirks that XML has - Investigate if JavaScript can be used (instead of Java) (and implement it) - Sorting (super cleverly with Tree Map ofc) - On-Demand Parser (would be awesome, probably to hard for 3 months) - Investigate on open source solutions for parsing flat files standards like EDI/EDIFACT or HL7/ORM formats and it's usability together with use-the-tree The project is very new (a couple of weeks old), it's a POC and there is nothing in production. Though it is confirmed to hit the point. So... I have already successfully contributed to open source (in this area) and there is constant change in the IT world (I see a clear decline of SOA and a push towards open source), so that I think that this project can have a significant success in the future. Again ... "use-the-tree" is the only thing which is left of SOA if you remove all the bad parts or the ones that have just failed... it is "SOA - the good parts" :). It provides strong guidance on how to do message transformation (to end up with the most efficient transformation code) as opposed to IBM's approach, which is "we provide you with every possibility and you better choose wisely". Xslt would look like it, if xslt would be done properly.
- smoyer 12y ago
  SOA as a buzz-word has fallen out of favor, but SOA as a means of segmenting a large problem to later solve it in "chunks" has been an accepted practice since the dawn of engineering. I'll agree that Microsoft, IBM and Oracle (Tibco, etc) would prefer that you were passing SOAP/XML over enterprise busses, but our layer's of RESTful services are still a "Service Oriented Architecture" ... the loss of statefullness was a big win.
  - mqsiuser 12y ago
    > I'll agree that Microsoft, IBM and Oracle (Tibco, etc) would prefer that you were passing SOAP/XML over enterprise busses They better say bye to this idea... it's hard to see why curly braces are good and angle brackets are evil... they get used differently. > Enterprise (Service) Bus Buzz word, snake oil There is no problem, there never was a problem: Build on existing (de-facto) standards and just don't reinvent the world (with WS-*). And do not use the beginning of the payload (like SOAP does). The solution was there already and the solution is HTTP and the 2048 characters of the URL (to encode a function call) and yes a machine readable data format, but NOT XML and yes, JSON is likely/probably good enought. You usually pay the bill for what you are doing Open Source is free (as in freedom) but also free of charge :)
    - AdrianRossouw 12y ago
      > yes a machine readable data format, but NOT XML and yes, JSON is likely/probably good enough I thought so too, until I realized json is terrible for binary. on it's own it can't reproduce the full capabilities of REST ie: http multi-part uploads. You need a performant streamable binary format too, and my vote for that is protobuf.
- FraaJad 12y ago
  Thanks for doing this. I have not used IBM's products, but I understand the motivation behind doing something like this. It will be a wonderful exercise in digging up super expensive programs like this and creating open source versions of it. They will rescue some hapless programmer and/or provide consulting opportunities for another. Do you have some programs that can benefit by having an open source version? It will be a neat exercise to learn new languages/platforms.

8 points by mqsiuser 12y ago

So I have been open sourcing the heart of a super expensive IBM software product, which IBM currently/today sells regularly for 100 tsd (Dollar or Euro, doesn't matter much) per CPU core (discounts of 40% to 95% usually apply). This product is IBM WebSphere Message Broker or now IBM Integration Bus (IIB) and was thought to be IBM's "SOA" offering. Let's say mildly that SOA failed and unfortunately the remaining parts are not "a lot" (also not worth that much). I was in fact able to understand them and it's been possible for me to open source the very most important parts in a POC. So the very most important part is the message processing and transformation language "ESQL". IBM implemented it in c and I implemented it in Java. I called the project "use-the-tree" and open sourced it on github. A deployed version is online at www.use-the-tree.com. Actually it turned out that I added a couple of improvements which makes my implementation superior to the one IBM has. There are open source ESB's and Broker's, but I did not find (and I think it is unlikely that there are) comparable proper implementations. I managed to work in this SOA trouble and Integration world for about 8 years and I am the contributor of important components to IIB which I open sourced as "GenericESQLUtilities". My contributions are confirmed (by IBM Hursley) and by the community (on www.mqseries.net) and are actually used. Basically I proved that xslt is a performance disaster and on the other hand I showed how to properly do message transformation and processing. I also just implemented stuff that IBM didn't know how to (implement), since they bought the product from another company (NEON) and certainly had some brain drain during that process. Main areas of future work are (also documented in the git hub wiki): - Add support for JSON - Add all quirks that XML has - Investigate if JavaScript can be used (instead of Java) (and implement it) - Sorting (super cleverly with Tree Map ofc) - On-Demand Parser (would be awesome, probably to hard for 3 months) - Investigate on open source solutions for parsing flat files standards like EDI/EDIFACT or HL7/ORM formats and it's usability together with use-the-tree The project is very new (a couple of weeks old), it's a POC and there is nothing in production. Though it is confirmed to hit the point. So... I have already successfully contributed to open source (in this area) and there is constant change in the IT world (I see a clear decline of SOA and a push towards open source), so that I think that this project can have a significant success in the future. Again ... "use-the-tree" is the only thing which is left of SOA if you remove all the bad parts or the ones that have just failed... it is "SOA - the good parts" :). It provides strong guidance on how to do message transformation (to end up with the most efficient transformation code) as opposed to IBM's approach, which is "we provide you with every possibility and you better choose wisely". Xslt would look like it, if xslt would be done properly.

smoyer 12y ago
SOA as a buzz-word has fallen out of favor, but SOA as a means of segmenting a large problem to later solve it in "chunks" has been an accepted practice since the dawn of engineering. I'll agree that Microsoft, IBM and Oracle (Tibco, etc) would prefer that you were passing SOAP/XML over enterprise busses, but our layer's of RESTful services are still a "Service Oriented Architecture" ... the loss of statefullness was a big win.
- mqsiuser 12y ago
  > I'll agree that Microsoft, IBM and Oracle (Tibco, etc) would prefer that you were passing SOAP/XML over enterprise busses They better say bye to this idea... it's hard to see why curly braces are good and angle brackets are evil... they get used differently. > Enterprise (Service) Bus Buzz word, snake oil There is no problem, there never was a problem: Build on existing (de-facto) standards and just don't reinvent the world (with WS-*). And do not use the beginning of the payload (like SOAP does). The solution was there already and the solution is HTTP and the 2048 characters of the URL (to encode a function call) and yes a machine readable data format, but NOT XML and yes, JSON is likely/probably good enought. You usually pay the bill for what you are doing Open Source is free (as in freedom) but also free of charge :)
  - AdrianRossouw 12y ago
    > yes a machine readable data format, but NOT XML and yes, JSON is likely/probably good enough I thought so too, until I realized json is terrible for binary. on it's own it can't reproduce the full capabilities of REST ie: http multi-part uploads. You need a performant streamable binary format too, and my vote for that is protobuf.
FraaJad 12y ago
Thanks for doing this. I have not used IBM's products, but I understand the motivation behind doing something like this. It will be a wonderful exercise in digging up super expensive programs like this and creating open source versions of it. They will rescue some hapless programmer and/or provide consulting opportunities for another. Do you have some programs that can benefit by having an open source version? It will be a neat exercise to learn new languages/platforms.