3 ms·

Yeah, I think about that enterprise software RFS as well. I'm working on a data warehouse for a bioinformatics research center right now, so we are somewhat pl

by tom_b 17y ago

Yeah, I think about that enterprise software RFS as well. I'm working on a data warehouse for a bioinformatics research center right now, so we are somewhat playing in the same place. Can you share what third party software you're using? I'm looking at some of the BI platforms now and comparing that with a ruby/sinatra backend hooked up to a decent jquery based web frontend developed in-house. And, I'm using this experience to decide if I can come up with something that would be worth submitting to YC's next round. It would start with the users building their own reports without needing IT, but there is some other stuff more on the backend with data mining and correlation that I think would be the real win. Oh, seeing joshwa response, I might fall into the hungry hacker category myself.

smokinn 17y ago
I've been looking into it a lot as well. It really depends what your BI is supposed to handle. Is it analytics with very wide but very flat tables that you don't join much (or at all)? Is it complex queries joining many tables together to respond to ad-hoc queries? I recently needed to support the first case and found that one of the best cost/performance solutions for us was infobright given that a lot of our existing base and reporting can stay as-is since we used heavily partitioned mysql myisam tables but it was getting slow (and crashy) and even slower every day. In the near future I plan on supporting the second case because I'm getting a little tired of having to write up new reports myself. I've automated pretty much all the frontend stuff so that generating new reports is mainly a question of plugging in the proper sql queries and massaging the resulting data appropriately (which may be why I find it so boring) and it'll only take me a day or two to write new reports on a new part of the business but I'd much rather pass that off to marketing and teach them how to do it themselves. For the second case, what I think is very promising and will be prototyping very soon is a combination of LucidDB and the Pentaho BI Suite. (Check out a demo here, it's pretty impressive: http://demo.pentaho.com/pentaho/ http://demo.pentaho.com/pentaho/ ) Truly complex predictive reports I'll still have to handle myself but those are interesting because they at least require problem solving. The rote historical analysis reports I'd prefer handing off for good and the other departments would love to handle it themselves since they wouldn't always have to wait until I had time to get them their data.
- tom_b 17y ago
  Believe it or not, Pentaho is what we have in place right now. For the ETL backend, I think it is a really nice tool. Flexible, intuitive, and measures up well to Informatica and other $$$ tools. But the reporting model, with the designer/xaction dev/publish quirkiness just bugs me. I find myself editing the XML definition files when I want to change stuff rather than firing up the tools. The documentation is a little lacking as well, but I have demanding standards there. I also don't particularly care for how business models show up in the GUI for user-driven report building. To be fair, that is probably somewhat related to a poor model without proper dimensional design put in place. First pass efforts (before my time here) basically copied the operational system schemas into the data warehouse (shudder).
  - smokinn 17y ago
    What database did you plug Pentaho into? Like I said earlier I was thinking of using LucidDB but according to this benchmark: http://www.mysqlperformanceblog.com/2010/01/07/star-schema-bechmark-infobright-infinidb-and-luciddb/ http://www.mysqlperformanceblog.com/2010/01/07/star-schema-b... it seems InfiniDB might be a better candidate.
    - tom_b 17y ago
      Caught this question only after a few days, sorry for the delay. We actually are using Oracle (we have a site license).

by tom_b 17y ago

smokinn 17y ago

I've been looking into it a lot as well. It really depends what your BI is supposed to handle. Is it analytics with very wide but very flat tables that you don't join much (or at all)? Is it complex queries joining many tables together to respond to ad-hoc queries? I recently needed to support the first case and found that one of the best cost/performance solutions for us was infobright given that a lot of our existing base and reporting can stay as-is since we used heavily partitioned mysql myisam tables but it was getting slow (and crashy) and even slower every day. In the near future I plan on supporting the second case because I'm getting a little tired of having to write up new reports myself. I've automated pretty much all the frontend stuff so that generating new reports is mainly a question of plugging in the proper sql queries and massaging the resulting data appropriately (which may be why I find it so boring) and it'll only take me a day or two to write new reports on a new part of the business but I'd much rather pass that off to marketing and teach them how to do it themselves. For the second case, what I think is very promising and will be prototyping very soon is a combination of LucidDB and the Pentaho BI Suite. (Check out a demo here, it's pretty impressive: http://demo.pentaho.com/pentaho/ http://demo.pentaho.com/pentaho/ ) Truly complex predictive reports I'll still have to handle myself but those are interesting because they at least require problem solving. The rote historical analysis reports I'd prefer handing off for good and the other departments would love to handle it themselves since they wouldn't always have to wait until I had time to get them their data.

tom_b 17y ago
Believe it or not, Pentaho is what we have in place right now. For the ETL backend, I think it is a really nice tool. Flexible, intuitive, and measures up well to Informatica and other $$$ tools. But the reporting model, with the designer/xaction dev/publish quirkiness just bugs me. I find myself editing the XML definition files when I want to change stuff rather than firing up the tools. The documentation is a little lacking as well, but I have demanding standards there. I also don't particularly care for how business models show up in the GUI for user-driven report building. To be fair, that is probably somewhat related to a poor model without proper dimensional design put in place. First pass efforts (before my time here) basically copied the operational system schemas into the data warehouse (shudder).
- smokinn 17y ago
  What database did you plug Pentaho into? Like I said earlier I was thinking of using LucidDB but according to this benchmark: http://www.mysqlperformanceblog.com/2010/01/07/star-schema-bechmark-infobright-infinidb-and-luciddb/ http://www.mysqlperformanceblog.com/2010/01/07/star-schema-b... it seems InfiniDB might be a better candidate.
  - tom_b 17y ago
    Caught this question only after a few days, sorry for the delay. We actually are using Oracle (we have a site license).