4 ms·

Probably related to http://news.ycombinator.com/item?id=1032875 http://news.ycombinator.com/item?id=1032875 (Tim Bray's "Doing it Wrong" screed on enterprise so

by joshwa 17y ago

Probably related to http://news.ycombinator.com/item?id=1032875 http://news.ycombinator.com/item?id=1032875 (Tim Bray's "Doing it Wrong" screed on enterprise software) (ps edw519 I was waiting for you to chime in on that thread)

edw519 17y ago
Hey joshwa, somehow I never even noticed that thread yesterday. Why? Because I was up to my earlobes implementing a business intelligence/data warehouse system in a very large SOX-compliant enterprise :-). For current requirements, this solution is excellent. It is third party software, installed on top of an existing ERP system, that enables the users to extract whatever data they need and build their own reports without submitting a ticket to IT. Everyone loves the prototypes and is dreaming about the possibilities. So instead of beating up on enterprise life, let me just share a little bit of yesterday, one typical enterprise day: - A 3rd party build just stopped at 98% complete with no error message. - Another build crashed with error messages I had never seen, so I had to open another ticket with our vendor - We ran out of disk space on a volume we didn't know the software was using. - I had to add additional data cleaning functions to remove heretofore unknown control characters in the enterprise data. - I inadvertantly named 44 files with the vendor's own naming convention, so now no one can tell whose files are whose. We had to reset our standards and rebuild. - Although this vendor has hundreds of installs, oddly, none of them are SOX compliant. The controls, audits, and duplication of data needed will more than double the resource requirements. Worse, I'll have to do an implementation that no one has ever done before with this software :-) - Today we start writing our own tools to handle the SOX compliance and satisfy the auditors. Some fun. I can go on and on. You get the idea. And I haven't even touched upon the usual enterprise culprits: the meetings, the politics, and the lack of project management. It kinda sucks to have to do triple work to get the same thing done. So whose fault is all of this? No one's. That's just the way it is. Every time I think of a better way to get things done in a large enterprise, someone has 5 good reasons why they are the way they are. It's wasted energy fighting that. As I've mentioned in previous posts (one example: http://news.ycombinator.com/item?id=760704 http://news.ycombinator.com/item?id=760704), I think the long term solution will be to eat away at the soft underbelly of the enterprise one bite at a time, with small apps, nimble software, and cloud computing that augments but doesn't replace (yet) current systems. I continue to ponder YC's RFS #6... [EDIT: I continue to ponder "Ideas We'd Like To Fund: Enterprise Software" whichever number that was :-)]
- Sukotto 17y ago
  is there really a RFS #6 or are you counting from 0? http://ycombinator.com/rfs.html http://ycombinator.com/rfs.html
  - fhars 17y ago
    Yeah, as we are discussing the enterprise he should probably have included a disclaimer that that was a forward looking statement.
- joshwa 17y ago
  Sounds pretty similar to my day, too: [redacted - list of gripes] (realized it's easy for my employer to find these posts. Trust me, they were funny/sad. And involved a certain company out of Armonk.) Re: YCRFS 6: do you mean #5 on this page? http://ycombinator.com/ideas.html http://ycombinator.com/ideas.html If you hit on anything good and need a hungry hacker with enterprise experience, you know where to find me. :)
- tom_b 17y ago
  Yeah, I think about that enterprise software RFS as well. I'm working on a data warehouse for a bioinformatics research center right now, so we are somewhat playing in the same place. Can you share what third party software you're using? I'm looking at some of the BI platforms now and comparing that with a ruby/sinatra backend hooked up to a decent jquery based web frontend developed in-house. And, I'm using this experience to decide if I can come up with something that would be worth submitting to YC's next round. It would start with the users building their own reports without needing IT, but there is some other stuff more on the backend with data mining and correlation that I think would be the real win. Oh, seeing joshwa response, I might fall into the hungry hacker category myself.
  - smokinn 17y ago
    I've been looking into it a lot as well. It really depends what your BI is supposed to handle. Is it analytics with very wide but very flat tables that you don't join much (or at all)? Is it complex queries joining many tables together to respond to ad-hoc queries? I recently needed to support the first case and found that one of the best cost/performance solutions for us was infobright given that a lot of our existing base and reporting can stay as-is since we used heavily partitioned mysql myisam tables but it was getting slow (and crashy) and even slower every day. In the near future I plan on supporting the second case because I'm getting a little tired of having to write up new reports myself. I've automated pretty much all the frontend stuff so that generating new reports is mainly a question of plugging in the proper sql queries and massaging the resulting data appropriately (which may be why I find it so boring) and it'll only take me a day or two to write new reports on a new part of the business but I'd much rather pass that off to marketing and teach them how to do it themselves. For the second case, what I think is very promising and will be prototyping very soon is a combination of LucidDB and the Pentaho BI Suite. (Check out a demo here, it's pretty impressive: http://demo.pentaho.com/pentaho/ http://demo.pentaho.com/pentaho/ ) Truly complex predictive reports I'll still have to handle myself but those are interesting because they at least require problem solving. The rote historical analysis reports I'd prefer handing off for good and the other departments would love to handle it themselves since they wouldn't always have to wait until I had time to get them their data.

by joshwa 17y ago

edw519 17y ago

Hey joshwa, somehow I never even noticed that thread yesterday. Why? Because I was up to my earlobes implementing a business intelligence/data warehouse system in a very large SOX-compliant enterprise :-). For current requirements, this solution is excellent. It is third party software, installed on top of an existing ERP system, that enables the users to extract whatever data they need and build their own reports without submitting a ticket to IT. Everyone loves the prototypes and is dreaming about the possibilities. So instead of beating up on enterprise life, let me just share a little bit of yesterday, one typical enterprise day: - A 3rd party build just stopped at 98% complete with no error message. - Another build crashed with error messages I had never seen, so I had to open another ticket with our vendor - We ran out of disk space on a volume we didn't know the software was using. - I had to add additional data cleaning functions to remove heretofore unknown control characters in the enterprise data. - I inadvertantly named 44 files with the vendor's own naming convention, so now no one can tell whose files are whose. We had to reset our standards and rebuild. - Although this vendor has hundreds of installs, oddly, none of them are SOX compliant. The controls, audits, and duplication of data needed will more than double the resource requirements. Worse, I'll have to do an implementation that no one has ever done before with this software :-) - Today we start writing our own tools to handle the SOX compliance and satisfy the auditors. Some fun. I can go on and on. You get the idea. And I haven't even touched upon the usual enterprise culprits: the meetings, the politics, and the lack of project management. It kinda sucks to have to do triple work to get the same thing done. So whose fault is all of this? No one's. That's just the way it is. Every time I think of a better way to get things done in a large enterprise, someone has 5 good reasons why they are the way they are. It's wasted energy fighting that. As I've mentioned in previous posts (one example: http://news.ycombinator.com/item?id=760704 http://news.ycombinator.com/item?id=760704), I think the long term solution will be to eat away at the soft underbelly of the enterprise one bite at a time, with small apps, nimble software, and cloud computing that augments but doesn't replace (yet) current systems. I continue to ponder YC's RFS #6... [EDIT: I continue to ponder "Ideas We'd Like To Fund: Enterprise Software" whichever number that was :-)]

Sukotto 17y ago
is there really a RFS #6 or are you counting from 0? http://ycombinator.com/rfs.html http://ycombinator.com/rfs.html
- fhars 17y ago
  Yeah, as we are discussing the enterprise he should probably have included a disclaimer that that was a forward looking statement.
joshwa 17y ago
Sounds pretty similar to my day, too: [redacted - list of gripes] (realized it's easy for my employer to find these posts. Trust me, they were funny/sad. And involved a certain company out of Armonk.) Re: YCRFS 6: do you mean #5 on this page? http://ycombinator.com/ideas.html http://ycombinator.com/ideas.html If you hit on anything good and need a hungry hacker with enterprise experience, you know where to find me. :)
tom_b 17y ago
Yeah, I think about that enterprise software RFS as well. I'm working on a data warehouse for a bioinformatics research center right now, so we are somewhat playing in the same place. Can you share what third party software you're using? I'm looking at some of the BI platforms now and comparing that with a ruby/sinatra backend hooked up to a decent jquery based web frontend developed in-house. And, I'm using this experience to decide if I can come up with something that would be worth submitting to YC's next round. It would start with the users building their own reports without needing IT, but there is some other stuff more on the backend with data mining and correlation that I think would be the real win. Oh, seeing joshwa response, I might fall into the hungry hacker category myself.
- smokinn 17y ago
  I've been looking into it a lot as well. It really depends what your BI is supposed to handle. Is it analytics with very wide but very flat tables that you don't join much (or at all)? Is it complex queries joining many tables together to respond to ad-hoc queries? I recently needed to support the first case and found that one of the best cost/performance solutions for us was infobright given that a lot of our existing base and reporting can stay as-is since we used heavily partitioned mysql myisam tables but it was getting slow (and crashy) and even slower every day. In the near future I plan on supporting the second case because I'm getting a little tired of having to write up new reports myself. I've automated pretty much all the frontend stuff so that generating new reports is mainly a question of plugging in the proper sql queries and massaging the resulting data appropriately (which may be why I find it so boring) and it'll only take me a day or two to write new reports on a new part of the business but I'd much rather pass that off to marketing and teach them how to do it themselves. For the second case, what I think is very promising and will be prototyping very soon is a combination of LucidDB and the Pentaho BI Suite. (Check out a demo here, it's pretty impressive: http://demo.pentaho.com/pentaho/ http://demo.pentaho.com/pentaho/ ) Truly complex predictive reports I'll still have to handle myself but those are interesting because they at least require problem solving. The rote historical analysis reports I'd prefer handing off for good and the other departments would love to handle it themselves since they wouldn't always have to wait until I had time to get them their data.