3 ms·

do you mind if I derail the thread for a little bit? [if yes] I currently work at a small-medium sized company, and we have one BI guy. Our ops team is ~5 peo

do you mind if I derail the thread for a little bit? [if yes] I currently work at a small-medium sized company, and we have one BI guy. Our ops team is ~5 people, and 1-2 of the more junior guys are tasked with doing everything necessary to support the BI guy. which basically means making sure the BI software is installed properly on the container, configured properly, has access to the db, and it comes up properly etc BI is a critical business function, so it's a pretty high priority, but the process is way less automated and skillfully crafted than the other parts of the system, mainly cause everyone hates it and only barely understands what the BI system (in this case, pentaho) actually does. it's no coincidence that the most junior engineers often end up doing it (with occasional but pretty infrequent help from the seniors when blocked) what i've been wondering is, what exactly does "a BI guy" need their system to do? I read a bit about ETL systems, but haven't done much research. I assume the job of an ETL system is to pull data from the production database(s), clean it up as necessary, and use that data to calculate whatever business metrics are required? is the software more about making it easy for the BI team to use an interface to craft their queries/dataflow/etc? basically my motivation is I've mulled over the idea of having the ETL system be written in-house. the benefits being potentially smoother and more native operation and in a way that's more seamless to integrate with monitoring, alerting, automated jobs etc. my guess is that such a project would end up being a waste of time in 98% of cases, especially because you need a usable frontend for the BI guy. so, if I may, what exactly is business intelligence / what do you do on a day-to-day basis? I'm pretty experienced with traditional finance/investing jobs, eg reading 10-q's, 10-k's, balance sheets etc, and using the data to calculate cash flow, opex, gross margin, and every other metric you could conceivably want. But I'd never heard of "business intelligence" until starting an internship several months ago. thanks, and no worries if you don't have time to respond / if this is too off-topic.

blihp 8y ago
The definition of what is BI varies by company and is one of the 'squishier' terms in IT becaise the needs and roles vary quite a bit. In a small company (which it sounds like yours is) one BI person does everything out of necessity. In a Fortune 500 company, a BI person would likely be on a reporting team and spend their days doing nothing but generating reports and performing analysis... there would be another team(s) to deal with ETL, and another to manage the Data Mart(s)/Data Warehouse(s), possibly another for Data Science etc etc. Your IT group sounds a bit dysfunctional, though not atypical: you admit that it's business critical but that the most junior people support it. I'm sure the business loves that (not to mention it's a bad career move for the senior technical people.) In your company, it sounds like BI is considered to be everything from ETL (which you've got the rough idea of) to providing the final deliverables (reports, analysis, dashboards etc.) to the business customers. Essentially what a BI person would want in that environment are tools with a high degree of leverage for working with the data, probably integrated (seamless workflow, unified metadata), and more geared towards individual productivity rather than group collaboration. Basically, if your BI guy stops showing up for work, the business would be flying blind. He's the person that managers and above go to in order to get the wider view of the business. The requirements fed to him would drive many IT people nuts: 'you know that one-time report you gave us last quarter? Well, we need that same report again this quarter... with these changes... and add the data from this spreadsheet to it. Oh, yeah, there are some gaps in my spreadsheet where we couldn't figure out where to get the numbers... can you fix that?' Regarding rolling your own ETL tool. Here's an easy way figure out if you really want to (hint: probably not) do this: offer to have your team take over some of the ETL workload from the BI person (i.e. pick a small project or two, learn how to use his tool, see how it works/what it does for him.) If he's good and busy, he'll jump at the offer. That's exactly what you'd be taking on anyway if you decided to roll your own and it's the best vantage point to collect requirements from. I think what you'll find, if you really have ETL/BI needs in your company, is that the requirements are changing far faster than your group would be able to handle. Good ETL/BI tools are generally around the midpoint of custom code (task specific, more structured, managed) and Excel (general purpose, rapid prototyping, free-form, flexible.) Fast turnaround time is the name of the game. At the end of it you'll probably find you would no more want to write your own ETL tools than you'd want to write your own DBMS. You might event find that, if you have good tools, you want your team to start using them to eliminate some of their custom code. I've been on all sides of the data business (I've rolled my own ETL and reporting tools and used most of the major commercial ones) and when given the option of a decent tool, the tool wins over custom code every time.
- __blockcipher__ 8y ago
  That was an awesome response - that helped a lot. Thanks for sharing your knowledge.

blihp 8y ago

The definition of what is BI varies by company and is one of the 'squishier' terms in IT becaise the needs and roles vary quite a bit. In a small company (which it sounds like yours is) one BI person does everything out of necessity. In a Fortune 500 company, a BI person would likely be on a reporting team and spend their days doing nothing but generating reports and performing analysis... there would be another team(s) to deal with ETL, and another to manage the Data Mart(s)/Data Warehouse(s), possibly another for Data Science etc etc. Your IT group sounds a bit dysfunctional, though not atypical: you admit that it's business critical but that the most junior people support it. I'm sure the business loves that (not to mention it's a bad career move for the senior technical people.) In your company, it sounds like BI is considered to be everything from ETL (which you've got the rough idea of) to providing the final deliverables (reports, analysis, dashboards etc.) to the business customers. Essentially what a BI person would want in that environment are tools with a high degree of leverage for working with the data, probably integrated (seamless workflow, unified metadata), and more geared towards individual productivity rather than group collaboration. Basically, if your BI guy stops showing up for work, the business would be flying blind. He's the person that managers and above go to in order to get the wider view of the business. The requirements fed to him would drive many IT people nuts: 'you know that one-time report you gave us last quarter? Well, we need that same report again this quarter... with these changes... and add the data from this spreadsheet to it. Oh, yeah, there are some gaps in my spreadsheet where we couldn't figure out where to get the numbers... can you fix that?' Regarding rolling your own ETL tool. Here's an easy way figure out if you really want to (hint: probably not) do this: offer to have your team take over some of the ETL workload from the BI person (i.e. pick a small project or two, learn how to use his tool, see how it works/what it does for him.) If he's good and busy, he'll jump at the offer. That's exactly what you'd be taking on anyway if you decided to roll your own and it's the best vantage point to collect requirements from. I think what you'll find, if you really have ETL/BI needs in your company, is that the requirements are changing far faster than your group would be able to handle. Good ETL/BI tools are generally around the midpoint of custom code (task specific, more structured, managed) and Excel (general purpose, rapid prototyping, free-form, flexible.) Fast turnaround time is the name of the game. At the end of it you'll probably find you would no more want to write your own ETL tools than you'd want to write your own DBMS. You might event find that, if you have good tools, you want your team to start using them to eliminate some of their custom code. I've been on all sides of the data business (I've rolled my own ETL and reporting tools and used most of the major commercial ones) and when given the option of a decent tool, the tool wins over custom code every time.

__blockcipher__ 8y ago
That was an awesome response - that helped a lot. Thanks for sharing your knowledge.