4 ms·

I'm a firm believer that data science should be pulled out of IT and put into the business. However, storage is so cheap that you should never NOT collect data

I'm a firm believer that data science should be pulled out of IT and put into the business. However, storage is so cheap that you should never NOT collect data if you can. It's better to dump it into a data warehouse and wait for somebody who can use it to come along than it is to never collect it. Then, and only then, should you look at your service levels and determine if there is a need for some sort of Hadoop or Spark infrastructure.

Terr_ 9y ago
> However, storage is so cheap that you should never NOT collect data if you can. Unless it involves customers and clients, in which case it's actually a sneaky liability... At least in terms of PR, if not legality.
acdha 9y ago
I like data too but that approach has several hazards. The most obvious is liability: if you store it, you have to be responsible for protecting against misuse. The second, however, is more subtle but often more damaging: people often assume that the data they have is the right data or complete so the drunkard's lampost problem is easy to fall into and people might not realize it as quickly because, hey, those numbers were based on so much data it took a day to run the query! Web analytics is notorious for that — people would make statements about performance, browser support, etc. from a bunch of log files and miss that this was skewed by bot traffic, confuse their server's response time as being a reasonable proxy for the user's perceived load-time, etc.
apohn 9y ago
>I'm a firm believer that data science should be pulled out of IT and put into the business. I say this from personal experience. The major risk here is that Data Scientists will constantly get pulled in as resources to support business fires and high visibility projects that should be handled by other people. Otherwise what happens is this: "I know you are working on that Data Science project that is scoped for 4 weeks, but can you do this Business Analyst thing the insert exec name here asked for by this Friday?" Data Scientists need to be shielded from that day to day analyst stuff, otherwise they'll never be able to do their jobs. One of the ways to do this is to put them in IT in a business facing consulting role. IT is typically (not always) is more focused on a proper solution, not day to day triage. So they can filter out projects that aren't appropriate.
- IgniteTheSun 9y ago
  This may end up repeating the mistakes of a decade or two ago: after a business leader gets two requests rejected, they are very unlikely to go back with a third request - they will simply outsource every future request. Soon the IT departments were being downsized because they couldn't get business leaders to use them instead of going outside.
RHSman2 9y ago
Data should be moved out of IT and put into Data. Data headed up by a Chief Data Officer should be a first class citizen of every company. Not f’in IT, not Buisness. Data.

Terr_ 9y ago

> However, storage is so cheap that you should never NOT collect data if you can. Unless it involves customers and clients, in which case it's actually a sneaky liability... At least in terms of PR, if not legality.

acdha 9y ago

I like data too but that approach has several hazards. The most obvious is liability: if you store it, you have to be responsible for protecting against misuse. The second, however, is more subtle but often more damaging: people often assume that the data they have is the right data or complete so the drunkard's lampost problem is easy to fall into and people might not realize it as quickly because, hey, those numbers were based on so much data it took a day to run the query! Web analytics is notorious for that — people would make statements about performance, browser support, etc. from a bunch of log files and miss that this was skewed by bot traffic, confuse their server's response time as being a reasonable proxy for the user's perceived load-time, etc.

apohn 9y ago

>I'm a firm believer that data science should be pulled out of IT and put into the business. I say this from personal experience. The major risk here is that Data Scientists will constantly get pulled in as resources to support business fires and high visibility projects that should be handled by other people. Otherwise what happens is this: "I know you are working on that Data Science project that is scoped for 4 weeks, but can you do this Business Analyst thing the insert exec name here asked for by this Friday?" Data Scientists need to be shielded from that day to day analyst stuff, otherwise they'll never be able to do their jobs. One of the ways to do this is to put them in IT in a business facing consulting role. IT is typically (not always) is more focused on a proper solution, not day to day triage. So they can filter out projects that aren't appropriate.

IgniteTheSun 9y ago
This may end up repeating the mistakes of a decade or two ago: after a business leader gets two requests rejected, they are very unlikely to go back with a third request - they will simply outsource every future request. Soon the IT departments were being downsized because they couldn't get business leaders to use them instead of going outside.

RHSman2 9y ago

Data should be moved out of IT and put into Data. Data headed up by a Chief Data Officer should be a first class citizen of every company. Not f’in IT, not Buisness. Data.