5 ms·

> The day people no longer have a reason to build systems using computers is the day these jobs will go away. Sooo, not too long after some clever person figur

by olefoo 12y ago

> The day people no longer have a reason to build systems using computers is the day these jobs will go away. Sooo, not too long after some clever person figures out how to automate 80% of the grunt work of building an analytics pipeline and their colleagues dumb down the 20% left so that the people inhabiting the C-suite can work it on their iPads...? I know at least three startups working that space, right now. And there are probably another 300 on the way.

dasil003 12y ago
This is like saying that all the programming jobs are going to go away because of some visual programming language just over the horizon. The fact is we are generating more and more data every day, and there are infinite ways to formulate and look at it. The tools that C-suite can use will never be powerful enough to do what a data scientist does because of the variety of data that exists. You can only dumb it down to the lowest common denominator, and the LCD will not be very useful unless the scope is extremely narrow.
- eli_gottlieb 12y ago
  >and there are infinite ways to formulate and look at it. Of course there aren't. The best way is to know exactly the causal process generating the data, thus enabling you to predict your variable-of-interest perfectly from the limited data. This isn't currently possible, of course, but there's one best way to interpret data, not an infinity.
  - dasil003 12y ago
    Of course there are, because there are infinite forms of data. And to say that there is always one best way to interpret data, well, I don't even know where to begin with such a naive declaration.
    - infinite8s 12y ago
      Philosophically, there is only one best way to interpret data - if you know the underlying causes then that provides the only explanatory power you need. A similar argument could be made to our understanding of multiple physical phenoma prior to Newton deriving His equations - your statement is equivalent to saying that there are multiple valid interpretations of how the planets revolve around the sun and apples fall from trees.
      - dasil003 12y ago
        The data we care about are not the direct result of physical laws, they are the result of chaotic systems both natural (eg. weather) and human (eg. stock markets). For your tortured analogy to bear any fruit you have to assume that we're just waiting to find a dead simple set of equations that explains all human endeavor, and after that finite memory and computation power won't be an issue because, hey, cause and effect.
foobarian 12y ago
Sometimes it's an arms race. The minute you automate away some analytics step, the smart analysts start building on top of that, raising the bar for the competition.
- deleted 12y ago
  [deleted]
jschwartzi 12y ago
Can you actually automate statistical analysis like that? I don't mean automating the actual math, I mean writing software to process raw data, decide what the meaningful statistical computations are, and then present the information output appropriately. As far as I'm aware doing probability and statistics correctly is all about framing the problem correctly. You can, for example, accidentally assume that two events are independent when they're really dependent. Is it even possible to detect that kind of mis-framing automatically?
- turnip1979 12y ago
  Yes .. it is possible. I may be getting the name wrong but I saw a demo from Ben Shneiderman's research group a few years ago where they did every possible statistical analysis on pairs of variables to look for high correlation. To their credit, they weren't selling the idea as automatic stats analysis. Their domain was health data and they were merely stating that their tool could give scientists an insight as to what factors to look at. Still ... it made me wonder where this could go.

by olefoo 12y ago

dasil003 12y ago

This is like saying that all the programming jobs are going to go away because of some visual programming language just over the horizon. The fact is we are generating more and more data every day, and there are infinite ways to formulate and look at it. The tools that C-suite can use will never be powerful enough to do what a data scientist does because of the variety of data that exists. You can only dumb it down to the lowest common denominator, and the LCD will not be very useful unless the scope is extremely narrow.

eli_gottlieb 12y ago
>and there are infinite ways to formulate and look at it. Of course there aren't. The best way is to know exactly the causal process generating the data, thus enabling you to predict your variable-of-interest perfectly from the limited data. This isn't currently possible, of course, but there's one best way to interpret data, not an infinity.
- dasil003 12y ago
  Of course there are, because there are infinite forms of data. And to say that there is always one best way to interpret data, well, I don't even know where to begin with such a naive declaration.
  - infinite8s 12y ago
    Philosophically, there is only one best way to interpret data - if you know the underlying causes then that provides the only explanatory power you need. A similar argument could be made to our understanding of multiple physical phenoma prior to Newton deriving His equations - your statement is equivalent to saying that there are multiple valid interpretations of how the planets revolve around the sun and apples fall from trees.
    - dasil003 12y ago
      The data we care about are not the direct result of physical laws, they are the result of chaotic systems both natural (eg. weather) and human (eg. stock markets). For your tortured analogy to bear any fruit you have to assume that we're just waiting to find a dead simple set of equations that explains all human endeavor, and after that finite memory and computation power won't be an issue because, hey, cause and effect.

foobarian 12y ago

Sometimes it's an arms race. The minute you automate away some analytics step, the smart analysts start building on top of that, raising the bar for the competition.

deleted 12y ago
[deleted]

jschwartzi 12y ago

Can you actually automate statistical analysis like that? I don't mean automating the actual math, I mean writing software to process raw data, decide what the meaningful statistical computations are, and then present the information output appropriately. As far as I'm aware doing probability and statistics correctly is all about framing the problem correctly. You can, for example, accidentally assume that two events are independent when they're really dependent. Is it even possible to detect that kind of mis-framing automatically?

turnip1979 12y ago
Yes .. it is possible. I may be getting the name wrong but I saw a demo from Ben Shneiderman's research group a few years ago where they did every possible statistical analysis on pairs of variables to look for high correlation. To their credit, they weren't selling the idea as automatic stats analysis. Their domain was health data and they were merely stating that their tool could give scientists an insight as to what factors to look at. Still ... it made me wonder where this could go.