5 ms·

You're absolutely right. That said, I don't buy that big-data is as revolutionary as the Internet. While in theory, every single business can collect data and o

by throwaway1979 14y ago

You're absolutely right. That said, I don't buy that big-data is as revolutionary as the Internet. While in theory, every single business can collect data and optimize based on what they see, this is way too complex for most businesses to deal with. While big data has certainly been critical for the business model of ad-based startups, I don't see it being used in other industries. People keep alluding to data-driven medicine and genetic analyses. These are some of the most complex information analyses domains, and yet, I don't see benefit commensurate with the big data hype. I'd love to hear counter examples though!

tgflynn 14y ago
This technology will change the world more than the Internet or any other technology in human history. You're right that adoption is very slow. I'm convinced that businesses could save trillions of dollars by applying existing weak AI to their problems. Why aren't they doing it ? For one thing there's a huge gulf between the average business person's understanding of what is possible and what actually is. On the other hand the people who understand the technology don't have domain experience in various businesses. You can't develop solutions if you don't know what the problems are and it's very hard to guess at what economically relevant problems exist in fields you've never worked in. There are probably other barriers as well. Domain experts are unlikely to champion technologies that may, well, replace them. Bayesian networks were developed in academia 20 years ago that outperformed doctors at medical diagnosis. Why aren't they being applied ? There are probably many reasons but I suspect conscious or unconscious resistance on the part of the medical community plays a significant role. As for a talent shortage, I don't buy it. I'm exactly the sort of person this article talks about, with a strong mathematical background, excellent implementation skills and real world experience in developing and applying machine learning algorithms that have made millions of dollars for my former employers. I have had a website and a LinkedIn profile for over a year that make this fairly clear. How many consulting inquiries have I had ? Exactly zero.
- zheng 14y ago
  Slightly off-topic, but your website breaks after visiting the RDMS page, as all the other links seem to be relative, so they attempt to go to pages such as /products/people.html
  - tgflynn 14y ago
    Sorry about that, thanks for letting me know. It should be fixed now.
- csomar 14y ago
  I have had a website and a LinkedIn profile for over a year that make this fairly clear It doesn't work like that. You LinkedIn profile might easily land you any job in Software development, but not consulting. In my opinion, if you want to do consulting for big corp. you should figure out what it takes to it. An attractive website and presentation, few buzzwords, client testimonials, business cards, and the other blablabla. Yes, it's irrelevant (and shitty) to what you are actually doing, but that's actually the world of consulting.
  - tgflynn 14y ago
    That's definitely not my world and one reason I left big-corp in the first place. EDIT ADDED: It seems like a really broken market if buyer decisions are completely orthogonal to the product being purchased.
    - tomjen3 14y ago
      I hope you are right. If you are, you have identified a potentially very, very lucrative option for a start-up. Broken markets can provide you a lot of money when you fix them.
      - lgieron 14y ago
        This market is already (at least partially) covered by small consulting shops which provide sales front for competent freelancers who don't feel like doing the whole corporate networking&sales ritual.
        ced 14y ago
        Where do I find these small consulting shops?
        lgieron 14y ago
        Well, the companies I know tend to do their recruiting based on word-of-mouth recomendations - so they have to know you from some previous job or you need to be recommended by someone etc. It's a tiny sample of a couple companies though, and that probably doesn't generalize to all "small consulting shops".
- dbecker 14y ago
  Whether or not you have these skills: potential employers need to SEE them. There are lots of pretenders out there, and employers are appropriately wary. Are you showing employers results on your web page that a worse ML practitioner can't match. Putting results from a kaggle competition on my LinkedIn page landed me my current job (and I am still contacted by potential employers every couple weeks). The employers of the world aren't stupid, but they aren't omniscient either. So you need to make it easy for them to see that you have the skills you claim.
  - tgflynn 14y ago
    Did you win the competition ? If not what was your rank, if you don't mind sharing ? I'd be really interested to know how much impact this could have.
    - dbecker 14y ago
      I was #4 at the time my current employer contacted me. I've continued in the competition with a small team of their employees, and we are currently #2. I don't know how sensitive the number of job contacts is to your ranking. My hunch is that a lower ranking would still establish credibility.
- Estragon 14y ago
  I'm convinced that businesses could save trillions of dollars by applying existing weak AI to their problems. Why are you convinced of this? Do you have any case studies with obvious broad applications? I have had a website and a LinkedIn profile for over a year that make this fairly clear. How many consulting inquiries have I had ? Exactly zero. Neither LinkedIn nor your ISP are responsible for marketing you. Have you been presenting your software at restaurant conferences? Do you have any case studies where it saved someone x% of their food costs?
- tgflynn 14y ago
  I couldn't reply to Estragon's comment below, so I'm replying here. I'm convinced of it primarily because I'm convinced that the majority of activities that human workers perform do not require general intelligence (ie. strong AI). This includes most manual tasks : cleaning, cooking, customer service in restaurants, etc, but also many tasks performed by office or even "knowledge" workers. A product my previous company developed replaced over 20K workers over a period of years. Few people even know it exists. Once you are aware of this and follow the news you see it happening in virtually all domains: e-discovery systems replacing attorney hours, automated news story generation, etc. One area that has been lagging is robotics but this will start to develop very quickly, especially with the new DARPA Robotics Challenge. Now to be clear many of the "Big Data" applications people are talking about may not fall into this human labor replacement category, but the underlying technologies are essentially the same.

tgflynn 14y ago

This technology will change the world more than the Internet or any other technology in human history. You're right that adoption is very slow. I'm convinced that businesses could save trillions of dollars by applying existing weak AI to their problems. Why aren't they doing it ? For one thing there's a huge gulf between the average business person's understanding of what is possible and what actually is. On the other hand the people who understand the technology don't have domain experience in various businesses. You can't develop solutions if you don't know what the problems are and it's very hard to guess at what economically relevant problems exist in fields you've never worked in. There are probably other barriers as well. Domain experts are unlikely to champion technologies that may, well, replace them. Bayesian networks were developed in academia 20 years ago that outperformed doctors at medical diagnosis. Why aren't they being applied ? There are probably many reasons but I suspect conscious or unconscious resistance on the part of the medical community plays a significant role. As for a talent shortage, I don't buy it. I'm exactly the sort of person this article talks about, with a strong mathematical background, excellent implementation skills and real world experience in developing and applying machine learning algorithms that have made millions of dollars for my former employers. I have had a website and a LinkedIn profile for over a year that make this fairly clear. How many consulting inquiries have I had ? Exactly zero.

zheng 14y ago
Slightly off-topic, but your website breaks after visiting the RDMS page, as all the other links seem to be relative, so they attempt to go to pages such as /products/people.html
- tgflynn 14y ago
  Sorry about that, thanks for letting me know. It should be fixed now.
csomar 14y ago
I have had a website and a LinkedIn profile for over a year that make this fairly clear It doesn't work like that. You LinkedIn profile might easily land you any job in Software development, but not consulting. In my opinion, if you want to do consulting for big corp. you should figure out what it takes to it. An attractive website and presentation, few buzzwords, client testimonials, business cards, and the other blablabla. Yes, it's irrelevant (and shitty) to what you are actually doing, but that's actually the world of consulting.
- tgflynn 14y ago
  That's definitely not my world and one reason I left big-corp in the first place. EDIT ADDED: It seems like a really broken market if buyer decisions are completely orthogonal to the product being purchased.
  - tomjen3 14y ago
    I hope you are right. If you are, you have identified a potentially very, very lucrative option for a start-up. Broken markets can provide you a lot of money when you fix them.
    - lgieron 14y ago
      This market is already (at least partially) covered by small consulting shops which provide sales front for competent freelancers who don't feel like doing the whole corporate networking&sales ritual.
      - ced 14y ago
        Where do I find these small consulting shops?
        lgieron 14y ago
        Well, the companies I know tend to do their recruiting based on word-of-mouth recomendations - so they have to know you from some previous job or you need to be recommended by someone etc. It's a tiny sample of a couple companies though, and that probably doesn't generalize to all "small consulting shops".
dbecker 14y ago
Whether or not you have these skills: potential employers need to SEE them. There are lots of pretenders out there, and employers are appropriately wary. Are you showing employers results on your web page that a worse ML practitioner can't match. Putting results from a kaggle competition on my LinkedIn page landed me my current job (and I am still contacted by potential employers every couple weeks). The employers of the world aren't stupid, but they aren't omniscient either. So you need to make it easy for them to see that you have the skills you claim.
- tgflynn 14y ago
  Did you win the competition ? If not what was your rank, if you don't mind sharing ? I'd be really interested to know how much impact this could have.
  - dbecker 14y ago
    I was #4 at the time my current employer contacted me. I've continued in the competition with a small team of their employees, and we are currently #2. I don't know how sensitive the number of job contacts is to your ranking. My hunch is that a lower ranking would still establish credibility.
Estragon 14y ago
I'm convinced that businesses could save trillions of dollars by applying existing weak AI to their problems. Why are you convinced of this? Do you have any case studies with obvious broad applications? I have had a website and a LinkedIn profile for over a year that make this fairly clear. How many consulting inquiries have I had ? Exactly zero. Neither LinkedIn nor your ISP are responsible for marketing you. Have you been presenting your software at restaurant conferences? Do you have any case studies where it saved someone x% of their food costs?
tgflynn 14y ago
I couldn't reply to Estragon's comment below, so I'm replying here. I'm convinced of it primarily because I'm convinced that the majority of activities that human workers perform do not require general intelligence (ie. strong AI). This includes most manual tasks : cleaning, cooking, customer service in restaurants, etc, but also many tasks performed by office or even "knowledge" workers. A product my previous company developed replaced over 20K workers over a period of years. Few people even know it exists. Once you are aware of this and follow the news you see it happening in virtually all domains: e-discovery systems replacing attorney hours, automated news story generation, etc. One area that has been lagging is robotics but this will start to develop very quickly, especially with the new DARPA Robotics Challenge. Now to be clear many of the "Big Data" applications people are talking about may not fall into this human labor replacement category, but the underlying technologies are essentially the same.