6 ms·

The common saying is "more data usually beats a better algorithm". However, this is only for short term progress. Long term progress in the field of AI clearl

• wangchow 10y ago
  Tons of resources dont matter if they aren't looking in the right direction. A small shop can make huge progress just need to think "okay they're all doing it this way", and then do the opposite most crazy thing ever. Honestly we dont have the hardware yet. AI needs something groundbreaking before it turns into more than a clever analytical tool.
  • chubot 10y ago
    Yes, the data and the algorithms don't really matter unless you have the right product. Otto apparently was a group of Googlers who thought the product was wrong, and went for the narrower self-driving trucks market (before being acquired). I imaginne that the winner of the self-driving car race will be determined by the product and not the technology per se. Tesla, Google, Uber, comma.ai, etc. all have quite different conceptions of what the product is.
• vonnik 10y ago
  Vanilla deep learning is so far ahead of what most companies use for analytics, that very little research is needed to offer a product with significant advantages over what's in practice now. In fact, I'd say most ML/DL/AI startups don't need to do research at all. They just need to spread more widely what exists now. I disagree with Andreessen that Google's Tensorflow propaganda on Udacity is "opening the kimono" or having a large effect on AI. TF isn't better than pre-existing tools in many ways. And the ways that Google uses it aren't really transferable in an online course, since Google does have the data, architectures, infrastructure and talent -- some of which it can't share by definition, and some of which it won't share, for obvious reasons. The smartest startups outsource research to universities and government funded bodies. It's a way of derisking themselves as investments. That's no different for AI. Theoretical research and product development are almost (but not quite) totally different things. Which is something that graduate students who think they want to do a startup should remember, because if they love research, they probably won't have much time for it. [Disclosure: I'm associated with a rival DL library that I won't plug here.]
  • radarsat1 10y ago
    > Vanilla deep learning is so far ahead of what most companies use for analytics, that very little research is needed to offer a product with significant advantages over what's in practice now But by "vanilla deep learning", I think you are referring to this vs research into new models and algorithms. I think "research" in the context of "research-heavy start-ups" is not about new algorithms, but about finding ways to use even "vanilla deep learning" on their available datasets. Quite a lot of papers at conferences, for example, are not about new algorithms, but about new ways to use old algorithms.
  • kough 10y ago
    Yeah, Tensorflow seems cool, but then you benchmark and realize that (with CUDA/CUDNN installed) it's 50% slower than Caffe.
    • mdonahoe 10y ago
      Was google too busy working with their custom asics to notice the speed difference on gpus?
• outsideline 10y ago
  You're right on many points. My observation is, when it comes to deeper Ai that trends towards AGI, the private sector seems lost in terms of the big ideas and applications. The VCs are also lost and have no idea on how to appreciate fundamentally groundbreaking R&D development ventures. To be honest, most of the funding and deeply interesting work for ground breaking Ai research lies in the public and defense sector. They have more of an idea and vision for what Ai can be applied to than the private sector. Leaps and bounds beyond. In the private sector, I get asked questions like 'Can it be used to predict if someone will click on an ad'. In the public sector, I get a 40 page technology acquisition pdf outlining a forward thinking application of Ai. People forget that self driving cars arose from Darpa research. The latest innovations in computer security utilizing Ai came from a Darpa competition. IBM's neuromorphic chip : darpa funding. Darpa and other defense programs also have a slew of Ai development projects open to researchers. Ultimately I find this a bit ridiculous given how vocal some individuals in the tech field are about the 'dangers of Ai'. They harp and harp about what could happen if more capable Ai gets developed but you don't see their money anywhere near the groups doing deeper development in Ai. Lastly.. Yes, better solutions and more deeply inspired approaches to Ai will trump hoards of data and compute power any-day. Thank goodness for the public sector pushing forward fundamental research and development in Ai.
  • Eliezer 10y ago
    > Ultimately I find this a bit ridiculous given how vocal some individuals in the tech field are about the 'dangers of Ai'. They harp and harp about what could happen if more capable Ai gets developed but you don't see their money anywhere near the groups doing deeper development in Ai. ...because if you think AGI is dangerous, you should fund groups trying to develop it? I am puzzled by the presentation of this proposition as if it is an absolutely obvious sequitur requiring no defense, such that failure to follow through on it is "ridiculous".
    • outsideline 10y ago
      Yep. "The best way to predict the future is to create it" Don't stretch your mind to far on this one. If someone or various groups develop AGI and you have no stake in them, you will have no say in how it gets used. For once, people need to be honest with themselves about this. You're not going to be able to dictate from a third party consortium of PhDs how someone should run their company or develop their software.
• tnecniv 10y ago
  > how does one draw the most talented researchers to work for a small group of people with minimal pay over working for a large company with a high salary and all the computational resources you might need? Offer them a PhD
• dave_sullivan 10y ago
  > how does one draw the most talented researchers to work for a small group of people with minimal pay over working for a large company with a high salary and all the computational resources you might need? And what kind of product could theoretical research produce in the short term that would keep a startup afloat long enough to come up with ground breaking ideas? I think the short answer is, you don't. Between Google, OpenAI, Facebook, Baidu, Microsoft, IBM, and every other startup that is putting "deep learning" in their pitch deck to see what happens, there's enough money being pumped into the area. 90% of the results of that money being spent will be made available to everyone in the world for free. IE, if you're investing for competitive advantage, you will lose. If you're not already on one of those teams or studying the subject, I don't see any opportunity for at least a year, maybe 2. To be clear, I'm talking about opportunities around starting a new company. If your business is consulting or you're working as an engineer for an existing company, there are many applications for deep learning, but none that stand on their own. In 2 years, content generation will be a big thing. Like pandora with purely algorithmically generated music, or games with assets that create themselves in a way far superior to what we see in the current generation of "procedural generation". The big cos will have very good versions of Siri, but no startup is going to be able to touch that near term.
• YeGoblynQueenne 10y ago
  >> doing more with less data is exactly the kind of problem that a startup in the field could solve with a clever idea. It will take much more than a "clever idea" to overcome the reliance on data of the current AI state of the art- that is to say, machine learning. If you think about it, machine learning algorithms are essentially clever search procedures for some optimum in a heap of data (that's optimisation algorithms, so about 90% of the field). You can always get smarter in the way you search, but unless you search far enough what you find won't generalise very well to unseen data. And if "unseen data" is the real world, you need immense amounts of data to get anywhere- hence, the reliance on big data. What we need is much more than a clever idea: it's a paradigm shift. There's probably some sort of way to make a system that can learn from very few examples and at a very short time, like we do ourselves. Machine learning is certainly not that way. We need to find another way. I'm not sure startups with clever ideas have a very good chance to just stumble on such an other way while looking for a way to do "Facebook with deep learning".
  • jfoutz 10y ago
    > There's probably some sort of way to make a system that can learn from very few examples and at a very short time, like we do ourselves. Maybe. I wonder if we can pick up identifying platypus with only a couple of examples, because we spent a long time identifying bird or cat as a toddler. We have a complicated model of the world, and we can take short cuts, because we just identify the differences compared to stuff we already know. You may be right, no one knows for sure. I think it's one of those P = NP kinda problems. Lots of clever people have thought about it for years. Maybe there's a trick. It seems unlikely there's a trick though. Absence of evidence starts to become evidence of absence.
  • dharma1 10y ago
    There's a lot of ongoing work in machine learning focusing on learning from few examples, online learning, more biologically plausible learning and so on. Most of it is done in the academia, and some of it is done at large corporations. Startups generally have a short runway before they run out of cash, which isn't long enough to sustain this kind of exploratory work.
  • 10y ago
• thomasahle 10y ago
  > how does one draw the most talented researchers to work for a small group of people with minimal pay Universities seem to be doing alright at this. Probably the trick is to offer an exciting problem and lots of freedom.
• agibsonccc 10y ago
  I have to fundamentally disagree with Marc here. Someone who actually DOES AI is Fei Fei Li at stanford. She's focusing on creating datasets like imagenet and more recently the visual genome: https://www.technologyreview.com/s/545906/next-big-test-for-ai-making-sense-of-the-world/ https://www.technologyreview.com/s/545906/next-big-test-for-... A lot of AI startups' best assets are their data which google also has plenty of. Major advancements in AI will come from less glamorous things like labeled data available for models to learn from. The code without the data is useless. The algorithms are only part of the equation here...and giving them away without the requisite data is useful, but not the critical path from profiting from AI. It's also not going to advance AI much. These AI labs publish subsets of the research they actually do (even if it is still a generous amount which is great). We can even see this from OpenAI's gym efforts. These environments are creating fundamental infrastructure for pushing the boundaries on reinforcement learning. That being said, research is a component of the problem, but even most "AI" startups just git clone some open source code and run something pretrained (say: opencv, kaldi for audio,..) and then wrap it in a nice gui. The main things these startups focus on is delivery of the product just like the rest of these startups, very few are actually building novel algorithms. A lot of it is just them collecting data. That being said, this is also why a lot of the novel research happens in the major for profit ad tech companies. They have the data to do research on and they can choose what to publish and what to profit from in products. Disclosure: I work at an AI startup.
• WilliamDhalgren 10y ago
  Truly doing much with little data may simply be impossible; its not a magical black box after all, but just as ideal a learner as possible - but if small data cannot give it confidence to confirm the existence of some subtile pattern, on what grounds is even an ideal learner supposed to believe in it? For example, no smarts would ever confirm the Higgs in the tevatron in its 10 year run even though the machine produced thousands of Higges; its a subtle signal and you just don't have the sample size to be more than 3 sigma sure there's something there (nor where exactly). What is key, however, is being able to leverage more than just explicit training sets, like with transfer learning and unsupervised learning, as well as making algorithms scale well with more data. The latter seems promising with deep learning, the former still a research thing. "more data beats better algorithm" is obviously false when the algorithms don't scale with data as well, as deep learning has convincingly demonstrated, being just as mediocre as anything else on small datasets, but being leaps above everything else on sufficiently large ones.
  • Xcelerate 10y ago
    > Truly doing much with little data may simply be impossible; its not a magical black box after all, but just as ideal a learner as possible This is interesting, and something that needs further explored IMO. I've been doing research on extracting mutual information from noisy, shifted copies of a ground truth signal, and it turns out there is a crossover point where recovering the ground truth essentially becomes impossible (in an information theoretic sense). What's interesting is that this crossover point is sharp and it looks a lot like a phase transition that one might see in statistical physics. We need more research that provides limits on what we are capable of predicting from a given dataset — an upper bound, in other words. This would let us know if it's worth it to spend time trying to get more predictivity out of a dataset, or if the data just simply doesn't contain sufficient enough information.
    • jandrewrogers 10y ago
      The concept of an upper bound on prediction for a given data set does exist, I've seen it discussed explicitly in some mid-1990s universal sequence prediction literature. I haven't seen the term (or concept) used in modern predictive algorithm literature but I believe the term of art is "prediction error complexity" which is conceptually similar to concepts like Kolmogorov complexity in algorithmic information theory.
• eli_gottlieb 10y ago
  >how does one draw the most talented researchers to work for a small group of people with minimal pay over working for a large company with a high salary and all the computational resources you might need? You mean how does one draw people to graduate school?