3 ms·

I also think this is kinda ridiculous. If anything I feel like the big consumer tech companies are disadvantaged because they need to deploy something that can

I also think this is kinda ridiculous. If anything I feel like the big consumer tech companies are disadvantaged because they need to deploy something that can scale to a billion people. They can only spend pennies per customer because the margins are so low. Sure they will have a research team for marketing purposes but when it comes to deployments they aren't doing anything too fancy. The work coming out of companies with higher profit margins per customer are doing much more novel work from what I have seen. All this is to say, I don't see universities getting shut out anytime soon. The necessary compute to contribute is pretty cheap and most universities either have a free cluster for students or are operating with large grants to pay for compute (or both).

madlag 7y ago
I don't think you have the right numbers in mind talking about the compute you need for AI. The prices are getting lower and lower of course, but you still need tons of money to train the kind of networks that make the news.
- theferalrobot 7y ago
  Example? I have yet to see something actually deployed by one of the big tech companies that could not be trained by students on a university cluster. I also think you underestimate grant funding. I worked at a state school a couple years back in a research lab that had over a million dollars in grant funds specifically for equipment and outside compute (not for salaries or new hires) and this is not at all abnormal.
  - solidasparagus 7y ago
    State of the art CV models (image, not video) can cost 3-figure dollars per training run. State of the art language models can cost 5-figure dollars per training run. There are a lot of variable in play here so your mileage will definitely vary (how much data, how long are you willing to wait, do you really need to train from scratch, etc) and these should only be considered very rough ballpark numbers. However, those are real numbers for SotA models on gold-standard benchmark datasets using cost-optimized cloud ML training resources. At 5-figures per training run, the list of people who can be innovators in the LM research space is very small (fine-tuning on top of a SotA LM is a different, more affordable matter).
    - theferalrobot 7y ago
      Sure but 3 figure and 5 figure runs certainly do not eliminate universities (see my above comment). Not to mention as I have said, most good universities will have clusters capable of training these that they maintain on premise drastically reducing that cost (and in a worst case just take longer to train).
      - solidasparagus 7y ago
        It really does. You've got to remember that a good SotA paper takes hundreds of training runs, at least. I can't go into detail about budgets, but suffice to say if you think $1M is a university compute budget that lets you be a competitive research team on the cutting edge, you are __severely__ underestimating the amount of compute that leading corporate researchers are using. Orders of magnitude off. On-prem is good for a bit until you're 18 months into your 3 year purchase cycle and you're on K80s while the major research leaders are running V100s and TPUs and you can't even fit the SotA model in your GPUs' memories any more. Longer to train can mean weeks or even months for one experiment - that iteration speed makes it so hard to stay on the cutting edge. And this is before considering things like neural architecture search and internet scale image/video/speech datasets where costs skyrocket. The boundary between corporate research and academia is incredibly porous and a big part of that is the cost of research (compute, but also things like data labelling and staffing ML talent).
        theferalrobot 7y ago
        Your goalposts moved a few figures. Furthermore, $1 million+ was not a university compute budget - that was money for a single lab on campus (at a general state school nonetheless) on a specific project. You still have yet to provide any concrete sources to back up your claims. We're talking about contributing to research here. If multi-million dollar training jobs are what it takes to be at the cutting edge you should be able to provide ample sources of that claim.
        solidasparagus 7y ago
- caenorst 7y ago
  I don't think you have the right numbers in mind talking about the networks used in academic works. The majority of network used in publications are good old references like VGG
- wbl 7y ago
  The resources academia has can be pretty big also. Top 500 doesnt have a lot of corporate machines on it.
jgalt212 7y ago
> big consumer tech companies are disadvantaged because they need to deploy something that can scale to a billion people pretty small group of companies. FAANG + maybe five more. So for everyone else, scaling is not an issue as you have described.

madlag 7y ago

I don't think you have the right numbers in mind talking about the compute you need for AI. The prices are getting lower and lower of course, but you still need tons of money to train the kind of networks that make the news.

theferalrobot 7y ago
Example? I have yet to see something actually deployed by one of the big tech companies that could not be trained by students on a university cluster. I also think you underestimate grant funding. I worked at a state school a couple years back in a research lab that had over a million dollars in grant funds specifically for equipment and outside compute (not for salaries or new hires) and this is not at all abnormal.
- solidasparagus 7y ago
  State of the art CV models (image, not video) can cost 3-figure dollars per training run. State of the art language models can cost 5-figure dollars per training run. There are a lot of variable in play here so your mileage will definitely vary (how much data, how long are you willing to wait, do you really need to train from scratch, etc) and these should only be considered very rough ballpark numbers. However, those are real numbers for SotA models on gold-standard benchmark datasets using cost-optimized cloud ML training resources. At 5-figures per training run, the list of people who can be innovators in the LM research space is very small (fine-tuning on top of a SotA LM is a different, more affordable matter).
  - theferalrobot 7y ago
    Sure but 3 figure and 5 figure runs certainly do not eliminate universities (see my above comment). Not to mention as I have said, most good universities will have clusters capable of training these that they maintain on premise drastically reducing that cost (and in a worst case just take longer to train).
    - solidasparagus 7y ago
      It really does. You've got to remember that a good SotA paper takes hundreds of training runs, at least. I can't go into detail about budgets, but suffice to say if you think $1M is a university compute budget that lets you be a competitive research team on the cutting edge, you are __severely__ underestimating the amount of compute that leading corporate researchers are using. Orders of magnitude off. On-prem is good for a bit until you're 18 months into your 3 year purchase cycle and you're on K80s while the major research leaders are running V100s and TPUs and you can't even fit the SotA model in your GPUs' memories any more. Longer to train can mean weeks or even months for one experiment - that iteration speed makes it so hard to stay on the cutting edge. And this is before considering things like neural architecture search and internet scale image/video/speech datasets where costs skyrocket. The boundary between corporate research and academia is incredibly porous and a big part of that is the cost of research (compute, but also things like data labelling and staffing ML talent).
      - theferalrobot 7y ago
        Your goalposts moved a few figures. Furthermore, $1 million+ was not a university compute budget - that was money for a single lab on campus (at a general state school nonetheless) on a specific project. You still have yet to provide any concrete sources to back up your claims. We're talking about contributing to research here. If multi-million dollar training jobs are what it takes to be at the cutting edge you should be able to provide ample sources of that claim.
        solidasparagus 7y ago
caenorst 7y ago
I don't think you have the right numbers in mind talking about the networks used in academic works. The majority of network used in publications are good old references like VGG
wbl 7y ago
The resources academia has can be pretty big also. Top 500 doesnt have a lot of corporate machines on it.

jgalt212 7y ago

> big consumer tech companies are disadvantaged because they need to deploy something that can scale to a billion people pretty small group of companies. FAANG + maybe five more. So for everyone else, scaling is not an issue as you have described.