3 ms·

My (basic) understanding is that 1/ there's some inductive bias (knowledge from researchers) 2/ data is definitely "compressed" in some ways 3/ since the model

by wiz21c 5y ago

My (basic) understanding is that 1/ there's some inductive bias (knowledge from researchers) 2/ data is definitely "compressed" in some ways 3/ since the model predicts better than the others, then it actually found some relationships in the data that were not found before. From what I understand, deep learning, although opaque and relying on tons of data, is a bit magical : although one would say "it's just probabilities", it actually does probabilities at a level where it actually figures some things. Plus, and that's very much a problem to me, Google does it at 100x the scale of a regular researcher. Since I just invested a year in studying data sciences, that worries me a lot : where am I suppose to work if, to produce meaningful results, you need way-too-expensive hardware...

thomasahle 5y ago
> where am I suppose to work if, to produce meaningful results, you need way-too-expensive hardware... I know how you feel, but I also think stories like this may be a wake-up call for some groups to invest more in hardware. The Baker and Tsinghua University groups are not small. They can afford more than 4 GPUs. Probably it's more about setting up a good pipeline. Once you get about 4 GPUs you need more than one machine to run them. Hopefully in the next years we'll see more open source tools to make it easy to "build your own GPU cloud".
- dumb1224 5y ago
  And to use all that GPU power exclusively you need some good strategy too. Speaking from a medium size biology research centre.
  - thomasahle 5y ago
    True. It may be that a smaller research center will only rarely be able to saturate the cluster. Maybe it doesn't matter, like how other lab equipment is not in constant use. Another option may be for centers to team up and have shared machines? Or maybe compute as a service will eventually be cheap enough for this not to matter...
    - dumb1224 5y ago
      Currently only those team who are heavy-deep-learning got special exclusive queues on the GPU nodes on the cluster. If many users want to use the GPUs at the same time it might need some planning. I don't know if it is a solved problem in the HPC field though.
sgt101 5y ago
>if, to produce meaningful results, you need way-too-expensive hardware... If you are in a team that is looking at problems that justify massive hardware (in the sense that solving them will pay back the capital and environmental cost) then you will have access to said hardware. Most (almost all) AI and Data Science teams are not working on that kind of problem though, and it's often the case that we are working on cloud infrastructures where GPU's and TPU's can be accessed on demand and $100's can be used to train pretty good models. Obviously models need to be trained many times so the crossover point from a few $100 to a few $1000 can be painful - but actually many/most engagements really only need models that cost <$100 to train. Also many of the interesting problems that are out there can utilize transfer learning over shared large pretrained models such as Resnet or GPT-2 (I know that in the dizzying paced modern world these no longer count as large or modern but they are examples...) So for images and natural language problems we can get round the intractable demand for staggeringly expensive compute. Imagine that you had got a degree in Aeronautical Engineering, you are watching the Apollo program and wondering how you will get a job at NASA or something similar... but there are lots of jobs at Boeing and Lockheed and Cessna and so on.
touisteur 5y ago
Most big labs have large computing clusters and upgrade them from time to time. We've almost always needed huge computing power in the scientific domain, no? Anyway these days I see a lot of industrial investment in middle size computing datacenters full of GPUs. Sure Google scale is not within reach but I'm sure there's room for scrapier algorithms and training methods to demonstrate feasibility and then paying a lump sum to AWS (or some ml-expert-for-cloud-training SME) for the final step. Anyway, I thought the expensive part was data acquisition and labeling? I like the 'surrogate network' approach of learning a very-expensive-to-compute simulation or model, that doesn't need data, but the output of a costly simulation.

by wiz21c 5y ago

thomasahle 5y ago

> where am I suppose to work if, to produce meaningful results, you need way-too-expensive hardware... I know how you feel, but I also think stories like this may be a wake-up call for some groups to invest more in hardware. The Baker and Tsinghua University groups are not small. They can afford more than 4 GPUs. Probably it's more about setting up a good pipeline. Once you get about 4 GPUs you need more than one machine to run them. Hopefully in the next years we'll see more open source tools to make it easy to "build your own GPU cloud".

dumb1224 5y ago
And to use all that GPU power exclusively you need some good strategy too. Speaking from a medium size biology research centre.
- thomasahle 5y ago
  True. It may be that a smaller research center will only rarely be able to saturate the cluster. Maybe it doesn't matter, like how other lab equipment is not in constant use. Another option may be for centers to team up and have shared machines? Or maybe compute as a service will eventually be cheap enough for this not to matter...
  - dumb1224 5y ago
    Currently only those team who are heavy-deep-learning got special exclusive queues on the GPU nodes on the cluster. If many users want to use the GPUs at the same time it might need some planning. I don't know if it is a solved problem in the HPC field though.

sgt101 5y ago

>if, to produce meaningful results, you need way-too-expensive hardware... If you are in a team that is looking at problems that justify massive hardware (in the sense that solving them will pay back the capital and environmental cost) then you will have access to said hardware. Most (almost all) AI and Data Science teams are not working on that kind of problem though, and it's often the case that we are working on cloud infrastructures where GPU's and TPU's can be accessed on demand and $100's can be used to train pretty good models. Obviously models need to be trained many times so the crossover point from a few $100 to a few $1000 can be painful - but actually many/most engagements really only need models that cost <$100 to train. Also many of the interesting problems that are out there can utilize transfer learning over shared large pretrained models such as Resnet or GPT-2 (I know that in the dizzying paced modern world these no longer count as large or modern but they are examples...) So for images and natural language problems we can get round the intractable demand for staggeringly expensive compute. Imagine that you had got a degree in Aeronautical Engineering, you are watching the Apollo program and wondering how you will get a job at NASA or something similar... but there are lots of jobs at Boeing and Lockheed and Cessna and so on.

touisteur 5y ago

Most big labs have large computing clusters and upgrade them from time to time. We've almost always needed huge computing power in the scientific domain, no? Anyway these days I see a lot of industrial investment in middle size computing datacenters full of GPUs. Sure Google scale is not within reach but I'm sure there's room for scrapier algorithms and training methods to demonstrate feasibility and then paying a lump sum to AWS (or some ml-expert-for-cloud-training SME) for the final step. Anyway, I thought the expensive part was data acquisition and labeling? I like the 'surrogate network' approach of learning a very-expensive-to-compute simulation or model, that doesn't need data, but the output of a costly simulation.