8 ms·

This has become increasingly important to me too. I am employed by a small (2-4 engineers at any time) company and I'm often disappointed because we're just so

by rychco 4y ago

This has become increasingly important to me too. I am employed by a small (2-4 engineers at any time) company and I'm often disappointed because we're just so far behind in manpower & technical expertise that we have to dramatically reduce the scope of any problem we want to tackle. On the other hand, I also worry about getting sucked into the bureaucracy of FAANG sized companies & not having any accountability or agency over what I work on. (I realize this is a sweeping generalization of FAANG, but some of my peers have had this experience even a few years into their jobs)

fxtentacle 4y ago
I'm surprised. Apart from DALLE, I haven't seen any AI approach that's off limits for 4 highly motivated people with 3090 GPUs. At that compute level, you should be able to at least replicate SOTA in optical flow, structure from motion, speech recognition, text to speech, translation, text summary, sentiment analysis, image classifications, image segmentation, and of course playing video games or optimizing processes with reinforcement learning. I mean thanks to KiCad even custom sensor hardware is cheap these days. Can you give more details about what you tried to do and why that wasn't possible?
- sigmoid10 4y ago
  Pretty much everything in the high end large language model area is off limits to people without access to a supercomputer (we're talking hundreds of A100s or several $100k in cloud computing equivalent). Open Source efforts like BigScience may open up downstream tasks for normal people, but the forefront of this research is no longer accessible to individuals.
  - fxtentacle 4y ago
    You might be surprised to hear that the KenLM language models that are used for speech recognition are actually trained on-disk using CPU. With a €149 monthly bare metal server, I could train my own LM on OSCAR DE and EN. Where I do agree with you is that transformer-style text generation models in the billion parameter range are off-limits for hobbyists. But that's only a tiny part of the useful applications of AI. And you can train them with gradient checkpointing, it's just 100x slower than what Google can do.
    - sigmoid10 4y ago
      KenLM ist not a neural network and instead a purely statistical n-gram model. So it's no surprise that it would be faster on a CPU in many cases. However, as soon as you have to deal with noisy data, KenLM gets blown out of the water by DL architectures like LSTM and, more recently, Transformers. There's a reason why purely statistical models have seen very little progress in the last 10 years (KenLM was published 11 years ago) and that reason is that this "noise" is basically just a consequence of the central limit theorem applied to data with a huge amount of nuance - much more than any human coded feature vector could ever account for.
- cellis 4y ago
  If you want to try to train a StyleGan for instance with image sizes of 1024 to acceptable quality, you need either a lot of GPUs or a lot of time, or both.
  - fxtentacle 4y ago
    That's an oddly specific example, because I was in fact training my own StyleGAN and using it to sell T-Shirts around 2018 as indivicia.com . A GTX 1080 TI was good enough for 300 DPI A4 prints (roughly 3500px on the longer edge) and I just trained a regular low-res StyleGAN model and then a styled 4x upscaler. Execution was hand-coded multicore C++ and took about a second per user upload.
    - cellis 4y ago
      Very cool usage of StyleGan! I think it’s a great idea but maybe allow poster prints? I wouldn’t put a lot of these on a shirt but family pics or nature photos would be nice to have styleized. But back to my original point: training stylegan takes a lot of resources at higher resolutions. For something like thispersondoesnotexist on stylegan 3 you not only need to have lots of quality data, you also need many cards for many days.
- rychco 4y ago
  Ah I wasn’t being clear, you’re actually correct when it comes to ML (which we use from time to time). My comment on the lack of manpower was mostly in regards to pure programming output or research efforts.

by rychco 4y ago

fxtentacle 4y ago

I'm surprised. Apart from DALLE, I haven't seen any AI approach that's off limits for 4 highly motivated people with 3090 GPUs. At that compute level, you should be able to at least replicate SOTA in optical flow, structure from motion, speech recognition, text to speech, translation, text summary, sentiment analysis, image classifications, image segmentation, and of course playing video games or optimizing processes with reinforcement learning. I mean thanks to KiCad even custom sensor hardware is cheap these days. Can you give more details about what you tried to do and why that wasn't possible?

sigmoid10 4y ago
Pretty much everything in the high end large language model area is off limits to people without access to a supercomputer (we're talking hundreds of A100s or several $100k in cloud computing equivalent). Open Source efforts like BigScience may open up downstream tasks for normal people, but the forefront of this research is no longer accessible to individuals.
- fxtentacle 4y ago
  You might be surprised to hear that the KenLM language models that are used for speech recognition are actually trained on-disk using CPU. With a €149 monthly bare metal server, I could train my own LM on OSCAR DE and EN. Where I do agree with you is that transformer-style text generation models in the billion parameter range are off-limits for hobbyists. But that's only a tiny part of the useful applications of AI. And you can train them with gradient checkpointing, it's just 100x slower than what Google can do.
  - sigmoid10 4y ago
    KenLM ist not a neural network and instead a purely statistical n-gram model. So it's no surprise that it would be faster on a CPU in many cases. However, as soon as you have to deal with noisy data, KenLM gets blown out of the water by DL architectures like LSTM and, more recently, Transformers. There's a reason why purely statistical models have seen very little progress in the last 10 years (KenLM was published 11 years ago) and that reason is that this "noise" is basically just a consequence of the central limit theorem applied to data with a huge amount of nuance - much more than any human coded feature vector could ever account for.
cellis 4y ago
If you want to try to train a StyleGan for instance with image sizes of 1024 to acceptable quality, you need either a lot of GPUs or a lot of time, or both.
- fxtentacle 4y ago
  That's an oddly specific example, because I was in fact training my own StyleGAN and using it to sell T-Shirts around 2018 as indivicia.com . A GTX 1080 TI was good enough for 300 DPI A4 prints (roughly 3500px on the longer edge) and I just trained a regular low-res StyleGAN model and then a styled 4x upscaler. Execution was hand-coded multicore C++ and took about a second per user upload.
  - cellis 4y ago
    Very cool usage of StyleGan! I think it’s a great idea but maybe allow poster prints? I wouldn’t put a lot of these on a shirt but family pics or nature photos would be nice to have styleized. But back to my original point: training stylegan takes a lot of resources at higher resolutions. For something like thispersondoesnotexist on stylegan 3 you not only need to have lots of quality data, you also need many cards for many days.
rychco 4y ago
Ah I wasn’t being clear, you’re actually correct when it comes to ML (which we use from time to time). My comment on the lack of manpower was mostly in regards to pure programming output or research efforts.