5 ms·
In short, this is a super cool approach to replace the discriminator in GANs with something that doesn't need to be trained and provably converges to the correc
by fxtentacle 6y ago
In short, this is a super cool approach to replace the discriminator in GANs with something that doesn't need to be trained and provably converges to the correct result.
"256×256 images cannot be done reliably without 8 V100 GPUs or more!"
That's quite sad because that means this approach is far out of reach for any hobby researcher and for most universities.
- deeeeeplearning 6y agoAWS has V100's available so most Universities with a decent budget should be able to swing this.
- belval 6y ago24$/hour for 8 V100 to be exact.
- sudosysgen 6y agoIt's interesting how renting 8 GPUs costing 7000$ each costs about the same as renting the services of the average US worker.
- smabie 6y ago24/hour comes to a salary of $49,920, working 8 hours a day, 5 days a week, no vacation, no holidays. Meanwhile the GPUs cost $56,000 total. So the numbers aren't really very far off.
- sudosysgen 6y agoThey kind of are, these GPUs last multiple years. Anyways, it's just an interesting observation.
- smabie 6y agoI mean you also have to take into account electricity, which I assume is pretty expensive.
- sudosysgen 6y agoIt's only about 2.5kW, so around 1-2$ per hour for the full system
- notsuoh 6y agoNot even that, spot pricing on an 8 GPU instance (the 16xl I believe, the larger one has the same number of GPUs but more memory per GPU) is something like $6/hr. I use this for personal projects sometimes, get all data in S3, a good launch template, and then spin up a spot instance and be super efficient about training quickly. I've even run evals on a separate, cheaper, machine so the 16xl can spend all its time training. It's still not "cheap", but $50 for 8 hours of training on a machine like that with $64k of GPUs on board is really not bad.
- bravura 6y agoIf you wrote a blog post about how to streamline this, I would read it and upvote it.
- fxtentacle 6y agoExcept that cloud-ified V100s are significantly less powerful than if you have direct access to the hardware. Last time I checked, in AWS they're actually external devices mapped in over GBit ethernet, which is significantly slower than the 8GB/s that PCIe x4 has.
- p1esk 6y agoI routinely switch between AWS 8x V100 instances and on-premise 8x V100 servers and I observe no difference in speed (time per epoch).
- Reelin 6y agoPresumably that depends on maximum PCIe bandwidth consumption before your workload bottlenecks elsewhere? A 2018 benchmark (https://www.pugetsystems.com/labs/hpc/PCIe-X16-vs-X8-with-4-x-Titan-V-GPUs-for-Machine-Learning-1167/ https://www.pugetsystems.com/labs/hpc/PCIe-X16-vs-X8-with-4-...) seems to indicate that x8 isn't generally a bottleneck for common (at the time) workloads. x8 is a far cry from the claimed gigabit ethernet though!
- p1esk 6y agoAWS is tricky in terms of how storage is provisioned - I don't remember details, but it's easy to put your datasets on storage that is connected to your GPU servers over 1Gb link. That could easily become a bottleneck. Datasets should live on Elastic Block Storage or something like that, over high speed links. Again, it's been a while since I looked into that, so I don't remember the details.
- Reelin 6y agoThe earlier comment claimed that the GPUs (!!!) were located elsewhere on the network; I suspect that the scenario you describe is what they intended to refer to. (IIRC AWS offers compute optimized instances with a volume that's guaranteed to be backed by blocks on a local NVMe drive.)
- marmaduke 6y agoI just checked (my university account with Dell) and a dual Quadro RTX8000 (so 96 GB GPU memory) is around 10k. A V100 is 32 GB so so a 6xRTX8000 could probably cut it at price of 30k. It’s not cheap but the price of a six month of a post doc in my country.
- alpineidyll3 6y agoI think there's a pretty high chance that tricks could be brought to bear to greatly reduce the costs of what's being done here (by OOMs). There's a lot of handles for tweaking. In particular the basic crux of this approach is a Monte-Carlo annealing of the score function. In every field of science, Monte-Carlo sampling entails a huge pre-factor cost, and in every case it only reaches it's greatest potential with importance sampling, which hasn't been applied here. This is basically the 'brute-force' version of some future approach which would replace the diffusion kernel with another process that allows one to avoid sampling huge volumes of function space which are irrelevant for your desired P(X). This would introduce dependence of the training process on the sampling pre-conditioner, but ultimately be required for highest performance. I suspect that these authors are already thinking of how to use invertible flows to this effect.
- fxtentacle 6y agoThe way I read it, they actually need the randomness introduced from full Monte-Carlo sampling to make sure that they explore the data distribution sufficiently far in every direction. So it might be that for this approach, importance sampling would distort the results and re-introduce problems like the chance for divergence that GANs are fighting with.
- alpineidyll3 6y agoThe miracle of importance sampling is that it doesn't formally ever prevent you from visiting the rare regions, it just might take longer. What can occur is that you will get what appears to be a well-converged estimate of an answer but really you weren't fully ergodic. Such is the curse of dimensionality. No free lunch theorem, yadda yadda.
- salty_biscuits 6y agoI've always had this idea that you could do a dumb version of annealing by just progressively increasing the size of samples used in the minibatches. SGD gives you noise on the gradient which should get smaller as you increase the batch size. If you just adjusted the batch size with some sort of MH accept/reject logic you should get a better optimzer for mode seeking than just vanilla SGD. Like all my ideas though it has probably been thought before or is just wrong :) too damn lazy to try it as well.
- twanvl 6y agoThis is not just replacing the discriminator. In this DSM-ALS method you learn a score function instead of a generator, and to generate a sample you need to evaluate the score function multiple times (it essentially gives you a direction to move in).
- scoopertrooper 6y agoOracle clouds are offering 8 A100s for $3.05 an hour! I can only assume it's some horrible trap, but it sounds like crazy good value! https://www.oracle.com/corporate/pressrelease/oracle-cloud-infrastructure-integrates-with-nvidia-a100-051420.html https://www.oracle.com/corporate/pressrelease/oracle-cloud-i...
- rajnathani 6y agoNo, they've listed "$3.05" per GPU. For 8 A100s that would be 8x$3.05 = $24.4/hour.
- scoopertrooper 6y agoAh missed that.