9 ms·
Score matching with Langevin Sampling: a new contender to GANs
- fxtentacle 6y agoIn short, this is a super cool approach to replace the discriminator in GANs with something that doesn't need to be trained and provably converges to the correct result. "256×256 images cannot be done reliably without 8 V100 GPUs or more!" That's quite sad because that means this approach is far out of reach for any hobby researcher and for most universities.
- deeeeeplearning 6y agoAWS has V100's available so most Universities with a decent budget should be able to swing this.
- belval 6y ago24$/hour for 8 V100 to be exact.
- sudosysgen 6y agoIt's interesting how renting 8 GPUs costing 7000$ each costs about the same as renting the services of the average US worker.
- smabie 6y ago24/hour comes to a salary of $49,920, working 8 hours a day, 5 days a week, no vacation, no holidays. Meanwhile the GPUs cost $56,000 total. So the numbers aren't really very far off.
- sudosysgen 6y agoThey kind of are, these GPUs last multiple years. Anyways, it's just an interesting observation.
- smabie 6y agoI mean you also have to take into account electricity, which I assume is pretty expensive.
- sudosysgen 6y agoIt's only about 2.5kW, so around 1-2$ per hour for the full system
- notsuoh 6y agoNot even that, spot pricing on an 8 GPU instance (the 16xl I believe, the larger one has the same number of GPUs but more memory per GPU) is something like $6/hr. I use this for personal projects sometimes, get all data in S3, a good launch template, and then spin up a spot instance and be super efficient about training quickly. I've even run evals on a separate, cheaper, machine so the 16xl can spend all its time training. It's still not "cheap", but $50 for 8 hours of training on a machine like that with $64k of GPUs on board is really not bad.
- bravura 6y agoIf you wrote a blog post about how to streamline this, I would read it and upvote it.
- fxtentacle 6y agoExcept that cloud-ified V100s are significantly less powerful than if you have direct access to the hardware. Last time I checked, in AWS they're actually external devices mapped in over GBit ethernet, which is significantly slower than the 8GB/s that PCIe x4 has.
- p1esk 6y agoI routinely switch between AWS 8x V100 instances and on-premise 8x V100 servers and I observe no difference in speed (time per epoch).
- Reelin 6y agoPresumably that depends on maximum PCIe bandwidth consumption before your workload bottlenecks elsewhere? A 2018 benchmark (https://www.pugetsystems.com/labs/hpc/PCIe-X16-vs-X8-with-4-x-Titan-V-GPUs-for-Machine-Learning-1167/ https://www.pugetsystems.com/labs/hpc/PCIe-X16-vs-X8-with-4-...) seems to indicate that x8 isn't generally a bottleneck for common (at the time) workloads. x8 is a far cry from the claimed gigabit ethernet though!
- p1esk 6y agoAWS is tricky in terms of how storage is provisioned - I don't remember details, but it's easy to put your datasets on storage that is connected to your GPU servers over 1Gb link. That could easily become a bottleneck. Datasets should live on Elastic Block Storage or something like that, over high speed links. Again, it's been a while since I looked into that, so I don't remember the details.
- Reelin 6y agoThe earlier comment claimed that the GPUs (!!!) were located elsewhere on the network; I suspect that the scenario you describe is what they intended to refer to. (IIRC AWS offers compute optimized instances with a volume that's guaranteed to be backed by blocks on a local NVMe drive.)
- marmaduke 6y agoI just checked (my university account with Dell) and a dual Quadro RTX8000 (so 96 GB GPU memory) is around 10k. A V100 is 32 GB so so a 6xRTX8000 could probably cut it at price of 30k. It’s not cheap but the price of a six month of a post doc in my country.
- alpineidyll3 6y agoI think there's a pretty high chance that tricks could be brought to bear to greatly reduce the costs of what's being done here (by OOMs). There's a lot of handles for tweaking. In particular the basic crux of this approach is a Monte-Carlo annealing of the score function. In every field of science, Monte-Carlo sampling entails a huge pre-factor cost, and in every case it only reaches it's greatest potential with importance sampling, which hasn't been applied here. This is basically the 'brute-force' version of some future approach which would replace the diffusion kernel with another process that allows one to avoid sampling huge volumes of function space which are irrelevant for your desired P(X). This would introduce dependence of the training process on the sampling pre-conditioner, but ultimately be required for highest performance. I suspect that these authors are already thinking of how to use invertible flows to this effect.
- fxtentacle 6y agoThe way I read it, they actually need the randomness introduced from full Monte-Carlo sampling to make sure that they explore the data distribution sufficiently far in every direction. So it might be that for this approach, importance sampling would distort the results and re-introduce problems like the chance for divergence that GANs are fighting with.
- alpineidyll3 6y agoThe miracle of importance sampling is that it doesn't formally ever prevent you from visiting the rare regions, it just might take longer. What can occur is that you will get what appears to be a well-converged estimate of an answer but really you weren't fully ergodic. Such is the curse of dimensionality. No free lunch theorem, yadda yadda.
- salty_biscuits 6y agoI've always had this idea that you could do a dumb version of annealing by just progressively increasing the size of samples used in the minibatches. SGD gives you noise on the gradient which should get smaller as you increase the batch size. If you just adjusted the batch size with some sort of MH accept/reject logic you should get a better optimzer for mode seeking than just vanilla SGD. Like all my ideas though it has probably been thought before or is just wrong :) too damn lazy to try it as well.
- twanvl 6y agoThis is not just replacing the discriminator. In this DSM-ALS method you learn a score function instead of a generator, and to generate a sample you need to evaluate the score function multiple times (it essentially gives you a direction to move in).
- scoopertrooper 6y agoOracle clouds are offering 8 A100s for $3.05 an hour! I can only assume it's some horrible trap, but it sounds like crazy good value! https://www.oracle.com/corporate/pressrelease/oracle-cloud-infrastructure-integrates-with-nvidia-a100-051420.html https://www.oracle.com/corporate/pressrelease/oracle-cloud-i...
- rajnathani 6y agoNo, they've listed "$3.05" per GPU. For 8 A100s that would be 8x$3.05 = $24.4/hour.
- scoopertrooper 6y agoAh missed that.
- toxik 6y agoI wish this essay wasn’t littered with emojis :(
- scott31 6y agoSame
- aperrien 6y agoI learned something new and valuable today. The emoji make no difference at all to me, as they do not affect the quality of what I have learned by any discernible measure.
- notsuoh 6y agoIf Geoff Hinton or Francois Chollet add emojis to their writing, it is visually less pleasing in my opinion but I agree with you overall. When it's someone I don't know though, it makes me trust the writing less because I can't strictly verify the contents of what is being discussed easily for work like this, so it does make a difference because I guess it's less "professional" in a way.
- sheikheddy 6y agoI've noticed it more and more in the ML research community, I think it's mostly the influence of twitter and medium articles. For a blog it really is fine though and I'm comfortable with language evolving in an imprecise fashion as long as the emojis don't try to do much more than add flavor.
- grp000 6y agoMost math heavy content is written very drily. Seeing emojis makes reading it feel more comfy and human. If someone maps the greek alphabet to emojis, you could enjoy the integration on a deeper, more intimate level. The math of emotion!
- nl 6y ago
- Fede_V 6y agoAlexia is one of the most original researchers working on GANs: she invented the relativistic discriminator (a brilliant and obvious idea in hindsight: https://arxiv.org/abs/1807.00734 https://arxiv.org/abs/1807.00734) which is one of the easiest tweaks you can make to instantly boost your GAN results.
- hyperbovine 6y agoThe better the idea, the more obvious it was in hindsight.
- derefr 6y agoThere's gotta be a better word for this than "obvious." The key thing about ideas that are "brilliant and obvious in hindsight" is that the world was already ready for them, and so nothing needed to change for them to happen; i.e., they don't have any prerequisites that aren't already in place. They "just" needed someone to actually notice that there were some pieces that could be fit together in a novel way. There's no word I know of that captures this idea of "the world being ready for" the idea, though. Is the idea "incremental" in hindsight? "Elegant" in hindsight? "Free" in hindsight?
- franklampard 6y agoIntuitive?
- derefr 6y agoThat's very close! But something that's intuitive in hindsight might still have required some hard work in development, rather than "just" a brilliant, novel idea. Photolithography, for example, is intuitive, and far simpler as a technique for constructing circuit boards than what came before it; but etching circuits using light projected onto light-activated chemicals isn't one of these "the world was ready for it, someone just needed to do it" ideas. Someone thought of it, then needed to do a whole lot of work to get it to happen, finding the right chemicals, experimenting with projection technologies, etc. After the fact, the idea of photolithography is extremely intuitive; but it wasn't a better-term-for-"obvious in retrospect" idea.
- rjeli 6y agoReminds me of JEMs from Grathwohl et al last year [0]. They train a generative model by treating it an an Energy Based Model using stochastic langevin gradients. I’m curious as to how it relates to the models in this post. [0] https://arxiv.org/abs/1912.03263 https://arxiv.org/abs/1912.03263
- natch 6y agoSomeday the field will figure out that not all images of interest are squares. That will be a great advance. I realize some people have hacked up personal branches of projects to support non-square rectangles but it really needs to become mainstream or else we’re going to stay stuck in this “AI square winter.”
- p1esk 6y agoI don't get it - is there anything preventing you using non-square images today?
- janhenr 6y agoNo, in production rectangle images are reshaped and resized to square ones.
- ebalit 6y agoIf you want to be able to run inference in batches, you have to normalize to a given input size. It often means a square input size if you want to minimally distort both vertical and horizontal images. You can also use variable input shape in production if you need to. It will be harder to optimize, especially for throughput, but it's not impossible.
- natch 6y agoJust the massive inconvenience of having to track down and implement the changes necessary to add non-square image support on most platforms. So if you think that is a minor thing that can be dismissed as nothing, then no. Or one can always crop, or resize, both of which throw away information, or pad, for which the underlying behavior is undocumented. I just don’t think any of these are great options. Do you? If the answer is “you can always write your own” that is true, but it’s just underlining my point that the problem is not yet solved.
- p1esk 6y agosupport on most platforms What do you mean by "platforms"?
- GistNoesis 6y agoFrom the Pascal Vincent link : "Beware that this usage differs slightly from traditional statistics terminology where score usually refers to the derivative of the log likelihood with respect to parameters,whereas here we are talking about a score with respect to the data."