gyang

7 ms·

1.
by gyang 5y ago
I think the concept makes sense. The basic insight, that the right batch size depends on the difficulty and noisiness of a task, is already used by teams. For example, the PaLM paper from last week increased its batch size throughout traini
2.
by gyang 5y ago
I think there remains an immense amount of such suboptimality still hanging from the tree, so to speak. For example, our recent paper "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer"[1] shows
3.
Tuning GPT-3 on a single GPU(twitter.com)
4 points by gyang 5y ago|0 comments
4.
On infinitely wide neural networks that exhibit feature learning(microsoft.com)
3 points by gyang 5y ago|0 comments
5.
Wide Neural Networks Are Feature Learners, Not Kernel Machines(arxiv.org)
5 points by gyang 6y ago|0 comments
6.
Validation loss is not a reliable signal for early stopping(github.com)
1 points by gyang 6y ago|0 comments
7.
In ML, when loss decreases, accuracy increases, right?(twitter.com)
1 points by gyang 6y ago|0 comments
8.
Provably Robust Deep Learning(decentdescent.org)
1 points by gyang 7y ago|0 comments
9.
by gyang 7y ago
Right, so your argument would work if you allow the layer widths to tend to infinity sequentially (so this corresponds to finite networks where each previous layer is much bigger than the next layer). This is the argument presented by Lee e
10.
by gyang 7y ago
Hi Radford, I'm very happy that this paper got your attention, but I'm regretful that I did not represent your research accurately! Indeed, in your thesis, section 2.3 is on "Priors for networks with more than one hidden laye
11.
by gyang 7y ago
I think in general, "training" a GP, i.e. doing GP inference (or kernel regression) is not done for speed reasons, but rather because they are sample efficient . More concretely, the practical folk wisdom regarding GPs is that wh
12.
by gyang 7y ago
Hi @zwaps, thanks so much for your interest! My answer to @throwlaplace seems relevant to your question, so let me copy it here and comment on your specific questions afterward. > The CLT would be a good guess at approaching this problem
13.
by gyang 7y ago
@fgabriel mentioned this below: if the network is parametrized in a certain way, then the GP evolves according to a linear equation (if trained with square loss). In this linear equation, a different kernel shows up, known as the Neural Tan
14.
by gyang 7y ago
Yes. I will have things to say about training, but that requires building up some theoretical foundations. This paper is the first step in laying it out. Stay tuned! :)
15.
by gyang 7y ago
Hi, author here. Thanks for your interest! The CLT would be a good guess at approaching this problem, and indeed it is the approach of prior works [1][2]. But in this paper, the key answer is actually law of large numbers, though CLT would
16.
by gyang 7y ago
Hi, the author here. Thanks for your interest! Let me try answering some of your questions. > This sounds important and interesting but isn't wide the key word here? Yes, width is very important for this result. Given the size of
17.
Wide Neural Networks of Any Architecture Are Gaussian Processes(arxiv.org)
144 points by gyang 7y ago|34 comments
18.
by gyang 9y ago
Hi, I'm the lead author on this paper. If you have any questions please feel free to ask!
19.
Lie-Access Neural Turing Machines: Manifolds as Memory Storage(lstm.seas.harvard.edu)
10 points by gyang 9y ago|1 comments