3 ms·
That's an interesting hypothesis, but the work under discussion here didn't make an attempt to test it. You make two claims: (1) That using an LLM (Palm2) some
by jcgrillo 3y ago
That's an interesting hypothesis, but the work under discussion here didn't make an attempt to test it. You make two claims:
(1) That using an LLM (Palm2) somehow "saved time" by avoiding "nonsensical programs"
(2) That there exists a "cold start problem" (which, presumably, the LLM somehow solves).
Yet not the paper, nor the blog post, nor the code on github support these.
- famouswaffles 3y agoWhat paper are you reading exactly ? There's a clear comparison https://imgur.com/a/xerOOLn https://imgur.com/a/xerOOLn
- jcgrillo 3y agoSame paper you are. This is comparing apples to oranges. Their "random" strategy is only capable of assembling very constrained python programs whereas the LLM is coming up with things that look like the sort of stuff a person might write, utilizing the whole language (modulo hallucinatory nonsense). So it's not a reasonable comparison at all. Instead, for the purpose of making something resembling a reasonable comparison, it would make a lot more sense to train an LLM on programs written in the same language they allow their "random" strategy to use. I wouldn't recommend trying this in python, as it's way too big a language. Something smaller and more fit for purpose would likely be much more tractable. There are multiple examples from the literature to choose from. EDIT: To be clear--it's obvious the LLM solution outperformed the hand-rolled solution. By a large margin. What would actually be an interesting scientific question, though, is to what extent is the success attributable to the LLM? That's not possible to answer given the results, because the hand rolled-solution and the LLM solution are literally speaking different languages.
- janalsncm 3y agoThe purpose of the ablation is to demonstrate the value-add of using an LLM. Comparing to another type of candidate generator wouldn’t be ablation anymore, it would just be another study. As I have mentioned previously, using an LLM addresses the cold start problem by immediately generating plausible-looking programs. The reason random token selection is limited to constrained programs is that valid programs are statistically unlikely to occur. And the ablation directly demonstrates that the LLM is better than random.
- janalsncm 3y agoThe fact that LLMs generate text which is better than random is not something the authors need to spell out. Reducing the cross entropy between predicted and target distributions is the entire purpose of training. Put another way: a flat probability distribution from an untrained network is equivalent to the random decoding you’re skeptical an LLM is better than. That’s not a criticism the authors’ peers would likely put forward.
- YeGoblynQueenne 3y ago>> Reducing the cross entropy between predicted and target distributions is the entire purpose of training. Nope. Test distribution. Not target distribution.
- janalsncm 3y agoI’m not sure why you felt the need to correct me. Target is the correct terminology. As a sanity check I even looked up the PyTorch doc which explicitly uses “target” in their example. https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html https://pytorch.org/docs/stable/generated/torch.nn.CrossEntr...
- YeGoblynQueenne 3y agoBecause the target distribution is not known, whatever PyTorch docs say. The test distribution is known and nobody knows what's its relation to the target distribution except a hope and a prayer.