3 ms·

CoT doesn't require it to be a few shot prompt though. That was in the original paper, as you point out, but even something as simple as a zero shot prompt foll

by kromem 3y ago

CoT doesn't require it to be a few shot prompt though. That was in the original paper, as you point out, but even something as simple as a zero shot prompt followed by "let's think step by step" has significant improvement: https://arxiv.org/abs/2205.11916 https://arxiv.org/abs/2205.11916 Do you still consider a zero-shot-CoT to be hinting? (And I don't mean the [0] example you shared - biasing results by continuing to ask for refinements in incorrect answers but not correct answers is an area where a number of papers have already had issues, but is not part of what I'm discussing - I mean in single prompt/response cycles). I agree that a consistently overlooked component of generators is their role as classifiers, though this currently seems to be overlooked more outside of research circles as all too common has become using GPT-4 to evaluate a study results in place of manual review (a bit disconcerting). I'm not entirely sure how we ended up on diffusion and GANs - I'd meant transmission from transformer to transformer like Orca or the work that followed. Though it was an interesting read nonetheless and I appreciated your writing it.

godelski 3y ago
Yeah the "zero shot" (why is everything "zero shot" these days and not matching the original definition?) CoT, I don't consider that hinting. If you're not giving models examples or leading it down the direction of the solution, the you're not hinting. Still lack of robustness but not that bad, especially since you can always append a user's prompt with things like "take a deep breath" or "think step by step" or "ELI5:" (prepend that one)[0]. But obviously I have issues with conclusions being taken from a lot of dataset results (see [0] and my other comment here [1]. Every time I ask or bring up [1] it gets left aside. Even in person. Not sure why...). Benchmarks are great and we need these tools, but I just think people are making too strong of conclusions from them. Plus, as implied earlier, I kinda don't give a shit about results when I have no information about the data you trained on. If I don't know the data, your results are meaningless to me (not necessarily the product, just benchmarks in papers[2]) especially as I see far too much data spoilage these days. Oh, yeah, I pushed into vision domain because I'm actually a vision researcher haha. So something I can talk about a bit better but of course these are all related. [0] I'm not sure I see too much discussion on this, but using internet speak really helps models. Models ate a lot of Reddit data and so likes those patterns like ELI5. But this is also where my critique about spoilage in things like LSAT/GRE/SAT/etc come from, because there are so many practice exam questions on Reddit. Same with a lot of homework problems and even code. The question is if it is robust outside these domains and it does not seem to be. Luckily that's a pretty large domain and also some of the most common things people want to know (I mean obviously since people ask more questions about things they want to know than not lol) [1] https://news.ycombinator.com/item?id=38091869 https://news.ycombinator.com/item?id=38091869 [2] I absolutely do not think we should be allowing papers into conferences when they exclusively use proprietary models and/or pretraining data. You can use those too, but you also gotta show results on publicly available data. Strong preference for making proprietary aspects of a work hidden till post review. I have a lot of respect for Carmen in this OR post because it demonstrates the dishonesty. If you don't know this is a google paper you're not qualified to review it, and if you do know, the review isn't blind. Not a great situation for our community that's suffering from high rates of noise and an increasing disdain for the review process (I for one have no faith left in it. Conference publication means nothing to me) https://openreview.net/forum?id=OpzV3lp3IMC&noteId=HXmrWV3lnh https://openreview.net/forum?id=OpzV3lp3IMC&noteId=HXmrWV3ln...

by kromem 3y ago

godelski 3y ago

Yeah the "zero shot" (why is everything "zero shot" these days and not matching the original definition?) CoT, I don't consider that hinting. If you're not giving models examples or leading it down the direction of the solution, the you're not hinting. Still lack of robustness but not that bad, especially since you can always append a user's prompt with things like "take a deep breath" or "think step by step" or "ELI5:" (prepend that one)[0]. But obviously I have issues with conclusions being taken from a lot of dataset results (see [0] and my other comment here [1]. Every time I ask or bring up [1] it gets left aside. Even in person. Not sure why...). Benchmarks are great and we need these tools, but I just think people are making too strong of conclusions from them. Plus, as implied earlier, I kinda don't give a shit about results when I have no information about the data you trained on. If I don't know the data, your results are meaningless to me (not necessarily the product, just benchmarks in papers[2]) especially as I see far too much data spoilage these days. Oh, yeah, I pushed into vision domain because I'm actually a vision researcher haha. So something I can talk about a bit better but of course these are all related. [0] I'm not sure I see too much discussion on this, but using internet speak really helps models. Models ate a lot of Reddit data and so likes those patterns like ELI5. But this is also where my critique about spoilage in things like LSAT/GRE/SAT/etc come from, because there are so many practice exam questions on Reddit. Same with a lot of homework problems and even code. The question is if it is robust outside these domains and it does not seem to be. Luckily that's a pretty large domain and also some of the most common things people want to know (I mean obviously since people ask more questions about things they want to know than not lol) [1] https://news.ycombinator.com/item?id=38091869 https://news.ycombinator.com/item?id=38091869 [2] I absolutely do not think we should be allowing papers into conferences when they exclusively use proprietary models and/or pretraining data. You can use those too, but you also gotta show results on publicly available data. Strong preference for making proprietary aspects of a work hidden till post review. I have a lot of respect for Carmen in this OR post because it demonstrates the dishonesty. If you don't know this is a google paper you're not qualified to review it, and if you do know, the review isn't blind. Not a great situation for our community that's suffering from high rates of noise and an increasing disdain for the review process (I for one have no faith left in it. Conference publication means nothing to me) https://openreview.net/forum?id=OpzV3lp3IMC&noteId=HXmrWV3lnh https://openreview.net/forum?id=OpzV3lp3IMC&noteId=HXmrWV3ln...