2 ms·
I have similar sentiment and looking at how mixed takes are, I think it depends on what you do. I write a lot of research code so I think it's unsurprising that
by godelski 3y ago
I have similar sentiment and looking at how mixed takes are, I think it depends on what you do. I write a lot of research code so I think it's unsurprising that GPT isn't too good here. But people I see that write code more in line with the "copy and paste from stackoverflow" style, get huge utility out of this. (This isn't a dis on that style, lots of work is repetitive and redundant).
So I changed how I use GPT (which I do through API. Much cheaper btw). I use it a lot like how I would use SO in the first place. Get outlines, understand how certain lines might work (noisy process here), generate generic chunks of code especially from modules I'm unfamiliar with. A lot of this can just be seen as cutting down time searching.
So, the most useful one: using it as a fuzzy search to figure out how to Google. This one is the most common pattern. Since everything on Google is so SEO optimized and Google clearly doesn't give a shit, I can ask GPT a question, get a noisy response that contains useful vernacular or keywords which I can then use to refine a Google search and actually filter out a decent amount of shit. I think people might read this comment and think that you should just build a LLM into Google, but no, what's going on is more complicated and requires the symbiosis. GPT is dumb, doesn't have context, but is good at being a lossy compression system. The whole reason this works is because I'm intelligent and __context aware__, and importantly, critical of relying on GPT's accuracy[0]. Much of this can't be easily conveyed to GPT and isn't just a matter of token length. So that said, the best way to actually improve this system is actually for Google to just get its shit together or some other search engine to replace them. Google, if you're listening, the best way you can make Google search better with LLMs is to: 1) stop enabling SEO bullshit, 2) throw bard into the side and have the LLM talk to you to help you refine a search. Hell, you can use a RL agent for 1 to just look how many times I back out from the links you send me or look at which links I actually use. Going to page 2 is a strong signal that you served shit.
[0] accuracy is going to highly depend on frequency of content. While they dedupe data for training, they don't do great semantic deduping (still an unsolved problem. Even in vision). So accuracy still depends on frequency and you can think of well known high frequency knowledge as having many different versions, or that augmentation is built in. You get lower augmentation rates with specific or niche expert knowledge as there's little baked in augmentation and your "test set" is much further from the distribution of training data.