3 ms·

He's not wrong. The key ingredient in ChatGPT is not brilliance but capital. It takes a lot of money not just to do the raw training but to get all the data tog

He's not wrong. The key ingredient in ChatGPT is not brilliance but capital. It takes a lot of money not just to do the raw training but to get all the data together and process everything. There's no break through insights that make ChatGPT work, just a lot of consolidated wealth. The hacker part of me finds AI less and less interesting for this reason. We're seeing what the limits of pouring resources into the problem are. The results are cool, but I think we'll very soon see progress bound since we're using most of the data we can and it doesn't look like adding even more parameters to these models might not yield that much of result. But this shift in AI means it's increasing something programmers consume rather than produce. As far as interesting AI goes, I found the series of posts on stuff people were doing with Prolog that showed up during the last month much more interesting.

woah 4y ago
There are a lot of things that require an amount of processing power that the average person doesn't yet possess. This doesn't mean that they aren't interesting. 50 years ago you would have said "these computer things aren't interesting, only big institutions and corporations have them".
sharemywin 4y ago
I find it the opposite. There were a lot of projects I didn't have the time to work on and got bored to fast learning. So, now I can tell chatgpt to build it for me. and I only have to debug the 1 or 2 things it can't do. I built a blogging platform, a voice based chatbot using gpt3, something that takes blog article and trims out all the garbage html so only the basic html is left. created a python api, and called it from shared hosting using php. something that trims the times out of youtube captions so I can post it into chatgpt to give me an outline. I just tell it what I want the function to do and it does it. It's helped me way up my own personal productivity and creativity.
visarga 4y ago
When language models run out of more trillions of words to train, there is one way ahead - we need to generate more. But wait, you might say, garbage in garbage out. It won't work. Normally it wouldn't, but we add an extra ingredient here. We get a validation signal. This is problem specific, but for code it would mean to integrate the LM with a compiler and runtime so it can iterate until it solves the task, step by step. For other tasks it could mean hooking the AI to simulators, games and robots to solve tasks. It is also possible to use LLMs as simulators of text. Basically doing Reinforcement Learning with a Language Model and not just for human preferences, but for problem solving on a grand scale. Collect data from problem solving, train on it, and iterate. It costs just electricity, but LLMs can make their own data. Anthropic's Constitutional AI which is RLAIF - reinforcement learning from AI feedback is proof it can be done.

woah 4y ago

There are a lot of things that require an amount of processing power that the average person doesn't yet possess. This doesn't mean that they aren't interesting. 50 years ago you would have said "these computer things aren't interesting, only big institutions and corporations have them".

sharemywin 4y ago

I find it the opposite. There were a lot of projects I didn't have the time to work on and got bored to fast learning. So, now I can tell chatgpt to build it for me. and I only have to debug the 1 or 2 things it can't do. I built a blogging platform, a voice based chatbot using gpt3, something that takes blog article and trims out all the garbage html so only the basic html is left. created a python api, and called it from shared hosting using php. something that trims the times out of youtube captions so I can post it into chatgpt to give me an outline. I just tell it what I want the function to do and it does it. It's helped me way up my own personal productivity and creativity.

visarga 4y ago

When language models run out of more trillions of words to train, there is one way ahead - we need to generate more. But wait, you might say, garbage in garbage out. It won't work. Normally it wouldn't, but we add an extra ingredient here. We get a validation signal. This is problem specific, but for code it would mean to integrate the LM with a compiler and runtime so it can iterate until it solves the task, step by step. For other tasks it could mean hooking the AI to simulators, games and robots to solve tasks. It is also possible to use LLMs as simulators of text. Basically doing Reinforcement Learning with a Language Model and not just for human preferences, but for problem solving on a grand scale. Collect data from problem solving, train on it, and iterate. It costs just electricity, but LLMs can make their own data. Anthropic's Constitutional AI which is RLAIF - reinforcement learning from AI feedback is proof it can be done.