4 ms·
“Content” requires as much, if not more, effort and expense than pretraining GPT-4. All you’re doing is redefining content, ie thoughts, ideas, movies, videos,
by addicted 3y ago
“Content” requires as much, if not more, effort and expense than pretraining GPT-4.
All you’re doing is redefining content, ie thoughts, ideas, movies, videos, literature, sounds, writing, etc as “raw data”. But that isn’t raw data. There was a ton of effort that went into creating the “content”. For example, a single Wikipedia page may have many hundreds of people, some who have done years of college level studies and original research, to produce a few thousand words of content. Others have done research using primary sources. All of them have had to use effort and ingenuity to craft those into actual high quality statements, which in itself was only possible in many cases due to years of training and education. Finally, they had to setup a validation process to produce useful output from this collaborative process which included loads of arguments etc to generate what you are calling “raw data”.
I’m not sure what makes GPT’s output is any less raw than all the effort that went into producing a single Wikipedia page? Further, Wikipedia actually goes out of its way to cite its sources. GPT is designed to go out of its way to obscure its sources.
The only thing GPT does, IOW, that apparently makes the data it uses is not to cite its sources, something that would at the very least lead to professional disgrace for the people who created the “raw data” GPT uses without thought, and would even lead to lawsuits and prosecution in many cases.
So besides going out of its way to obscure the source of its data, what makes GPT’s output less raw than the output people have spent billions of man hours creating?
- ta988 3y agoExcept that the content already exists and there is no cost to maintain it. If GPT incurred a non negligible cost on the content owners by accessing their resources it may have been different but that's not the case. The only thing that content owners may be able to complain about is that potentially ChatGPT/DallE may reduce their potential income and this would have to be proven. I have not stopped buying books or art of any kind since I use ChatGPT/DallE. And low quality automated content producers existed before OpenAI and were already diluting the attention to more carefully produced content (as can be seen with videos on youtube).
- xiphias2 3y agoIt seems like you have no idea how much effort it takes to write a book. Quite often it contains the experience of a life of a person condensed to a few hundred pages. ChatGPT gives easier access to the knowledge contained in tens of thousands of these books. As for me I have been reading less and less books as more wisdom is accessable on the internet in better forms (now GPT). I'm not against what OpenAI is doing as it moves humanity forward, but like you said I won't stop using ChatGPT just because ByteDance scrapes it.
- dougb5 3y agoThat's great to hear that there's no cost to maintaining content! I'll tell AWS they've been overcharging me :)
- ta988 3y agoNot what I am saying. I am saying it is much much smaller than inference/model running cost. Easy exercise How many books do you store in 1GB How much does it cost a year to store it and have OpenAI gather it once. How much does it cost to run a GPT4 level model that will output 1GB. That's my point here that's all. It is a huge cost for OpenAI to run a system that produces dynamic content. And it is not comparable to the cost of storing static content. I didn't talk about the cost of producing the original data. And I do not talk about training costs.
- dougb5 3y agoSure, but your comment said "maintain", not "store". Even if storage were free, and even if you discount the value of the initial creation to zero, there are still nontrivial serving costs associated with many sites. What I share with people on the Web may look like a static byte sequence to the robots consuming it, but it takes a lot of work to compute those bytes (in the moment, I mean). Aggregated over the whole web, no, that is not smaller than OpenAI's expenditures.
- leereeves 3y agoIf cost is your primary concern, shouldn't you support ByteDance's efforts to reduce inference costs by distilling the model? (while at the same time reducing future costs for everyone by distributing the capability more widely to prevent monopolization)