6 ms·
It will be interesting to see how long anyone can dominate. AI might actually result in actually the opposite. Meaning that the era of years and decades of domi
by 13years 4y ago
It will be interesting to see how long anyone can dominate. AI might actually result in actually the opposite. Meaning that the era of years and decades of dominance is not sustainable any longer.
Why would this be? Because AI allows you to copy your competitor in ways that was never possible before. It essentially becomes a skill and technology replicator. For example, look how Alpaca was able to somewhat replicate the multi million dollar models with $600 dollar cost.
- crop_rotation 4y agoBut Alpaca was possible only after Llama, and Llama is not gonna happen with 600 or 60000 USD.
- bogwog 4y agoI don't have the link, but someone estimated that training a Llama clone from scratch would cost around ~$80k. That's pocket change for most corporations and some individuals.
- nickthegreek 4y agoAnd low enough to be crowdsourced for the right project.
- acchow 4y agoThe data that you train your model on is certainly a huge moat...
- sacnoradhq 4y agoThe other moat is tech to capture feedback with actionable metadata from users.
- 13years 4y agoIt is a moat to be the first. The question is can you defend that moat.
- kmeisthax 4y agoYes, but you don't own that data, it's scraped off the Internet. Either anyone can scrape data and train an AI with it, or nobody can do that: there is no special privilege or advantage for OpenAI.
- weixiyen 4y agowhat about RLHF at scale?
- alchemist1e9 4y agoCould we discuss the datasets more? I’d really like to start acquiring them. I know about The Pile and Common Crawl. Are their any resources for individuals, that happen to have large amounts of storage, in regards to acquiring the best datasets being used?
- kmeisthax 4y agoThe Pile and Common Crawl are where everyone gets their data from. I did lie a bit; OpenAI does have private datasets. For example, GPT-3 was trained on Internet data, but InstructGPT/ChatGPT was further finetuned through their own dataset of question/answer pairs. This was augmented in a really clever way: the initial set of valid completions was written by humans, but then OpenAI made more AI models to augment their training data before finetuning GPT-3 on it. Even then, there's techniques for taking the output of one large language model and using it to finetune another. So just having a publicly-available AI chatbot is enough to get it to generate a dataset that you can then train another AI chatbot on. I personally have been scraping Wikimedia Commons for public-domain imagery to train an art generator bot on; but this is mostly because I want an art generator that isn't full of pending copyright infringement lawsuits.
- alchemist1e9 4y agoInteresting. I think the raw original source data would be good to have a copy of if possible and over time tools to curate and sample it should get better. What I can’t get really pinned down is how much storage one needs to have your own copy of all of the most valuable raw data. At initial estimate it seemed like less than 1PB was enough for sure if focusing on text and not images/video. It even looked like maybe 200-400 TB might have been more than enough. I need to investigate more. I need numbers like 900GB for pile, then 2.5TB for a meta curated text only common crawl. So I’m assuming there is at least 10 of these in same ballpark that are important to have a copy of. Any thoughts on the size requirements? Focusing on text.
- hiddencost 4y agoI'm excited for home assistant's voice efforts these years. Packaging speech and an assistant into open source hobby electronics could be incredibly powerful. https://www.home-assistant.io/blog/2022/12/20/year-of-voice/ https://www.home-assistant.io/blog/2022/12/20/year-of-voice/