3 ms·

I don't think anyone serious thinks or talks in terms of AGI. The feverishly simplistic idea of the singularity is quite silly. Most notably, neural networks a

I don't think anyone serious thinks or talks in terms of AGI. The feverishly simplistic idea of the singularity is quite silly. Most notably, neural networks alone will not reach any kind of AGI. Start adding the capacity to read from massive knowledge stores, and a place to keep long term information (i.e., memory, probably also in a database), plus a feedback loop for the model to learn and improve? Plus the ability to call APIs? Now you're talking. I think all of those pieces are close to doable right now, maybe with a latency of 5s. If one of the big players puts that in place in a way that is well measured and they can iterate on, I think we'll start to see some really incredible advances.

hiddencost 4y ago
One really exciting place things might improve is in data cleaning. Right now preprocessing your data and putting it in a format that can be learned efficiently and without bias is a huge pain // risk. This next generation is allowing us to largely ignore a lot of that work. Similarly, transfer learning is finally good. And the models are generalist, few shot learners. As a consequence, individuals with minimal expertise can set up a world class system to solve niche problems. That's really exciting and it's going to get easier.
- hiddencost 4y ago
  Cross-language learning (what was referred to as an "interlingua" in the 90s) means we're seeing some stunning advances in low resource languages. It used to be that everyone ignored languages other than English, and then provides them with mediocre support. I think we're at a point where there's very little excuse not to launch in many languages at once.
localhost 4y ago
The gpt_index project looks very promising in this area. "At its core, GPT Index is about: 1. loading in external data (@NotionHQ, @Slack, .txt, etc.) 2. Building indices over that data 3. Inputting a prompt -> getting an output!" https://twitter.com/jerryjliu0/status/1608632335695745024 https://twitter.com/jerryjliu0/status/1608632335695745024
- amelius 4y ago
  Interesting. How are these indexes stored and how are they fed into the transformer model so that GPT can use them? Does this require an additional training step?
  - localhost 4y ago
    Based on my reading of the docs (I haven't looked at the code), it appears that it uses the existing langchain[1] (which I have used and is excellent) library for constructing longer prompts through a technique known as prompt chaining. So for example, summarizing long documents would involve a map/reduce style effort where you get the LLM to summarize chunks of the document (with some degree of overlap between chunks) and then getting the model to summarize the summaries. For answering "queries", it appears like it iterates over the documents in the store, i.e., NOT using it like an index, and feeding each document as part of the context into the LLM. [1] https://github.com/hwchase17/langchain https://github.com/hwchase17/langchain

hiddencost 4y ago

One really exciting place things might improve is in data cleaning. Right now preprocessing your data and putting it in a format that can be learned efficiently and without bias is a huge pain // risk. This next generation is allowing us to largely ignore a lot of that work. Similarly, transfer learning is finally good. And the models are generalist, few shot learners. As a consequence, individuals with minimal expertise can set up a world class system to solve niche problems. That's really exciting and it's going to get easier.

hiddencost 4y ago
Cross-language learning (what was referred to as an "interlingua" in the 90s) means we're seeing some stunning advances in low resource languages. It used to be that everyone ignored languages other than English, and then provides them with mediocre support. I think we're at a point where there's very little excuse not to launch in many languages at once.

localhost 4y ago

The gpt_index project looks very promising in this area. "At its core, GPT Index is about: 1. loading in external data (@NotionHQ, @Slack, .txt, etc.) 2. Building indices over that data 3. Inputting a prompt -> getting an output!" https://twitter.com/jerryjliu0/status/1608632335695745024 https://twitter.com/jerryjliu0/status/1608632335695745024

amelius 4y ago
Interesting. How are these indexes stored and how are they fed into the transformer model so that GPT can use them? Does this require an additional training step?
- localhost 4y ago
  Based on my reading of the docs (I haven't looked at the code), it appears that it uses the existing langchain[1] (which I have used and is excellent) library for constructing longer prompts through a technique known as prompt chaining. So for example, summarizing long documents would involve a map/reduce style effort where you get the LLM to summarize chunks of the document (with some degree of overlap between chunks) and then getting the model to summarize the summaries. For answering "queries", it appears like it iterates over the documents in the store, i.e., NOT using it like an index, and feeding each document as part of the context into the LLM. [1] https://github.com/hwchase17/langchain https://github.com/hwchase17/langchain