> Until NLP can create a short-term memory model Transformers can fit about 4000 tokens in the context window. That's a lot of dialogue rounds. The problem is

> Until NLP can create a short-term memory model Transformers can fit about 4000 tokens in the context window. That's a lot of dialogue rounds. The problem is these models are too expensive to offer at scale. We almost never get to see the latest AI in big corporation products. It is always AI from a couple generations ago that is cheaper to run. When we get the latest and greatest, like GPT-3 and DallE, then it's super expensive to use, not free.