5 ms·

Mine: In November 2023, a surprise AI model will be release that is 10x as powerful as GPT-4, and consumes 5x less resources. One individual will be able to ru

• ActorNightly 3y ago
  Highly doubtful. Transformers are at their limit. In general, first there was simply (weight * value) . Then people experimented with a trainable parameter that would select a set of weights for a given input. Transformers essentially turned this 1 dimentional discrete selection into a n dimentional multiplication, where the weights get modified by a product of values and a trainable parameter, i.e (weight * value) * value To get further, 2 things need to happen. First, there needs to be investigation into being able to learn the network architecture itself. For example, what if the transformer learned parameters were instead computed at inference time? I.e ( ( weight * value ) * value ) * value. What if this was abstracted to some form where there was a learned parameters on where to get those values from? Secondly, there needs to be some differentiable method of storing data between inference runs, so instead of input->output, you have input+memory -> output.
• satvikpendem 3y ago
  > 1 month later the first pixar scale movie is released, made from a guys basement. I give it another year or two, RunwayML's models are good but not as good as they could be yet.