4 ms·
I think the model does have short-term memory, but not long-term memory, if I'm understanding it correctly. From https://aibusiness.com/document.asp?doc_id=777
by upwardbound 4y ago
I think the model does have short-term memory, but not long-term memory, if I'm understanding it correctly.
From https://aibusiness.com/document.asp?doc_id=777540 https://aibusiness.com/document.asp?doc_id=777540 : "LaMDA2 was trained on Google’s Pathways Language Model (PaLM), which has 540 billion parameters."
And then looking at the PaLM paper https://arxiv.org/pdf/2204.02311.pdf https://arxiv.org/pdf/2204.02311.pdf on page 5 we have: "PaLM uses a standard Transformer model architecture (Vaswani et al., 2017) in a decoder-only setup (i.e., each timestep can only attend to itself and past timesteps)"