4 ms·

The GPT architecture can't really produce a 300+ page report right now, because that doesn't even come close to fitting within its window. Of course you can jus

by jerf 4y ago

The GPT architecture can't really produce a 300+ page report right now, because that doesn't even come close to fitting within its window. Of course you can just keep running it and concatenating, but I wouldn't really call that a 300+ page report. It'll either wander off topic completely once it loses the prompt, or be very repetitive because it'll have no idea whatsoever what it said before its window. I gather that with the current architecture it becomes exponentially more difficult to grow the window, but I'm not sure if that's what a computer scientist would call "exponential" or if it was the common colloquial usage that calls x^2 "exponential growth"; would actually be interested if anyone could clarify that. With the current expense though, any of the common complexity classes beyond O(n log n) isn't very feasible at the moment though.

famouswaffles 4y ago
Attention used to be quadratic. But there have been linear attention advances. In fact, gpt-4 is almost certainly using some linear attention variant, maybe flash attention. Then there's this https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning https://hazyresearch.stanford.edu/blog/2023-03-27-long-learn... I don't think a 300 page context window is too far away
ModernMech 4y ago
You can prompt it for an outline, and then prompt it to go section by section. It’s a lot more work than “write 300 pages!” It’s actually more like working with a junior copy editor that you have to constantly correct, so you keep wondering what you’re even paying them for. But at $20 a month it’s worth it for that.
firatsarlar 4y ago
I believe LLMs could generate limitless page reports. Not a single prompt of course. Pick a topic, outline it, grow the outline with subsections. Use a generic summary and LLM indexing to feed context every time you want a subsection, if you want do not stop there use text to image to add drawings, you have 1000 pages thing. Yes, currently it is a thing not an article or research paper. Simple tool and intention relation.

by jerf 4y ago

famouswaffles 4y ago

Attention used to be quadratic. But there have been linear attention advances. In fact, gpt-4 is almost certainly using some linear attention variant, maybe flash attention. Then there's this https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning https://hazyresearch.stanford.edu/blog/2023-03-27-long-learn... I don't think a 300 page context window is too far away

ModernMech 4y ago

You can prompt it for an outline, and then prompt it to go section by section. It’s a lot more work than “write 300 pages!” It’s actually more like working with a junior copy editor that you have to constantly correct, so you keep wondering what you’re even paying them for. But at $20 a month it’s worth it for that.

firatsarlar 4y ago

I believe LLMs could generate limitless page reports. Not a single prompt of course. Pick a topic, outline it, grow the outline with subsections. Use a generic summary and LLM indexing to feed context every time you want a subsection, if you want do not stop there use text to image to add drawings, you have 1000 pages thing. Yes, currently it is a thing not an article or research paper. Simple tool and intention relation.