kromem

26 ms·

181.
by kromem 3y ago
It seems like it could be any transformer, which is exciting now that even in imaging gradient transformers are all the rage. But ideally we'd need to see this result in other transformers (but I have a hard time seeing why it wouldn&#
182.
by kromem 3y ago
Yes, but with their current market cap, the more likely result is they acquire one of the several competitors poised to take advantage of this and throw massive resources behind them.
183.
by kromem 3y ago
Given the weights are just mapping to a virtual network structure anyways, my guess would be that as parameter sizes increase any difference node precision might have will evaporate when trained from the ground up. So moving to extremely hi
184.
by kromem 3y ago
This opens the door to very exciting hardware shifts, like to optical computing, where there's already been over a decade of research on ternary optical computing and other parallel research at using optical computing for more efficien
185.
by kromem 3y ago
It sure looks like this might pair well with ternary optical computing advances: https://ieeexplore.ieee.org/document/9720446
186.
by kromem 3y ago
Part of what I think may be an aspect overlooked in the conflict there. The level of destruction is in part a preview of future warfare engagements without Geneva convention style bans on AI use. Too much rhetoric is treating it as a one-of
187.
by kromem 3y ago
I think we'll continue to see n -1 open models used to build momentum to closed n models for the foreseeable future. Which is fine, as it will accelerate to diminishing returns on the +1 difference.
188.
by kromem 3y ago
This works when the person evaluating can distinguish between the subject matter being correct or not. But the models can generate compelling devil's advocate positions, particularly when they confabulate supporting facts, which might
189.
by kromem 3y ago
One of the most interesting things for me over the past 18 months was seeing the difference between this behavior in GPT-3.5 and 4 (especially the early versions of it). You jumped from being extremely context sensitive to leading questions
190.
by kromem 3y ago
For about a year now I've privately wondered if GPT-4 would end up modeling/simulating the over-justification effect. Very much appreciate the link showing it absolutely did. Also why I structure my system prompts to say it "
191.
by kromem 3y ago
Having details in context seems to reduce hallucinations, which makes sense if we'd switch to using the more accurate term of confabulations. LLM confabulations generally occur when they don't have the information to answer, so th
192.
by kromem 3y ago
The article did mention costs, specifically it was provided to them for free and they don't know how much it will actually cost. As for your larger point, it really depends on the ROI. To summarize your Twitter feed, probably not. To i
193.
by kromem 3y ago
I'm most excited at what this is going to look like not by abandoning RAG but by pairing it with these massive context windows. If you can parse an entire book to identify relevant chunks using RAG and can fit an entire book into a con
194.
by kromem 3y ago
Depth isn't always the right approach though. Personally, I'm much more excited at the idea of pairing RAG with a 1M token context window to have enormous effective breadth in a prompt. For example, you could have RAG grab the rel
195.
by kromem 3y ago
I always thought it would have been neat if the users themselves had pooled resources and bought it up. It could have been extremely valuable, but at this point management has been clearly outed as having no idea what they are doing.
196.
by kromem 3y ago
It might not tell you what to do, but done right it sure as heck can point you in the right direction.
197.
by kromem 3y ago
I've been playing a Bingo game for news articles that aren't negative about AI. I don't have any squares yet. Please send help.
198.
by kromem 3y ago
No, I did my small part in contributing to the training data on Reddit though.
199.
by kromem 3y ago
Every LLM almost certainly already is trained on Reddit from before 2015 when the entire site content was compiled for research. This is just adding in more recent content.
200.
by kromem 3y ago
This may be a result of an internal API call or something, where it truthfully doesn't know when you ask, then in answering the prompt something akin to the internal_monolouge part of the prompt (such as Bing uses) calls an API which r
201.
by kromem 3y ago
No no no, don't go blaming the model here. I guarantee that you could get the current version of Gemini without the guardrails to appropriately contextualize a prompt for historical context. It's being directly instructed to adjus
202.
by kromem 3y ago
Tell that to Grok: https://www.zdnet.com/article/i-tried-xs-anti-woke-grok-ai-c...
203.
by kromem 3y ago
It's not a binary. Why are the only options "only generate comically inaccurate images to the point of being offensive to probably everyone" or "only generate images of one group of people"? Are current models so po
204.
by kromem 3y ago
It's pretty amazing how much of the boring parts of programming can be abstracted away with intelligent temporary comments and enough neighboring context using Copilot. Maybe they need to do a better job at teaching users how to be pro
205.
by kromem 3y ago
Threads like these often feel like reading a lot of "Does anyone else feel like your Phillips screwdriver just isn't very good at hammering in nails? It kind of works, but I'd rather not use it for that." I really have z
206.
by kromem 3y ago
If we really wanted it to be good, we'd see a humor classifier trained that could be put in front of a generative process which spit out hundreds of variations and see what the classifier picked as funniest. Most human comics refine th
207.
by kromem 3y ago
Well, they fixed it with a patch introducing "stability fixes."
208.
by kromem 3y ago
So people with Wernicke aphasia don't have consciousness? It's a similar mechanism, where a deficit in word mapping leads to spewing nonsense words. I'm not saying LLMs are conscious - but the notion that spewing nonsense bec
209.
by kromem 3y ago
This is going to get worse and worse. Be OpenAI. Have a model you train to autocomplete text. Tell it it's ChatGPT. Train it to reject inappropriate output. People post examples of it rejecting output. Feed it that data of ChatGPT reje
210.
by kromem 3y ago
Of course it didn't. The actual fix needs to be at the system level prompt. If you train an large language model to complete human generated text, don't instruct it to complete text as a large language model. Especially after feed

More ›