7 ms·
It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of
by sigmoid10 3y ago
It has become common knowledge that GPT4 (and also 3.5) have problems with deterministic outputs (even at T=0). So what we're seeing here is just the effect of random sampling, not any actual change to the model itself. If you scroll down, you'll see other close attempts by the exact same model that could already be counted as a win depending on who you ask.
Edit: This comment section is a super fascinating case study on the inherent flaws in human cognition. Especially when it comes to seeing patterns in random noise. The fact that some people believe that the model really has to have changed in the past few days is amazing, because if you've kept up with the GPT architecture and the way OpenAI does things (especially on the API), it is incredibly obvious that nothing has happened. But people who want to believe that something has happened will definitely also start to see something.
- bhouston 3y ago100%. This person is trying to find patterns in random noise and believes they are meaningful. The original post hurts my head with its bad logic.
- calderknight 3y agoI don't see the bad logic.
- bhouston 3y agoHe said he is "Asking GPT-4 to draw a unicorn every day to track changes in the model." The variance he is seeing in the output is primarily the product of random chance, rather than changes in the model. Specifically this "unicorn" that he found today is likely just random chance and there was no changes in the model between yesterday and today that lead to it arising. If he wanted to track changes in the model for real, he would have to ask multiple questions per day and try to infer some type of distribution characterization and then see if that changes over time. That is much more complex and not what he is doing. This is just a curious experiment that doesn't mean much.
- Teever 3y agoyou're looking at it from the wrong time scale. The difference over months or years is what is interesting.
- calderknight 3y agoIt takes a picture from GPT every day, so we'll be able to see if this was a fluke by looking at future days' outputs. I think it will work to track changes in the model.
- bhouston 3y agoYes, this makes sense. You are agreeing with me. In order to see if it is just a fluke or not, you need a lot of samples so you can characterize the distribution yourself and try to see if it changed.
- calderknight 3y agoOK but I'm still not seeing any logic error.
- bhouston 3y agoThat a single sample per day can track changes meaningfully when the noise floor is above the signal strength. And also the fact that today's unicorn-ish sample is meaningful at all. It is a fun experiment though.
- MauranKilom 3y agoIf you look closely, you can see that the model version is attached to each image, and the original blog post clearly states that asking the same model over multiple days is how they intend to track the variance. No bad logic there imo.
- vasco 3y agoIf you look at the examples from April, only 2 or 3 can be counted as unicorns. If in 3 months it's the reverse and only 2 or 3 can't be counted as unicorns, that would show a progressive improvement in the model. I agree we shouldn't take much from day N-1 to day N as there will be a lot of variance, but this can show us progressive improvement over model updates. Perhaps in a few months half of the generated pictures will look like unicorns, perhaps in more months they will all be unicorns but 2 or 3 will look way more detailed instead of drawn by a 4 year old, etc. We just need to wait longer for the signal to break through the noise.
- brabel 3y agoI believe the logic is fine. You seem to think you need multiple data points from the same version of the model (i.e. multiple samples per day at least) would be necessary to judge the actual performance on each particular day. That's worse logic. How would you visualize the very large sample you would get? Even with the current 118 samples (one per day) it's already difficult to find a pattern. Would you "average" the samples?? That would not help IMO, you would need to average the score of each image, which requires either manually doing it or finding a reliable algorithm to do it automatically, but good luck with that. So, a sample per day which allows clearly visualizing any change in the results over months and years is a valuable thing to do and I find it hard to improve on the methodology. You just need to keep in mind that one single picture from the sample is not enough, no one is going to disagree with that... but that doesn't make it "bad logic" and it's pretty thoughtless to say so.
- bhouston 3y agoI agree that trying to determine the distribution of these drawings is hard because it isn't a simple floating point number in its current form. But maybe you could covert it to a linear monotonic measure? You could pass it to an image recognition model and see record the degree to which it thinks it is an: animal horse unicorn Basically if it fails to be a unicorn, see if it is a horse and if it fails to be a horse check if it is an animal. This gives you some type of linear measure if you place these three measures along the same axis adjacently. Then you can transform each image to a floating point number and then characterize the distribution.
- itsoktocry 3y ago>You could pass it to an image recognition model and see record the degree to which it thinks it is... We don't care what another algorithm "thinks". We want to see if what it draws is humanly interpretable as a unicorn.
- bhouston 3y agoThen you could use mechanical turk to have them rate each image to figure out how close it is to a Unicorn...
- nurettin 3y agoWhat if, instead of your ridiculous strawman, they believe in waiting to get consistency?
- imdsm 3y agoI could also believe in not having to have beliefs and just letting it run until I either die or run out of money, and the former may not result in an immediate shuttering of the project either
- Klugistiono 3y agoThe 'random noise' from a prompt "Draw a unicorn in svg" should still return Unicorns. This is absolutly fine and it should start showing unicorn like drawings over a longer period and potentially finetuned ones over a longer period of time when the model changes.
- throwaway154 3y agoExpecting a convergent series is bad logic.
- itsoktocry 3y ago>The original post hurts my head with its bad logic. Huh? What "original post"? This is an experiment, today the model drew something resembling a unicorn. Tomorrow we will see how the experiment goes again. I see no associated analysis, so what makes your "head hurt".
- lcnPylGDnU4H9OF 3y agohttps://adamkdean.co.uk/posts/gpt-unicorn-a-daily-exploration-of-gpt-4s-image-generation-capabilities https://adamkdean.co.uk/posts/gpt-unicorn-a-daily-exploratio... > The idea behind GPT Unicorn is quite simple: every day, GPT-4 will be asked to draw a unicorn in SVG format. This daily interaction with the model will allow us to observe changes in the model over time, as reflected in the output.
- imdsm 3y agoSure, each day it will draw a unicorn. Each time the model changes, we'll have a new group of drawings. No they're not drawn at T=0 but even at T=0 GPT-4 is not deterministic. This isn't science -- this is just a bit of fun.
- imdsm 3y agoI'm sorry it hurts your head. I'm happy to sponsor a packet of paracetamol or some water if that helps. Ultimately, this is fun, not science. I'm just happy that after all these attempts, it finally got to a unicorn.
- Bootvis 3y agoIt got to a unicorn quite easily when the output was set to Tikz as I’ve done here: https://bobjansen.net/drawing-with-chatgpt/ https://bobjansen.net/drawing-with-chatgpt/
- YeGoblynQueenne 3y agoThat looks nothing like a unicorn, but the three triangles at the top right in the third picture look like the mask of some evil manga villain alien robot.
- YeGoblynQueenne 3y agoWell done for being a good sport, but I'm willing to bet that tomorrow's shape will not resemble a unicorn and you'll have to figure out how that works with your assumption that the model is improving.
- imdsm 3y agoI think we're going to need to wait a few more years, at least, to see any improvement. I expect to see 4 new models a year, before GPT-5 arrives. I'll just keep using the latest model and we can all reconvene in 1, 2, 5, 10 years.
- pietroppeter 3y agoIs there some reference or explanation on why the model it is non deterministic at temperature 0?
- Frummy 3y agohttps://news.ycombinator.com/item?id=37006224 https://news.ycombinator.com/item?id=37006224
- pietroppeter 3y agoThanks!
- luc4sdreyer 3y agoI'm not aware of anything concrete by OpenAI, but others have offered possible explanations. One idea is that the cause is batched inference in sparse MoE (mixture of experts) models. https://152334h.github.io/blog/non-determinism-in-gpt-4/ https://152334h.github.io/blog/non-determinism-in-gpt-4/ HN discussion: https://news.ycombinator.com/item?id=37006224 https://news.ycombinator.com/item?id=37006224
- davrosthedalek 3y agoSo in some sense the spectre attack for AI?
- awestroke 3y agoNo
- rav 3y agoOne important source of non-determinism is from using massive parallelism together with floating point arithmetic. In real math, a sum of numbers has an exact value that doesn't change if you change which order the numbers are added up in, but floating point arithmetic addition is not associative in the same way as real math, and parallelism can cause numbers to be added in a different order from execution to execution, which is one cause of non-determinism.
- motoboi 3y agoAgreed. But have you seen the original talk? I believe he's trying to find an unicorn similar in style to the one generated by the original researcher. It's so sad that openai has a far more capable model internally that it can't give open access to because of safety (or any other argument).
- christkv 3y agoIs it possible that inference cost is so high it’s viable?
- ActivePattern 3y agoI suspect the model is fundamentally the same underneath, but that various tricks like quantization are being performed in the deployed model to improve inference speed/cost at the expense of output quality.
- imdsm 3y agoBear in mind I chose SVG rather than TiKZ
- MauranKilom 3y agoYes, nothing about GPT4 changed today. But that's not the goal of the project (although I can't speak for the intentions of the submitter here). Currently there are two different GPT4 models represented in the samples, with quite significant quality difference between them. The quality (and variance in quality within a single model!) is interesting to see in such a comparison.
- sigmoid10 3y agoThis variance also exists among outputs from the same model. Just scroll down a bit and you'll see drastic quality differences with the exact same model.
- imdsm 3y ago> But that's not the goal of the project (although I can't speak for the intentions of the submitter here). (submitter here) You're correct, it's not the goal of the project. It would be fair to say there is no goal other than to ask GPT to draw a unicorn every day, and through it, create a talking point and potential fun for people who follow along.
- worldsayshi 3y agoEven if a single version of gpt4 would be deterministic any change done to the model would probably introduce enough noise to make it impossible to make any conclusions on a few samples?
- throwuwu 3y agoIf the model understood the spacial relationships as well as the one that produced the original drawings of a unicorn then variance in the choice of the next token should produce many similar but somewhat different images of unicorns. None of the images until today bear any resemblance to the original images.
- imdsm 3y agoWhat I find interesting is that there are aspects of what a "unicorn" represents, horns, limbs etc, in some of the drawings. Today is the first time I've seen it create something that actually looks like a representation of a unicorn.
- deleted 3y ago[deleted]
- rcme 3y agoI'm not sure what you mean by random sampling. If I sample a random SVG, I wouldn't expect to look like anything, let alone roughly like a unicorn.
- sigmoid10 3y agoI mean random sampling in the sense how autoregressive language models like GPT generate sequences using token probabilities. It's not a random svg, but the text sequence that is used to draw it suffers from inherent non-determinism in the underlying model.
- rcme 3y agoRelying on token probabilities seems like the exact opposite of random
- esafak 3y ago"Random" is a loose term. We're discussing a technical subject. Talk about the distribution the random numbers are sampled from.
- sigmoid10 3y agoThe neural network just generates a set of probabilities for all tokens. The actual next token is then sampled from this set, which is always random for T>0 (and in the case of GPT4 even for T=0, because of the way the model itself works).
- deleted 3y ago[deleted]
- theonlybutlet 3y agoIf he removed the word changes it would've made sense. See what the odds are of it producing a unicorn. So far it's roughly 1 in 118 based on 1 test a day.
- goodpoint 3y ago> This comment section is a super fascinating case study on the inherent flaws in human cognition Like most of HN.
- sheepscreek 3y agoAgree with this except for one data point. OpenAI does enhance/tweak/do something with the models at different levels. This can be determined by: 1. A change in the current model number (eg. gpt-3.5-turbo-0613) 2. On ChatGPT UI, the date at the bottom (eg. August 2023) So it isn’t correct to say “it is incredibly obvious that nothing has happened”. Not that obvious to me. A bit like how you can never tell for sure if Coca Cola has tweaked their formula, or McDonalds has changed the recipe for its signature sauce. Only in this case, the model number going up or the date becoming more recent leads credence to something having changed.
- sigmoid10 3y agoThe ChatGPT UI is indeed a wildcard. But it is irrelevant here because according to the github repo this page queries the API and OpenAI guarantees it doesn't change models with version number information (like gpt-4-0613, which is mentioned in the latest images). So this "experiment" would make a lot more sense if it was only run once every few months when the API actually offers new model versions and then generate a bunch of images for every model, instead of generating one single image every day (which is meaningless due to non-deterministic noise, even if the model had somehow changed since yesterday). That is also how it was done in the original study during the development of GPT4. I don't know how this experiment came up with its logic, unless they had a gross misunderstanding about how these models actually work (which admittedly seems common among tech interested folks here).
- imdsm 3y agoWhen I last spoken to Logan, he confirmed that there are no changes between the API models, so 0314 and 0613 are it. That's two models so far that I've collected SVGs for. With regard to batch vs daily -- it makes no different in terms of output, but by going daily, I don't need to track model changes and generate a new batch. Also it's fun to see each daily unicorn.
- sheepscreek 3y agoThat’s a good summary and it makes a lot of sense.
- noduerme 3y agoI have not kept up with GPT architecture, other than noticing that other people have noticed that T=0 is clearly not deterministic for these things (and that that results from a bug, not an intentional feature). This much was obvious when the supposedly genius idiots rolled out GPT-3. It's wonderful to see the whole world bend over and just take it up the ass from a bunch of people who can't figure out why their code can't produce the same result twice; but it's quite natural for there to be a big cheerleading section on HN for any new technology that's (1) brilliant in theory, (2) deeply anti-human in practice, and (3) just needs a couple more revisions before it "fixes" a bunch of stuff.
- QuadmasterXLII 3y agoWe’re pretty sure the nondeterminism is batching + mixture of experts + contention for specific experts
- noduerme 3y agoIf by batching you mean bad code that fails to sort or relies on hardware to best-guess how things sort, then sure, that's called a bug. Also, "we're pretty sure" is rather self-important while also admitting total, abject failure to produce a deterministic result. You shouldn't blame yourself. A lot of people had the same feeling after staking their life on the revolutionary properties of NFTs.
- n2d4 3y agoYour incorrect assumption here is that determinism comes with no tradeoffs. There are a few outsider analyses on the topic, for example, this one on sparse MoEs [1]. If OpenAI uses sparse MoE as described, then determinism would be possible but inefficient. Even if it's not sparse MoE, chances are high that the non-determinism is introduced somewhere purely as a performance optimization. The article speculates that OpenAI knows this well and hides it to protect the model internals. [1] https://152334h.github.io/blog/non-determinism-in-gpt-4/ https://152334h.github.io/blog/non-determinism-in-gpt-4/
- 3y ago
- itsoktocry 3y ago>This comment section is a super fascinating case study on the inherent flaws in human cognition. Especially when it comes to seeing patterns in random noise. The fact that some people believe that the model really has to have changed in the past few days is amazing You need only to look at the discourse around the Tesla FSD superusers to see this: they report a glitch at an intersection one day, then believe the next day it was "fixed" by the AI.
- olabyne 3y agoThats even worse for autonomous cars, there is so such data and noise there is no way to reproduce the issue, it's complete chaos. Whereas with a LLM if we control the seed we can 100% reproduce the same result
- noduerme 3y ago>> if we control the seed we can 100% reproduce the same result No, that's the problem. You can't. You should be able to, but you can't. If you could, they wouldn't be scary. But we have Temperature Zero, different results. Because no one gave enough of a shit when coding them, and no one gives enough of a shit to try to fix the issue. This is what in any other industry would be called gross negligence.
- jquery 3y agoA lot of stuff behind the scenes is going on to batch and route queries to GPT-4 models that are in perturbed states already[1]. This isn't gross negligence, this is basic capitalism. If you want sole access to a GPT-4 MoE cluster starting fresh, it's gonna cost you. 1. https://152334h.github.io/blog/non-determinism-in-gpt-4/ https://152334h.github.io/blog/non-determinism-in-gpt-4/
- noduerme 3y agoInteresting article. I can see how it makes sense for OpenAI or someone with a LLM to take advantage of any entropy that presented itself, as a shortcut to non-repetitive answers. I'm not sure if you're saying that these LLMs take on new characteristics as they get more randomized? Or just that it would be hard to get your hands on a fresh one to test the determinism of?
- asddubs 3y agoyeah, see also image-2023-04-25 which is way earlier and comes really close, surrounded by garbage