6 ms·
This makes perfect sense and is expected. But you have to know, none of the images are memorized. It's a coincidence. I can explain. Think of the analogy of li
by kilgnad 4y ago
This makes perfect sense and is expected. But you have to know, none of the images are memorized. It's a coincidence. I can explain.
Think of the analogy of linear regression. You have X,Y 2D space. X is the input (the English sentence), Y is the output (the image).
You use training data, which is a bunch of coordinates in this 2D space (each coordinate representing an <English, image> pair) to generate a best fit line (aka the machine learning model).
The properties of that line fit exactly what's going on here. That trend line will not touch most of the coordinates, but it likely will touch a few. When you take an X from the coordinate that touches the line and feed it into the equation of that line (aka model) you will get a Y (an image) that matches the coordinate Y exactly because the coordinate and the line intersect. Makes sense.
This is why an image appears to be memorized. But really it's not memorized. The equation of that line is the ONLY thing memorized here.
Why does it happen more when the image appears multiple times in the training data?
Well this is exactly what happens in least squares analysis. The line is generated from an equation that involves the averages of all the training data (aka coordinates) and if you have many samples of the same coordinate (aka image) you will skew the average and therefore the line towards touching that specific coordinate.
There is no complete technical fix for this issue. With enough training data, EVEN when you get rid of duplicates, the line will likely touch or be very close to a coordinate.
If you think about it, for a 2D coordinate system you can select a bunch of coordinates that generate a line that the never touches a single coordinate. But this defeats the purpose of machine learning. You're suppose to sample data you have no knowledge about and derive a model from it.
If you can already pre-select images that form a model that never touches a coordinate, it means you already have an understanding of the model and can likely just manually tune all the weights by hand to get what you want. As regular humans, we don't have the super-human ability to do this. We can only do it for the simplest 2D example and all the higher dimensional stuff can only be understood by analogy and random sampling.
One thing you can do is just add more data to the training set. That will move the trendline and could shift it away from something it touches. But at the same time it could move it towards touching a new coordinate.
- kgwgk 4y ago> It's a coincidence. https://en.wikipedia.org/wiki/Pierre_Menard,_Author_of_the_Quixote https://en.wikipedia.org/wiki/Pierre_Menard,_Author_of_the_Q...
- FeepingCreature 4y agoIt kind of sounds like you're just giving a detailed technical explanation of how the network memorized some of the images.
- kilgnad 4y agoHow so? I have an equation that makes a number: y = 2x + 3 And I have (x,y) = (1,5) I'm saying the y = 2x + 3 is the ONLY thing stored in memory. The (1,5) is not stored anywhere. (1,5) is training data. y=2x+3 is the model. This makes sense.
- WithinReason 4y agoIf the model captures (almost) all information in the training sample, that is memorization. In your example 2x + 3 memorized the point (1,5) because you can recall 5 as f(1).
- kilgnad 4y agoThis doesn't make sense. F(X) itself is not (1,5). You have to apply F for it to work. Paint and paint brushes can be thought of as functions. You apply the right inputs (brush strokes) and you get an output that is a painting. Therefore could you say that all paintings in the world are encoded into the paintbrush? No. You can't. Not until you use the paintbrush to copy something.
- WithinReason 4y agoIf I ask you to recite a poem you're a function f("recite poem") which returns the memorized poem. It's not even possible to have memorisation without a function.
- kilgnad 4y agoBut it's not memorized. The "poem" in the case of the machine learning model is constructed from a general algorithm. Similar to how 2x+1 has no concept of 3. When you input a 1, it does a *2 operation and a +1 operation and produces a 3. Nothing is memorized. 3 arises as a side effect of unrelated transformation operations.
- sdwr 4y agoThat doesn't sound right at all. In 2D space a line bisects the plane. There's no way of moving from one side to the other without touching it. The possibility space of images is much, much larger than 2D.
- kilgnad 4y agoMachine learning is an extension of the concept of linear regression in 2-D space to 9999999-D space. They also make the line more then just a "line" the equation produces more of a best fit complex curve in multi-dimensional space. This is the actuality of what programmers in this area do. Much of that space consists of pairs of data that aren't relevant. So to understand it in terms of the analogy of the line in 2D space... The only thing that's relevant are points near the line. For example if we use the line to represent housing costs(Y) over time(X). You can take a bunch of housing prices sold over time and plot it on the graph as dots. Linear regression will form a line that best fits those dots. Not EVERY single section on the plane matters though. Only the dots matter and the space that's very near the trend line have anything meaningful in terms of data.
- Kye 4y agoNow imagine you're an expert witness on the stand in the case that eventually settles this. I don't think this will be persuasive even if it's technically sound. How you end up with a copyrighted work doesn't seem relevant: the model still produced something eerily similar to a copyrighted work. "Copyright laundering" will be the phrase of the era. Throw Picasso in with a thousand reproductions, wash it with DeviantArt, get Picasso back out. Does it matter than it's algorithmically derived rather than a stroke-by-stroke reproduction? There's probably already ample case law around reproductions to deal with this.
- deleted 4y ago[deleted]
- kilgnad 4y agoIt's technically sound though. Let me give you an example. y = f(x) = x + 2 This is what's memorized. But with that equation you can plug in a specific x. to get: 3 = f(1) = 1 + 2 The training set here would be (1,3). What is memorized is y = f(x) = x + 2. You can literally see that (1,3) is NOT in the model EVEN though that model CAN produce a (1,3) given the right input. I think the technical part of this is sound. People will just take the face value explanation which is I see a 3! therefore a THREE was memorized. But technically a 3 WAS not memorized. This is categorically true. You are Right. The persuasive part of this argument is not very good though as you can see by this thread.
- owlbite 4y agoIf I have 10 points in 2d space, I can always hit them all exactly so long as I have a polynomial of degree 10 or higher. This is overfitting, which is indistinguishable from memorization.
- kilgnad 4y agoOver-fitting is a known problem and deliberately avoided. This does not prevent the curve from intersecting an actual training set. I mean technically, if there are many copies of the same data set pair, THEN you can call it over-fitting. But removing that does not fully remove the problem. The curve will still intersect datapoints EVEN without over fitting. Also over-fitting is not memorization.
- babel_ 4y agoBecause the dual of a structure is clearly not another representation of the structure. Isomorphism means nothing. /s Your intuition is correct, fitting for the data will lead to this arising. However, you misunderstand the interpretation. What is being fit is exactly an encoding of the dataset (a dual of it, like switching from edges of a cube to faces), so the "coincidence" is which specific images show up with extremely high similarity after similar prompting; likely meaning nothing related showed up in the testing, so it never got pushed around and settled nearby. For many images, that encoding was lossy and sufficiently moved that they are not matches to the training data, however for some they have ended up essentially encoded in these "lines" (as your analogy goes) and can therefore be reconstructed with not that much effort.
- kilgnad 4y ago>Because the dual of a structure is clearly not another representation of the structure. Isomorphism means nothing. /s But it's not an isomorphism. I have a function that has a range that covers every single number in existence. Does that mean that function is A copy of every single number in existence? No. Is that function an isomorphism of every number in existence? No mathematician would use the term "isomorphism" in such a way. You could say that if F(3) = 4, then (4, F(3)) are two things that are isomorphic. But, again, no mathematician would call F(X) isomorphic to 4. Technically in terms of law, what this would mean is that USING the model to generate a COPY would be illegal. The model itself does not hold a copy until function application is executed. Therefore existence of the model itself IS not illegal. To say it's illegal is to say that a copy of the mona lisa exists in paint and paint brushes. F(X) is the paint and paint brush. F(3) is the painting. I will say this, I don't think most people in the jury will be technical enough to get this distinction and the technical part isn't even really important in my opinion. The laws should be made based off of what's better for society/humanity not what's technically valid. Still my comment here is on what is technically happening which SHOULD remain completely separate from the politics surrounding it.
- babel_ 4y agoI mentioned isomorphism in reference to representation, which was kinda the point. And, not to belabour the point, but a function F that has an isomorphism to the real numbers means that F does indeed encode all numbers (and, when defined as a set, exactly contains a copy of each). That's literally the point of isomorphism. To demonstrate simply, the integers have an isomorphism to the natural numbers: you count them in an alternating pattern. That's an isomorphism between them (and it need not be relation preserving). So, with that, you can interpret all integers as natural numbers; we say they're the same up to their isomorphism. Isomorphisms also exist for the real numbers with other structures, via bijective maps, and so forth. I'm not going to respond to the other stuff since it's nothing to do with what I said or responded to (even when it was a misinterpretation).
- dogleash 4y agoThat just sounds like memorization with extra steps.
- kilgnad 4y agohttps://news.ycombinator.com/item?id=34613042 https://news.ycombinator.com/item?id=34613042