13 ms·

Edge detection doesn’t explain line drawing

246 points by KqAmJQ7 3y ago

SanjayMehta 3y ago
Whilst reading Dr Seuss to kids I observed that even at a very early age they can mimic the expressions on the cartoon characters’ faces, even when they’ve never seen that drawing before. An easy experiment to reproduce.
cheschire 3y ago
I have a different hypothesis. I think line drawings are the representation of vectors. The details we focus on are the vectors with the highest magnitude of change, but that change is not always visual. For example, momentum often has an outsized representation in line drawings. Edges can be a high magnitude change as well, but it's not the only thing.
- cropher 3y ago
  Interesting; can you elaborate? Also could you clarify what you mean by changes that are "not always visual"? How would non-visual information exist in a purely visual medium?
  - cheschire 3y ago
    The example I mentioned was momentum. That's not visual information, that's extrapolated information of position over time. It can be represented in line drawings as motion lines in comics, for example. Interestingly, by simply implying motion, I hypothesize that the brain deprioritizes processing detail on the object that is implied to be moving, and focuses instead on the interactions that will follow. If I were a researcher, my contrived test of this would be to simply have people recreate drawings of "static" objects, and have others recreate drawings of objects implied to be in motion. Other non-visual information would be emotions. The shape of eyes and mouth lines are highly critical to passing emotion. I suspect that people's interpretation of emotion directly impacts how strong the emotional representation of those parts of the face would be drawn. For example if a test subject is told to draw the face of a model in front of them, but they are told the person is experiencing an emotion, I hypothesize that the group of people who are told the person is happy would more frequently bias their interpretation of the eyes, eyebrows, and corners of the mouth towards a "happy" representation than those who are told the person is experiencing great inner turmoil. To be clear though, I'm not saying we only draw based on non-visual information. I'm saying the sum total of all vectors has an influence on the drawing. Colors, in my opinion, have as much of an impact as edges. And it would be interesting to compare the drawings of a person with less common color sensitivities to more common color sensitivities.
- plaidfuji 3y ago
  You’re on to something here. In the past I did a bunch of work on extracting line drawings from images, and the fundamental goal was always to vectorize the lines - then you get an abstract representation of the figure that you can scale up or down.
cnity 3y ago
As someone (probably like many here) who graduated from a university which taught computer vision and peripheral neuroscience courses, with such titles as "Computational Neuroscience of Vision", I always felt that trying to understand the human brain as a kind of algorithm was a bit of an artefact of computer scientists as they approach biology. The truth is the visual cortex is vast, and not sufficient to explain the human classification and perception of objects visually. Never mind individual neurons or edge perception. Edge detection is an interesting isolated example for study and learning, but you will never come close to explaining human recognition and cognition in such simple terms.
- hprotagonist 3y ago
  ah, someone who paid attention in lecture! (incidentally; there’s a fairly deep literature of historians of science that have carefully documented that we describe ourselves as analogous to the most sophisticated technology of the day: see “to lose one’s temper”, “to blow a gasket”, “i got my wires crossed”, “sorry, cache miss”, … as metaphors and idioms of mental state through the centuries that reflect the cool tech of the time in which they were coined )
  - pavlov 3y ago
    And also the universe itself is often seen through the lens of contemporary technology. Are we living on an island that floats on a giant turtle’s back? Or are the heavens like giant clockworks? Or maybe it’s all a computer simulation? These cosmological speculations are separated by thousands of years, but they are all simply a reflection of what the person finds most awe-inspiring in their everyday life.
    - CuriouslyC 3y ago
      The difference is turtles all the way down was literal, but heavenly clockwork and computer program perspectives are clear metaphors.
      - 3y ago
amelius 3y ago
The human brain also triggers on soft features. It's a combination.
zellyn 3y ago
Wouldn't a simple explanation of line drawings be that segmentation of images into shapes is an important part of vision, using whatever information is available? The reason it would be somewhat invariant across color and lighting is that those change so much. (eg. we can see by moonlight or faint illumination, when color signal is absent) In some cases, simple segmentation fails (like with the shadowed face in the article), and you have to rely on other features.
smokel 3y ago
Line drawings are just easy to produce, historically. This makes them embedded in our culture, and kids learn to read them from an early age. The same skepticism could be raised towards letters and words. If we had invented the photo camera before the paint brush or pen, things might have looked different.
smokel 3y ago
Here's a link to the paper that contains the "my hypothesis". The link in the article runs into a paywall. https://arxiv.org/pdf/2002.06260.pdf https://arxiv.org/pdf/2002.06260.pdf
simias 3y ago
Ten years or so ago I was working on a video chip that had an upscaler feature. While prototyping and simulating it, we first started by applying a mathematically-correct (i.e. information preserving) FIR filter to do the upscale. Then we compared the result with other solutions and found that ours looked worse. We asked our colleagues to blind-test it and they all picked third-party-scaled images over ours. At first we assumed that we must have had a bug somewhere because the Fourrier transform told us that our approach was optimal, but after more testing everything matched the expected output. Yet it looked worse. So we started reverse-engineering the other solutions and, long story short, what they did better is that they added some form of edge-enhancement to the upscaling. Information-theory-wise it actually degraded the image, but subjectively the sharper outlines were just so much nicer to look at and looked correct-er. You felt like you could more easily tell the details even though, again, in a mathematical sense you actually lost information that way. I don't think it makes a lot of sense to reduce human vision to edge detection (we can still make sense of a blurry image like this one after all: https://static0.makeuseofimages.com/wordpress/wp-content/uploads/2013/01/blurred-city.jpg https://static0.makeuseofimages.com/wordpress/wp-content/upl... ) but it's clear to me from empirical evidence that edge-detection is a core aspect of how we parse visual stimuli. As such I'm a bit confused as to why the author seems to see this as a binary proposition. That being said, I could just be misunderstanding completely the point the author is trying to make.
- JoeAltmaier 3y ago
  The difference between the data in the image, and the information? If for instance you upscaled text so large that it became blurry and unrecognizable, you lost information. Our cortext is all about interpreting what we see. Almost before our brain proper has the data, nerves have begun extracting information (edges etc). Probably because it was the difference between hitting and missing the animal with the spear. Or seeing or missing the tiger in the grass.
  - simias 3y ago
    Precisely! I also find it interesting how, from an information theory standpoint, audio processing and image processing are effectively the same thing (audio resampling is fundamentally 1D image scaling for instance) but because humans process sounds very differently from images we end up doing things pretty differently. For instance when we want to subjectively make images more attractive we tend increase contrast and sharpness, whereas for sound we tend to compress it, effectively reducing "audio contrast".
deleted 3y ago
[deleted]
a2128 3y ago
My completely uninformed hypothesis is that it is edge detection, but on a depth map rather than a color map. A figure in the article even shows a situation where a depth sketch succeeds while edge detection on a color map fails
- regularfry 3y ago
  Kinda both, would be my guess. The brain reconstructs depth from visual information, so where the visual edge can be interpreted as a depth cue, you're faking out that mechanism. It also explains why the pure edge detection image just looks noisy, especially in the hair: most of the edges are effectively incidental colour shifts that don't provide depth information.
jaked89 3y ago
Line drawings work because they represent the centers of symmetry for surfaces and volumes. They trigger the same center-neurons that the original shape would. A hand could appear on you retina in different sizes and orientations. According to distance, the size will grow and shrink. But the center of symmetry will stay the same. This goes further. There's also a center of symmetry between edges, and higher-level features as well. Our brain has no issue detecting these.
JoeAltmaier 3y ago
There was a science fiction story of a race of aliens that had to draw what they found instead of photographing it. A photograph was flat and uninformative to them, almost completely lacking in meaning. Their science was based on interpretation at the root. An interesting concept to put in a story anyway.
bschne 3y ago
> Problem #1: What about all the other features? Is this necessarily a problem in the argument? If we consider e.g. color, and construct a blurry image that has a color distribution very similar to a real face, but doesn't fire the edge detection in the same way, we can still recognize it as a face. This could just mean that a sufficiently close match to other examples of the same class on any one of the strong "dimensions" of the image our brain processes is sufficient to make us recognize it, no? The author makes it sound like if edge detection is what makes us recognize line drawings, this implies that our brain discards every other feature in attempts to recognize visual input, but I don't think that's a sound conclusion.
- CapsAdmin 3y ago
  This stood out to me as well. If I close my eyes and walk into a wall I will still perceive the wall. Our various senses help build an understanding of what's going on. It's not like we fail to understand anything once you remove one part of the system. Maybe this person is trying to say something deeper or more nuanced and I fail to understand the meaning behind it.
- bduffany 3y ago
  The "Problem #2" that they mention also has an easy solution. If the "intermediate" variable / internal representation is just "the input image but with edges only," then of course you can see internal representations. When you compute that intermediate variable for the line drawing, it will just happen to behave like the identity function for that particular case. So if you have already filtered out non-edges then the transformation is basically a no-op. The "types" mismatching as they mention is not a concern because the type is just "image" i.e. a big vector of HSL values or something. Edge detection is just a convolution filter so it's going to have approximately the same type as the input.
- armoredkitten 3y ago
  Yes that stuck out to me as well. The author seems to be setting up a straw man, as if people are arguing that the brain can't distinguish between a line drawing and the real thing (or a line drawing and a photograph). The other information isn't necessarily discarded. It's just used to identify that this is a line drawing and not the real thing. It's still remarkable that just the lines themselves (I make no claim as to whether it really is edge detection or something else) are still enough information to be able to identify the representation, but it doesn't mean the brain is discarding the other information.
coldtea 3y ago
>A classic answer to this question is what I will call the Lines-As-Edges hypothesis. It says that drawings simulate natural images because line features activate edge receptors in the human visual system. Why does the explanation need to go to that direction? Line drawings have a liking to the thing being depicted. They're a crude representation of it (compared to a photograph or a photo-realistic oil painting) but are nonetheless a represenation. That's why they work, in the sense of people understanding what they show: they share similar patterns with the things being show. And we are pattern matches. In this case the patterns are edge patterns, but it could just as well be non-edge patterns. Imagine a color drawing of a human face, where the ink drawn edges have been removed, and it's just blocks of color for the head, the eyes, the pupils, the mouth, the nostrils, ears, etc. We could still tell it's a face, even if we applied some gaussian blur to those blocks. >The most basic statement of the problem with Lines-As-Edges is that the human visual system isn’t just an edge detector. You can see colors, you can see absolute intensities. You can tell the difference between a thin black line and the silhouette of a dark object against a light background; we have both kinds of receptors in the primary visual cortex, as well as others. Yet Lines-As-Edge supposes that the vision system discards all of this other information present in an image, for just this one special case. Why? Isn't this taking things backwards? It's not the vision system which "discards all of this other information present in an image" in our regular operation. Rather, it's the line drawing with does away with (discards) all of this other information and only focuses on a thing's edges. In other words, our visual system has capabilities A, B, C (say edge detection, color, 3d placement, etc). And, a line drawing gives it only A - which is still enough. When our visual system also gets B and C, it can perceive objects even better. But for merely identifying something, A is apparently enough. What I described here is totally compatible with Lines-As-Edges hypothesis, and makes sense too, so I don't see where the author's issue is, and why he thinks the fact that "the human visual system isn’t just an edge detector" invalidates the lines-as-edges hypothesis. >Now you get a sense of the color of the object, and not just its outlines. How would one generalize Lines-As-Edges to account for these different types of depiction? The visual system is no longer ignoring everything aside some gradients; it’s now paying attention to some colors (and not others). Yeah, so? It just means that the visual system can work with less or with more (and multi-type) information. Lines-as-edges is a hypothesis for why line drawings "work" (are recognizable as the thing). The hypothesis doesn't say however that edges are the only thing the visual system can understand. So, there's no need to "generalize Lines-As-Edges to account for these different types of depiction". Lines-as-edges explains how line drawings are understood, period. Line drawings with shading and color, add additional information, aside from the edges. That's fine: no proponent of Lines-As-Edge ever said that the visual system only works with edges. Just that it works with edges when interpreting a line drawing which only offers edges.
henry_pulver 3y ago
This is fascinating! I remember reading that most optical illusions don't work on people raised in remote tribes in the Amazon, as their visual perception has been 'fine-tuned' for jungle contours, instead of the straight lines in the west. Is it possible that we _learn_ how to perceive line drawings in our early years?
- lacrimacida 3y ago
  I think so and this is true about music as well as other arts.
wzdd 3y ago
I don't understand the essay at all, perhaps because I'm not a domain expert. Edges are clearly a very strong signal in the visual system -- as the article points out, edge detection is one of the first things that happens in the visual pathway from the eyes. However, edges are clearly not the only signal, as the article demonstrates with cross-hatching, colour, and so on (and is also obvious to presumably all sighted humans). If you remove edges as a signal -- say, by taking the coloured apple in the article, removing the lines, and blurring the colours around the silhouette -- you'd probably still recognise an apple, but not as quickly. For the same reason, if you defocus your eyes (or take off your glasses -- the popularity of these is a strong indicator that contour matters), you'll have more difficulty navigating, even though many signals (colour, shade, depth) are still present. Clearly edges are important, but are not the only thing we're working with. Optical illusions also don't invalidate the hypothesis, because they almost by definition rarely occur in nature. Similarly, objects in extreme shadow don't invalidate it either, because, e.g., we are quite capable of recognising half a lion as a lion. I think possibly there's a difference in interpretation here. The claim the author has an issue with is (his phrasing): "the lines in a line drawing are drawn at natural image edges, where an edge receptor would fire. These lines activate the same edge receptor cells that the natural image would. Hence, the line drawing produces a cortical response that is very similar to that of some natural image, and thus you perceive the drawing and the photograph in roughly the same way." But the paper the author links to doesn't say that. It says "The likely explanation is that lines trigger a neural response that has evolved to deal with natural scenes." It's not claiming that line drawings and photographs are perceived in "roughly the same way", only that we evolved to recognise outlines and line drawings are outlines or at least contain outlines. So problem 1 (what about all the other features), problem 3 (what is the benefit), problem 4 (visual art isn't just line drawings), and problem 5 (edge detection is not a line drawing algorithm) don't seem to really address the hypothesis. That just leaves problem 2, "we can't see internal representations". I'm not sure what the statement is here. The author writes "The idea is that we have neurons that activate for object contours and similar, and that line drawings directly activate these neurons. Lines-As-Edges is a special case of this hypothesis. I don’t understand this claim at all." But that claim, as a hypothesis, seems very reasonable? The author seems to be saying that you can't bypass all the neuronal machinery to get directly to the contour-recognising neurons. But that's not true if all you're bypassing is the stuff to isolate the feature of interest (edges, in this case).
Nevermark 3y ago
The article sheds doubt on edge detection as being the major reason why we can interpret drawings as what they attempt to portray, because drawings are often missing shading and color. Drawings also often drop a great amount of detail, and proper proportion information, and remain recognizable. Think of cave paintings or Picasso’s bull. But if you consider image interpretation as a competitive classification, then missing information doesn’t present a problem. If color, shading, detail & proportion are missing, then they are missing from all possible pattern interpretations equally. That leaves the final classification problem relatively unchanged despite all the missing info. EDIT: in fact, if my hypothesis is true we should be able to see patterns with even less information! For instance, dropping all internal detail, and even most shape. As in seeing a face profile on the side of a cloud that otherwise looks nothing like a human head. Or dropping edges completely. Perhaps seeing an object where there are only stars creating points of light. Please post your anecdotal experiments at 404experimentreports.com!
cvoss 3y ago
The brain's ability to do "shadow removal" is really impressive. You can see it happening in the final example depicted in the article. I think this ability is what was used to explain "the dress". It also comes up in a classic optical illusion [0] where a checkerboard of light and dark gray squares has an object sitting on it and casting a shadow. The shadowed light gray squares are exactly the same shade as the unshadowed dark gray squares, but if you point this out to someone, they'll have a really hard time accepting it because they are so good at unconsciously accounting for shadows. [0] https://en.wikipedia.org/wiki/Checker_shadow_illusion https://en.wikipedia.org/wiki/Checker_shadow_illusion
- aftbit 3y ago
  Hey thanks, this one is new to me. 100% it is impossible to convince myself that those two squares are the same color. I have a camera looking over my driveway with motion detection that triggers all the time because of the shadow of a tree waving in the wind. My current weekend project is to use FastSAM to detect cars / animals / people in the driveway instead of just looking at a threshold of changed pixels. 1: https://github.com/CASIA-IVA-Lab/FastSAM https://github.com/CASIA-IVA-Lab/FastSAM
  - thehappypm 3y ago
    It's because the squares are not the same color. The alternating squares are different colors... in absence of a shadow. Your brain is correctly adjusting for the shadow to correctly identify the "true" color.
mistercow 3y ago
Are there any studies on what, if any, animals respond to line drawings (and to what extent)? In particular, do chimpanzees and other apes closely related to humans? It seems somewhat plausible that “lines-as-edges” was a foot in the door for something that specifically evolved in humans as our ancestors began using paintings and drawings for communication. Maybe it was initially just hijacking edge detection so that some images could be conveyed through drawings, and over time, and that developed into a kind of “grammar” for artificially depicting more nuanced images in media where realism was impossible. This would be similar to how recognizing and interpreting different kinds of basic vocalizations (as many animals can) developed into a much more sophisticated mechanism for developing complex language.
drcode 3y ago
when a plane is receding from us, it is more likely to contain color variation, which or eyes sees as a line. it is more likely to contain color variation simply because a plane receding from us has more surface area relative to a unit of space in our visual cortex, versus a plane we are viewing at the position of a normal vector. the edges of a 3d object tend to be receding planes, hence we evolved to detect this
timacles 3y ago
As a person that has been practicing and studying line drawing for 15+ years, this article seems like its way off the mark. Its just not asking the right questions. The brain uses all kinds of context sensitive cues to try to link what its looking at to whats its already seen before. If you happen to look at something thats completely new, it doesnt matter if its a line drawing, 3d 4k image or whatever, your brain will be confused. OTOH, when you're looking at something that you've seen before the brain will do all kinds of tricks and cheats to make that thing as real as possible. Anyone that draws will know you can use plain black and white lines to depict everything from textures, depth, color etc.. Line drawing works because literally anything will work. You could see a distorted, flat black silhouette of one of your family members, and you will instantly know who it is. Its your brain that "makes" things work because its job is to take incomplete information and make it fit an existing mental model. Line drawings carry so much visual information about objects that its really not that impressive your brain can "figure" out what its looking at. It can do so much more with so much less.
- regularfry 3y ago
  The last picture in the article is an interesting counter-example: a line drawing that's constructed in a fairly natural way that is nevertheless almost impossible to interpret. I'm sure there are "line art rules" that picture breaks which are almost second nature to someone who's been in the game for as long as you have; the interesting thing is how that translates to the visual cortex and why those rules work.
  - timacles 3y ago
    That one isn't a line drawing, its a shadow relief drawing with depth removed. Then with with outlines applied to the shadows. Even then, the only reason why its so "confusing" to us is because they applied some kind of random black shape to the background which extends above his head to the left. The black background blends with the man's body making an amorphous blob. If it was removed you'd be able to instantly tell thats a bald mans head. You are correct about the "line art rules" they're all just basic visual rules that the brain uses that artists have been playing with for 100s of years. Its no different than those illusions that look like a Grandma or young woman at the same time. All it takes is one conspicuously placed shape and your whole perspective can be thrown off. No different from how camo works: https://en.wikipedia.org/wiki/Disruptive_coloration https://en.wikipedia.org/wiki/Disruptive_coloration Anyway, my point being, this isn't some new phenomenon, its a well understood at this point.
AndrewKemendo 3y ago
>Yet Lines-As-Edge supposes that the vision system discards all of this other information present in an image, for just this one special case. Why? The "other information" is not discarded, it's just not processed yet I think this is a misrepresentation of how the visual system works and views the vision system as more "batch" than the continuous process it is. So if you think about it as a time series problem, when light hits the retina then "inference and processing" starts with a kind of "fast and rough" inference and then proceeds to fill in details and contextual coherency follows I'd have to go pull out the textbooks but if you look at visual interpretation sequence it's something like: Movement > edges > color > details So your visual system acting as an object detector - from sensor to inference - makes inferences about movement first, then infers the edges, then infers color and finally additional details in the last few ms Nothing is "discarded" it's just less relevant in the first pass and additional refinements happen mostly sequentially - this all happens in nearly imperceptible time.
uglycoyote 3y ago
Very frustrating article to read. The article is setting up a straw man and attacking it. He is acting like everyone else thinks: 1) edges are the only important features in images and 2) line drawings can only represent edges. Who are these brainless absolutists that he is attacking? Then he's acting like he is the only one with other bright ideas that nobody will listen to. I think it is obvious to anyone who thinks about this that: 1) edges are a useful feature for recognizing objects in images but not the only useful feature 2) lines in line drawings can and often do represent edges, but there are a lot of other things they can represent. Light and shading and texture of various kinds. It would be fine to write an article that goes in to depth on the different nuances, but it is annoying that this author pretends that most other experts have naive and simplistic views, with "uncritical certainty", and "no one seems to question it", and the author "has a hard time convincing them otherwise". It is a very condescending tone that comes off sounding like the author is presenting themselves as some brilliant but misunderstood outcast, and the only one who can see the light of truth. we could do without the drama!
- arketyp 3y ago
  The author comes across as ignorant at best, but then to present his own work as the Realism Hypothesis of Hertzmann, it leans more towards arrogance.
  - 6D794163636F756 3y ago
    There is a subset of the tech bro that believes everything can be reduced to a problem with clearly defined taxonomy and as such every problem can be solved by an engineer with no subject knowledge. This article very much reads like one of those people wrote it.
    - wpietri 3y ago
      Yeah, I have been guilty of that myself sometimes. This XKCD is a reminder to me: https://xkcd.com/793/ https://xkcd.com/793/
deleted 3y ago
[deleted]
bambax 3y ago
Figure 4 in the article [1] is absolutely fascinating and seems to prove conclusively that the edge detection hypothesis is completely misguided: when one retains only the edges (B) of the original image (A), all meaning is lost and the resulting image is unrecognizable. What A has that B lacks is shapes. There is obviously a rotund shape of the face that is reconstructed by the brain, that one can almost see, although technically it's not actually present in A. Not present, yet visible. Same for the eyes, smile, cheekbones. It must trigger some kind of pattern-matching in the brain. This problem seems to be similar to the famous optical illusion of the old woman / young woman [2] that works well also in strict black and white. In this optical illusion there is surprisingly little information on the image, yet it can trigger two very different representations (that one can see alternatively, but never at the same time). I think the brain tries to fit the shapes it sees in one box or another, and when it finds a box it builds a whole concept around it. I also think that boxes have to pre-exist or be learned: they can't be inferred from an image, if one has never encountered the original representation in the wild. One evidence for this is children are completely blind to optical illusions that have one "innocent" representation and one involving some kind of nudity or sexual activity, while adults tend to see the NSFW version first. And so, to come back at the original question of why line drawing works, I think it's because it triggers concepts. The word "square" or "circle" are unambiguous and designate precise geometric shapes (provided one has learned the relevant concepts of square and circle). Same with shapes, except that there are an infinite amount of different shapes that we can "discuss" using the language of line drawings. [1] https://aaronhertzmann.com/images/howtodraw/sayim.jpg https://aaronhertzmann.com/images/howtodraw/sayim.jpg [2] https://cdn.mos.cms.futurecdn.net/rQkQZ6pDZbEHz23rxckWPm-320-80.jpg https://cdn.mos.cms.futurecdn.net/rQkQZ6pDZbEHz23rxckWPm-320...
oaktowner 3y ago
I am not a computer vision specialist, but I wanted to add an anecdote from years of parenting. One thing that amazed me about babies was how early they understand line drawings. Long before a baby can talk, it knows what a dog is (and can imitate a dog sound), and can also identify a dog in a photo, in a realistic drawing, and in a very simple line drawing. It seems so easy, but in reality those things all look so very, very different. And after 6ish months, babies have mastered that recognition.
golol 3y ago
Eh. To me it looks like the authors extends "lines-as-edges" explanation of why we can understand line drawings to the claim that the visual cortex works almost only on edge detection, which is obviously bunk and so easily refuted by him. Then later he shows a shillouette image we can understand even though the edges themselves are not understandable, and somehow this is supposed to be evidence against lines as edges. No, it just means that we have other methods of understanding images: shillouette, color, texture etc. In my opinion these kinds of arguments trying to decompose the brains functioning into a couple of distinct techniques become quite obviously pointless once you look at the activation patterns of neural networks. Just looking at the features that neurons detect throughout the different layers of an inage classifying neural network tells you more than these kinds of papers ever will. You see that edges, shillouettes, circles, circles with holes, textures, shinyness patterns, grid patterns etc. up to complex patterns and then real things like heads or arms are detected. There is some manifold of all the images a being is somehow likely to see on this world and it has a complicated structure. You can extract the major features of the geomatry of this manifold and you come across the usual patterns. At simple complexity you find things like edges and textures, at higher complexity things like eyes or appendages. You try to find these features in an image and hope one of them works well. Maybe edges work, maybe shillouettes work or maybe both or neither. Look I know ANNs and NNs are quite different, but the experimental evidence with NNs shows that what I described above, a mix of feature detectors that just approximate the structures of the data to deeper and deeper detail and are just all somehow applied to see what works, is much more plausible than some constructed algorithm a neuroscientist or philosopher would write down.
- kortex 3y ago
  > Look I know ANNs and NNs are quite different, but the experimental evidence with NNs shows that what I described above, a mix of feature detectors that just approximate the structures of the data to deeper and deeper detail and are just all somehow applied to see what works, is much more plausible than some constructed algorithm a neuroscientist or philosopher would write down. Precisely. Deep CNNs were directly inspired by studying the structure and behavior of visual cortical tissue. The algorithm is just recursive feature detection, where the visual elements are transformed into abstractions, based on both sensory input and projected expectations. If anything, that's the real exciting part of the visual system, why do illusions occur, how does expectation affect perception, etc. Not "how do line drawings evoke similar concepts to images". That's just bypassing the first few layers of filters. Basic ass tiny MNIST nets can do this.
porphyra 3y ago
Drawing lines around semantic segmentation, such as using Segment Anything [1] seems to make a lot more sense than just doing edge detection on image brightness. [1] https://segment-anything.com/ https://segment-anything.com/
radicaldreamer 3y ago
Figuring this out is one of the central questions of phenomenology... how is it that after seeing a couple of trees, we can recognize almost any kind of tree from any angle? What is intrinsic about a tree that makes this possible? What (or is) there a pure form of a tree?
- mrguyorama 3y ago
  Since definitions aren't concrete, and language is explicitly a popularity contest, you are not learning "what makes a tree a tree", but rather "what people call a tree" You are re-deriving an existing definition, not an actual physical reality. This is a great example actually, because there is no genetic or biologic definition of a tree, and what we call a tree is matched better by some idealized representation of info like "has a trunk maybe". Also that the definition you develop is rarely rigorously tested in practice, so you only need about 80%, and basic pattern matching can get you that really easily.
lookdangerous 3y ago
A drawing of a cup has 'cup-ness', as far as our mind is concerned. It's not about the micro mechanisms of edge detection or color or whatever, it's about how we recognize the quality of a cup in real cups and things that push the margins for what a cup is and in drawings of a cup.
bazzargh 3y ago
Huh, right up my alley... a while back I was playing with trying to turn photos into sketches ( https://hachyderm.io/@bazzargh/109928618521729073 https://hachyderm.io/@bazzargh/109928618521729073 ); one of the effects I found really noticeable with my super-naive approach was that it tended to overemphasise very dark areas in a way we don't see. Then last week I saw this, which also does some fill in shading (but, using ML)-worth the click, it's a project that makes it appear as if paper held over an object causes it to be sketched...but in fact it's all post-processing... https://www.youtube.com/watch?v=vArIkzYtW6I https://www.youtube.com/watch?v=vArIkzYtW6I This got me back to wondering about the shading problem, and I ended up down a rabbithole reading papers like https://www.yorku.ca/rfm/pub/2021annrev.pdf https://www.yorku.ca/rfm/pub/2021annrev.pdf (review article on state of the art), https://www.frontiersin.org/articles/10.3389/fpsyg.2022.915672/full#B35 https://www.frontiersin.org/articles/10.3389/fpsyg.2022.9156... (recent markov model, with links to code) about lightness and brightness perception, and how our mental model of how the scene is lit can explain a bunch of optical illusions. While I was going for a charcoal effect, this approach https://openprocessing.org/sketch/486307 https://openprocessing.org/sketch/486307 - scribble using brightness as 'gravity' on the pen - is pretty nice. Anyhoo, I'm not going to critique the article, because I'm a total amateur just doing this for fun, but it _is_ fun, and very satisfying when you get the computer to draw something that looks hand drawn instead of just a sobel filter.
brap 3y ago
This was a very interesting read. From childhood we instinctively draw lines, but I never questioned how we produce them from real objects, and why they work so well at representing real objects. It’s one of those things we just take for granted.
eternityforest 3y ago
Re. problem 2, Running edge detection twice will still give a recognizable result. Maybe some functions DO let you take the result of an internal layer, feed it to the input, and not get complete nonsense
rajnathani 3y ago
> Why do line drawings work? Why is it that we can immediately recognize objects in line drawings, even though they are not a phenomenon from our natural world. This is wrong, as humans have been drawing and viewing art for tens of thousands of years. Also, for other animals, a form of edge detection is present in for example the tiger paw mark that a tiger leaves on the bark of a tree to mark territory, or simply paw/foot marks on the ground.

246 points by KqAmJQ7 3y ago

Whilst reading Dr Seuss to kids I observed that even at a very early age they can mimic the expressions on the cartoon characters’ faces, even when they’ve never seen that drawing before. An easy experiment to reproduce.

cheschire 3y ago

I have a different hypothesis. I think line drawings are the representation of vectors. The details we focus on are the vectors with the highest magnitude of change, but that change is not always visual. For example, momentum often has an outsized representation in line drawings. Edges can be a high magnitude change as well, but it's not the only thing.

cropher 3y ago
Interesting; can you elaborate? Also could you clarify what you mean by changes that are "not always visual"? How would non-visual information exist in a purely visual medium?
- cheschire 3y ago
  The example I mentioned was momentum. That's not visual information, that's extrapolated information of position over time. It can be represented in line drawings as motion lines in comics, for example. Interestingly, by simply implying motion, I hypothesize that the brain deprioritizes processing detail on the object that is implied to be moving, and focuses instead on the interactions that will follow. If I were a researcher, my contrived test of this would be to simply have people recreate drawings of "static" objects, and have others recreate drawings of objects implied to be in motion. Other non-visual information would be emotions. The shape of eyes and mouth lines are highly critical to passing emotion. I suspect that people's interpretation of emotion directly impacts how strong the emotional representation of those parts of the face would be drawn. For example if a test subject is told to draw the face of a model in front of them, but they are told the person is experiencing an emotion, I hypothesize that the group of people who are told the person is happy would more frequently bias their interpretation of the eyes, eyebrows, and corners of the mouth towards a "happy" representation than those who are told the person is experiencing great inner turmoil. To be clear though, I'm not saying we only draw based on non-visual information. I'm saying the sum total of all vectors has an influence on the drawing. Colors, in my opinion, have as much of an impact as edges. And it would be interesting to compare the drawings of a person with less common color sensitivities to more common color sensitivities.
plaidfuji 3y ago
You’re on to something here. In the past I did a bunch of work on extracting line drawings from images, and the fundamental goal was always to vectorize the lines - then you get an abstract representation of the figure that you can scale up or down.

cnity 3y ago

As someone (probably like many here) who graduated from a university which taught computer vision and peripheral neuroscience courses, with such titles as "Computational Neuroscience of Vision", I always felt that trying to understand the human brain as a kind of algorithm was a bit of an artefact of computer scientists as they approach biology. The truth is the visual cortex is vast, and not sufficient to explain the human classification and perception of objects visually. Never mind individual neurons or edge perception. Edge detection is an interesting isolated example for study and learning, but you will never come close to explaining human recognition and cognition in such simple terms.

hprotagonist 3y ago
ah, someone who paid attention in lecture! (incidentally; there’s a fairly deep literature of historians of science that have carefully documented that we describe ourselves as analogous to the most sophisticated technology of the day: see “to lose one’s temper”, “to blow a gasket”, “i got my wires crossed”, “sorry, cache miss”, … as metaphors and idioms of mental state through the centuries that reflect the cool tech of the time in which they were coined )
- pavlov 3y ago
  And also the universe itself is often seen through the lens of contemporary technology. Are we living on an island that floats on a giant turtle’s back? Or are the heavens like giant clockworks? Or maybe it’s all a computer simulation? These cosmological speculations are separated by thousands of years, but they are all simply a reflection of what the person finds most awe-inspiring in their everyday life.
  - CuriouslyC 3y ago
    The difference is turtles all the way down was literal, but heavenly clockwork and computer program perspectives are clear metaphors.
    - 3y ago

amelius 3y ago

The human brain also triggers on soft features. It's a combination.

zellyn 3y ago

Wouldn't a simple explanation of line drawings be that segmentation of images into shapes is an important part of vision, using whatever information is available? The reason it would be somewhat invariant across color and lighting is that those change so much. (eg. we can see by moonlight or faint illumination, when color signal is absent) In some cases, simple segmentation fails (like with the shadowed face in the article), and you have to rely on other features.

smokel 3y ago

Line drawings are just easy to produce, historically. This makes them embedded in our culture, and kids learn to read them from an early age. The same skepticism could be raised towards letters and words. If we had invented the photo camera before the paint brush or pen, things might have looked different.

smokel 3y ago

Here's a link to the paper that contains the "my hypothesis". The link in the article runs into a paywall. https://arxiv.org/pdf/2002.06260.pdf https://arxiv.org/pdf/2002.06260.pdf

simias 3y ago

Ten years or so ago I was working on a video chip that had an upscaler feature. While prototyping and simulating it, we first started by applying a mathematically-correct (i.e. information preserving) FIR filter to do the upscale. Then we compared the result with other solutions and found that ours looked worse. We asked our colleagues to blind-test it and they all picked third-party-scaled images over ours. At first we assumed that we must have had a bug somewhere because the Fourrier transform told us that our approach was optimal, but after more testing everything matched the expected output. Yet it looked worse. So we started reverse-engineering the other solutions and, long story short, what they did better is that they added some form of edge-enhancement to the upscaling. Information-theory-wise it actually degraded the image, but subjectively the sharper outlines were just so much nicer to look at and looked correct-er. You felt like you could more easily tell the details even though, again, in a mathematical sense you actually lost information that way. I don't think it makes a lot of sense to reduce human vision to edge detection (we can still make sense of a blurry image like this one after all: https://static0.makeuseofimages.com/wordpress/wp-content/uploads/2013/01/blurred-city.jpg https://static0.makeuseofimages.com/wordpress/wp-content/upl... ) but it's clear to me from empirical evidence that edge-detection is a core aspect of how we parse visual stimuli. As such I'm a bit confused as to why the author seems to see this as a binary proposition. That being said, I could just be misunderstanding completely the point the author is trying to make.

JoeAltmaier 3y ago
The difference between the data in the image, and the information? If for instance you upscaled text so large that it became blurry and unrecognizable, you lost information. Our cortext is all about interpreting what we see. Almost before our brain proper has the data, nerves have begun extracting information (edges etc). Probably because it was the difference between hitting and missing the animal with the spear. Or seeing or missing the tiger in the grass.
- simias 3y ago
  Precisely! I also find it interesting how, from an information theory standpoint, audio processing and image processing are effectively the same thing (audio resampling is fundamentally 1D image scaling for instance) but because humans process sounds very differently from images we end up doing things pretty differently. For instance when we want to subjectively make images more attractive we tend increase contrast and sharpness, whereas for sound we tend to compress it, effectively reducing "audio contrast".

deleted 3y ago

[deleted]

a2128 3y ago

My completely uninformed hypothesis is that it is edge detection, but on a depth map rather than a color map. A figure in the article even shows a situation where a depth sketch succeeds while edge detection on a color map fails

regularfry 3y ago
Kinda both, would be my guess. The brain reconstructs depth from visual information, so where the visual edge can be interpreted as a depth cue, you're faking out that mechanism. It also explains why the pure edge detection image just looks noisy, especially in the hair: most of the edges are effectively incidental colour shifts that don't provide depth information.

jaked89 3y ago

Line drawings work because they represent the centers of symmetry for surfaces and volumes. They trigger the same center-neurons that the original shape would. A hand could appear on you retina in different sizes and orientations. According to distance, the size will grow and shrink. But the center of symmetry will stay the same. This goes further. There's also a center of symmetry between edges, and higher-level features as well. Our brain has no issue detecting these.

JoeAltmaier 3y ago

There was a science fiction story of a race of aliens that had to draw what they found instead of photographing it. A photograph was flat and uninformative to them, almost completely lacking in meaning. Their science was based on interpretation at the root. An interesting concept to put in a story anyway.

bschne 3y ago

> Problem #1: What about all the other features? Is this necessarily a problem in the argument? If we consider e.g. color, and construct a blurry image that has a color distribution very similar to a real face, but doesn't fire the edge detection in the same way, we can still recognize it as a face. This could just mean that a sufficiently close match to other examples of the same class on any one of the strong "dimensions" of the image our brain processes is sufficient to make us recognize it, no? The author makes it sound like if edge detection is what makes us recognize line drawings, this implies that our brain discards every other feature in attempts to recognize visual input, but I don't think that's a sound conclusion.

CapsAdmin 3y ago
This stood out to me as well. If I close my eyes and walk into a wall I will still perceive the wall. Our various senses help build an understanding of what's going on. It's not like we fail to understand anything once you remove one part of the system. Maybe this person is trying to say something deeper or more nuanced and I fail to understand the meaning behind it.
bduffany 3y ago
The "Problem #2" that they mention also has an easy solution. If the "intermediate" variable / internal representation is just "the input image but with edges only," then of course you can see internal representations. When you compute that intermediate variable for the line drawing, it will just happen to behave like the identity function for that particular case. So if you have already filtered out non-edges then the transformation is basically a no-op. The "types" mismatching as they mention is not a concern because the type is just "image" i.e. a big vector of HSL values or something. Edge detection is just a convolution filter so it's going to have approximately the same type as the input.
armoredkitten 3y ago
Yes that stuck out to me as well. The author seems to be setting up a straw man, as if people are arguing that the brain can't distinguish between a line drawing and the real thing (or a line drawing and a photograph). The other information isn't necessarily discarded. It's just used to identify that this is a line drawing and not the real thing. It's still remarkable that just the lines themselves (I make no claim as to whether it really is edge detection or something else) are still enough information to be able to identify the representation, but it doesn't mean the brain is discarding the other information.

coldtea 3y ago

>A classic answer to this question is what I will call the Lines-As-Edges hypothesis. It says that drawings simulate natural images because line features activate edge receptors in the human visual system. Why does the explanation need to go to that direction? Line drawings have a liking to the thing being depicted. They're a crude representation of it (compared to a photograph or a photo-realistic oil painting) but are nonetheless a represenation. That's why they work, in the sense of people understanding what they show: they share similar patterns with the things being show. And we are pattern matches. In this case the patterns are edge patterns, but it could just as well be non-edge patterns. Imagine a color drawing of a human face, where the ink drawn edges have been removed, and it's just blocks of color for the head, the eyes, the pupils, the mouth, the nostrils, ears, etc. We could still tell it's a face, even if we applied some gaussian blur to those blocks. >The most basic statement of the problem with Lines-As-Edges is that the human visual system isn’t just an edge detector. You can see colors, you can see absolute intensities. You can tell the difference between a thin black line and the silhouette of a dark object against a light background; we have both kinds of receptors in the primary visual cortex, as well as others. Yet Lines-As-Edge supposes that the vision system discards all of this other information present in an image, for just this one special case. Why? Isn't this taking things backwards? It's not the vision system which "discards all of this other information present in an image" in our regular operation. Rather, it's the line drawing with does away with (discards) all of this other information and only focuses on a thing's edges. In other words, our visual system has capabilities A, B, C (say edge detection, color, 3d placement, etc). And, a line drawing gives it only A - which is still enough. When our visual system also gets B and C, it can perceive objects even better. But for merely identifying something, A is apparently enough. What I described here is totally compatible with Lines-As-Edges hypothesis, and makes sense too, so I don't see where the author's issue is, and why he thinks the fact that "the human visual system isn’t just an edge detector" invalidates the lines-as-edges hypothesis. >Now you get a sense of the color of the object, and not just its outlines. How would one generalize Lines-As-Edges to account for these different types of depiction? The visual system is no longer ignoring everything aside some gradients; it’s now paying attention to some colors (and not others). Yeah, so? It just means that the visual system can work with less or with more (and multi-type) information. Lines-as-edges is a hypothesis for why line drawings "work" (are recognizable as the thing). The hypothesis doesn't say however that edges are the only thing the visual system can understand. So, there's no need to "generalize Lines-As-Edges to account for these different types of depiction". Lines-as-edges explains how line drawings are understood, period. Line drawings with shading and color, add additional information, aside from the edges. That's fine: no proponent of Lines-As-Edge ever said that the visual system only works with edges. Just that it works with edges when interpreting a line drawing which only offers edges.

henry_pulver 3y ago

This is fascinating! I remember reading that most optical illusions don't work on people raised in remote tribes in the Amazon, as their visual perception has been 'fine-tuned' for jungle contours, instead of the straight lines in the west. Is it possible that we _learn_ how to perceive line drawings in our early years?

lacrimacida 3y ago
I think so and this is true about music as well as other arts.

wzdd 3y ago

I don't understand the essay at all, perhaps because I'm not a domain expert. Edges are clearly a very strong signal in the visual system -- as the article points out, edge detection is one of the first things that happens in the visual pathway from the eyes. However, edges are clearly not the only signal, as the article demonstrates with cross-hatching, colour, and so on (and is also obvious to presumably all sighted humans). If you remove edges as a signal -- say, by taking the coloured apple in the article, removing the lines, and blurring the colours around the silhouette -- you'd probably still recognise an apple, but not as quickly. For the same reason, if you defocus your eyes (or take off your glasses -- the popularity of these is a strong indicator that contour matters), you'll have more difficulty navigating, even though many signals (colour, shade, depth) are still present. Clearly edges are important, but are not the only thing we're working with. Optical illusions also don't invalidate the hypothesis, because they almost by definition rarely occur in nature. Similarly, objects in extreme shadow don't invalidate it either, because, e.g., we are quite capable of recognising half a lion as a lion. I think possibly there's a difference in interpretation here. The claim the author has an issue with is (his phrasing): "the lines in a line drawing are drawn at natural image edges, where an edge receptor would fire. These lines activate the same edge receptor cells that the natural image would. Hence, the line drawing produces a cortical response that is very similar to that of some natural image, and thus you perceive the drawing and the photograph in roughly the same way." But the paper the author links to doesn't say that. It says "The likely explanation is that lines trigger a neural response that has evolved to deal with natural scenes." It's not claiming that line drawings and photographs are perceived in "roughly the same way", only that we evolved to recognise outlines and line drawings are outlines or at least contain outlines. So problem 1 (what about all the other features), problem 3 (what is the benefit), problem 4 (visual art isn't just line drawings), and problem 5 (edge detection is not a line drawing algorithm) don't seem to really address the hypothesis. That just leaves problem 2, "we can't see internal representations". I'm not sure what the statement is here. The author writes "The idea is that we have neurons that activate for object contours and similar, and that line drawings directly activate these neurons. Lines-As-Edges is a special case of this hypothesis. I don’t understand this claim at all." But that claim, as a hypothesis, seems very reasonable? The author seems to be saying that you can't bypass all the neuronal machinery to get directly to the contour-recognising neurons. But that's not true if all you're bypassing is the stuff to isolate the feature of interest (edges, in this case).

Nevermark 3y ago

The article sheds doubt on edge detection as being the major reason why we can interpret drawings as what they attempt to portray, because drawings are often missing shading and color. Drawings also often drop a great amount of detail, and proper proportion information, and remain recognizable. Think of cave paintings or Picasso’s bull. But if you consider image interpretation as a competitive classification, then missing information doesn’t present a problem. If color, shading, detail & proportion are missing, then they are missing from all possible pattern interpretations equally. That leaves the final classification problem relatively unchanged despite all the missing info. EDIT: in fact, if my hypothesis is true we should be able to see patterns with even less information! For instance, dropping all internal detail, and even most shape. As in seeing a face profile on the side of a cloud that otherwise looks nothing like a human head. Or dropping edges completely. Perhaps seeing an object where there are only stars creating points of light. Please post your anecdotal experiments at 404experimentreports.com!

cvoss 3y ago

The brain's ability to do "shadow removal" is really impressive. You can see it happening in the final example depicted in the article. I think this ability is what was used to explain "the dress". It also comes up in a classic optical illusion [0] where a checkerboard of light and dark gray squares has an object sitting on it and casting a shadow. The shadowed light gray squares are exactly the same shade as the unshadowed dark gray squares, but if you point this out to someone, they'll have a really hard time accepting it because they are so good at unconsciously accounting for shadows. [0] https://en.wikipedia.org/wiki/Checker_shadow_illusion https://en.wikipedia.org/wiki/Checker_shadow_illusion

aftbit 3y ago
Hey thanks, this one is new to me. 100% it is impossible to convince myself that those two squares are the same color. I have a camera looking over my driveway with motion detection that triggers all the time because of the shadow of a tree waving in the wind. My current weekend project is to use FastSAM to detect cars / animals / people in the driveway instead of just looking at a threshold of changed pixels. 1: https://github.com/CASIA-IVA-Lab/FastSAM https://github.com/CASIA-IVA-Lab/FastSAM
- thehappypm 3y ago
  It's because the squares are not the same color. The alternating squares are different colors... in absence of a shadow. Your brain is correctly adjusting for the shadow to correctly identify the "true" color.

mistercow 3y ago

Are there any studies on what, if any, animals respond to line drawings (and to what extent)? In particular, do chimpanzees and other apes closely related to humans? It seems somewhat plausible that “lines-as-edges” was a foot in the door for something that specifically evolved in humans as our ancestors began using paintings and drawings for communication. Maybe it was initially just hijacking edge detection so that some images could be conveyed through drawings, and over time, and that developed into a kind of “grammar” for artificially depicting more nuanced images in media where realism was impossible. This would be similar to how recognizing and interpreting different kinds of basic vocalizations (as many animals can) developed into a much more sophisticated mechanism for developing complex language.

drcode 3y ago

when a plane is receding from us, it is more likely to contain color variation, which or eyes sees as a line. it is more likely to contain color variation simply because a plane receding from us has more surface area relative to a unit of space in our visual cortex, versus a plane we are viewing at the position of a normal vector. the edges of a 3d object tend to be receding planes, hence we evolved to detect this

timacles 3y ago

As a person that has been practicing and studying line drawing for 15+ years, this article seems like its way off the mark. Its just not asking the right questions. The brain uses all kinds of context sensitive cues to try to link what its looking at to whats its already seen before. If you happen to look at something thats completely new, it doesnt matter if its a line drawing, 3d 4k image or whatever, your brain will be confused. OTOH, when you're looking at something that you've seen before the brain will do all kinds of tricks and cheats to make that thing as real as possible. Anyone that draws will know you can use plain black and white lines to depict everything from textures, depth, color etc.. Line drawing works because literally anything will work. You could see a distorted, flat black silhouette of one of your family members, and you will instantly know who it is. Its your brain that "makes" things work because its job is to take incomplete information and make it fit an existing mental model. Line drawings carry so much visual information about objects that its really not that impressive your brain can "figure" out what its looking at. It can do so much more with so much less.

regularfry 3y ago
The last picture in the article is an interesting counter-example: a line drawing that's constructed in a fairly natural way that is nevertheless almost impossible to interpret. I'm sure there are "line art rules" that picture breaks which are almost second nature to someone who's been in the game for as long as you have; the interesting thing is how that translates to the visual cortex and why those rules work.
- timacles 3y ago
  That one isn't a line drawing, its a shadow relief drawing with depth removed. Then with with outlines applied to the shadows. Even then, the only reason why its so "confusing" to us is because they applied some kind of random black shape to the background which extends above his head to the left. The black background blends with the man's body making an amorphous blob. If it was removed you'd be able to instantly tell thats a bald mans head. You are correct about the "line art rules" they're all just basic visual rules that the brain uses that artists have been playing with for 100s of years. Its no different than those illusions that look like a Grandma or young woman at the same time. All it takes is one conspicuously placed shape and your whole perspective can be thrown off. No different from how camo works: https://en.wikipedia.org/wiki/Disruptive_coloration https://en.wikipedia.org/wiki/Disruptive_coloration Anyway, my point being, this isn't some new phenomenon, its a well understood at this point.

AndrewKemendo 3y ago

>Yet Lines-As-Edge supposes that the vision system discards all of this other information present in an image, for just this one special case. Why? The "other information" is not discarded, it's just not processed yet I think this is a misrepresentation of how the visual system works and views the vision system as more "batch" than the continuous process it is. So if you think about it as a time series problem, when light hits the retina then "inference and processing" starts with a kind of "fast and rough" inference and then proceeds to fill in details and contextual coherency follows I'd have to go pull out the textbooks but if you look at visual interpretation sequence it's something like: Movement > edges > color > details So your visual system acting as an object detector - from sensor to inference - makes inferences about movement first, then infers the edges, then infers color and finally additional details in the last few ms Nothing is "discarded" it's just less relevant in the first pass and additional refinements happen mostly sequentially - this all happens in nearly imperceptible time.

uglycoyote 3y ago

Very frustrating article to read. The article is setting up a straw man and attacking it. He is acting like everyone else thinks: 1) edges are the only important features in images and 2) line drawings can only represent edges. Who are these brainless absolutists that he is attacking? Then he's acting like he is the only one with other bright ideas that nobody will listen to. I think it is obvious to anyone who thinks about this that: 1) edges are a useful feature for recognizing objects in images but not the only useful feature 2) lines in line drawings can and often do represent edges, but there are a lot of other things they can represent. Light and shading and texture of various kinds. It would be fine to write an article that goes in to depth on the different nuances, but it is annoying that this author pretends that most other experts have naive and simplistic views, with "uncritical certainty", and "no one seems to question it", and the author "has a hard time convincing them otherwise". It is a very condescending tone that comes off sounding like the author is presenting themselves as some brilliant but misunderstood outcast, and the only one who can see the light of truth. we could do without the drama!

arketyp 3y ago
The author comes across as ignorant at best, but then to present his own work as the Realism Hypothesis of Hertzmann, it leans more towards arrogance.
- 6D794163636F756 3y ago
  There is a subset of the tech bro that believes everything can be reduced to a problem with clearly defined taxonomy and as such every problem can be solved by an engineer with no subject knowledge. This article very much reads like one of those people wrote it.
  - wpietri 3y ago
    Yeah, I have been guilty of that myself sometimes. This XKCD is a reminder to me: https://xkcd.com/793/ https://xkcd.com/793/

deleted 3y ago

[deleted]

bambax 3y ago

Figure 4 in the article [1] is absolutely fascinating and seems to prove conclusively that the edge detection hypothesis is completely misguided: when one retains only the edges (B) of the original image (A), all meaning is lost and the resulting image is unrecognizable. What A has that B lacks is shapes. There is obviously a rotund shape of the face that is reconstructed by the brain, that one can almost see, although technically it's not actually present in A. Not present, yet visible. Same for the eyes, smile, cheekbones. It must trigger some kind of pattern-matching in the brain. This problem seems to be similar to the famous optical illusion of the old woman / young woman [2] that works well also in strict black and white. In this optical illusion there is surprisingly little information on the image, yet it can trigger two very different representations (that one can see alternatively, but never at the same time). I think the brain tries to fit the shapes it sees in one box or another, and when it finds a box it builds a whole concept around it. I also think that boxes have to pre-exist or be learned: they can't be inferred from an image, if one has never encountered the original representation in the wild. One evidence for this is children are completely blind to optical illusions that have one "innocent" representation and one involving some kind of nudity or sexual activity, while adults tend to see the NSFW version first. And so, to come back at the original question of why line drawing works, I think it's because it triggers concepts. The word "square" or "circle" are unambiguous and designate precise geometric shapes (provided one has learned the relevant concepts of square and circle). Same with shapes, except that there are an infinite amount of different shapes that we can "discuss" using the language of line drawings. [1] https://aaronhertzmann.com/images/howtodraw/sayim.jpg https://aaronhertzmann.com/images/howtodraw/sayim.jpg [2] https://cdn.mos.cms.futurecdn.net/rQkQZ6pDZbEHz23rxckWPm-320-80.jpg https://cdn.mos.cms.futurecdn.net/rQkQZ6pDZbEHz23rxckWPm-320...

oaktowner 3y ago

I am not a computer vision specialist, but I wanted to add an anecdote from years of parenting. One thing that amazed me about babies was how early they understand line drawings. Long before a baby can talk, it knows what a dog is (and can imitate a dog sound), and can also identify a dog in a photo, in a realistic drawing, and in a very simple line drawing. It seems so easy, but in reality those things all look so very, very different. And after 6ish months, babies have mastered that recognition.

golol 3y ago

Eh. To me it looks like the authors extends "lines-as-edges" explanation of why we can understand line drawings to the claim that the visual cortex works almost only on edge detection, which is obviously bunk and so easily refuted by him. Then later he shows a shillouette image we can understand even though the edges themselves are not understandable, and somehow this is supposed to be evidence against lines as edges. No, it just means that we have other methods of understanding images: shillouette, color, texture etc. In my opinion these kinds of arguments trying to decompose the brains functioning into a couple of distinct techniques become quite obviously pointless once you look at the activation patterns of neural networks. Just looking at the features that neurons detect throughout the different layers of an inage classifying neural network tells you more than these kinds of papers ever will. You see that edges, shillouettes, circles, circles with holes, textures, shinyness patterns, grid patterns etc. up to complex patterns and then real things like heads or arms are detected. There is some manifold of all the images a being is somehow likely to see on this world and it has a complicated structure. You can extract the major features of the geomatry of this manifold and you come across the usual patterns. At simple complexity you find things like edges and textures, at higher complexity things like eyes or appendages. You try to find these features in an image and hope one of them works well. Maybe edges work, maybe shillouettes work or maybe both or neither. Look I know ANNs and NNs are quite different, but the experimental evidence with NNs shows that what I described above, a mix of feature detectors that just approximate the structures of the data to deeper and deeper detail and are just all somehow applied to see what works, is much more plausible than some constructed algorithm a neuroscientist or philosopher would write down.

kortex 3y ago
> Look I know ANNs and NNs are quite different, but the experimental evidence with NNs shows that what I described above, a mix of feature detectors that just approximate the structures of the data to deeper and deeper detail and are just all somehow applied to see what works, is much more plausible than some constructed algorithm a neuroscientist or philosopher would write down. Precisely. Deep CNNs were directly inspired by studying the structure and behavior of visual cortical tissue. The algorithm is just recursive feature detection, where the visual elements are transformed into abstractions, based on both sensory input and projected expectations. If anything, that's the real exciting part of the visual system, why do illusions occur, how does expectation affect perception, etc. Not "how do line drawings evoke similar concepts to images". That's just bypassing the first few layers of filters. Basic ass tiny MNIST nets can do this.

porphyra 3y ago

Drawing lines around semantic segmentation, such as using Segment Anything [1] seems to make a lot more sense than just doing edge detection on image brightness. [1] https://segment-anything.com/ https://segment-anything.com/

radicaldreamer 3y ago

Figuring this out is one of the central questions of phenomenology... how is it that after seeing a couple of trees, we can recognize almost any kind of tree from any angle? What is intrinsic about a tree that makes this possible? What (or is) there a pure form of a tree?

mrguyorama 3y ago
Since definitions aren't concrete, and language is explicitly a popularity contest, you are not learning "what makes a tree a tree", but rather "what people call a tree" You are re-deriving an existing definition, not an actual physical reality. This is a great example actually, because there is no genetic or biologic definition of a tree, and what we call a tree is matched better by some idealized representation of info like "has a trunk maybe". Also that the definition you develop is rarely rigorously tested in practice, so you only need about 80%, and basic pattern matching can get you that really easily.

lookdangerous 3y ago

A drawing of a cup has 'cup-ness', as far as our mind is concerned. It's not about the micro mechanisms of edge detection or color or whatever, it's about how we recognize the quality of a cup in real cups and things that push the margins for what a cup is and in drawings of a cup.

bazzargh 3y ago

Huh, right up my alley... a while back I was playing with trying to turn photos into sketches ( https://hachyderm.io/@bazzargh/109928618521729073 https://hachyderm.io/@bazzargh/109928618521729073 ); one of the effects I found really noticeable with my super-naive approach was that it tended to overemphasise very dark areas in a way we don't see. Then last week I saw this, which also does some fill in shading (but, using ML)-worth the click, it's a project that makes it appear as if paper held over an object causes it to be sketched...but in fact it's all post-processing... https://www.youtube.com/watch?v=vArIkzYtW6I https://www.youtube.com/watch?v=vArIkzYtW6I This got me back to wondering about the shading problem, and I ended up down a rabbithole reading papers like https://www.yorku.ca/rfm/pub/2021annrev.pdf https://www.yorku.ca/rfm/pub/2021annrev.pdf (review article on state of the art), https://www.frontiersin.org/articles/10.3389/fpsyg.2022.915672/full#B35 https://www.frontiersin.org/articles/10.3389/fpsyg.2022.9156... (recent markov model, with links to code) about lightness and brightness perception, and how our mental model of how the scene is lit can explain a bunch of optical illusions. While I was going for a charcoal effect, this approach https://openprocessing.org/sketch/486307 https://openprocessing.org/sketch/486307 - scribble using brightness as 'gravity' on the pen - is pretty nice. Anyhoo, I'm not going to critique the article, because I'm a total amateur just doing this for fun, but it _is_ fun, and very satisfying when you get the computer to draw something that looks hand drawn instead of just a sobel filter.

brap 3y ago

This was a very interesting read. From childhood we instinctively draw lines, but I never questioned how we produce them from real objects, and why they work so well at representing real objects. It’s one of those things we just take for granted.

eternityforest 3y ago

Re. problem 2, Running edge detection twice will still give a recognizable result. Maybe some functions DO let you take the result of an internal layer, feed it to the input, and not get complete nonsense

rajnathani 3y ago

> Why do line drawings work? Why is it that we can immediately recognize objects in line drawings, even though they are not a phenomenon from our natural world. This is wrong, as humans have been drawing and viewing art for tens of thousands of years. Also, for other animals, a form of edge detection is present in for example the tiger paw mark that a tiger leaves on the bark of a tree to mark territory, or simply paw/foot marks on the ground.