5 ms·

The Night Watch: The Missing Pieces

170 points by BjoernKW 5y ago

DFHippie 5y ago
A vaguely similar anecdote: My grandfather, a taciturn, unsentimental Quaker mechanical engineer, was given the job of preparing a Methodist church to be a Quaker meetinghouse. The church had tall stained glass windows, but Quaker meetings are un- or minimally adorned. Well, Dan was a competent, efficient fellow. In short order the meeting was fit for occupancy again. The stained glass windows were gone except for a rose window. Some time after some of the Methodists came by asking what had become of the tall stained glass windows. They were memorial windows and they wanted to keep them as memorials. No one knew. Questions went around. Eventually someone asked Dan. He said they were under the gravel in the parking lot. What! Why are they there!? He just said, "they were ugly." I hope the spare pieces of the Night Watch weren't ugly, and if they were, the job of fitting them to the space wasn't given to some 18th century Dan.
- hutzlibu 5y ago
  Or the elderly women, who "restored" a historical painting. https://en.wikipedia.org/wiki/Ecce_Homo_(Mart%C3%ADnez_and_Gim%C3%A9nez) https://en.wikipedia.org/wiki/Ecce_Homo_(Mart%C3%ADnez_and_G...
  - Aeolun 5y ago
    This will forever remain hilarious.
- MomoXenosaga 5y ago
  It is entirely possible that at the time the Nightwatch was considered ugly. Let us not forget Rembrandt died an impoverished man. How art is appreciated changes with the times.
  - coyotespike 5y ago
    To be fair, Rembrandt was famous and very well-employed in his own day, he just spent money like it was going out of style. Had he saved a little he'd have died rich.
dominicjj 5y ago
That is brilliant. I saw the Night Watch at the end of a busy day and it was pretty much the only thing I had time for in the Rijks Museum. I will have to return to Amsterdam and see the expanded version.
- BrandoElFollito 5y ago
  The first time I went to the Rijks Museum I almost missed it. I saw on the left side of the main gallery (this is from memory, 30 years back) a small painting (say, 40x20 cm) which had a name similar (or the same) as The Night Watch and I thought that it is a great deal for such a nice but average painting. Then I saw some people standing in the middle of the gallery watching something and I discovered the painting :)
  - BjoernKW 5y ago
    The impact also depends on the direction from which one approaches the painting. If you enter from the room to the right hand side of the Night Watch it's almost easy to miss, not least because the ship models in the rooms that precede it are quite impressive, too. From the Great Hall and the Eregalerij (Gallery of Honour) the effect is positively awe-inspiring. I've once heard someone describe the Rijksmuseum as a cathedral of Enlightenment. The Night Watch is the centrepiece that cathedral was built around.
    - BrandoElFollito 5y ago
      I do not remember how I got there but I remember that there was a superbe exhibition of the activities on ships of the ~17th century. I vividly remember the doctor's equipment (just a few) and realized that being injured on a boat at that time was really bad. I think there was also an exhibition on the history of doll houses
  - prawn 5y ago
    I like how strong your memories can be of art in context. I can remember times I rounded a corner decades ago to first see a major piece, how many people were around, what the room was like, etc. I remember the second time I saw Guernica I had my brother with me, who hadn't seen it before. I picked an intentional path to it so his focus would be in one direction initially and then he'd turn and see it for the first time from a great position. My first viewing I'd approached from a pretty dull angle.
defaultname 5y ago
Very interesting watch. What sort of NN would be used to warp two compositionally similar images to match each other in structure? In the example the copy painting had almost all of the same general components, laid out with fairly significant differences, and they warped it to match quite precisely.
- danbruc 5y ago
  Without a neural network one would probably use optical flow [1] and I guess you could train a neural network to predict optical flow and then use that flow to align the two inputs. I have no answer to your specific question, what kind of neural network architecture would be best suited but given the task it seems reasonable to assume some form of convolutional neural network. [1] https://en.wikipedia.org/wiki/Optical_flow https://en.wikipedia.org/wiki/Optical_flow
- jvanderbot 5y ago
  I'm taking your question without context, so sorry if it doesn't make sense: Warping images to match based on structure / features is common in much of image processing and requires no NN since we know how to do it. (But may be enhanced by NN). The basic idea is: 1. Find features in images A and B (SURF / SIFT are common). However, maybe a NN could find interesting features here? 2. Match features so you know what feature from A is where in B. Again, a NN could potentially do this "better". 3. Compute a homography (translation between planes of features) or procrustes transform (or other types of transformations) 4. For each pixel (p) in the target frame (B), take the inverse of the transformation to find the pixel of the original frame (A) (which is usually a fractional pixel because it won't line up) 5. interpolate between nearby pixels in (A) to get the "true" pixel value at (p) in (B). Perhaps a NN could improve this interpolation? We've seen this in super resolution, for example. You can easily reproduce these steps with just a few function calls in OpenCV. Try it out. https://opencv.org/ https://opencv.org/ Up to step 3 is very helpful to answer questions like: "How much did the camera move between captures of image A and B. So now you can solve structure from motion, visual odometry, etc. This is typically how quadrotors navigate with cameras.
  - 5y ago
stefanpie 5y ago
Robert Erdmann also gave a keynote talk at PyCon US 2021 about how they used Python for most of digitization and processing of the Night Watch (aka Operation Night Watch). Fascinating talk with tones of technical details. Link: https://www.youtube.com/watch?v=z_hm5oX7ZlE&list=PL2Uw4_HvXqvYk1Y5P8kryoyd83L_0Uk5K https://www.youtube.com/watch?v=z_hm5oX7ZlE&list=PL2Uw4_HvXq...
ffhhj 5y ago
Let's make those paintings Ships of Theseus: recreate the missing parts around the paintings up to universe scale.
- remram 5y ago
  Like Theseus recreated the universe around his ship?
Daub 5y ago
Art teacher speaking. Cropping is surprisingly easily to detect in an aesthetic image. Its kind of like reading a half-finished book. Aesthetic images are designed for an defined format. If that format is changed nearing completion, the geometry feels wrong. It is even easier in paintings, where there are overt brush marks. The structure of a brush mark is strongly framed by the format of the canvas. Anyone interested in the recreation of famous incomplete paintings, should check out Manet's The Execution of Emperor Maximilian. This is a painting Manet gave up on, and chopped up into smaller studies. https://www.nationalgallery.org.uk/media/33986/n-3294-00-000069-hd.jpg https://www.nationalgallery.org.uk/media/33986/n-3294-00-000... There are sketches that indicate the format he was aiming for, but Manet was a famous experimenter, so we will probably never know.
appleflaxen 5y ago
For anyone else who wants context before watching: this is about the use of AI to reconstruct missing pieces of a Rembrandt painting ("Night Watch") using a contemporaneous copy from Lundens. After the Lundens painting was made, the Rembrandt original was cropped aggressively (~30 cm on a side) in order to fit onto a particular wall where it hung. Now that it is considered a masterpiece, they are trying to computationally estimate what those cropped sections looked like using Lunden's copy.
shadeslayer_ 5y ago
Anyone else thought this was about the Night's Watch in the Game of Thrones books/novels? I'll see myself out..
- astrange 5y ago
  Could've at least thought it was about a Dutch translation of the Terry Pratchett book.
bruce343434 5y ago
I was hoping for a before/after comparison picture, but I can't find it.
ElViajero 5y ago
This is a cool project. The part that I am not so sure about is why don't they just use an artist to paint the missing part? What is the advantage of using an AI? My guess, even that I hope not, is that people thinks that AI is going to be more impartial, more accurate, less biased. The reality is that an AI, as used in this example, is going to be over-fitting. An artist can understand Lunden and Rembrant art and use that knowledge to produce something closer to the intention of the artist and the copy. Cool project, but a dangerous precedent if people thinks that AI provides unbiased reality, that using AI you can show "how it really looked" as something more accurate that any human con achieve. Constructing something from missing data is always guesswork, whenever the intelligence is natural or artificial.
- treve 5y ago
  They mention in the video that they could ask a person to make the changes with Photoshop, but the reason they give is: "Then it would be that artist's work, not Rembrandt". That statement rubs me the wrong way a bit though, because whether you ask a person or a computer model to copy Rembrandt's style, in each case it's still a copy, and I would argue in each case there's still a(n) artist(s) mimicking Rembrandt, just using different tools. > My guess, even that I hope not, is that people thinks that AI is going to be more impartial, more accurate, less biased. The reality is that an AI, as used in this example, is going to be over-fitting. > Cool project, but a dangerous precedent if people thinks that AI provides unbiased reality, that using AI you can show "how it really looked" as something more accurate that any human con achieve. Constructing something from missing data is always guesswork, whenever the intelligence is natural or artificial. I think this is an excellent take.
- floatrock 5y ago
  I'm not gonna anthropomorphize the AI and say "the AI is the artist", but the AI is definitely an art style. I see it as just an old-meets-new mashup, craftfully executed. Any artist/technique you use isn't going to be the original Rembrant, so the choice of what artist/technique you use is just a question of aesthetics and taste. Here we're just exploring a new technique of the 21st century.
- johncalvinyoung 5y ago
yk 5y ago
Using a neural network to complete lost parts of a canonical masterpiece reminds me of nothing as much as using some kind of necromancer to ask the spirit of the artists to complete the picture. And of course depending on the quality of the scholar of the dark arts, we expect a similar quality of the result. So the thing is, the completion is a modern artifact, and fundamentally in both cases it captures the expertise of the reconstructor, and not some kind of essence of the painting.

170 points by BjoernKW 5y ago

A vaguely similar anecdote: My grandfather, a taciturn, unsentimental Quaker mechanical engineer, was given the job of preparing a Methodist church to be a Quaker meetinghouse. The church had tall stained glass windows, but Quaker meetings are un- or minimally adorned. Well, Dan was a competent, efficient fellow. In short order the meeting was fit for occupancy again. The stained glass windows were gone except for a rose window. Some time after some of the Methodists came by asking what had become of the tall stained glass windows. They were memorial windows and they wanted to keep them as memorials. No one knew. Questions went around. Eventually someone asked Dan. He said they were under the gravel in the parking lot. What! Why are they there!? He just said, "they were ugly." I hope the spare pieces of the Night Watch weren't ugly, and if they were, the job of fitting them to the space wasn't given to some 18th century Dan.

hutzlibu 5y ago
Or the elderly women, who "restored" a historical painting. https://en.wikipedia.org/wiki/Ecce_Homo_(Mart%C3%ADnez_and_Gim%C3%A9nez) https://en.wikipedia.org/wiki/Ecce_Homo_(Mart%C3%ADnez_and_G...
- Aeolun 5y ago
  This will forever remain hilarious.
MomoXenosaga 5y ago
It is entirely possible that at the time the Nightwatch was considered ugly. Let us not forget Rembrandt died an impoverished man. How art is appreciated changes with the times.
- coyotespike 5y ago
  To be fair, Rembrandt was famous and very well-employed in his own day, he just spent money like it was going out of style. Had he saved a little he'd have died rich.

dominicjj 5y ago

That is brilliant. I saw the Night Watch at the end of a busy day and it was pretty much the only thing I had time for in the Rijks Museum. I will have to return to Amsterdam and see the expanded version.

BrandoElFollito 5y ago
The first time I went to the Rijks Museum I almost missed it. I saw on the left side of the main gallery (this is from memory, 30 years back) a small painting (say, 40x20 cm) which had a name similar (or the same) as The Night Watch and I thought that it is a great deal for such a nice but average painting. Then I saw some people standing in the middle of the gallery watching something and I discovered the painting :)
- BjoernKW 5y ago
  The impact also depends on the direction from which one approaches the painting. If you enter from the room to the right hand side of the Night Watch it's almost easy to miss, not least because the ship models in the rooms that precede it are quite impressive, too. From the Great Hall and the Eregalerij (Gallery of Honour) the effect is positively awe-inspiring. I've once heard someone describe the Rijksmuseum as a cathedral of Enlightenment. The Night Watch is the centrepiece that cathedral was built around.
  - BrandoElFollito 5y ago
    I do not remember how I got there but I remember that there was a superbe exhibition of the activities on ships of the ~17th century. I vividly remember the doctor's equipment (just a few) and realized that being injured on a boat at that time was really bad. I think there was also an exhibition on the history of doll houses
- prawn 5y ago
  I like how strong your memories can be of art in context. I can remember times I rounded a corner decades ago to first see a major piece, how many people were around, what the room was like, etc. I remember the second time I saw Guernica I had my brother with me, who hadn't seen it before. I picked an intentional path to it so his focus would be in one direction initially and then he'd turn and see it for the first time from a great position. My first viewing I'd approached from a pretty dull angle.

defaultname 5y ago

Very interesting watch. What sort of NN would be used to warp two compositionally similar images to match each other in structure? In the example the copy painting had almost all of the same general components, laid out with fairly significant differences, and they warped it to match quite precisely.

danbruc 5y ago
Without a neural network one would probably use optical flow [1] and I guess you could train a neural network to predict optical flow and then use that flow to align the two inputs. I have no answer to your specific question, what kind of neural network architecture would be best suited but given the task it seems reasonable to assume some form of convolutional neural network. [1] https://en.wikipedia.org/wiki/Optical_flow https://en.wikipedia.org/wiki/Optical_flow
jvanderbot 5y ago
I'm taking your question without context, so sorry if it doesn't make sense: Warping images to match based on structure / features is common in much of image processing and requires no NN since we know how to do it. (But may be enhanced by NN). The basic idea is: 1. Find features in images A and B (SURF / SIFT are common). However, maybe a NN could find interesting features here? 2. Match features so you know what feature from A is where in B. Again, a NN could potentially do this "better". 3. Compute a homography (translation between planes of features) or procrustes transform (or other types of transformations) 4. For each pixel (p) in the target frame (B), take the inverse of the transformation to find the pixel of the original frame (A) (which is usually a fractional pixel because it won't line up) 5. interpolate between nearby pixels in (A) to get the "true" pixel value at (p) in (B). Perhaps a NN could improve this interpolation? We've seen this in super resolution, for example. You can easily reproduce these steps with just a few function calls in OpenCV. Try it out. https://opencv.org/ https://opencv.org/ Up to step 3 is very helpful to answer questions like: "How much did the camera move between captures of image A and B. So now you can solve structure from motion, visual odometry, etc. This is typically how quadrotors navigate with cameras.
- 5y ago

stefanpie 5y ago

Robert Erdmann also gave a keynote talk at PyCon US 2021 about how they used Python for most of digitization and processing of the Night Watch (aka Operation Night Watch). Fascinating talk with tones of technical details. Link: https://www.youtube.com/watch?v=z_hm5oX7ZlE&list=PL2Uw4_HvXqvYk1Y5P8kryoyd83L_0Uk5K https://www.youtube.com/watch?v=z_hm5oX7ZlE&list=PL2Uw4_HvXq...

ffhhj 5y ago

Let's make those paintings Ships of Theseus: recreate the missing parts around the paintings up to universe scale.

remram 5y ago
Like Theseus recreated the universe around his ship?

Daub 5y ago

Art teacher speaking. Cropping is surprisingly easily to detect in an aesthetic image. Its kind of like reading a half-finished book. Aesthetic images are designed for an defined format. If that format is changed nearing completion, the geometry feels wrong. It is even easier in paintings, where there are overt brush marks. The structure of a brush mark is strongly framed by the format of the canvas. Anyone interested in the recreation of famous incomplete paintings, should check out Manet's The Execution of Emperor Maximilian. This is a painting Manet gave up on, and chopped up into smaller studies. https://www.nationalgallery.org.uk/media/33986/n-3294-00-000069-hd.jpg https://www.nationalgallery.org.uk/media/33986/n-3294-00-000... There are sketches that indicate the format he was aiming for, but Manet was a famous experimenter, so we will probably never know.

appleflaxen 5y ago

For anyone else who wants context before watching: this is about the use of AI to reconstruct missing pieces of a Rembrandt painting ("Night Watch") using a contemporaneous copy from Lundens. After the Lundens painting was made, the Rembrandt original was cropped aggressively (~30 cm on a side) in order to fit onto a particular wall where it hung. Now that it is considered a masterpiece, they are trying to computationally estimate what those cropped sections looked like using Lunden's copy.

shadeslayer_ 5y ago

Anyone else thought this was about the Night's Watch in the Game of Thrones books/novels? I'll see myself out..

astrange 5y ago
Could've at least thought it was about a Dutch translation of the Terry Pratchett book.

bruce343434 5y ago

I was hoping for a before/after comparison picture, but I can't find it.

ElViajero 5y ago

This is a cool project. The part that I am not so sure about is why don't they just use an artist to paint the missing part? What is the advantage of using an AI? My guess, even that I hope not, is that people thinks that AI is going to be more impartial, more accurate, less biased. The reality is that an AI, as used in this example, is going to be over-fitting. An artist can understand Lunden and Rembrant art and use that knowledge to produce something closer to the intention of the artist and the copy. Cool project, but a dangerous precedent if people thinks that AI provides unbiased reality, that using AI you can show "how it really looked" as something more accurate that any human con achieve. Constructing something from missing data is always guesswork, whenever the intelligence is natural or artificial.

treve 5y ago
They mention in the video that they could ask a person to make the changes with Photoshop, but the reason they give is: "Then it would be that artist's work, not Rembrandt". That statement rubs me the wrong way a bit though, because whether you ask a person or a computer model to copy Rembrandt's style, in each case it's still a copy, and I would argue in each case there's still a(n) artist(s) mimicking Rembrandt, just using different tools. > My guess, even that I hope not, is that people thinks that AI is going to be more impartial, more accurate, less biased. The reality is that an AI, as used in this example, is going to be over-fitting. > Cool project, but a dangerous precedent if people thinks that AI provides unbiased reality, that using AI you can show "how it really looked" as something more accurate that any human con achieve. Constructing something from missing data is always guesswork, whenever the intelligence is natural or artificial. I think this is an excellent take.
floatrock 5y ago
I'm not gonna anthropomorphize the AI and say "the AI is the artist", but the AI is definitely an art style. I see it as just an old-meets-new mashup, craftfully executed. Any artist/technique you use isn't going to be the original Rembrant, so the choice of what artist/technique you use is just a question of aesthetics and taste. Here we're just exploring a new technique of the 21st century.
johncalvinyoung 5y ago

yk 5y ago

Using a neural network to complete lost parts of a canonical masterpiece reminds me of nothing as much as using some kind of necromancer to ask the spirit of the artists to complete the picture. And of course depending on the quality of the scholar of the dark arts, we expect a similar quality of the result. So the thing is, the completion is a modern artifact, and fundamentally in both cases it captures the expertise of the reconstructor, and not some kind of essence of the painting.