4 ms·
a Film is a collection of shots, scenes and sequences. Shots are fairly easy to detect. Most editing software comes with built in shot detection. This approach
by Daub 7y ago
a Film is a collection of shots, scenes and sequences. Shots are fairly easy to detect. Most editing software comes with built in shot detection.
This approach promises to detect scenes. I’m not enough of a coder to know if it is valid, but I can certainly imagine that it is a solvable problem. Most scenes are defined by location, and movies makers like each locatation to be distinct from the preceding one.
To my mind, sequences are where the action is at. Sequences are defined by narrative development... These can usually be defined by a trained human, but are probably tough to computationly define.
There is also a huge difference between film genres. The scenes and sequences of Finding Nemo are pretty easy to define. But you try the same approach on an art house film like The Scent of Green Papaya, and see how far you get.
https://www.helpingwritersbecomeauthors.com/movie-storystructure/finding-nemo/ https://www.helpingwritersbecomeauthors.com/movie-storystruc...
- ekianjo 7y agoIf you mix pictures and audio the scene detection is bound to be more accurate since there is usually patterns there in most movies.
- Breakthrough 7y agoAuthor of the program here. This is definitely something that is worth while considering, although I'm not too sure if a correlation always exists between the audio and picture patterns (e.g. in a music video, scene transitions may not be exactly synchronized with the audio). I've also seen people combine the cuts detected by PySceneDetect with subtitle (.srt) information to generate a more comprehensive list of scenes, which are sets of shots joined together by some context (to allow for jumping forward to a particular scene by it's description).
- mrob 7y agoDoes this program really detect scenes by the film-making definition? In video encoding terminology, shots are commonly called "scenes". E.g. x264's "scenecut" option really sets the sensitivity for detecting new shots, so that I-frames can be inserted there to avoid the inefficiency of encoding differences from a different shot.
- Breakthrough 7y agoAuthor of the program here. You are correct, the program detects shots rather than scenes, but I didn't want to give the impression that this project was related to the existing ShotDetect program. I felt that the documentation explained this well enough, but I'm open to considering an alternative project name if anyone has a suggestion.
- samstave 7y agoYeah. It would be kind of awesome if we could “hash tag” scenes such that at some point we could search fot “remember that #action scene in movie X where the [something happens]” and it can pull up that clip of the scene...
- IanCal 7y agoSearching over subtitles works well for some of this, if you did transcribed audio-description you'd probably have something pretty cool.
- Breakthrough 7y agoI've seen a proof of concept which combined the output of PySceneDetect with subtitle information and computer vision to allow you to do something like "go to the scene with the big castle" or something similar. I can't remember what it's called off the top of my head, but it seemed like a pretty cool concept. Disclaimer: I'm the author of PySceneDetect.
- samstave 7y agoGod i love HN
- bryanrasmussen 7y agoYou could get pretty close to that by allowing people to bookmark and tag timestamps in films. In fact I'm pretty sure we are close to being able to do it, but implementations lag because it would be a feature for services that do not generally compete on features but only on content.
- onion2k 7y agoDetecting scenes seems incredibly difficult to me. Even in a basic edit you could have back to back shots looking at very different parts of a single set. Take the "Jack Rabbit Slims" scene from Pulp Fiction - Vincent Vega walking through the diner, the car table head to head scene about the milkshake with linking cuts around the diner when they discuss the two Marilyns, and the beginning of the Twist competition are three scenes that use the same location, actors, background, etc. Finding the boundaries would be very hard.
- joshvm 7y agoOne approach might be to estimate the intrinsic parameters of the camera based on keypoint tracking. Assuming the camera was adjusted between shots, you might be able to get that to work. Or you could make some kind of generic feature vector from each frame that includes contrast, etc. The interiors are the same, but the camera focus/fov will likely be different. Nowadays just take an annotated movie and throw it into an encoder/decoder network to assess if a frame is from the same scene?
- Breakthrough 7y agoSurprisingly, this uses a fairly simple heuristic to accomplish just that (HSV deltas). I'm the author of the program, and it actually works pretty well with Pulp Fiction as a whole. The one area where PySceneDetect does struggle with currently, however, is dealing with sudden/rapid flashes, or momentary obstructions to the viewing scene.
- s_Hogg 7y agoThe example they show in the repo seems to indicate that it detects shots rather than scenes
- Breakthrough 7y agoAuthor of the program here. Yes, the program detects shots rather than scenes, but I didn't want to give the impression that this project was related to the existing ShotDetect program. I felt that the documentation explained this well enough, but I'm open to considering an alternative project name if anyone has a suggestion.
- danabrams 7y agoBeyond all that, good filmmakers often use edits to manipulate or confound expectations about time and space. However, there are scenes in the narrative sense, but also scene main the way an AD making a schedule will see them, that is, scene headings (sluglines), which would be easier to differentiate and detect.
- IanCal 7y agoI had fun some years ago working on some sentiment analysis, and found if you took subtitles you could often see where general changes happened. Splitting that over where the histogram jumped you could more reasonably split things over camera cuts. Plot an exponentially smoothed sentiment and you could see patterns for types of content (sadness then uplift at the end of current affairs shows for example). I built it for auto-summarisation of video and categorisation as a few day hack and it was OK (better than I expected, not good enough to use). Totally failed at sports though, as the commentary usually has more varied speech when there's little happening.