8 ms·

NTSB report: Decryption of images from the Titan submersible camera [pdf] (2024)

191 points by bmurray7jhu 11mo ago

jipsy 11mo ago
[dead]
bmurray7jhu 11mo ago
Report on unrecoverable SSDs: https://data.ntsb.gov/Docket/Document/docBLOB?ID=19169363&FileExtension=pdf&FileName=DCA23FM036%20Electronic%20Devices3_party%20review-Rel.pdf https://data.ntsb.gov/Docket/Document/docBLOB?ID=19169363&Fi... Full docket: https://data.ntsb.gov/Docket/?NTSBNumber=DCA23FM036 https://data.ntsb.gov/Docket/?NTSBNumber=DCA23FM036
dwohnitmok 11mo ago
I'm confused. Why are decryption keys in NVRAM? That seems to negate the purpose of at-rest encryption if you can retrieve keys from the device even after shutdown.
- daemonologist 11mo ago
  Well they're encrypting an SD card, so this still defends against its being removed from the camera and stolen or left in a bar or something. But honestly from the rest of the story it sounds like the camera manufacturer was selling their pressure housing moreso than the off-the-shelf camera hardware inside, and was not particularly concerned with whether/how the storage was encrypted.
- tremon 11mo ago
  What would you say is the threat model that leads one to auto-encrypt sdcards? For a machine that needs to boot unattended, what would you do with disk decryption keys?
  - dwohnitmok 11mo ago
    > What would you say is the threat model that leads one to auto-encrypt sdcards? Jumping off of daemonologist's point, the most convincing model I could think of is one where you are consistently removing the SD Card, likely because you are swapping multiple SD cards back and forth. But this still seems bad! In that case your multiple SD cards all share the same decryption key (or I guess less egregiously all at least share the same physical vulnerability of that device's nonvolatile memory). > For a machine that needs to boot unattended, what would you do with disk decryption keys? This depends entirely on what you mean by unattended. Here's some candidates: 1. The machine has a network connection and you would like to boot it over the network. In that case you have a small unencrypted bit that handles the boot sequence and then pass the decryption key over the wire, store in volatile memory, and wipe on shutdown or logout. 2. The machine is regularly serviced/booted by an in-person operator. You give the decryption key or passphrase to the operator to decrypt and boot the machine. 3. The machine is meant to autonomously reboot itself due to e.g. error conditions. In extreme cases when the machine detects that there is a large probability of physical compromise, it shuts down. In this case for the former I would imagine some power supply, e.g. a backup battery, that keeps the volatile memory up when rebooting and in case of large probability of physical compromise there is a forced shutdown and/or wipe of the volatile memory. 4. An operator needs to periodically boot up the machine, but the operator should not be able to access any previous data the machine has recorded (data from the moment operator arrives forward is assumed leaked to the operator because e.g. the operator can just position their own sensor in front of the machine and collect data). In that case asymmetric encryption should be used and the encryption keys should be stored on the machine, but the decryption keys should not, effectively turning the machine into a write-only device from the operator's perspective. The only case I could imagine where you want the machine to boot unattended and you'd keep the decryption keys in nonvolatile memory on the device is if you want an operator to be able to boot the device without knowledge of the decryption key and also to be able to view historical data previously recorded by the device. But in that case there is no functional difference between that and a device that is unencrypted! In both cases you boot up the device without a password and voila all the data is in front of you ready to be read! And indeed that's basically what happened here (modulo the difficulties associated with the device being damaged). And in that case encryption seems like pure overhead for no real security benefit.
cbsks 11mo ago
Wow. SubC’s software engineering needs some work. They thought the camera’s file system was unencrypted, when it was encrypted. They didn’t know where the keys were to decrypt it. It turned out the key was written unencrypted to a UFS storage device. There was a file written to /mnt/nas/Stills, which indicates that the camera was to writing to a remote file system that wasn’t mounted.
- userbinator 11mo ago
  They thought the camera’s file system was unencrypted, when it was encrypted. Unfortunately this situation is likely to get more common in the future as the "security" crowd keep pushing for encryption-by-default with no regard to whether the user wants or is even aware of it. Encryption is always a tradeoff; it trades the possibility of unauthorised access with the possibility of even the owner losing access permanently. IMHO this tradeoff needs careful consideration and not blind application.
  - jiggawatts 11mo ago
    This is why I always shake my head when the Reddit armchair security experts say "The data wasn't even encrypted!? Amateur hour!" in response to some PII leak. Sure, sure buddy, I'll encrypt all of my PII data so nobody can access it... including the web application server. Okay, fine, I'll decrypt it on the fly with a key in some API server... now the web server had unencrypted access to it, which sounds bad, but that's literally the only way that it can process and serve the data to users in a meaningful way! Now if someone hacks the web app server -- the common scenario -- then the attacker has unencrypted access! I can encrypt the database, but at what layer? Storage? Cloud storage is already encrypted! Backups? Yeah, sure, but then what happens in a disaster? Who's got the keys? Are they contactable at 3am? Etc, etc... It's not only not as simple as ticking an "encrypted: yes" checkbox, it's maximally difficult, with a very direct tradeoff between accessibility and protection. The sole purpose of encrypting data is to prevent access!
    - londons_explore 11mo ago
      I like the approach of mega.nz... Server stores encrypted blobs. Server doesn't have the keys. Entire application is on the client, and just downloads and decrypts what it needs. Obviously your entire application stack needs to be developed with that approach in mind, and some things like 'make a hyperlink to share this' get much more complex.
watersb 11mo ago
The SD card on the camera was intact but encrypted. Decrypting the data required a key stored on a separate SOM board, but the SOM was damaged. The investigation team delivered the SOM and SD card to the camera manufacturer in Newfoundland, and they were able to decrypt the card. They found a couple of images, but No data with a timestamp after May 16th was found on the camera, so it is likely that none of the data recorded on the SD Card were of the accident voyage or dive. After all that work... If you're interested in data recovery, you will enjoy reading this report, about 10 pages, clearly written. The technical language mentioned they didn't see a LUKS header on the card so they figured it was a custom dm_crypt setup.
- cloudbonsai 11mo ago
  > No data with a timestamp after May 16th was found on the camera, so it is likely that none of the data recorded on the SD Card were of the accident voyage or dive. Evidently the camera data was recorded to an external SSD card in the mission computer when the accident occurred. The investigation team actually managed to salvage the PC as well: https://data.ntsb.gov/Docket/Document/docBLOB?ID=19169363&FileExtension=pdf&FileName=DCA23FM036%20Electronic%20Devices3_party%20review-Rel.pdf https://data.ntsb.gov/Docket/Document/docBLOB?ID=19169363&Fi... Sadly it turned into a compressed ball of metal...
  - djmips 11mo ago
    That's a striking image! Thanks for sharing - that really hits home on the pressures involved.
  - londons_explore 11mo ago
    Pretty sure tech exists to recover data from flash memory with cracked dies... I guess they decided it wasn't worth pursuing.
    - dwohnitmok 11mo ago
      > Pretty sure tech exists to recover data from flash memory with cracked dies... If you have anymore on this would love to see any relevant materials.
rozab 11mo ago
What's with the entire dev board crammed in there? Is that... normal? What board is it?
- daemonologist 11mo ago
  It appears to be a Teensy 3.2 The "carrier" that everything rides on within the housing is clearly FDM printed as well. I assume these cameras (rated to 6,000 meters) are rather low volume products.
  - buildbot 11mo ago
    It honestly makes sense. You are paying for the pressure engineering, and can take advantage of an off the shelf camera system. Maybe use a special lens or filter or something but why bother customizing the software/hardware of the camera much. They probably should still know what it's doing though...
  - analog31 11mo ago
    I came here to say "that's definitely a Teensy 3.2" The black cable goes dangerously close to the pushbutton of death. ;-)
- duskwuff 11mo ago
  The small board on the left is unmistakably a Teensy 3.2: https://www.pjrc.com/store/teensy32.html https://www.pjrc.com/store/teensy32.html As to what it's doing in there, I have no idea.
  - djmips 11mo ago
    Well, I'll be darned! I wonder if Paul Stoffregen knows about that! edit: probably? It was posted at the Teensy forum about a month ago. https://forum.pjrc.com/index.php?threads/the-deepest-teensy.77396/ https://forum.pjrc.com/index.php?threads/the-deepest-teensy....
- whalesalad 11mo ago
  Looks like a pi zero
daemonologist 11mo ago
Previous discussion (October 17th): https://news.ycombinator.com/item?id=45613898 https://news.ycombinator.com/item?id=45613898 Also a good video from Scott Manley: https://youtu.be/qMUjCZ7MMWQ https://youtu.be/qMUjCZ7MMWQ
pseudolus 11mo ago
There's a fascinating and redacted interview with an "anonymous" subject about the disaster. To say the least it's an unsuccessful attempt to hide the identity of the individual: "Q. So how did you get yourself started into submersible operations? A. Well, I'm sure you're familiar with my film Titanic. When I set down the path to make that film, the first thing that I did was arrange to be introduced to the head of the submersible program at the P.P. Shirshov Institute in Moscow, a guy named..." https://media.defense.gov/2025/Sep/17/2003800984/-1/-1/0/CG-115_INTERVIEW-DEEP-SEA-EXPLORER_REDACTED.PDF https://media.defense.gov/2025/Sep/17/2003800984/-1/-1/0/CG-...
- vlovich123 11mo ago
  Let’s hear from L Simpson. No, that’s too specific. Let’s hear from Lisa S.
  - vlovich123 11mo ago
    Took me forever to find the actual quote - ChatGPT and Gemini kept trying to gaslight me that it’s not a real quote or that Willie said it to Bart until I gave the exact quote (at which point normal search engines were fine): > A certain agitator, for privacy's sake, let's call her Lisa S. No, that's too obvious, let's say L. Simpson. Lisa the Vegetarian
    - edoceo 11mo ago
      When I grow up I'm going to Bovine University!
    - maybelsyrup 11mo ago
      In the future, Frinkiac [0] is your friend [0] https://frinkiac.com/ https://frinkiac.com/
    - monkpit 11mo ago
      If you google “Lisa S L Simpson” it’s the first thing that comes up… why make it harder with AI?
Zak 11mo ago
> Removed SD card. The manufacturer of the camera had requested certain components of the device be redacted. Portions of this image have been redacted. And so it is, but anyone who has ever seen a Sandisk SD card knows what they're looking at. I can even tell it's not the fastest V90 speed. The things companies try ineffectually to keep out of public view are weird.
- vayup 11mo ago
  Especially when anyone can buy the product off the shelf, remove the casing to see what they are trying to redact in these images.
- kldg 10mo ago
  yeah, I'd guess the person who made this decision just wanted to cover their butt. it got a good laugh out of me reading the caption, though. "hmmm. who could use this distinct branding so many people are familiar with? it looks a bit orange; maybe it's Home Depot."
deleted 11mo ago
[deleted]
JCM9 11mo ago
Amusing that the bits the “manufacturer asked to be redacted” in the images appear to be the identifiers for common off-the-shelf electronic components, including a standard memory card. Is that really super secret IP?
- rasz 11mo ago
  It is if you are a camera manufacturer. Another example https://www.cined.com/whats-inside-a-red-mini-mag-the-controversy-jarred-lands-statement/ https://www.cined.com/whats-inside-a-red-mini-mag-the-contro...
- gosub100 11mo ago
  could be a PR / brand identity management thing. They dont want their slogan to be come "The official Storage Medium of Deadly Disasters".
alhirzel 11mo ago
Crazy that it's pretty much a 3D printed assembly internally, and the manufacturer didn't know how it worked. No way that would pass any kind of vibration test.

191 points by bmurray7jhu 11mo ago

jipsy 11mo ago

[dead]

Report on unrecoverable SSDs: https://data.ntsb.gov/Docket/Document/docBLOB?ID=19169363&FileExtension=pdf&FileName=DCA23FM036%20Electronic%20Devices3_party%20review-Rel.pdf https://data.ntsb.gov/Docket/Document/docBLOB?ID=19169363&Fi... Full docket: https://data.ntsb.gov/Docket/?NTSBNumber=DCA23FM036 https://data.ntsb.gov/Docket/?NTSBNumber=DCA23FM036

dwohnitmok 11mo ago

I'm confused. Why are decryption keys in NVRAM? That seems to negate the purpose of at-rest encryption if you can retrieve keys from the device even after shutdown.

daemonologist 11mo ago
Well they're encrypting an SD card, so this still defends against its being removed from the camera and stolen or left in a bar or something. But honestly from the rest of the story it sounds like the camera manufacturer was selling their pressure housing moreso than the off-the-shelf camera hardware inside, and was not particularly concerned with whether/how the storage was encrypted.
tremon 11mo ago
What would you say is the threat model that leads one to auto-encrypt sdcards? For a machine that needs to boot unattended, what would you do with disk decryption keys?
- dwohnitmok 11mo ago
  > What would you say is the threat model that leads one to auto-encrypt sdcards? Jumping off of daemonologist's point, the most convincing model I could think of is one where you are consistently removing the SD Card, likely because you are swapping multiple SD cards back and forth. But this still seems bad! In that case your multiple SD cards all share the same decryption key (or I guess less egregiously all at least share the same physical vulnerability of that device's nonvolatile memory). > For a machine that needs to boot unattended, what would you do with disk decryption keys? This depends entirely on what you mean by unattended. Here's some candidates: 1. The machine has a network connection and you would like to boot it over the network. In that case you have a small unencrypted bit that handles the boot sequence and then pass the decryption key over the wire, store in volatile memory, and wipe on shutdown or logout. 2. The machine is regularly serviced/booted by an in-person operator. You give the decryption key or passphrase to the operator to decrypt and boot the machine. 3. The machine is meant to autonomously reboot itself due to e.g. error conditions. In extreme cases when the machine detects that there is a large probability of physical compromise, it shuts down. In this case for the former I would imagine some power supply, e.g. a backup battery, that keeps the volatile memory up when rebooting and in case of large probability of physical compromise there is a forced shutdown and/or wipe of the volatile memory. 4. An operator needs to periodically boot up the machine, but the operator should not be able to access any previous data the machine has recorded (data from the moment operator arrives forward is assumed leaked to the operator because e.g. the operator can just position their own sensor in front of the machine and collect data). In that case asymmetric encryption should be used and the encryption keys should be stored on the machine, but the decryption keys should not, effectively turning the machine into a write-only device from the operator's perspective. The only case I could imagine where you want the machine to boot unattended and you'd keep the decryption keys in nonvolatile memory on the device is if you want an operator to be able to boot the device without knowledge of the decryption key and also to be able to view historical data previously recorded by the device. But in that case there is no functional difference between that and a device that is unencrypted! In both cases you boot up the device without a password and voila all the data is in front of you ready to be read! And indeed that's basically what happened here (modulo the difficulties associated with the device being damaged). And in that case encryption seems like pure overhead for no real security benefit.

cbsks 11mo ago

Wow. SubC’s software engineering needs some work. They thought the camera’s file system was unencrypted, when it was encrypted. They didn’t know where the keys were to decrypt it. It turned out the key was written unencrypted to a UFS storage device. There was a file written to /mnt/nas/Stills, which indicates that the camera was to writing to a remote file system that wasn’t mounted.

userbinator 11mo ago
They thought the camera’s file system was unencrypted, when it was encrypted. Unfortunately this situation is likely to get more common in the future as the "security" crowd keep pushing for encryption-by-default with no regard to whether the user wants or is even aware of it. Encryption is always a tradeoff; it trades the possibility of unauthorised access with the possibility of even the owner losing access permanently. IMHO this tradeoff needs careful consideration and not blind application.
- jiggawatts 11mo ago
  This is why I always shake my head when the Reddit armchair security experts say "The data wasn't even encrypted!? Amateur hour!" in response to some PII leak. Sure, sure buddy, I'll encrypt all of my PII data so nobody can access it... including the web application server. Okay, fine, I'll decrypt it on the fly with a key in some API server... now the web server had unencrypted access to it, which sounds bad, but that's literally the only way that it can process and serve the data to users in a meaningful way! Now if someone hacks the web app server -- the common scenario -- then the attacker has unencrypted access! I can encrypt the database, but at what layer? Storage? Cloud storage is already encrypted! Backups? Yeah, sure, but then what happens in a disaster? Who's got the keys? Are they contactable at 3am? Etc, etc... It's not only not as simple as ticking an "encrypted: yes" checkbox, it's maximally difficult, with a very direct tradeoff between accessibility and protection. The sole purpose of encrypting data is to prevent access!
  - londons_explore 11mo ago
    I like the approach of mega.nz... Server stores encrypted blobs. Server doesn't have the keys. Entire application is on the client, and just downloads and decrypts what it needs. Obviously your entire application stack needs to be developed with that approach in mind, and some things like 'make a hyperlink to share this' get much more complex.

watersb 11mo ago

The SD card on the camera was intact but encrypted. Decrypting the data required a key stored on a separate SOM board, but the SOM was damaged. The investigation team delivered the SOM and SD card to the camera manufacturer in Newfoundland, and they were able to decrypt the card. They found a couple of images, but No data with a timestamp after May 16th was found on the camera, so it is likely that none of the data recorded on the SD Card were of the accident voyage or dive. After all that work... If you're interested in data recovery, you will enjoy reading this report, about 10 pages, clearly written. The technical language mentioned they didn't see a LUKS header on the card so they figured it was a custom dm_crypt setup.

cloudbonsai 11mo ago
> No data with a timestamp after May 16th was found on the camera, so it is likely that none of the data recorded on the SD Card were of the accident voyage or dive. Evidently the camera data was recorded to an external SSD card in the mission computer when the accident occurred. The investigation team actually managed to salvage the PC as well: https://data.ntsb.gov/Docket/Document/docBLOB?ID=19169363&FileExtension=pdf&FileName=DCA23FM036%20Electronic%20Devices3_party%20review-Rel.pdf https://data.ntsb.gov/Docket/Document/docBLOB?ID=19169363&Fi... Sadly it turned into a compressed ball of metal...
- djmips 11mo ago
  That's a striking image! Thanks for sharing - that really hits home on the pressures involved.
- londons_explore 11mo ago
  Pretty sure tech exists to recover data from flash memory with cracked dies... I guess they decided it wasn't worth pursuing.
  - dwohnitmok 11mo ago
    > Pretty sure tech exists to recover data from flash memory with cracked dies... If you have anymore on this would love to see any relevant materials.

rozab 11mo ago

What's with the entire dev board crammed in there? Is that... normal? What board is it?

daemonologist 11mo ago
It appears to be a Teensy 3.2 The "carrier" that everything rides on within the housing is clearly FDM printed as well. I assume these cameras (rated to 6,000 meters) are rather low volume products.
- buildbot 11mo ago
  It honestly makes sense. You are paying for the pressure engineering, and can take advantage of an off the shelf camera system. Maybe use a special lens or filter or something but why bother customizing the software/hardware of the camera much. They probably should still know what it's doing though...
- analog31 11mo ago
  I came here to say "that's definitely a Teensy 3.2" The black cable goes dangerously close to the pushbutton of death. ;-)
duskwuff 11mo ago
The small board on the left is unmistakably a Teensy 3.2: https://www.pjrc.com/store/teensy32.html https://www.pjrc.com/store/teensy32.html As to what it's doing in there, I have no idea.
- djmips 11mo ago
  Well, I'll be darned! I wonder if Paul Stoffregen knows about that! edit: probably? It was posted at the Teensy forum about a month ago. https://forum.pjrc.com/index.php?threads/the-deepest-teensy.77396/ https://forum.pjrc.com/index.php?threads/the-deepest-teensy....
whalesalad 11mo ago
Looks like a pi zero

daemonologist 11mo ago

Previous discussion (October 17th): https://news.ycombinator.com/item?id=45613898 https://news.ycombinator.com/item?id=45613898 Also a good video from Scott Manley: https://youtu.be/qMUjCZ7MMWQ https://youtu.be/qMUjCZ7MMWQ

pseudolus 11mo ago

There's a fascinating and redacted interview with an "anonymous" subject about the disaster. To say the least it's an unsuccessful attempt to hide the identity of the individual: "Q. So how did you get yourself started into submersible operations? A. Well, I'm sure you're familiar with my film Titanic. When I set down the path to make that film, the first thing that I did was arrange to be introduced to the head of the submersible program at the P.P. Shirshov Institute in Moscow, a guy named..." https://media.defense.gov/2025/Sep/17/2003800984/-1/-1/0/CG-115_INTERVIEW-DEEP-SEA-EXPLORER_REDACTED.PDF https://media.defense.gov/2025/Sep/17/2003800984/-1/-1/0/CG-...

vlovich123 11mo ago
Let’s hear from L Simpson. No, that’s too specific. Let’s hear from Lisa S.
- vlovich123 11mo ago
  Took me forever to find the actual quote - ChatGPT and Gemini kept trying to gaslight me that it’s not a real quote or that Willie said it to Bart until I gave the exact quote (at which point normal search engines were fine): > A certain agitator, for privacy's sake, let's call her Lisa S. No, that's too obvious, let's say L. Simpson. Lisa the Vegetarian
  - edoceo 11mo ago
    When I grow up I'm going to Bovine University!
  - maybelsyrup 11mo ago
    In the future, Frinkiac [0] is your friend [0] https://frinkiac.com/ https://frinkiac.com/
  - monkpit 11mo ago
    If you google “Lisa S L Simpson” it’s the first thing that comes up… why make it harder with AI?

Zak 11mo ago

> Removed SD card. The manufacturer of the camera had requested certain components of the device be redacted. Portions of this image have been redacted. And so it is, but anyone who has ever seen a Sandisk SD card knows what they're looking at. I can even tell it's not the fastest V90 speed. The things companies try ineffectually to keep out of public view are weird.

vayup 11mo ago
Especially when anyone can buy the product off the shelf, remove the casing to see what they are trying to redact in these images.
kldg 10mo ago
yeah, I'd guess the person who made this decision just wanted to cover their butt. it got a good laugh out of me reading the caption, though. "hmmm. who could use this distinct branding so many people are familiar with? it looks a bit orange; maybe it's Home Depot."

deleted 11mo ago

[deleted]

JCM9 11mo ago

Amusing that the bits the “manufacturer asked to be redacted” in the images appear to be the identifiers for common off-the-shelf electronic components, including a standard memory card. Is that really super secret IP?

rasz 11mo ago
It is if you are a camera manufacturer. Another example https://www.cined.com/whats-inside-a-red-mini-mag-the-controversy-jarred-lands-statement/ https://www.cined.com/whats-inside-a-red-mini-mag-the-contro...
gosub100 11mo ago
could be a PR / brand identity management thing. They dont want their slogan to be come "The official Storage Medium of Deadly Disasters".

alhirzel 11mo ago

Crazy that it's pretty much a 3D printed assembly internally, and the manufacturer didn't know how it worked. No way that would pass any kind of vibration test.