4 ms·

Scaleway: Incident SSD was stolen during a secure transport between datacenters

99 points by mot2ba 5y ago

KronisLV 5y ago
Google Translate link into English: https://translate.google.com/translate?sl=auto&tl=en&u=https://blog.scaleway.com/incident-securitaire-video-youtube/ https://translate.google.com/translate?sl=auto&tl=en&u=https... Edit: seems like the blog itself can be read in English with the button at the bottom, yet the articles themselves aren't necessarily translated after clicking on it. Localization is hard.
jve 5y ago
So, did the YouTuber extract any data from that drive or not? If not, I suspect there wouldn't be a headline? + The text starts with: "Over a year ago, an SSD was stolen...". I may be wrong of course. But would gladly know more info, if anyone has it.
- malka 5y ago
  You can find the first of 3 videos from this youtuber about this subject here: https://youtu.be/vt8PyQ2PGxI https://youtu.be/vt8PyQ2PGxI It's in French though
- deleted 5y ago
  [deleted]
- mot2ba 5y ago
  Yep, the French vlogger Micode [0] asked their Twitter followers to identify the data on an SSD they purchased on the local classified ads website leboncoin. [0]: https://web.archive.org/web/20210531091659/https://twitter.com/Micode/status/1395640486715662336 https://web.archive.org/web/20210531091659/https://twitter.c... [1]: https://www.youtube.com/watch?v=vt8PyQ2PGxI https://www.youtube.com/watch?v=vt8PyQ2PGxI [2]: https://www.youtube.com/watch?v=aOBVZUL1iBA https://www.youtube.com/watch?v=aOBVZUL1iBA [3]: https://www.youtube.com/watch?v=xf_cKTlOYLo https://www.youtube.com/watch?v=xf_cKTlOYLo
- drouar_b 5y ago
  For those who don't speak french, this Youtuber bought a second-handed SSD to do some forensics and raise awareness. He managed to recover the partition, inside he found qcow2 image of a company, he managed to mount it and recover the full data, including source code of their app, a ssh key and S3 credentials of the company
  - jve 5y ago
    Thanks. Is there any info whether drive was formatted or not?
    - drouar_b 5y ago
      The drive had a quick format, so he managed to recover the partition with testdisk
      - foxpurple 5y ago
        These days every single drive should have full disk encryption. And then you can format by simply securely wiping the key part.
        quietbritishjim 5y ago
        SSDs work in an odd way: they have more capacity than they claim, and when you overwrite some data you're really just writing to some of that spare space while the old version is just marked as free [1, 2]. Admittedly, at that point it's going to be very hard to retrieve the old data (you'd need to bypass the remapping logic presented by the controller circuitry), but it's theoretically possible. If you're paranoid, it's best to rewrite the whole thing even if it's encrypted, ideally several times (at that point a little original data could technically be left over but it's unlikely and it's not going to be feasible to get any of it if it was encrypted). [1] https://databasearchitects.blogspot.com/2021/06/what-every-programmer-should-know-about.html https://databasearchitects.blogspot.com/2021/06/what-every-p... [2] https://news.ycombinator.com/item?id=27572218 https://news.ycombinator.com/item?id=27572218
- sirk390 5y ago
  Yes a running VM from an a corporate customer containing all source code, SSH keys, access keys to github repos, AWS keys and much more.
Moonlight_TC 5y ago
Scaleway's claim that customers were immediately informed («prévenu la clientèle potentiellement impactée») appears to be false. Affected customers were not notified until June 2021. [1] Scaleway did not publish their blog post until after the 3 part video series by Micode, despite being aware of the incident since May 2021. [2] [1] https://www.lowendtalk.com/discussion/172819/scaleway-ssd-with-customer-data-purchased-on-classified-ads-website-by-french-vlogger https://www.lowendtalk.com/discussion/172819/scaleway-ssd-wi... [2] https://twitter.com/Micode/status/1395640486715662336 https://twitter.com/Micode/status/1395640486715662336
adriancarrieres 5y ago
They collaborated so much with Micode that they threatened him when he disclosed the SSD was from a cloud provider (without giving away its name) As for the other things, at the end of the 2nd video he did succeed on extracting and gaining access to the data, with full code source, AWS and Facebook (bot account) credentials (among others). The exploration of the (redacted) data is in part 3
belter 5y ago
File Carving: https://resources.infosecinstitute.com/topic/file-carving/ https://resources.infosecinstitute.com/topic/file-carving/
formerly_proven 5y ago
And why exactly is the bare storage of these not encrypted? I would expect, at the very least, that the data is encrypted using customer-specific keys.
- gberger 5y ago
  The article doesn't say whether the disk was encrypted or not. Where did you find this info?
  - lrem 5y ago
    If it was encrypted, then the news wouldn't be about someone recovering a quick formatted partition. It would be about someone breaking encryption.
  - quietbritishjim 5y ago
    > The article doesn't say whether the disk was encrypted or not. So we can conclude that it was not. If the disk was encrypted (with decent passphrase/key) there is 0% chance that the article wouldn't mention it. They also wouldn't have bothered to get the disk back from the YouTuber. Why would they if the data is unreadable anyway?
- deleted 5y ago
  [deleted]
dsign 5y ago
They ublish all their blog entries in English, but this one is in French. Why? I would understand if the information were addressed to French stakeholders, but this information is most relevant to (perhaps international, perhaps prospective) customers.
angauber 5y ago
This raise a question, why weren't they using encryption on their qcow2 volumes ?
zenexer 5y ago
I’m having a little trouble following since I don’t speak French, but based on other comments here, it sounds like: 1. French YouTuber Micode bought a used SSD from leboncoin. 2. Micode wanted to demonstrate that data should always be properly wiped from used drives. 3. The SSD Micode obtained had been quick-formatted and was never encrypted, so it was trivial to recover the data on it. 4. Micode asked his followers on Twitter to try to identify the source of the drive. 5. It was eventually identified as belonging to Scaleway, and it contained important data from a Scaleway customer’s VM, including an SSH key, source code, and an S3 secret. 6. Scaleway threatened Micode, who now claims to have wiped the data. 7. Scaleway published a blog post claiming the drive was stolen while being transported between datacenters. Unless I’m missing something that was lost in translation, I call bullshit on #7. They also seem to be claiming customers were notified immediately, but it that doesn’t appear to be the case. This just seems like they sold an old drive that should’ve been encrypted (it wasn’t) and wiped (it wasn’t). Whether that sale was authorized is a matter of debate—one former employee said they wouldn’t be surprised if someone just decided to walk out of the building with decommissioned hardware.
- sofixa 5y ago
  > This just seems like they sold an old drive that should’ve been encrypted (it wasn’t) and wiped (it wasn’t I find that unlikely ( unless as you said it was a rogue employee) - they're certified HDS ( hosting healthcare data) so i find it improbable they aren't supposed to have a special disk decommissioning process to follow, including secure wiping.
  - orwin 5y ago
    Yeah, our procedure for decommissioned SSD and HDD (imposed to us by HDS requirement) where 3-pass shred on the disk (logical wiping), and either pierce the HDD or microwave the SSD, with photographic proof send to our client if asked. The 3-pass shred was done on the virtual disk each time a client left, and was done on the physical during maintenance. [edit] And concerning encryption, it was at a premium, and i think only one of our client asked for it (this had a few issue). "hot" logs where not encrypted, but during the logrotation we tried to implement this (this was not technically audited btw, i think it was OK for Deloitte not to see our code)
malka 5y ago
Now this has utterly disappeared from the front page. wtf happened ?
MisterTea 5y ago
Sounds like someone forgot to run 'dd if=/dev/zero of=/dev/sdX'

99 points by mot2ba 5y ago

Google Translate link into English: https://translate.google.com/translate?sl=auto&tl=en&u=https://blog.scaleway.com/incident-securitaire-video-youtube/ https://translate.google.com/translate?sl=auto&tl=en&u=https... Edit: seems like the blog itself can be read in English with the button at the bottom, yet the articles themselves aren't necessarily translated after clicking on it. Localization is hard.

jve 5y ago

So, did the YouTuber extract any data from that drive or not? If not, I suspect there wouldn't be a headline? + The text starts with: "Over a year ago, an SSD was stolen...". I may be wrong of course. But would gladly know more info, if anyone has it.

malka 5y ago
You can find the first of 3 videos from this youtuber about this subject here: https://youtu.be/vt8PyQ2PGxI https://youtu.be/vt8PyQ2PGxI It's in French though
deleted 5y ago
[deleted]
mot2ba 5y ago
Yep, the French vlogger Micode [0] asked their Twitter followers to identify the data on an SSD they purchased on the local classified ads website leboncoin. [0]: https://web.archive.org/web/20210531091659/https://twitter.com/Micode/status/1395640486715662336 https://web.archive.org/web/20210531091659/https://twitter.c... [1]: https://www.youtube.com/watch?v=vt8PyQ2PGxI https://www.youtube.com/watch?v=vt8PyQ2PGxI [2]: https://www.youtube.com/watch?v=aOBVZUL1iBA https://www.youtube.com/watch?v=aOBVZUL1iBA [3]: https://www.youtube.com/watch?v=xf_cKTlOYLo https://www.youtube.com/watch?v=xf_cKTlOYLo
drouar_b 5y ago
For those who don't speak french, this Youtuber bought a second-handed SSD to do some forensics and raise awareness. He managed to recover the partition, inside he found qcow2 image of a company, he managed to mount it and recover the full data, including source code of their app, a ssh key and S3 credentials of the company
- jve 5y ago
  Thanks. Is there any info whether drive was formatted or not?
  - drouar_b 5y ago
    The drive had a quick format, so he managed to recover the partition with testdisk
    - foxpurple 5y ago
      These days every single drive should have full disk encryption. And then you can format by simply securely wiping the key part.
      - quietbritishjim 5y ago
        SSDs work in an odd way: they have more capacity than they claim, and when you overwrite some data you're really just writing to some of that spare space while the old version is just marked as free [1, 2]. Admittedly, at that point it's going to be very hard to retrieve the old data (you'd need to bypass the remapping logic presented by the controller circuitry), but it's theoretically possible. If you're paranoid, it's best to rewrite the whole thing even if it's encrypted, ideally several times (at that point a little original data could technically be left over but it's unlikely and it's not going to be feasible to get any of it if it was encrypted). [1] https://databasearchitects.blogspot.com/2021/06/what-every-programmer-should-know-about.html https://databasearchitects.blogspot.com/2021/06/what-every-p... [2] https://news.ycombinator.com/item?id=27572218 https://news.ycombinator.com/item?id=27572218
sirk390 5y ago
Yes a running VM from an a corporate customer containing all source code, SSH keys, access keys to github repos, AWS keys and much more.

Moonlight_TC 5y ago

Scaleway's claim that customers were immediately informed («prévenu la clientèle potentiellement impactée») appears to be false. Affected customers were not notified until June 2021. [1] Scaleway did not publish their blog post until after the 3 part video series by Micode, despite being aware of the incident since May 2021. [2] [1] https://www.lowendtalk.com/discussion/172819/scaleway-ssd-with-customer-data-purchased-on-classified-ads-website-by-french-vlogger https://www.lowendtalk.com/discussion/172819/scaleway-ssd-wi... [2] https://twitter.com/Micode/status/1395640486715662336 https://twitter.com/Micode/status/1395640486715662336

adriancarrieres 5y ago

They collaborated so much with Micode that they threatened him when he disclosed the SSD was from a cloud provider (without giving away its name) As for the other things, at the end of the 2nd video he did succeed on extracting and gaining access to the data, with full code source, AWS and Facebook (bot account) credentials (among others). The exploration of the (redacted) data is in part 3

belter 5y ago

File Carving: https://resources.infosecinstitute.com/topic/file-carving/ https://resources.infosecinstitute.com/topic/file-carving/

formerly_proven 5y ago

And why exactly is the bare storage of these not encrypted? I would expect, at the very least, that the data is encrypted using customer-specific keys.

gberger 5y ago
The article doesn't say whether the disk was encrypted or not. Where did you find this info?
- lrem 5y ago
  If it was encrypted, then the news wouldn't be about someone recovering a quick formatted partition. It would be about someone breaking encryption.
- quietbritishjim 5y ago
  > The article doesn't say whether the disk was encrypted or not. So we can conclude that it was not. If the disk was encrypted (with decent passphrase/key) there is 0% chance that the article wouldn't mention it. They also wouldn't have bothered to get the disk back from the YouTuber. Why would they if the data is unreadable anyway?
deleted 5y ago
[deleted]

dsign 5y ago

They ublish all their blog entries in English, but this one is in French. Why? I would understand if the information were addressed to French stakeholders, but this information is most relevant to (perhaps international, perhaps prospective) customers.

angauber 5y ago

This raise a question, why weren't they using encryption on their qcow2 volumes ?

zenexer 5y ago

I’m having a little trouble following since I don’t speak French, but based on other comments here, it sounds like: 1. French YouTuber Micode bought a used SSD from leboncoin. 2. Micode wanted to demonstrate that data should always be properly wiped from used drives. 3. The SSD Micode obtained had been quick-formatted and was never encrypted, so it was trivial to recover the data on it. 4. Micode asked his followers on Twitter to try to identify the source of the drive. 5. It was eventually identified as belonging to Scaleway, and it contained important data from a Scaleway customer’s VM, including an SSH key, source code, and an S3 secret. 6. Scaleway threatened Micode, who now claims to have wiped the data. 7. Scaleway published a blog post claiming the drive was stolen while being transported between datacenters. Unless I’m missing something that was lost in translation, I call bullshit on #7. They also seem to be claiming customers were notified immediately, but it that doesn’t appear to be the case. This just seems like they sold an old drive that should’ve been encrypted (it wasn’t) and wiped (it wasn’t). Whether that sale was authorized is a matter of debate—one former employee said they wouldn’t be surprised if someone just decided to walk out of the building with decommissioned hardware.

sofixa 5y ago
> This just seems like they sold an old drive that should’ve been encrypted (it wasn’t) and wiped (it wasn’t I find that unlikely ( unless as you said it was a rogue employee) - they're certified HDS ( hosting healthcare data) so i find it improbable they aren't supposed to have a special disk decommissioning process to follow, including secure wiping.
- orwin 5y ago
  Yeah, our procedure for decommissioned SSD and HDD (imposed to us by HDS requirement) where 3-pass shred on the disk (logical wiping), and either pierce the HDD or microwave the SSD, with photographic proof send to our client if asked. The 3-pass shred was done on the virtual disk each time a client left, and was done on the physical during maintenance. [edit] And concerning encryption, it was at a premium, and i think only one of our client asked for it (this had a few issue). "hot" logs where not encrypted, but during the logrotation we tried to implement this (this was not technically audited btw, i think it was OK for Deloitte not to see our code)

malka 5y ago

Now this has utterly disappeared from the front page. wtf happened ?

MisterTea 5y ago

Sounds like someone forgot to run 'dd if=/dev/zero of=/dev/sdX'