6 ms·

SSD belonging to Euro-cloud Scaleway was stolen,then turned up on YouTube

90 points by ronalbarbaren 5y ago

fulafel 5y ago
So was there cleartext data on it?
- hdjjhhvvhga 5y ago
  Yes, the article clearly says the data was unencrypted.
detaro 5y ago
2 days ago: https://news.ycombinator.com/item?id=27957471 https://news.ycombinator.com/item?id=27957471
- mot2ba 5y ago
  I posted that thread and I don't know why it was penalty by mod, completely out of front page HN.
terom 5y ago
Apparently with plaintext customer data. > Lechelle said Scaleway worked with the YouTuber to recover the disk. The French-language video creator has written to Scaleway with assurances they have not copied the information contained on the disk. It is said some customer data was on the drive, unencrypted, including the source code and SSH keys of an Italian VPS provider.
- terom 5y ago
  Based on links in the other hackernews thread, it was quick-formatted, but a filesystem containing qemu disk images was recovered: [1] The YouTube video is in French without English CC subtitles, but there's screenshots of exploring what I assume would be one of the qcow images: [2] * 03:59 `/root/.ssh` with SSH id_rsa (private) keys, (public) authorized_keys and (hashed) known_hosts * 07:14 server binaries * 08:42 server logs * 12:59 AWS/S3 secret access keys * 13:33 Source code... in QBASIC ?! must be very valuable legacy code :D * 17:30 brute-forcing the ssh known_hosts * etc [1] https://mobile.twitter.com/Micode/status/1395640486715662336 https://mobile.twitter.com/Micode/status/1395640486715662336 [2] https://www.youtube.com/watch?v=xf_cKTlOYLo https://www.youtube.com/watch?v=xf_cKTlOYLo
  - TonyTrapp 5y ago
    Note: QB64 (which is mentioned in the video) is a modern-day reimplementation of QBasic. It doesn't have to be legacy. Some people still prefer using Basic as their "scripting language" for quick tasks where other people would be using Python or JS these days.
    - alyandon 5y ago
      Although I never cared much for the language itself, I continued to write small programs in QB for many years because of its ability to produce an EXE file that ran in DOS and Windows without any dependencies.
  - rasz 5y ago
    >it was quick-formatted are there OSes that perform quick format without issuing TRIM command to the SSD? TRIM would vanish the data in less than 15 minutes.
    - djrogers 5y ago
      Yes, 15 minutes is not 'quick' by most standards - quick formats take seconds, not minutes.
      - rasz 5y ago
        up to ~15 minutes is not the time it takes to quick format, its the number Iv seen during testing of TRIM, the time it took to internally erase translation table after format command. There should be NO DATA on the drive if it was really formatted on OS supporting TRIM.
    - BBC-vs-neolibs 5y ago
      Not if you turn the computer off right after deleting the files.
    - fomine3 5y ago
      I believe parted won't issue TRIM for whole disk/partition after erasing partition table or partition. Possibly mkfs.xfs issues TRIM? I don't know. Anyway, what they should at least is just issue `blkdiscard /dev/sdb`, or betterly Secure Erase (on SATA) or Sanitize (on NVMe).
jnwatson 5y ago
Why in the world do they ship unencrypted drives around? Drive encryption is pretty standard these days, isn't it?
- atonse 5y ago
  With the big cloud provides its a trivial checkbox. Can anyone speak to the effort in doing it at the OS level?
  - tpxl 5y ago
    There's a checkbox to encrypt the disks when installing some linux distros.
    - jsiepkes 5y ago
      > There's a checkbox to encrypt the disks when installing some linux distros. While I'm not going to defend moving around unencrypted disks it is not as simple as that. The difficult part is not the encryption. The difficult part is the key management of the encryption. You need to solve issue like ensuring you don't lose access to the keys (and thereby access to the data), securely providing keys to people and machines that need access, dealing with key revocation when people leave the company, etc. Though I guess in this case using a simple (long enough) passphrase instead of some form of keys could have saved them from this particular threat.
      - atonse 5y ago
        I thought any of these providers always encrypted with their own key whether you encrypted on top of that or not. But I guess not. Yet another reason to just stick with the big providers. They seem to have pretty serious compliance needs and encrypt no matter what. (Yet you should still encrypt your assets, but I believe they all encrypt their volumes anyway and have pretty strict policies on how to dispose this kind of hardware)
      - rkeene2 5y ago
        When I ran a cloud provider, we used a hardware security module on storage systems which had public/private key pair which could be used to decrypt the header on the disk which contained a copy of the symmetric cipher key. Each header had the symmetric cipher key encrypted with multiple different public keys, including a fallback whose private key only existed in on paper in a vault. Each system could reach out to other storage systems to ask them to use their private key to decrypt the header (for example if their hardware security module had failed), but in some configurations this would require an operator to intervene to enter in passphrase to unlock the hardware security module to authorize the action. This means that: 1. The symmetric cipher key could always be recovered, even from paper backup 2. Having physical access to a disk or any set of disks did not allow you to read the data 3. Having physical access to a disk and a hardware security module did not allow you to read the disk (unless you knew the passphrase, which was always present, and user set) 4. Having physical access to disks, servers, and hardware security module may not allow you to read the data (in the more secure configuration where passphrases were not cached on disk -- but this meant that rebooting required an operator to manually enter the password at boot) 5. The set of valid public keys could be changed frequently (this was indeed automated and only the set of currently active hardware security modules could decrypt the current disk header) Of course you could always short circuit this by making a copy of the symmetric key when you did have access (e.g., `dmsetup table --showkeys`) but without putting some more hardware in the fast/hot path that was unavoidable. The symmetric key could not easily be changed, without rewriting the entire disk (though since it was a storage system designed to accommodate multiple failures, this wasn't that hard, but we didn't do it automatically) The hardware security modules we used were FIPS 140-2 validated and physically connected to the racks (though it would be possible to cut them away). It would also be possible to spy on the APDUs sent to the modules to capture the decrypted data, since there was not mutual authentication (it was in the works though).
vajenetehais 5y ago
On their website they refer to multiple certification [0]. One of them is the CISPE and on their website, it is stated: "Requirement for CISP: (a) Security measures The CISP will implement and maintain appropriate technical and organisational measures for the CISP’s data centre facilities, servers, networking equipment and host software systems that are within the CISP’s control and are used to provide the CISP’s service (the CISP Network). Those technical and organisational measures should (a) be designed *to help customers secure personal data against unauthorised processing and accidental or unlawful loss, access or disclosure*, and (b) address the security responsibilities of the CISP as set out in Annex A (Security Responsibilities)." [1] I do not know about others certifications, but this situation seems to be a clear violation of one of the requirement for CISP. Answer to this requirement is disk encryption. Moreover they are authorized to store medical records and data. I can't imagine that they do this without providing proper disk encryption. in the light of this event, I'm not sure they qualify for all these ceertifications. [0] https://www.scaleway.com/fr/a-propos/ https://www.scaleway.com/fr/a-propos/ [1] https://cispe.cloud/code-of-conduct/ https://cispe.cloud/code-of-conduct/
- madaxe_again 5y ago
  It’s a total violation of about a third of the controls in ISO27001 Annex A. Whoever certified them needs looking at, as their certifications are valueless. It’s possible scaleway aren’t certified, and are lying about it. Maybe they had an auditor in, maybe not, maybe they have a boatload of nonconformities that they’re not documenting (which you must, if you have them).
raverbashing 5y ago
The things that are not clear to me are: - Why was the disk being replaced (SMART warnings? HW upgrade? It failed in some way) - Why was the quick format recoverable? Was TRIM issued or the HW didn't support it? - As a follow up, I'm guessing the HW didn't support HW encryption? - Why are they relying on 3rd parties for the destruction of data? Meanwhile, it seems that HDs don't leave Google datacenters in one piece
- dracodoc 5y ago
  The disk could be simply transferred to another data center physically so they can move data faster/cheaper than transferring the data by internet.
  - capableweb 5y ago
    Reminds me of the following quote: > Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. - Andrew S. Tanenbaum
Nextgrid 5y ago
> The CEO said recovering the disk helped the authorities to advance their investigations into the heist, and meant the company felt able to publicly disclose the theft. Aka they could no longer sweep it under the rug.
Moonlight_TC 5y ago
The timeline of events in Scaleway's blog post is very dubious. If the SSD was stolen over one year ago why do they only acknowledge it now? In March 2021 [1] they were writing about how great their security was: > We are proud of our data centers and their security. We consider that we have implemented the best solutions to protect your most valuable asset: your data. We are well aware of the huge responsibility this represents. There can be no compromises when it comes to your data. Why were customers whose data was leaked only informed in June 2021? The delay of over 1 year is a huge GDPR issue. The timeline of the incident from public sources: 21 May 2021: Micode tweets a screenshot of the directory listing [2] 26 May 2021: First video on the subject [3] 6 June 2021: Scaleway customer is notified their data was leaked [4] 21 July 2021: Second video on the subject [5] 24 July 2021: Third video on the subject [6] 24 July 2021: Scaleway releases a French blog post stating "Over a year ago, an SSD was stolen" [7] Storing the data unencrypted is bad, but IMHO Scaleway's handling of the incident creates much bigger questions about their credibility. [1] https://blog.scaleway.com/how-we-protect-your-data/ https://blog.scaleway.com/how-we-protect-your-data/ [2] https://mobile.twitter.com/Micode/status/1395640486715662336 https://mobile.twitter.com/Micode/status/1395640486715662336 [3] https://www.youtube.com/watch?v=vt8PyQ2PGxI https://www.youtube.com/watch?v=vt8PyQ2PGxI [4] https://www.lowendtalk.com/discussion/comment/3258386/#Comment_3258386 https://www.lowendtalk.com/discussion/comment/3258386/#Comme... [5] https://www.youtube.com/watch?v=aOBVZUL1iBA https://www.youtube.com/watch?v=aOBVZUL1iBA [6] https://www.youtube.com/watch?v=xf_cKTlOYLo https://www.youtube.com/watch?v=xf_cKTlOYLo [7] https://blog.scaleway.com/incident-securitaire-video-youtube/ https://blog.scaleway.com/incident-securitaire-video-youtube... Previous discussion: https://news.ycombinator.com/item?id=27957471 https://news.ycombinator.com/item?id=27957471
throwaway382985 5y ago
I worked at scaleway. This whole post is basically bullshit, "secure" transporation meant taking a random electric car (renault zoe) at the company headquarters or at DC5 and throwing hardware in the trunk. No security in place whatsoever, servers laying in datacenter hallways fully loaded with disks, accessible to anyone. Most of the company had access to the back office where they could just access customers' data without any kind of logging. (Internally this was called sudo mode on the online console, we had basically to click on a button to log as any customer). The "funniest" was the corporate vpn network being shared by the internal datacenter network which meant any employee had, for instance, full access to all the home made switches management interfaces which had no access control whatsoever, it happened that employees stumbled accross this and wondering what it was (LOL), there was even a button to upload and flash (!) a firmware. The upper-management was terribly incompetent and would discard any kind of issue that was not affecting sales in the immediate term as not important (security or otherwise).
- emteycz 5y ago
  What about GDPR, did it change anything?
- Abishek_Muthian 5y ago
  There was a post in the scaleway user forum(~2020) asking 'Who has access to the data on the disk' to which the founder or CEO replied that the Engineers do have access but the datacenters are very secure as it's thoroughly monitored. I remember this because I used fscrypt after seeing the post, Scaleway seems to have removed the entire user forum sometime last year. I stopped using their service after they removed their ARM severs with short notice, But this is more scary. Unfortunately this is the reality when it comes to AWS[1] or other top-tier cloud providers vs inexpensive ones. The cost saving has to happen somewhere. [1] https://aws.amazon.com/blogs/security/importance-of-encryption-and-how-aws-can-help/ https://aws.amazon.com/blogs/security/importance-of-encrypti...
- madaxe_again 5y ago
  But they’re ISO27001 certified? Didn’t they have security controls for transit of protected information? What about human resource security? What about their access control policy and network access control policy? Do you know who their certifier is, as they don’t say, so I can ensure I never, ever trust them?
hulitu 5y ago
How can one steal from a cloud ? /s

90 points by ronalbarbaren 5y ago

fulafel 5y ago

So was there cleartext data on it?

hdjjhhvvhga 5y ago
Yes, the article clearly says the data was unencrypted.

detaro 5y ago

2 days ago: https://news.ycombinator.com/item?id=27957471 https://news.ycombinator.com/item?id=27957471

mot2ba 5y ago
I posted that thread and I don't know why it was penalty by mod, completely out of front page HN.

terom 5y ago

Apparently with plaintext customer data. > Lechelle said Scaleway worked with the YouTuber to recover the disk. The French-language video creator has written to Scaleway with assurances they have not copied the information contained on the disk. It is said some customer data was on the drive, unencrypted, including the source code and SSH keys of an Italian VPS provider.

terom 5y ago
Based on links in the other hackernews thread, it was quick-formatted, but a filesystem containing qemu disk images was recovered: [1] The YouTube video is in French without English CC subtitles, but there's screenshots of exploring what I assume would be one of the qcow images: [2] * 03:59 `/root/.ssh` with SSH id_rsa (private) keys, (public) authorized_keys and (hashed) known_hosts * 07:14 server binaries * 08:42 server logs * 12:59 AWS/S3 secret access keys * 13:33 Source code... in QBASIC ?! must be very valuable legacy code :D * 17:30 brute-forcing the ssh known_hosts * etc [1] https://mobile.twitter.com/Micode/status/1395640486715662336 https://mobile.twitter.com/Micode/status/1395640486715662336 [2] https://www.youtube.com/watch?v=xf_cKTlOYLo https://www.youtube.com/watch?v=xf_cKTlOYLo
- TonyTrapp 5y ago
  Note: QB64 (which is mentioned in the video) is a modern-day reimplementation of QBasic. It doesn't have to be legacy. Some people still prefer using Basic as their "scripting language" for quick tasks where other people would be using Python or JS these days.
  - alyandon 5y ago
    Although I never cared much for the language itself, I continued to write small programs in QB for many years because of its ability to produce an EXE file that ran in DOS and Windows without any dependencies.
- rasz 5y ago
  >it was quick-formatted are there OSes that perform quick format without issuing TRIM command to the SSD? TRIM would vanish the data in less than 15 minutes.
  - djrogers 5y ago
    Yes, 15 minutes is not 'quick' by most standards - quick formats take seconds, not minutes.
    - rasz 5y ago
      up to ~15 minutes is not the time it takes to quick format, its the number Iv seen during testing of TRIM, the time it took to internally erase translation table after format command. There should be NO DATA on the drive if it was really formatted on OS supporting TRIM.
  - BBC-vs-neolibs 5y ago
    Not if you turn the computer off right after deleting the files.
  - fomine3 5y ago
    I believe parted won't issue TRIM for whole disk/partition after erasing partition table or partition. Possibly mkfs.xfs issues TRIM? I don't know. Anyway, what they should at least is just issue `blkdiscard /dev/sdb`, or betterly Secure Erase (on SATA) or Sanitize (on NVMe).

jnwatson 5y ago

Why in the world do they ship unencrypted drives around? Drive encryption is pretty standard these days, isn't it?

atonse 5y ago
With the big cloud provides its a trivial checkbox. Can anyone speak to the effort in doing it at the OS level?
- tpxl 5y ago
  There's a checkbox to encrypt the disks when installing some linux distros.
  - jsiepkes 5y ago
    > There's a checkbox to encrypt the disks when installing some linux distros. While I'm not going to defend moving around unencrypted disks it is not as simple as that. The difficult part is not the encryption. The difficult part is the key management of the encryption. You need to solve issue like ensuring you don't lose access to the keys (and thereby access to the data), securely providing keys to people and machines that need access, dealing with key revocation when people leave the company, etc. Though I guess in this case using a simple (long enough) passphrase instead of some form of keys could have saved them from this particular threat.
    - atonse 5y ago
      I thought any of these providers always encrypted with their own key whether you encrypted on top of that or not. But I guess not. Yet another reason to just stick with the big providers. They seem to have pretty serious compliance needs and encrypt no matter what. (Yet you should still encrypt your assets, but I believe they all encrypt their volumes anyway and have pretty strict policies on how to dispose this kind of hardware)
    - rkeene2 5y ago
      When I ran a cloud provider, we used a hardware security module on storage systems which had public/private key pair which could be used to decrypt the header on the disk which contained a copy of the symmetric cipher key. Each header had the symmetric cipher key encrypted with multiple different public keys, including a fallback whose private key only existed in on paper in a vault. Each system could reach out to other storage systems to ask them to use their private key to decrypt the header (for example if their hardware security module had failed), but in some configurations this would require an operator to intervene to enter in passphrase to unlock the hardware security module to authorize the action. This means that: 1. The symmetric cipher key could always be recovered, even from paper backup 2. Having physical access to a disk or any set of disks did not allow you to read the data 3. Having physical access to a disk and a hardware security module did not allow you to read the disk (unless you knew the passphrase, which was always present, and user set) 4. Having physical access to disks, servers, and hardware security module may not allow you to read the data (in the more secure configuration where passphrases were not cached on disk -- but this meant that rebooting required an operator to manually enter the password at boot) 5. The set of valid public keys could be changed frequently (this was indeed automated and only the set of currently active hardware security modules could decrypt the current disk header) Of course you could always short circuit this by making a copy of the symmetric key when you did have access (e.g., `dmsetup table --showkeys`) but without putting some more hardware in the fast/hot path that was unavoidable. The symmetric key could not easily be changed, without rewriting the entire disk (though since it was a storage system designed to accommodate multiple failures, this wasn't that hard, but we didn't do it automatically) The hardware security modules we used were FIPS 140-2 validated and physically connected to the racks (though it would be possible to cut them away). It would also be possible to spy on the APDUs sent to the modules to capture the decrypted data, since there was not mutual authentication (it was in the works though).

vajenetehais 5y ago

On their website they refer to multiple certification [0]. One of them is the CISPE and on their website, it is stated: "Requirement for CISP: (a) Security measures The CISP will implement and maintain appropriate technical and organisational measures for the CISP’s data centre facilities, servers, networking equipment and host software systems that are within the CISP’s control and are used to provide the CISP’s service (the CISP Network). Those technical and organisational measures should (a) be designed *to help customers secure personal data against unauthorised processing and accidental or unlawful loss, access or disclosure*, and (b) address the security responsibilities of the CISP as set out in Annex A (Security Responsibilities)." [1] I do not know about others certifications, but this situation seems to be a clear violation of one of the requirement for CISP. Answer to this requirement is disk encryption. Moreover they are authorized to store medical records and data. I can't imagine that they do this without providing proper disk encryption. in the light of this event, I'm not sure they qualify for all these ceertifications. [0] https://www.scaleway.com/fr/a-propos/ https://www.scaleway.com/fr/a-propos/ [1] https://cispe.cloud/code-of-conduct/ https://cispe.cloud/code-of-conduct/

madaxe_again 5y ago
It’s a total violation of about a third of the controls in ISO27001 Annex A. Whoever certified them needs looking at, as their certifications are valueless. It’s possible scaleway aren’t certified, and are lying about it. Maybe they had an auditor in, maybe not, maybe they have a boatload of nonconformities that they’re not documenting (which you must, if you have them).

raverbashing 5y ago

The things that are not clear to me are: - Why was the disk being replaced (SMART warnings? HW upgrade? It failed in some way) - Why was the quick format recoverable? Was TRIM issued or the HW didn't support it? - As a follow up, I'm guessing the HW didn't support HW encryption? - Why are they relying on 3rd parties for the destruction of data? Meanwhile, it seems that HDs don't leave Google datacenters in one piece

dracodoc 5y ago
The disk could be simply transferred to another data center physically so they can move data faster/cheaper than transferring the data by internet.
- capableweb 5y ago
  Reminds me of the following quote: > Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. - Andrew S. Tanenbaum

Nextgrid 5y ago

> The CEO said recovering the disk helped the authorities to advance their investigations into the heist, and meant the company felt able to publicly disclose the theft. Aka they could no longer sweep it under the rug.

Moonlight_TC 5y ago

The timeline of events in Scaleway's blog post is very dubious. If the SSD was stolen over one year ago why do they only acknowledge it now? In March 2021 [1] they were writing about how great their security was: > We are proud of our data centers and their security. We consider that we have implemented the best solutions to protect your most valuable asset: your data. We are well aware of the huge responsibility this represents. There can be no compromises when it comes to your data. Why were customers whose data was leaked only informed in June 2021? The delay of over 1 year is a huge GDPR issue. The timeline of the incident from public sources: 21 May 2021: Micode tweets a screenshot of the directory listing [2] 26 May 2021: First video on the subject [3] 6 June 2021: Scaleway customer is notified their data was leaked [4] 21 July 2021: Second video on the subject [5] 24 July 2021: Third video on the subject [6] 24 July 2021: Scaleway releases a French blog post stating "Over a year ago, an SSD was stolen" [7] Storing the data unencrypted is bad, but IMHO Scaleway's handling of the incident creates much bigger questions about their credibility. [1] https://blog.scaleway.com/how-we-protect-your-data/ https://blog.scaleway.com/how-we-protect-your-data/ [2] https://mobile.twitter.com/Micode/status/1395640486715662336 https://mobile.twitter.com/Micode/status/1395640486715662336 [3] https://www.youtube.com/watch?v=vt8PyQ2PGxI https://www.youtube.com/watch?v=vt8PyQ2PGxI [4] https://www.lowendtalk.com/discussion/comment/3258386/#Comment_3258386 https://www.lowendtalk.com/discussion/comment/3258386/#Comme... [5] https://www.youtube.com/watch?v=aOBVZUL1iBA https://www.youtube.com/watch?v=aOBVZUL1iBA [6] https://www.youtube.com/watch?v=xf_cKTlOYLo https://www.youtube.com/watch?v=xf_cKTlOYLo [7] https://blog.scaleway.com/incident-securitaire-video-youtube/ https://blog.scaleway.com/incident-securitaire-video-youtube... Previous discussion: https://news.ycombinator.com/item?id=27957471 https://news.ycombinator.com/item?id=27957471

throwaway382985 5y ago

I worked at scaleway. This whole post is basically bullshit, "secure" transporation meant taking a random electric car (renault zoe) at the company headquarters or at DC5 and throwing hardware in the trunk. No security in place whatsoever, servers laying in datacenter hallways fully loaded with disks, accessible to anyone. Most of the company had access to the back office where they could just access customers' data without any kind of logging. (Internally this was called sudo mode on the online console, we had basically to click on a button to log as any customer). The "funniest" was the corporate vpn network being shared by the internal datacenter network which meant any employee had, for instance, full access to all the home made switches management interfaces which had no access control whatsoever, it happened that employees stumbled accross this and wondering what it was (LOL), there was even a button to upload and flash (!) a firmware. The upper-management was terribly incompetent and would discard any kind of issue that was not affecting sales in the immediate term as not important (security or otherwise).

emteycz 5y ago
What about GDPR, did it change anything?
Abishek_Muthian 5y ago
There was a post in the scaleway user forum(~2020) asking 'Who has access to the data on the disk' to which the founder or CEO replied that the Engineers do have access but the datacenters are very secure as it's thoroughly monitored. I remember this because I used fscrypt after seeing the post, Scaleway seems to have removed the entire user forum sometime last year. I stopped using their service after they removed their ARM severs with short notice, But this is more scary. Unfortunately this is the reality when it comes to AWS[1] or other top-tier cloud providers vs inexpensive ones. The cost saving has to happen somewhere. [1] https://aws.amazon.com/blogs/security/importance-of-encryption-and-how-aws-can-help/ https://aws.amazon.com/blogs/security/importance-of-encrypti...
madaxe_again 5y ago
But they’re ISO27001 certified? Didn’t they have security controls for transit of protected information? What about human resource security? What about their access control policy and network access control policy? Do you know who their certifier is, as they don’t say, so I can ensure I never, ever trust them?

hulitu 5y ago

How can one steal from a cloud ? /s