4 ms·

> Still no email blast from Vercel alerting users, which is concerning. On the one hand, I get that it's a Sunday, and the CEO can't just write a mass email wi

by cowsup 5mo ago

> Still no email blast from Vercel alerting users, which is concerning. On the one hand, I get that it's a Sunday, and the CEO can't just write a mass email without approval from legal or other comms teams. But on the other hand... It's Sunday. Unless you're tuned-in to social media over the weekend, your main provider could be undergoing a meltdown while you are completely unaware. Many higher-up folks check company email over the weekend, but if they're traveling or relaxing, social media might be the furthest thing from their mind. It really bites that this is the only way to get critical information.

loloquwowndueo 5mo ago
> the CEO can't just write a mass email without approval from legal or other comms teams. They can be brought in to do their job on a Sunday for an event of this relevance. They can always take next Friday off or something.
eclipticplane 5mo ago
Has anyone actually gotten an email from Vercel confirming their secrets were accessed? Right now we're all operating under the hope (?) that since we haven't (yet?) gotten an email, we're not completely hosed.
- loloquwowndueo 5mo ago
  Hope-based security should not be a thing. Did you rotate your secrets? Did you audit your platform for weird access patterns? Don’t sit waiting for that vercel email.
  - eclipticplane 5mo ago
    Of course rotated. But we don't even know when the secrets were stolen vs we were told, so we're missing a ton of info needed to _fully_ triage.
  - lelanthran 5mo ago
    > Did you rotate your secrets? For most secrets they are under your control so, sure, go ahead and rotate them, allowing the old version to continue being used in parallel with the new version for 30 minutes or so. For other secrets, rotation involves getting a new secret from some upstream provider and having some services (users of that secret) fail while the secret they have in cache expires. For example, if your secret is a Stripe key; generating a new key should invalidate the old one (not too sure, I don't use Stripe), at which point the services with the cached secret will fail until the expiry.
- ItsClo688 5mo ago
  nope...I feel u, the "Hope-based security" is exactly what Vercel is forcing on its users right now by prioritizing social media over direct notification. If the attacker is moving with "surprising velocity," every hour of delay on an email blast is another hour the attacker has to use those potentially stolen secrets against downstream infrastructure. Using Twitter/X as a primary disclosure channel for a "sophisticated" breach is amateur hour. If legal is the bottleneck for a mass email during an active compromise, then your incident response plan is fundamentally broken.
gk1 5mo ago
> On the one hand, I get that it's a Sunday, and the CEO can't just write a mass email without approval from legal or other comms teams This is not how things work. In a crisis like this there is a war room with all stakeholders present. Doesn’t matter if it’s Sunday or 3am or Christmas. And for this company specifically, Guillermo is not one to defer to comms or legal.
- huflungdung 5mo ago
  [dead]
- brobdingnagians 5mo ago
  If he's not one to defer to Comms or legal, maybe this one is so bad that he's acting differently then he normally would
refulgentis 5mo ago
I'm going down with the ship over on X.com the Everything App. There's a parcel of very important tech people that are running some playbook where posting to X.com is sufficient enough to be unimpeachable on communication, despite its rather beleaguered state and traffic.
nnurmanov 5mo ago
Usually, companies have procedures for such events. But most do not.
- gnabgib 5mo ago
  Usually have procedures, but most don't? Say again
  - wombatpm 5mo ago
    The disaster plan says there is a process, but it has never been used and is probably outdated. Chances are the social media strategy requires posting on the Facebook and updating key Circles on Google+
  - nnurmanov 5mo ago
    Yes, they say we have backup procedures, but have they ever tested that the backups work? They write procedures to please auditors:)
steve1977 5mo ago
> the CEO can't just write a mass email without approval from legal or other comms teams Wouldn't the CEO be... you know... the chief executive?
- hvb2 5mo ago
  Sure, and the reason he is is because he DOES check stuff like this before sending it out. Top leaders excel because they assemble a team around them they trust. You can't do everything yourself, you need to delegate. And having people in those positions also means you shouldn't be acting alone or those people will not stick around
  - steve1977 5mo ago
    I disagree. In a crisis, a leader should take the lead and make decisions. If he/she is not able to that on their own, they are in the wrong place. Now I will agree that there are many executives like the ones you describe. But they are not top leaders.
    - scott_w 5mo ago
      So you’re telling me a CEO must also be a practicing lawyer? Because any other option is how you guarantee your company gets sued into oblivion.
      - steve1977 5mo ago
        First of all, I would expect a top leader to be prepared for scenarios like this (including templates of customer communication). And yeah, I would expect a CEO to have enough legal knowledge to handle such a situation (customer communication) on his own. But I also have to mentioned that I'm not in the US. Not every country has the litigation system of the US where you can basically destroy a company because you as the customer are too dumb to not spill hot coffee over yourself.
        5mo ago

by cowsup 5mo ago

loloquwowndueo 5mo ago

> the CEO can't just write a mass email without approval from legal or other comms teams. They can be brought in to do their job on a Sunday for an event of this relevance. They can always take next Friday off or something.

eclipticplane 5mo ago

Has anyone actually gotten an email from Vercel confirming their secrets were accessed? Right now we're all operating under the hope (?) that since we haven't (yet?) gotten an email, we're not completely hosed.

loloquwowndueo 5mo ago
Hope-based security should not be a thing. Did you rotate your secrets? Did you audit your platform for weird access patterns? Don’t sit waiting for that vercel email.
- eclipticplane 5mo ago
  Of course rotated. But we don't even know when the secrets were stolen vs we were told, so we're missing a ton of info needed to _fully_ triage.
- lelanthran 5mo ago
  > Did you rotate your secrets? For most secrets they are under your control so, sure, go ahead and rotate them, allowing the old version to continue being used in parallel with the new version for 30 minutes or so. For other secrets, rotation involves getting a new secret from some upstream provider and having some services (users of that secret) fail while the secret they have in cache expires. For example, if your secret is a Stripe key; generating a new key should invalidate the old one (not too sure, I don't use Stripe), at which point the services with the cached secret will fail until the expiry.
ItsClo688 5mo ago
nope...I feel u, the "Hope-based security" is exactly what Vercel is forcing on its users right now by prioritizing social media over direct notification. If the attacker is moving with "surprising velocity," every hour of delay on an email blast is another hour the attacker has to use those potentially stolen secrets against downstream infrastructure. Using Twitter/X as a primary disclosure channel for a "sophisticated" breach is amateur hour. If legal is the bottleneck for a mass email during an active compromise, then your incident response plan is fundamentally broken.

gk1 5mo ago

> On the one hand, I get that it's a Sunday, and the CEO can't just write a mass email without approval from legal or other comms teams This is not how things work. In a crisis like this there is a war room with all stakeholders present. Doesn’t matter if it’s Sunday or 3am or Christmas. And for this company specifically, Guillermo is not one to defer to comms or legal.

huflungdung 5mo ago
[dead]
brobdingnagians 5mo ago
If he's not one to defer to Comms or legal, maybe this one is so bad that he's acting differently then he normally would

refulgentis 5mo ago

I'm going down with the ship over on X.com the Everything App. There's a parcel of very important tech people that are running some playbook where posting to X.com is sufficient enough to be unimpeachable on communication, despite its rather beleaguered state and traffic.

nnurmanov 5mo ago

Usually, companies have procedures for such events. But most do not.

gnabgib 5mo ago
Usually have procedures, but most don't? Say again
- wombatpm 5mo ago
  The disaster plan says there is a process, but it has never been used and is probably outdated. Chances are the social media strategy requires posting on the Facebook and updating key Circles on Google+
- nnurmanov 5mo ago
  Yes, they say we have backup procedures, but have they ever tested that the backups work? They write procedures to please auditors:)

steve1977 5mo ago

> the CEO can't just write a mass email without approval from legal or other comms teams Wouldn't the CEO be... you know... the chief executive?

hvb2 5mo ago
Sure, and the reason he is is because he DOES check stuff like this before sending it out. Top leaders excel because they assemble a team around them they trust. You can't do everything yourself, you need to delegate. And having people in those positions also means you shouldn't be acting alone or those people will not stick around
- steve1977 5mo ago
  I disagree. In a crisis, a leader should take the lead and make decisions. If he/she is not able to that on their own, they are in the wrong place. Now I will agree that there are many executives like the ones you describe. But they are not top leaders.
  - scott_w 5mo ago
    So you’re telling me a CEO must also be a practicing lawyer? Because any other option is how you guarantee your company gets sued into oblivion.
    - steve1977 5mo ago
      First of all, I would expect a top leader to be prepared for scenarios like this (including templates of customer communication). And yeah, I would expect a CEO to have enough legal knowledge to handle such a situation (customer communication) on his own. But I also have to mentioned that I'm not in the US. Not every country has the litigation system of the US where you can basically destroy a company because you as the customer are too dumb to not spill hot coffee over yourself.
      - 5mo ago