8 ms·

At my first gig, I had "god" level access to our production database. All I learned is that nobody should have this level of access unless it is some sort of t

At my first gig, I had "god" level access to our production database. All I learned is that nobody should have this level of access unless it is some sort of temporary break glass situation. It is extremely dangerous and even experienced engineers can cause irreparable data loss or some other bad outcome. In our case, some engineer accidentally sent around 10,000 invoices to customers that shouldn't have gotten them. There are far better data access patterns. In the case of US gov data, I don't see why the DOGE team would need anything more than a read replica to query. It could even be obfuscated in some way to protect citizens' identities.

neycoda 2y ago
Why should they even have read access? They're not a legal government institution, and they're being led by a private citizen that's not been elected or appointed by Congress to access our data in agencies that were made by Congress under particular rules to keep these kinds of snoops out.
cratermoon 2y ago
I loathe working places where they just give you all the permissions because it's "easier". One risk is if something does happen, and they don't have exceptional tracing and logging, (and let's be honest, at an organization sloppy enough to hand out privileges like candy, what's the chance of that?) it's difficult or impossible to pin down the source to any individual. As a result, both responsibility and suspicion is diffuse.
- JohnFen 2y ago
  I agree. Good access controls and being prevented from accessing things that I don't need access to protect me as an employee just as much as the data itself.
- FigurativeVoid 2y ago
  I have so many horror stories from there. When they did decide to lock down the database, the DB admin only locked in down in the sql server client most people used. If you used some other client, you still had access. _sigh_
  - tomrod 2y ago
    What DB system operates that way, that's nuts.
  - cratermoon 2y ago
    My favorite security anti-pattern! Locking the main doors while leaving all the windows wide open.
- FigurativeVoid 2y ago
  A second thought. It leads to lazy application development. Whenever you have production intervention that happens more than a few times, you should just make a feature that does it safely via application code.
  - Tobani 2y ago
    I've definitely worked in places where "Move fast and break things" tended to focus on breaking things. There would be bugs that we didn't fix because "We can just fix the database when it happens." It would take 2hours to fix a bug that would cause of 10's of hours of weekly support request, but the focus would always be on building new features, of which 10% got any real usage.
- TransAtlToonz 2y ago
  The appropriate restrictions are relative to the size and momentum of the organization. It's easy to spend months setting up safeguards rather than working on product development that won't proportionally return. Of course, this involves being honest with yourself about risk and reward, and we all have implicit incentives to disregard the risk until we get burned and learn to factor that in.
- justin66 2y ago
  It's not just about the risk. It signifies that you're not dealing with an experienced database administration staff. (At a startup that might just mean one guy, but that's better than zero.
- alsoforgotmypwd 2y ago
  Meta completely restricted graph data access to requiring a specific business purpose and managerial approval tied an articulable, concrete task need.
r00fus 2y ago
Ah, I remember a time 30 years ago when I logged accidentally into the PROD database (forgot to add the suffix "1" to the connection ID), thinking it was a Dev instance, and then issued a "truncate table CUSTOMERS"... the reaction came within 75 seconds - and restore from backing took several hours.
- archon810 2y ago
  That's not bad. I wonder how many companies would not be able to recover from such an event in days / weeks / ever.
- raffraffraff 2y ago
  That's when you pray that the engineering team didn't fix the replication lag.
TrackerFF 2y ago
Never mind the direct risks, if you have "god mode" to basically any government thing, you instantly become the target of foreign intel/military operations. You can bet good money that there are entire teams, if not divisions, working around the clock to exploit this situation.
- netsharc 2y ago
  I can imagine Chinese and Russian hackers laughing at the DOGE l33t hackers. And if I was advising the Ukranians I'd tell them to try to exploit it too, hey, if you're fighting 2 superpowers with another 1 quietly backing the fight against you, you need all the help you can get.
simpaticoder 2y ago
I've worked with older governmental systems, and chances are they are running a wide variety of systems, some of which, the oldest and most critical, are probably written in COBOL running on IBM mainframe hardware. In those environments, there is no real distinction between "database" and "application". COBOL systems are very file- and batch-oriented, and are "monolithic" in the extremist sense. The technology itself makes it impossible to give read only access to such systems.
- neycoda 2y ago
  These old systems need to be upgraded but Congress never approves the financing or execution of it because they're too divided and won't increase taxes on anyone especially the rich and giant corporations to do it.
- kvakerok 2y ago
  You can absolutely give read only access in COBOL systems. That's just lazy administration and IT security on a shoestring budget.
- jart 2y ago
  [flagged]
  - discreteevent 2y ago
    >They don't even know how to build a website that works. What percentage of people who know how to make a "website" do you think could make an automated tax system? >the tech industry has been the beating heart of this country Agriculture? Construction? The heart means something without which you can't function. How did people in the 1950s survive?
    - jart 2y ago
      The agriculture industry is a skeleton crew for something that's largely been automated by tech: https://justine.lol/tmp/agriculture.jpg https://justine.lol/tmp/agriculture.jpg There's not much of a construction industry either, since the government doesn't let us build anything except sprawl.
Zefiroj 2y ago
There's a good balance between preventing accidents and reducing friction. One person having "god-mode" access isn't usually that terrible.
godelski 2y ago
> It is extremely dangerous and even experienced engineers can cause irreparable data loss or some other bad outcome It is literally why we never log in as root. HERE BE DRAGONS I don't know an admin who hasn't, on multiple occasions, unintentionally caused irreparable damage. It is easy to do even with the best of intentions and with extreme levels of care. Any one trying to rush through a dragon's den is only going to get burned. Considering how many dragons' dens they are running into, I do not question "if" damage has been done, but "what".
- amy214 2y ago
  I remember having some kind of C programming bug where output filenames got scrambled (string memory error probably). And output files in the same folder as the source code. That seems innocuous, but remember then some of the output files might have the character "?" or even "*". So imagine trying to remove these files and going an asterisk too far. All gone!
manfre 2y ago
I've had a company give me full admin access to their cloud account. Thankfully, I learned the lesson earlier in my career and immediately created myself of more mundane user. Break glass access is important, but definitely not as the usual level of access. > I don't see why the DOGE team would need anything more than a read replica to query. They shouldn't need more than limited read access. The fact that they have more access, very likely demanded and not accidentally given, is due to their intent to do more than simply query data.
erulabs 2y ago
Ultimately someone has root permissions. Re: federal agencies, in the United States, that someone is clearly, constitutionally, the President. Article II of the constitution vests all power of the executive in the person of the President. The President has authority to appoint agents. That same article _does also_ say the President has to "take Care that the Laws be faithfully executed", but the "Care" there is highly debated. But the idea that the President doesn't have the right to appoint Musk to get root access to federal agencies seems legally incorrect. I'm not make a value judgement on this, it's just how it is. At a startup, the founder ultimately has root access to the database, no matter what the technical controls. Now, maybe it's stupid, and maybe it should be some other way, but to my mind the other way is that Congress gets together and writes a law saying "the executive cannot get root access to X, Y, Z". In absence of that law, the executive can do whatever they want. Not to be THAT GUY, but "an append-only database which cannot be modified by anyone" is something HN has spent the past 10 years saying is completely useless...
- netsharc 2y ago
  And Trump can launch the nukes to blow up the world too... but building a system where he can just click a button to do so would be idiotic. Same idea with giving godmode to the guy who thinks carrying a sink and saying "Let that sink in" is hilariously clever.
- Rapzid 2y ago
  The power rests with the office. There is an important but nuanced distinction there.

neycoda 2y ago

Why should they even have read access? They're not a legal government institution, and they're being led by a private citizen that's not been elected or appointed by Congress to access our data in agencies that were made by Congress under particular rules to keep these kinds of snoops out.

cratermoon 2y ago

I loathe working places where they just give you all the permissions because it's "easier". One risk is if something does happen, and they don't have exceptional tracing and logging, (and let's be honest, at an organization sloppy enough to hand out privileges like candy, what's the chance of that?) it's difficult or impossible to pin down the source to any individual. As a result, both responsibility and suspicion is diffuse.

JohnFen 2y ago
I agree. Good access controls and being prevented from accessing things that I don't need access to protect me as an employee just as much as the data itself.
FigurativeVoid 2y ago
I have so many horror stories from there. When they did decide to lock down the database, the DB admin only locked in down in the sql server client most people used. If you used some other client, you still had access. _sigh_
- tomrod 2y ago
  What DB system operates that way, that's nuts.
- cratermoon 2y ago
  My favorite security anti-pattern! Locking the main doors while leaving all the windows wide open.
FigurativeVoid 2y ago
A second thought. It leads to lazy application development. Whenever you have production intervention that happens more than a few times, you should just make a feature that does it safely via application code.
- Tobani 2y ago
  I've definitely worked in places where "Move fast and break things" tended to focus on breaking things. There would be bugs that we didn't fix because "We can just fix the database when it happens." It would take 2hours to fix a bug that would cause of 10's of hours of weekly support request, but the focus would always be on building new features, of which 10% got any real usage.
TransAtlToonz 2y ago
The appropriate restrictions are relative to the size and momentum of the organization. It's easy to spend months setting up safeguards rather than working on product development that won't proportionally return. Of course, this involves being honest with yourself about risk and reward, and we all have implicit incentives to disregard the risk until we get burned and learn to factor that in.
justin66 2y ago
It's not just about the risk. It signifies that you're not dealing with an experienced database administration staff. (At a startup that might just mean one guy, but that's better than zero.
alsoforgotmypwd 2y ago
Meta completely restricted graph data access to requiring a specific business purpose and managerial approval tied an articulable, concrete task need.

r00fus 2y ago

Ah, I remember a time 30 years ago when I logged accidentally into the PROD database (forgot to add the suffix "1" to the connection ID), thinking it was a Dev instance, and then issued a "truncate table CUSTOMERS"... the reaction came within 75 seconds - and restore from backing took several hours.

archon810 2y ago
That's not bad. I wonder how many companies would not be able to recover from such an event in days / weeks / ever.
raffraffraff 2y ago
That's when you pray that the engineering team didn't fix the replication lag.

TrackerFF 2y ago

Never mind the direct risks, if you have "god mode" to basically any government thing, you instantly become the target of foreign intel/military operations. You can bet good money that there are entire teams, if not divisions, working around the clock to exploit this situation.

netsharc 2y ago
I can imagine Chinese and Russian hackers laughing at the DOGE l33t hackers. And if I was advising the Ukranians I'd tell them to try to exploit it too, hey, if you're fighting 2 superpowers with another 1 quietly backing the fight against you, you need all the help you can get.

simpaticoder 2y ago

I've worked with older governmental systems, and chances are they are running a wide variety of systems, some of which, the oldest and most critical, are probably written in COBOL running on IBM mainframe hardware. In those environments, there is no real distinction between "database" and "application". COBOL systems are very file- and batch-oriented, and are "monolithic" in the extremist sense. The technology itself makes it impossible to give read only access to such systems.

neycoda 2y ago
These old systems need to be upgraded but Congress never approves the financing or execution of it because they're too divided and won't increase taxes on anyone especially the rich and giant corporations to do it.
kvakerok 2y ago
You can absolutely give read only access in COBOL systems. That's just lazy administration and IT security on a shoestring budget.
jart 2y ago
[flagged]
- discreteevent 2y ago
  >They don't even know how to build a website that works. What percentage of people who know how to make a "website" do you think could make an automated tax system? >the tech industry has been the beating heart of this country Agriculture? Construction? The heart means something without which you can't function. How did people in the 1950s survive?
  - jart 2y ago
    The agriculture industry is a skeleton crew for something that's largely been automated by tech: https://justine.lol/tmp/agriculture.jpg https://justine.lol/tmp/agriculture.jpg There's not much of a construction industry either, since the government doesn't let us build anything except sprawl.

Zefiroj 2y ago

There's a good balance between preventing accidents and reducing friction. One person having "god-mode" access isn't usually that terrible.

godelski 2y ago

> It is extremely dangerous and even experienced engineers can cause irreparable data loss or some other bad outcome It is literally why we never log in as root. HERE BE DRAGONS I don't know an admin who hasn't, on multiple occasions, unintentionally caused irreparable damage. It is easy to do even with the best of intentions and with extreme levels of care. Any one trying to rush through a dragon's den is only going to get burned. Considering how many dragons' dens they are running into, I do not question "if" damage has been done, but "what".

amy214 2y ago
I remember having some kind of C programming bug where output filenames got scrambled (string memory error probably). And output files in the same folder as the source code. That seems innocuous, but remember then some of the output files might have the character "?" or even "*". So imagine trying to remove these files and going an asterisk too far. All gone!

manfre 2y ago

I've had a company give me full admin access to their cloud account. Thankfully, I learned the lesson earlier in my career and immediately created myself of more mundane user. Break glass access is important, but definitely not as the usual level of access. > I don't see why the DOGE team would need anything more than a read replica to query. They shouldn't need more than limited read access. The fact that they have more access, very likely demanded and not accidentally given, is due to their intent to do more than simply query data.

erulabs 2y ago

Ultimately someone has root permissions. Re: federal agencies, in the United States, that someone is clearly, constitutionally, the President. Article II of the constitution vests all power of the executive in the person of the President. The President has authority to appoint agents. That same article _does also_ say the President has to "take Care that the Laws be faithfully executed", but the "Care" there is highly debated. But the idea that the President doesn't have the right to appoint Musk to get root access to federal agencies seems legally incorrect. I'm not make a value judgement on this, it's just how it is. At a startup, the founder ultimately has root access to the database, no matter what the technical controls. Now, maybe it's stupid, and maybe it should be some other way, but to my mind the other way is that Congress gets together and writes a law saying "the executive cannot get root access to X, Y, Z". In absence of that law, the executive can do whatever they want. Not to be THAT GUY, but "an append-only database which cannot be modified by anyone" is something HN has spent the past 10 years saying is completely useless...

netsharc 2y ago
And Trump can launch the nukes to blow up the world too... but building a system where he can just click a button to do so would be idiotic. Same idea with giving godmode to the guy who thinks carrying a sink and saying "Let that sink in" is hilariously clever.
Rapzid 2y ago
The power rests with the office. There is an important but nuanced distinction there.