5 ms·

> The shell the tests ran in unknowingly had a DATABASE_URL environment variable set as our production database. It was an old terminal window in a tmux session

> The shell the tests ran in unknowingly had a DATABASE_URL environment variable set as our production database. It was an old terminal window in a tmux session that had been used for inspecting production data many days before. The developer returned to this window and executed the test suite with the DATABASE_URL still set. I was expecting something like this. I remember, I configured my terminal windows to change their background when I'm on production systems [2], after around I read about gitlab database incident [1]. 1 : https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/ https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-... 2 : http://www.drinchev.com/blog/ssh-and-terminal-background/ http://www.drinchev.com/blog/ssh-and-terminal-background/

cjbprime 9y ago
Though note that wouldn't help here, at least according to a common sense reading of this part of the post: > we connected our development environment to a production database with write access i.e. they weren't logged in to a production machine, they were logged in to a development machine that was permitted to connect directly with write access to the production database.
- geofft 9y ago
  Since it's a local shell doing something like this is a lot easier - just change $PS1 to add some colors when certain variables are set. I use something like this today on my personal machine to distinguish between my personal and work email addresses in $EMAIL (for silly firewall reasons it's easier to originate work-owned OSS on my personal machine), and on my work machine to tell me which production zone I'm talking to - but I don't have it color when I have any variable set at all, I probably should do that.
mohitmun 9y ago
checked out this https://github.com/drinchev/phook https://github.com/drinchev/phook. Really awesome!!
jsmeaton 9y ago
I did a similar thing as a result of the gitlab incident. I use a custom iterm2 profile called "prod" that sshs into the production "shell box" and sets up a badge + bright colour. https://imgur.com/a/QdbMM https://imgur.com/a/QdbMM It's the most convenient way for me to get a production shell: CMD+O -> "prod" -> Enter Drops me straight where I need to be.
- bpicolo 9y ago
  TIL about iterm profiles. That's going to be handy. :)
konschubert 9y ago
How come it was enough to set a different database URL in the development server and BAM you're connected to the production database? Do they share the same password? Or is the password in the URL?
- aprdm 9y ago
  The database URL usually has the username and password as well. For example: postgresql://[user[:password]@][netloc][:port][/dbname][?param1=value1&...]
  - astura 9y ago
    [deleted]
    - detaro 9y ago
      No, its a URL. Connection strings have a different format, as clearly shown in the example in your link.
  - anonacct37 9y ago
    For anything using libpq a .pgpase file can help separate creds in source/conf from now machine level ACLS.
    - anonacct37 9y ago
      *.pgpass
philip1209 9y ago
I've similarly started to rely on Google Cloud Shell as the only way to access our production database. That way, firewalls never have to allow external access, and it's very clear whether I'm on production. Adding a Metabase instance with read-only credentials has helped this a lot - because executing one-off data queries is easy (and safe) now.

cjbprime 9y ago

Though note that wouldn't help here, at least according to a common sense reading of this part of the post: > we connected our development environment to a production database with write access i.e. they weren't logged in to a production machine, they were logged in to a development machine that was permitted to connect directly with write access to the production database.

geofft 9y ago
Since it's a local shell doing something like this is a lot easier - just change $PS1 to add some colors when certain variables are set. I use something like this today on my personal machine to distinguish between my personal and work email addresses in $EMAIL (for silly firewall reasons it's easier to originate work-owned OSS on my personal machine), and on my work machine to tell me which production zone I'm talking to - but I don't have it color when I have any variable set at all, I probably should do that.

mohitmun 9y ago

checked out this https://github.com/drinchev/phook https://github.com/drinchev/phook. Really awesome!!

jsmeaton 9y ago

I did a similar thing as a result of the gitlab incident. I use a custom iterm2 profile called "prod" that sshs into the production "shell box" and sets up a badge + bright colour. https://imgur.com/a/QdbMM https://imgur.com/a/QdbMM It's the most convenient way for me to get a production shell: CMD+O -> "prod" -> Enter Drops me straight where I need to be.

bpicolo 9y ago
TIL about iterm profiles. That's going to be handy. :)

konschubert 9y ago

How come it was enough to set a different database URL in the development server and BAM you're connected to the production database? Do they share the same password? Or is the password in the URL?

aprdm 9y ago
The database URL usually has the username and password as well. For example: postgresql://[user[:password]@][netloc][:port][/dbname][?param1=value1&...]
- astura 9y ago
  [deleted]
  - detaro 9y ago
    No, its a URL. Connection strings have a different format, as clearly shown in the example in your link.
- anonacct37 9y ago
  For anything using libpq a .pgpase file can help separate creds in source/conf from now machine level ACLS.
  - anonacct37 9y ago
    *.pgpass

philip1209 9y ago

I've similarly started to rely on Google Cloud Shell as the only way to access our production database. That way, firewalls never have to allow external access, and it's very clear whether I'm on production. Adding a Metabase instance with read-only credentials has helped this a lot - because executing one-off data queries is easy (and safe) now.