4 ms·
Am I the only person who's shocked by the idea that one of Scribd's developers was repeatedly taking down the website by pushing code live without testing it or
by kevingadd 17y ago
Am I the only person who's shocked by the idea that one of Scribd's developers was repeatedly taking down the website by pushing code live without testing it or understanding the queries it ran?
I guess maybe he's just exaggerating for entertainment value... It's good to see that he's sharing his mistakes for others to learn from, and going into detail on why things went wrong. However, it's really depressing to not see any attention paid to the root causes - WTF wasn't he testing his code?
- deleted 17y ago[deleted]
- mos1 17y agoI am completely unsurprised, and I'm not being sarcastic. They probably think that they're using an "agile" methodology.
- patio11 17y agoHalf of these are things that I could see doing quite easily -- for example, although I happen to know that "select count(*) from users where lower(users.user_name) = 'patio11'" will result in a full table scan on MySQL, that category of mistake is easy to make and has bitten me a time or three. And it is easy to miss if your testing methodology doesn't include "Now run it against a data set about as big as the production DB." So that would be a pattern to look out for both when coding and, critically, when designing processes to make sure this sort of thing doesn't happen to you. Then there's "code that throws an exception anywhere you execute it every time you execute it was pushed to production." This points to a rather different type of process error.
- bendtheblock 17y agoI know you're making a general point here, but for the example given, I think it's generally best to refresh your dev data from production regularly (maybe after each large release). That way you're always testing on data 'close' to the current production state, which would help you avoid any data-related issues like this.
- patio11 17y agoI think that is a good idea if you're able to get away with it. Scribd can. We can't do it at my day job. There are hundreds of different ways to break the personal information privacy law if we screw up anonymization of test data. For example, supposing we just take the naive approach and overwrite all names, addresses, emails, phone numbers, etc etc etc. Should be fine, right? Except, uh oh, the data is still personally identifiable: the data will tell you that a female student who took CS103 and English 101 last year was given a semester of medical leave. If a copy of that dataset leaks and someone in the department realizes "Hey wait, the only person in that double major is Hanako... medical leave... Hanako was pregnant last year?!?", then our company just made the front page news, we made our customer look horrible (and likely owe them and Hanako several tens of millions of yen in we're-so-sorry money), and we just broke the information privacy law something fierce. Incidentally, engineers not treating test data with the same "This CD is nuclear waste" precautions we treat the production data set is a frequent cause of breaches like this. Somebody decides to work from home for the day, gets his laptop stolen, bam front page news. I nearly got in severe trouble for leaving a printout of the student roster on the printer fifteen feet from a door somebody could tailgate through -- the only thing that saved my keister was that I could show that the student roster I printed out was fake. (Lesson learned about producing good test data: don't produce too good test data.)
- bendtheblock 17y agoWhy is this being down voted? The privacy concerns are valid but are the symptom of a different problem, which is controlling machine/network access. Previously I've worked in banking tech and this procedure was always followed (with data scrubbing), but it was also low risk since we were already on a secure network in a secure building. For a startup with sensitive live data though, I can see why it's best to keep prod data out of test environments and therefore lower the chances of laptops being left unlocked or print outs lost that contain sensitive data. It's clearly a cheaper and better solution to keeping everyone in a secure building and limiting their remote access.
- gizmo 17y agoMy vote goes to entertainment value, because the stories don't completely add up. When he was implementing the case-insensitive logins he would, of course, first have to run a query to see if there are any users whose login names collide without case sensitivity. That query would run (according to the slides) really slowly, because it would have to essentially do something like: SELECT a.id, b.id, a.username FROM users AS a, users AS B WHERE a.id != b.id AND a.username = b.username AND BINARY(a.username) != BINARY(b.username) At that point he'd realize the query is taking forever, and use mysql-top or something equivalent to kill the query. Edit: I realize that most likely the username column had a UNIQUE index already, making this query unnecessary. -- In his example of find_in_batches where he has to make some changes that applies to all users he makes it sound like the User.all is the big problem because it does a `SELECT * FROM users`. But even when you break up the work in parts you have to be careful, because by the time you're done processing a few of the existing users may have changed already. So as your database becomes too large to do your updates in one big atomic transaction block you almost always have to follow a process like this: 1. find set of rows that match the problem (in batches) 2. apply the (expensive) fix/migration on those rows 3. go back to 1 until the result set is empty 4. do (1) and (2) in a big transaction (but since there are only a few rows, that's fine) Of course there are a hundred ways to upgrade a live database safely, but there always must be some kind of process in place otherwise your database is going to go down all the time.
- deleted 17y ago[deleted]