25 ms·

Netflix Billing Migration to AWS

219 points by hepha1979 10y ago

merb 10y ago
They used a DRBD replicated MySQL. Wonder why they used MySQL over PostgresSQL then.. Would be great to know if they looked that up aswell.
- setheron 10y ago
  What does DRBD have to do with MySQL vs PostgresSQL ? DRBD is just block device replication
  - merb 10y ago
    Actually you could do DRBD with PostgreSQL, too. There is a Master-Master Replication with MySQL that PostgreSQL don't has, but since they didn't used that it would be great to hear we they've choosen MySQL over PostgreSQL. Would be great if they would share that, too.
    - deleted 10y ago
      [deleted]
    - gaius 10y ago
      DRBD and what database you use are orthogonal surely?
      - ecnahc515 10y ago
        The usage of DRDB somewhat eliminates the need for built in master/master replication, since its handled at the block layer. The two disks are always in sync, thus you don't need mysql's master/master setup. One of the main features missing from Postgres is that feature. Since it's not being used in their Mysql setup many people might ask why they wouldn't just use Postgres.
  - gdulli 10y ago
    I think it's just a reflex for (ex-MySQL) Postgres users to ask that anytime they see someone using MySQL. If you have significant experience with both you know the quality of life is different between the two.
    - merb 10y ago
      Actually I never used MySQL really. But sometimes it's great to know why decisions are made. Actually I don't think they said, "well lets use MySQL over Oracle" especially since MySQL is a Oracle product, too. There would've been a way to use MariaDB aswell. And I guess anything with license costs fall out already (they explained why in the article). Edit: My guess would be that they still keep galera in mind, but since they didn't shared they why, one could only guess. And Transaction Wraparound maybe.
      - rimantas 10y ago
        MySQL had useable replication years beofre PG. Maybe that's why.
      - beachstartup 10y ago
        it's probably because they still had oracle software and support contracts in place, and in-house knowledge of mysql.
    - bpicolo 10y ago
      Really? Because Amazon, Netflix, Facebook, and Google all have stake in the MySQL ring (google least so). MySQL has a lot of good knowns at scale, too.
      - gdulli 10y ago
        All companies old enough to have a lot of legacy infrastructure from before it was common knowledge that Postgres is better. It would be more curious why MySQL would be adopted today than 10-15 years ago.
        10y ago
- yeukhon 10y ago
  The interesting thing is cross-region read replication has existed for MySQL since in 2013, whereas cross-region read replication for PG has been out a few days out officially. https://aws.amazon.com/blogs/aws/cross-region-read-replicas-for-amazon-rds-for-mysql/ https://aws.amazon.com/blogs/aws/cross-region-read-replicas-... https://aws.amazon.com/about-aws/whats-new/2016/06/amazon-rds-for-postgresql-now-supports-cross-region-read-replicas/ https://aws.amazon.com/about-aws/whats-new/2016/06/amazon-rd... I am interested in non-read replica replication...
- frik 10y ago
  In the last 15 years every news (on /. and HN) that mentions MySQL is interrupted with such questions from Postgres fanboys. How about: MySQL is the very best solution for such tasks and Postgres isn't the universal holy grail? Wouldn't it be better to try to convert people over from (Postgres direct competitors) MSSQL, OracleDB, DB2 to Postgres?
  - deathanatos 10y ago
    Frankly, no; I've used all of Postgres, MySQL, MSSQL, and SQLite, and MySQL is the worst of the lot. No, PostgreSQL isn't a holy grain, but any user of MySQL would overall be better served by PostgreSQL; especially on AWS where the replication is so simple to set up, which IMO is the only good argument I've heard in favor of MySQL (but I hear even that is fraught with peril). MySQL is so completely riddled with bugs and utterly baffling behavior — it is the PHP of relational DBs. What I remember off the top of my head includes attempting to subtract datetimes causing the punctuation to be removed from the datetimes and the resulting "integers" to be subtracted, FK integrity being violated in a number of easy-to-hit corner cases, the "utf8" encoding not being able to encode UTF-8 (and the default being latin1…), GROUP BY allowing obviously (i.e., catchable to the parser) broken queries, `SELECT * FROM table` on <10k row tables taking minutes in some cases, the SQL dialect swapping the words "key" and "index" inappropriately; Read [1] if you want more. Thus far the only other tool I've seen come close to this level of insanity is MongoDB. The thing about a tool so willfully discards any sort of reasoned approach to its topic area is that the people who use it — who inevitably are not well versed in how a relational database works "in theory" — cannot derive from its behavior its rules, because its behavior is irrational, bordering on psychotic, and the people using it tweak query after query while having no understanding of why one query might work better until some abomination that someone usually works is crafted; `-- Don't touch`. A good tool — I believe, somewhere deep inside me — will teach the novice user. MySQL will not; it will drive you insane. [1]: http://grimoire.ca/mysql/choose-something-else http://grimoire.ca/mysql/choose-something-else
    - Matt3o12_ 10y ago
      It should be noted that most Database supported by cloud providers are actually MySQL (compatible) database. While Amazon supports a managed Postgres Database as well, I would bet that their own MySQL database – Amazon Aurora – works best because it integrated best into the AWS eco system. Same goes for Google, unlike Amazon, the only managed Database they offer is a MySQL database (called Cloud SQL). I'm not disagreeing with your arguments (quite the opposite in fact), but that might be an important point to use MySQL over Postgres.
crisopolis 10y ago
At least they ditched Oracle (licensed)...
- emcrazyone 10y ago
  I worked at a Fortune-5 that was heavily invested in Oracle. Oracle has a nasty licensing model where they charge you per core regardless of if that core is a physical one or not (hyper-threading). While I was there, it suites told all the engineering managers that Oracle was out and the going forward solution was Microsoft SQL which, as I understand, has more relaxed licensing model. Another thing I'm wondering about is I would figure Netflix to be big enough to have SAN storage. Just about every large company I worked at always used SAN replication technologies instead of open source stuff. And it's not a debate about open source solution vs. commercial. It's more about support. Large companies want a throat to grab when things break bad.
  - matt_wulfeck 10y ago
    Netflix doesn't have SAN because it doesn't run anything outside of AWS. AWS provides the throat to hold in your analogy.
    - emcrazyone 10y ago
      I was pondering about the netflix data center prior to the AWS migration. "Migration of Billing infrastructure from Netflix Data Center(DC) to AWS Cloud was part of a broader initiative." Sorry, should have been more clear. As a follow up. Many projects I was involved with did use Oracle replication but it was a rolling log file type that was purposely delayed to account for mistakes. Rolling replication happen across geographic locations while SAN replication dealt with the hot fail-over situations locally.
  - c17r 10y ago
    MS SQL used to be per socket pricing. With 2012 they switched to per core. That was a sad day.
hetfeld 10y ago
Dropped Oracle, using MySQL. Why not use PostgreSQL instead?
- eterm 10y ago
  Is there a version of poe's law for hacker news / slashdot? A large company describes their very real efforts and shares their experience and knowledge, and the first comment is "Why not use <preferred thing> instead?".
  - Rodeoclash 10y ago
    Because knowing how they worked through the decision to come to a conclusion on using a particular product is valuable. Postgres is usually the first choice for applications like this so knowing why they chose something else over that could influence others that need to make a similar decision.
    - thinkingkong 10y ago
      When you say usually you probably mean "currently, usually" or "in the last x years usually". Netflix has super smart people. They arent just picking random tools off the shelf and implementing things without constraints for the sake of it.
      - angersock 10y ago
        Right, which is why asking how they came to that decision (instead of using a DB dating back over three decades of development) is useful and reasonable.
    - otterley 10y ago
      > Postgres is usually the first choice for applications like this Do you have any data to substantiate this claim?
- justinhj 10y ago
  Assuming MySQL meets their needs there's no hugely compelling reason to choose PostgreSQL over MySQL.
ForHackernews 10y ago
Does this mean Amazon can mine Netflix's data to improve their Prime Video services?
- justicezyx 10y ago
  There is explicit policy inside AWS to not access any user data.
  - mgrennan 10y ago
    And Sodden is wrong. The government is not collecting your data.
    - moosingin3space 10y ago
      Different scenarios. Netflix could sue Amazon for breach of contract if there were evidence of them mining their data.
- alttab 10y ago
  No, but they can probably look at how Netflix is utilizing AWS and make that better.
- deleted 10y ago
  [deleted]
- ascendantlogic 10y ago
  AWS is a gigantic cash cow for Amazon now. If they were caught with their hands in a competitor's cookie jar the fallout would cost them in the billions. They probably wouldn't risk it, but this is just me hypothesizing.
- thramp 10y ago
  Customer data is sacrosanct within Amazon. Cannot touch it without the customer’s consent. source: I work for Amazon Web Services.
mikikian 10y ago
Dropped Oracle, using MySQL. Why not use AWS Aurora instead?
- nemothekid 10y ago
  FTA: While our subscription processing was using data in our Cassandra datastore, our payment processor needed ACID capabilities of an RDBMS to process charge transactions. We still had a multi-terabyte database that would not fit in AWS RDS with TB limitations.
- jon-wood 10y ago
  Somewhere else in the comments here one off the engineers on this project said Aurora was still in beta at the time but they went with MySQL so they had a reasonably simple migration path later.
ww520 10y ago
DRBD works very well for high availability, especially good to provide failover for master database, since that usually requires fast failover like under 10 seconds. Five or six years ago, did couple HA setup with DRBD along with Linux HA and virtual IP. The failover work great.
- kbenson 10y ago
  I was under the impression the accepted way to fail-over with mysql and DRBD was to fail out the old server, and then start repairing tables, because you couldn't be sure they were in a valid state. That's about a decade old info though. I always just set up master/master replication with mysql. You can get free distributed reads that way if you architect your application right.
  - Rapzid 10y ago
    DRBD supports flushes and barriers. You just switch the primary and startup the database. InnoDB will recover as necessary and you are good to go. This is assuming you are using all InnoDB tables(you likely should be).
    - kbenson 10y ago
      > This is assuming you are using all InnoDB tables(you likely should be). Ah, that's probably it. ~2004/2005 InnoDB wasn't nearly as popular as it is today, IIRC (but it was there, and was used). In any case, multiple master with failover setups always worked well in my experience, as long as you take care to track the replication state.
  - ww520 10y ago
    DRBD produces two mirror copies of data. When the primary failed, the standby has whatever data the primary has just before the crash. When the standby starts up, the RDBMS goes through the normal recovery and bootup. It's same as the primary crashed and being started up again. MySQL's master/master has a number of complication and problems: 1. data loss due to async nature of replication, 2. update conflict on same data on multiple masters, 3. two masters mean two IP so all the clients need to know how to fail over to different IP, 4. complication in adding or removing master. With DRBD, the disks are mirrored so that's no chance of data loss once a transaction is committed. There's only one master so no complicate conflict resolution. Linux HA's virtual IP means the standby will take over the primary IP so the clients don't need to know there's a server failover. Adding or removing standby is easy. DRBD will sync the disks automatically, no downtime on the primary.
tzakrajs 10y ago
How many times is Netflix going to finalize its transition to AWS?
- deleted 10y ago
  [deleted]
- yeukhon 10y ago
  They won't 100% be on AWS, but probably close to 95%. Netflix owns its own media CDN.
  - dexterdog 10y ago
    So almost all of their bandwidth is handled outside of AWS. I would wager a guess that bandwidth is quite a bit more than 5% of their ops bill.
    - yeukhon 10y ago
      I think that's where you have to draw line between owning your own infrastructure and using a public cloud in the architecture.
adrianggg 10y ago
I got really excited because I read the heading as Netflix migration costs paid for by AWS. I thought they worked out deal to get a free tier during migration. Wow...oh...nevermind... :-)
- ryanmerket 10y ago
  I work for AWS. I believe we do offer some migration assistance for bigger startups. Hit me up if you want to learn more: rmerket@amazon.com
partiallypro 10y ago
I find it to be a peculiar business decision to completely (as much as you can) migrate to one of your largest competitors' cloud service. It seems like Microsoft is the only, of the larger cloud providers, that doesn't -really- compete with Netflix. (Google has YouTube, and I guess even Microsoft has a much smaller Windows Store presence.) Even if Amazon can't access the raw data, they could see how you're utilizing it to improve their own video service, and they get the benefit of billing you and potentially using their pricing leverage to squeeze your margins. I feel like Apple's approach of utilizing multiple providers makes more sense (though they do this for uptime and redundancy.) Maybe I'm being a pessimist.
- truncate 10y ago
  As per this article, Netflix only migrated their billing infrastructure not their video streaming service.
  - luhn 10y ago
    They only migrated their billing infrastructure because the rest of Netflix is already in AWS. As far as I'm aware, the only non-AWS Netflix component remaining is their CDN. (And it will likely remain that way.)
  - niftich 10y ago
    That's because they've already migrated their videostreaming infrastructure to AWS (and some in Google Cloud Storage as a failsafe). This Ars Technica article from February summarizes the multi-year effort: http://arstechnica.com/information-technology/2016/02/netflix-finishes-its-massive-migration-to-the-amazon-cloud/ http://arstechnica.com/information-technology/2016/02/netfli...
    - ihsw 10y ago
      Just to be clear -- their video streams are still served up via Netflix's OpenConnect appliances. The surrounding infrastructure -- everything "outside" of the play button -- finished moving to AWS. Their billing system has now joined its siblings in living on AWS.
cia48621793 10y ago
What if the Netflix side on AWS was hacked say like their security credentials was leaked?
innocenat 10y ago
Am I the only one who initially thought that Netflix is passing the bill for migration to AWS?
Annatar 10y ago
Considering how much code and data was interacting with Oracle, one of our objectives was to disintegrate our giant Oracle based solution into a services based architecture. Some of our APIs needed to be multi-region and highly available. So we decided to split our data into multiple data stores. Subscriber data was migrated to Cassandra data store. Our payment processing integration needed ACID transaction. Hence all relevant data was migrated to MYSQL. Considering that Cassandra is not ACID compliant with her "eventual consistency", and that MySQL is notorious for corrupting data and not functioning correctly, I am compelled to wonder just what kind of people work at Netflix. And who gets the idea to go to AWS and pay the full virtualization on Linux performance penalty? Now, I've done Oracle engineering at some very large databases (hundreds of millions of rows, OLTP and DWH), and I know that Oracle is a smoking fast database when the right people develop on it. Also makes me wonder what kind of code they had running, and what kind of people selected it, when they managed to gum up what is essentially the Bugatti Veyron of databases. Given this information from "Netflix" I won't be considering them as a potential employer any time soon. It has to be a mess over there.
- tedivm 10y ago
  The majority of the corruption issues with MySQL comes from using the innodb engine. AWS built their own MySQL engine called Aurora that I would be shocked if Netflix wasn't using. It's designed for distributed workloads and should be harder to corrupt. I'll admit I'm confused about picking Cassandra as well, but not for the same reasons you are. They're only storing subscriber data (billing address, subscription type, etc). That data is going to remain static for months at a time. When changed the only potential problem that could occur is the billing process using old data, but I'm guessing their system is smart enough to try again in thirty minutes. Oracle may be fast, but it's also expensive. This is billing, which means batched jobs running in the background- they don't care how long each individual transaction takes, and I'm positive it's going to be cheaper to roll up more servers to compensate than it is to pay Oracle's licensing fees.
  - ec109695 10y ago
    Amazon doesn't offer instances large enough on RDS, so they are managing their own. Do you have any sources for the corruption comparison between Aurora vs InnoDB?
buro9 10y ago
Thanks for writing this up, I'm just at the tail end of having re-architected the CloudFlare billing system, also a subscription system written in Java with a MySQL back-end, but fronted by a Go API that insulates the rest of the business from the internals of the billing system. The blog post covers a lot of the high level stuff really well, but I'm interested to learn whether you experienced any issues along the way, what they were, how you dealt with them. In CloudFlare's case our migration was made more complex by also adding PayPal, and changing our processing gateway. Both of which created risk that we've had to work hard to understand and mitigate, i.e. how different gateways may, with the same card, return different results, etc.
- pamonrails 10y ago
  We do a lot of migrations from custom systems or SaaS integrations to Kill Bill (open-source subscription billing and payments platform) [0] and we've summarized our strategy and usual pain points in our migration guide [1]. You might find it useful. Happy to chat offline too if you want to go into specifics. [0] http://killbill.io/ http://killbill.io/ [1] http://docs.killbill.io/0.16/migration_guide.html http://docs.killbill.io/0.16/migration_guide.html
back_beyond 10y ago
This thread is missing Google's PR team and links to its SRE book
jason46 10y ago
Is this why my daughter can't watch Young Justice? I've noticed quite a few titles show unavailable. Curious if Netflix is positioning to sell to amazon.

219 points by hepha1979 10y ago

merb 10y ago

They used a DRBD replicated MySQL. Wonder why they used MySQL over PostgresSQL then.. Would be great to know if they looked that up aswell.

setheron 10y ago
What does DRBD have to do with MySQL vs PostgresSQL ? DRBD is just block device replication
- merb 10y ago
  Actually you could do DRBD with PostgreSQL, too. There is a Master-Master Replication with MySQL that PostgreSQL don't has, but since they didn't used that it would be great to hear we they've choosen MySQL over PostgreSQL. Would be great if they would share that, too.
  - deleted 10y ago
    [deleted]
  - gaius 10y ago
    DRBD and what database you use are orthogonal surely?
    - ecnahc515 10y ago
      The usage of DRDB somewhat eliminates the need for built in master/master replication, since its handled at the block layer. The two disks are always in sync, thus you don't need mysql's master/master setup. One of the main features missing from Postgres is that feature. Since it's not being used in their Mysql setup many people might ask why they wouldn't just use Postgres.
- gdulli 10y ago
  I think it's just a reflex for (ex-MySQL) Postgres users to ask that anytime they see someone using MySQL. If you have significant experience with both you know the quality of life is different between the two.
  - merb 10y ago
    Actually I never used MySQL really. But sometimes it's great to know why decisions are made. Actually I don't think they said, "well lets use MySQL over Oracle" especially since MySQL is a Oracle product, too. There would've been a way to use MariaDB aswell. And I guess anything with license costs fall out already (they explained why in the article). Edit: My guess would be that they still keep galera in mind, but since they didn't shared they why, one could only guess. And Transaction Wraparound maybe.
    - rimantas 10y ago
      MySQL had useable replication years beofre PG. Maybe that's why.
    - beachstartup 10y ago
      it's probably because they still had oracle software and support contracts in place, and in-house knowledge of mysql.
  - bpicolo 10y ago
    Really? Because Amazon, Netflix, Facebook, and Google all have stake in the MySQL ring (google least so). MySQL has a lot of good knowns at scale, too.
    - gdulli 10y ago
      All companies old enough to have a lot of legacy infrastructure from before it was common knowledge that Postgres is better. It would be more curious why MySQL would be adopted today than 10-15 years ago.
      - 10y ago
yeukhon 10y ago
The interesting thing is cross-region read replication has existed for MySQL since in 2013, whereas cross-region read replication for PG has been out a few days out officially. https://aws.amazon.com/blogs/aws/cross-region-read-replicas-for-amazon-rds-for-mysql/ https://aws.amazon.com/blogs/aws/cross-region-read-replicas-... https://aws.amazon.com/about-aws/whats-new/2016/06/amazon-rds-for-postgresql-now-supports-cross-region-read-replicas/ https://aws.amazon.com/about-aws/whats-new/2016/06/amazon-rd... I am interested in non-read replica replication...
frik 10y ago
In the last 15 years every news (on /. and HN) that mentions MySQL is interrupted with such questions from Postgres fanboys. How about: MySQL is the very best solution for such tasks and Postgres isn't the universal holy grail? Wouldn't it be better to try to convert people over from (Postgres direct competitors) MSSQL, OracleDB, DB2 to Postgres?
- deathanatos 10y ago
  Frankly, no; I've used all of Postgres, MySQL, MSSQL, and SQLite, and MySQL is the worst of the lot. No, PostgreSQL isn't a holy grain, but any user of MySQL would overall be better served by PostgreSQL; especially on AWS where the replication is so simple to set up, which IMO is the only good argument I've heard in favor of MySQL (but I hear even that is fraught with peril). MySQL is so completely riddled with bugs and utterly baffling behavior — it is the PHP of relational DBs. What I remember off the top of my head includes attempting to subtract datetimes causing the punctuation to be removed from the datetimes and the resulting "integers" to be subtracted, FK integrity being violated in a number of easy-to-hit corner cases, the "utf8" encoding not being able to encode UTF-8 (and the default being latin1…), GROUP BY allowing obviously (i.e., catchable to the parser) broken queries, `SELECT * FROM table` on <10k row tables taking minutes in some cases, the SQL dialect swapping the words "key" and "index" inappropriately; Read [1] if you want more. Thus far the only other tool I've seen come close to this level of insanity is MongoDB. The thing about a tool so willfully discards any sort of reasoned approach to its topic area is that the people who use it — who inevitably are not well versed in how a relational database works "in theory" — cannot derive from its behavior its rules, because its behavior is irrational, bordering on psychotic, and the people using it tweak query after query while having no understanding of why one query might work better until some abomination that someone usually works is crafted; `-- Don't touch`. A good tool — I believe, somewhere deep inside me — will teach the novice user. MySQL will not; it will drive you insane. [1]: http://grimoire.ca/mysql/choose-something-else http://grimoire.ca/mysql/choose-something-else
  - Matt3o12_ 10y ago
    It should be noted that most Database supported by cloud providers are actually MySQL (compatible) database. While Amazon supports a managed Postgres Database as well, I would bet that their own MySQL database – Amazon Aurora – works best because it integrated best into the AWS eco system. Same goes for Google, unlike Amazon, the only managed Database they offer is a MySQL database (called Cloud SQL). I'm not disagreeing with your arguments (quite the opposite in fact), but that might be an important point to use MySQL over Postgres.

crisopolis 10y ago

At least they ditched Oracle (licensed)...

emcrazyone 10y ago
I worked at a Fortune-5 that was heavily invested in Oracle. Oracle has a nasty licensing model where they charge you per core regardless of if that core is a physical one or not (hyper-threading). While I was there, it suites told all the engineering managers that Oracle was out and the going forward solution was Microsoft SQL which, as I understand, has more relaxed licensing model. Another thing I'm wondering about is I would figure Netflix to be big enough to have SAN storage. Just about every large company I worked at always used SAN replication technologies instead of open source stuff. And it's not a debate about open source solution vs. commercial. It's more about support. Large companies want a throat to grab when things break bad.
- matt_wulfeck 10y ago
  Netflix doesn't have SAN because it doesn't run anything outside of AWS. AWS provides the throat to hold in your analogy.
  - emcrazyone 10y ago
    I was pondering about the netflix data center prior to the AWS migration. "Migration of Billing infrastructure from Netflix Data Center(DC) to AWS Cloud was part of a broader initiative." Sorry, should have been more clear. As a follow up. Many projects I was involved with did use Oracle replication but it was a rolling log file type that was purposely delayed to account for mistakes. Rolling replication happen across geographic locations while SAN replication dealt with the hot fail-over situations locally.
- c17r 10y ago
  MS SQL used to be per socket pricing. With 2012 they switched to per core. That was a sad day.

hetfeld 10y ago

Dropped Oracle, using MySQL. Why not use PostgreSQL instead?

eterm 10y ago
Is there a version of poe's law for hacker news / slashdot? A large company describes their very real efforts and shares their experience and knowledge, and the first comment is "Why not use <preferred thing> instead?".
- Rodeoclash 10y ago
  Because knowing how they worked through the decision to come to a conclusion on using a particular product is valuable. Postgres is usually the first choice for applications like this so knowing why they chose something else over that could influence others that need to make a similar decision.
  - thinkingkong 10y ago
    When you say usually you probably mean "currently, usually" or "in the last x years usually". Netflix has super smart people. They arent just picking random tools off the shelf and implementing things without constraints for the sake of it.
    - angersock 10y ago
      Right, which is why asking how they came to that decision (instead of using a DB dating back over three decades of development) is useful and reasonable.
  - otterley 10y ago
    > Postgres is usually the first choice for applications like this Do you have any data to substantiate this claim?
justinhj 10y ago
Assuming MySQL meets their needs there's no hugely compelling reason to choose PostgreSQL over MySQL.

ForHackernews 10y ago

Does this mean Amazon can mine Netflix's data to improve their Prime Video services?

justicezyx 10y ago
There is explicit policy inside AWS to not access any user data.
- mgrennan 10y ago
  And Sodden is wrong. The government is not collecting your data.
  - moosingin3space 10y ago
    Different scenarios. Netflix could sue Amazon for breach of contract if there were evidence of them mining their data.
alttab 10y ago
No, but they can probably look at how Netflix is utilizing AWS and make that better.
deleted 10y ago
[deleted]
ascendantlogic 10y ago
AWS is a gigantic cash cow for Amazon now. If they were caught with their hands in a competitor's cookie jar the fallout would cost them in the billions. They probably wouldn't risk it, but this is just me hypothesizing.
thramp 10y ago
Customer data is sacrosanct within Amazon. Cannot touch it without the customer’s consent. source: I work for Amazon Web Services.

mikikian 10y ago

Dropped Oracle, using MySQL. Why not use AWS Aurora instead?

nemothekid 10y ago
FTA: While our subscription processing was using data in our Cassandra datastore, our payment processor needed ACID capabilities of an RDBMS to process charge transactions. We still had a multi-terabyte database that would not fit in AWS RDS with TB limitations.
jon-wood 10y ago
Somewhere else in the comments here one off the engineers on this project said Aurora was still in beta at the time but they went with MySQL so they had a reasonably simple migration path later.

ww520 10y ago

DRBD works very well for high availability, especially good to provide failover for master database, since that usually requires fast failover like under 10 seconds. Five or six years ago, did couple HA setup with DRBD along with Linux HA and virtual IP. The failover work great.

kbenson 10y ago
I was under the impression the accepted way to fail-over with mysql and DRBD was to fail out the old server, and then start repairing tables, because you couldn't be sure they were in a valid state. That's about a decade old info though. I always just set up master/master replication with mysql. You can get free distributed reads that way if you architect your application right.
- Rapzid 10y ago
  DRBD supports flushes and barriers. You just switch the primary and startup the database. InnoDB will recover as necessary and you are good to go. This is assuming you are using all InnoDB tables(you likely should be).
  - kbenson 10y ago
    > This is assuming you are using all InnoDB tables(you likely should be). Ah, that's probably it. ~2004/2005 InnoDB wasn't nearly as popular as it is today, IIRC (but it was there, and was used). In any case, multiple master with failover setups always worked well in my experience, as long as you take care to track the replication state.
- ww520 10y ago
  DRBD produces two mirror copies of data. When the primary failed, the standby has whatever data the primary has just before the crash. When the standby starts up, the RDBMS goes through the normal recovery and bootup. It's same as the primary crashed and being started up again. MySQL's master/master has a number of complication and problems: 1. data loss due to async nature of replication, 2. update conflict on same data on multiple masters, 3. two masters mean two IP so all the clients need to know how to fail over to different IP, 4. complication in adding or removing master. With DRBD, the disks are mirrored so that's no chance of data loss once a transaction is committed. There's only one master so no complicate conflict resolution. Linux HA's virtual IP means the standby will take over the primary IP so the clients don't need to know there's a server failover. Adding or removing standby is easy. DRBD will sync the disks automatically, no downtime on the primary.

tzakrajs 10y ago

How many times is Netflix going to finalize its transition to AWS?

deleted 10y ago
[deleted]
yeukhon 10y ago
They won't 100% be on AWS, but probably close to 95%. Netflix owns its own media CDN.
- dexterdog 10y ago
  So almost all of their bandwidth is handled outside of AWS. I would wager a guess that bandwidth is quite a bit more than 5% of their ops bill.
  - yeukhon 10y ago
    I think that's where you have to draw line between owning your own infrastructure and using a public cloud in the architecture.

adrianggg 10y ago

I got really excited because I read the heading as Netflix migration costs paid for by AWS. I thought they worked out deal to get a free tier during migration. Wow...oh...nevermind... :-)

ryanmerket 10y ago
I work for AWS. I believe we do offer some migration assistance for bigger startups. Hit me up if you want to learn more: rmerket@amazon.com

partiallypro 10y ago

I find it to be a peculiar business decision to completely (as much as you can) migrate to one of your largest competitors' cloud service. It seems like Microsoft is the only, of the larger cloud providers, that doesn't -really- compete with Netflix. (Google has YouTube, and I guess even Microsoft has a much smaller Windows Store presence.) Even if Amazon can't access the raw data, they could see how you're utilizing it to improve their own video service, and they get the benefit of billing you and potentially using their pricing leverage to squeeze your margins. I feel like Apple's approach of utilizing multiple providers makes more sense (though they do this for uptime and redundancy.) Maybe I'm being a pessimist.

truncate 10y ago
As per this article, Netflix only migrated their billing infrastructure not their video streaming service.
- luhn 10y ago
  They only migrated their billing infrastructure because the rest of Netflix is already in AWS. As far as I'm aware, the only non-AWS Netflix component remaining is their CDN. (And it will likely remain that way.)
- niftich 10y ago
  That's because they've already migrated their videostreaming infrastructure to AWS (and some in Google Cloud Storage as a failsafe). This Ars Technica article from February summarizes the multi-year effort: http://arstechnica.com/information-technology/2016/02/netflix-finishes-its-massive-migration-to-the-amazon-cloud/ http://arstechnica.com/information-technology/2016/02/netfli...
  - ihsw 10y ago
    Just to be clear -- their video streams are still served up via Netflix's OpenConnect appliances. The surrounding infrastructure -- everything "outside" of the play button -- finished moving to AWS. Their billing system has now joined its siblings in living on AWS.

cia48621793 10y ago

What if the Netflix side on AWS was hacked say like their security credentials was leaked?

innocenat 10y ago

Am I the only one who initially thought that Netflix is passing the bill for migration to AWS?

Annatar 10y ago

Considering how much code and data was interacting with Oracle, one of our objectives was to disintegrate our giant Oracle based solution into a services based architecture. Some of our APIs needed to be multi-region and highly available. So we decided to split our data into multiple data stores. Subscriber data was migrated to Cassandra data store. Our payment processing integration needed ACID transaction. Hence all relevant data was migrated to MYSQL. Considering that Cassandra is not ACID compliant with her "eventual consistency", and that MySQL is notorious for corrupting data and not functioning correctly, I am compelled to wonder just what kind of people work at Netflix. And who gets the idea to go to AWS and pay the full virtualization on Linux performance penalty? Now, I've done Oracle engineering at some very large databases (hundreds of millions of rows, OLTP and DWH), and I know that Oracle is a smoking fast database when the right people develop on it. Also makes me wonder what kind of code they had running, and what kind of people selected it, when they managed to gum up what is essentially the Bugatti Veyron of databases. Given this information from "Netflix" I won't be considering them as a potential employer any time soon. It has to be a mess over there.

tedivm 10y ago
The majority of the corruption issues with MySQL comes from using the innodb engine. AWS built their own MySQL engine called Aurora that I would be shocked if Netflix wasn't using. It's designed for distributed workloads and should be harder to corrupt. I'll admit I'm confused about picking Cassandra as well, but not for the same reasons you are. They're only storing subscriber data (billing address, subscription type, etc). That data is going to remain static for months at a time. When changed the only potential problem that could occur is the billing process using old data, but I'm guessing their system is smart enough to try again in thirty minutes. Oracle may be fast, but it's also expensive. This is billing, which means batched jobs running in the background- they don't care how long each individual transaction takes, and I'm positive it's going to be cheaper to roll up more servers to compensate than it is to pay Oracle's licensing fees.
- ec109695 10y ago
  Amazon doesn't offer instances large enough on RDS, so they are managing their own. Do you have any sources for the corruption comparison between Aurora vs InnoDB?

buro9 10y ago

Thanks for writing this up, I'm just at the tail end of having re-architected the CloudFlare billing system, also a subscription system written in Java with a MySQL back-end, but fronted by a Go API that insulates the rest of the business from the internals of the billing system. The blog post covers a lot of the high level stuff really well, but I'm interested to learn whether you experienced any issues along the way, what they were, how you dealt with them. In CloudFlare's case our migration was made more complex by also adding PayPal, and changing our processing gateway. Both of which created risk that we've had to work hard to understand and mitigate, i.e. how different gateways may, with the same card, return different results, etc.

pamonrails 10y ago
We do a lot of migrations from custom systems or SaaS integrations to Kill Bill (open-source subscription billing and payments platform) [0] and we've summarized our strategy and usual pain points in our migration guide [1]. You might find it useful. Happy to chat offline too if you want to go into specifics. [0] http://killbill.io/ http://killbill.io/ [1] http://docs.killbill.io/0.16/migration_guide.html http://docs.killbill.io/0.16/migration_guide.html

back_beyond 10y ago

This thread is missing Google's PR team and links to its SRE book

jason46 10y ago

Is this why my daughter can't watch Young Justice? I've noticed quite a few titles show unavailable. Curious if Netflix is positioning to sell to amazon.