4 ms·

Man, I really have the impression that my video-on-demand service has a better understanding of ensuring availability and risk-management than either my bank or

by herge 12y ago

Man, I really have the impression that my video-on-demand service has a better understanding of ensuring availability and risk-management than either my bank or any government service. Could you imagine say a utility company creating a title called 'Chaos Engineer'.

hortonew 12y ago
It's so true. The entertainment/consumer shopping industries seems to have surpassed banks/governments in availability. It's also pretty ridiculous knowing that some of these companies are tasked with keeping our data/money safe, and yet they still don't allow extremely complex passwords.
- tracker1 12y ago
  Scaling financial transactions requires real sharding, but you can't eliminate ACID from the mix... certain functionality just doesn't work like that. In the end, one system needs to be responsible for a given account... best case you could have is only some accounts/users are affected. It's a different kind of problem. Facebook, for example uses Cassandra with a very wide distribution of nodes, with an immediate ack on receive... you can't do that with a bank... transactional payment systems won't tolerate it. You could do a few things to alleviate this issue... read-only replicas, etc... even then it's a matter of failing soft (disabling only those systems/services that are unavailable instead of the whole thing).
- deleted 12y ago
  [deleted]
Kronopath 12y ago
Can you imagine how difficult that would be to justify in a bank or government service? It would be very hard to convince someone in a position of power at an institution like that that deliberately introducing random failures and problems into your production services would be a good idea. Even if it is a good idea, it would be hard to make that argument. If Netflix's system fails, the worst thing that could happen is you don't get to watch Orange is the New Black. If your bank's system fails, you may lose payments, fail to pay your bills on time, or worse.
- opendais 12y ago
  You run a staging environment to mirror production [with fewer nodes per PoP] and throw things at it. The only real difference is the load on staging would be generated artificially [e.g. ACH, credit card processing] using faked staging-only accounts. You even have a fake banking website for security audits that is a clone of production with the fake accounts. No risk to production and probably 80% of the benefits. Of course its probably 125% of the effort.
  - diminoten 12y ago
    Show me a staging environment that's as robust as a production environment...
    - deeviant 12y ago
      My last company I worked at, they had like 30% of the staging environment running production stuff. They also had only one deploy target, which is drum roll production, which meant all of staging AND some of Dev/QA was live and could interact with the production environment. Also fun to see the production load at 300% of expected, burning, only to find out some event, data roll up, query, whatever was being run in duplicate on 10 different machines because somebody forget to to manually edit the configs after they rolled out a new version to staging/QA. Although I think this would qualify as "chaos engineering", I don't think it fits with what netflix is going for. Yeah, ok that had nothing to do with the OP, sorry, I just had to vent.
      - diminoten 12y ago
        That really sucks, I'm sorry. The kinds of things I work on now are entire "environments" we sell, so "production" testing is more along the lines of "go plug the box in and do stuff to it".
    - opendais 12y ago
      1) I stated it as a requirement. 2) The projects I work at on $DAY_JOB that I control all have 3 hardware nodes for HA in both staging & production. Staging is under artificial load that plays back the same series of events on dummy data. [e.g. Job A runs -> Completes -> Puts a mock job in Staging's Job Queue -> runs on Staging with mocked data -> I randomly restart a node to see what happens] Buying 3 1Us is $750 for a staging environment. If you have 1 DC availability, you can just put them on your Local LAN (Free). http://www.ebay.com/itm/Sun-Oracle-x4140-1U-Server-2x-Opteron-2431-Six-Core-32GB-SSD-SAS-RAID-2x-146GB-/141396271527?pt=COMP_EN_Servers&hash=item20ebdfe1a7 http://www.ebay.com/itm/Sun-Oracle-x4140-1U-Server-2x-Optero... Now, you suddenly have a rough approximation of a 3 Node / 1 DC environment. But what if you have multiple DCs, 3 Nodes per DC, Hot/Cold Loadbalancers and have DNS Failover Between Them?! Say, you run a LBx2+Nodex3/LBx2+Nodex3/LBx2+Nodex3 across 3 DCs. Assuming you automate the build, you could just pay by hour. But if not... Linode's Failover IPs let you do the Hot/Cold LB thing [$60 for 3 DCs worth]. 3 Nodes per DC [$40 Nodes for 4096 MB is probably more than enough to test most issues]. $360 So for $420/month, you could replicate a pretty big real world setup. What if you have 25 servers like stackoverflow and a hot/cold DC setup that is all Linux? http://highscalability.com/blog/2014/7/21/stackoverflow-update-560m-pageviews-a-month-25-servers-and-i.html http://highscalability.com/blog/2014/7/21/stackoverflow-upda... Linode's Failover IPs let you do the Hot/Cold LB thing [$40 for 2 DCs worth]. 13 Nodes per DC [$40 Nodes for 4096 MB is probably more than enough to test most issues]. $520 $560/month you can basically duplicate the 2 coast hot/cold setup with 26 servers. But its expensive! ...unless this is a one-man operation, not really. A fully loaded junior engineer is a minimum of $120k. A robust staging environment is very useful and isn't going to cost more than 1 engineer month to setup. Yes, you might not use the same hardware. However, it is close enough and honestly...the same hardware 1:1 with real servers [based on the ebay servers] and a rack at Fremont w/ Hurricane Electric is $600/month. 13 servers is $3,250 in hardware. Get someone else on the East Coast in like Buffalo, NY and you can get as close as your budget affords. I doubt you'd hit 1 Junior Engineer Year worth of $$. Not really Free. Power and A/C are things you'll pay for but its not a notable cost [less than $100/month].
deleted 12y ago
[deleted]
angersock 12y ago
4chan has better availability and higher traffic than healthcare.gov. Think about that.
elblanco 12y ago
I feel like there's lots of space for b2b disruption in banking. Banks really shouldn't be in the business of building secure and robust software. They don't manufacture their own vaults either.

by herge 12y ago

hortonew 12y ago

It's so true. The entertainment/consumer shopping industries seems to have surpassed banks/governments in availability. It's also pretty ridiculous knowing that some of these companies are tasked with keeping our data/money safe, and yet they still don't allow extremely complex passwords.

tracker1 12y ago
Scaling financial transactions requires real sharding, but you can't eliminate ACID from the mix... certain functionality just doesn't work like that. In the end, one system needs to be responsible for a given account... best case you could have is only some accounts/users are affected. It's a different kind of problem. Facebook, for example uses Cassandra with a very wide distribution of nodes, with an immediate ack on receive... you can't do that with a bank... transactional payment systems won't tolerate it. You could do a few things to alleviate this issue... read-only replicas, etc... even then it's a matter of failing soft (disabling only those systems/services that are unavailable instead of the whole thing).
deleted 12y ago
[deleted]

Kronopath 12y ago

Can you imagine how difficult that would be to justify in a bank or government service? It would be very hard to convince someone in a position of power at an institution like that that deliberately introducing random failures and problems into your production services would be a good idea. Even if it is a good idea, it would be hard to make that argument. If Netflix's system fails, the worst thing that could happen is you don't get to watch Orange is the New Black. If your bank's system fails, you may lose payments, fail to pay your bills on time, or worse.

opendais 12y ago
You run a staging environment to mirror production [with fewer nodes per PoP] and throw things at it. The only real difference is the load on staging would be generated artificially [e.g. ACH, credit card processing] using faked staging-only accounts. You even have a fake banking website for security audits that is a clone of production with the fake accounts. No risk to production and probably 80% of the benefits. Of course its probably 125% of the effort.
- diminoten 12y ago
  Show me a staging environment that's as robust as a production environment...
  - deeviant 12y ago
    My last company I worked at, they had like 30% of the staging environment running production stuff. They also had only one deploy target, which is drum roll production, which meant all of staging AND some of Dev/QA was live and could interact with the production environment. Also fun to see the production load at 300% of expected, burning, only to find out some event, data roll up, query, whatever was being run in duplicate on 10 different machines because somebody forget to to manually edit the configs after they rolled out a new version to staging/QA. Although I think this would qualify as "chaos engineering", I don't think it fits with what netflix is going for. Yeah, ok that had nothing to do with the OP, sorry, I just had to vent.
    - diminoten 12y ago
      That really sucks, I'm sorry. The kinds of things I work on now are entire "environments" we sell, so "production" testing is more along the lines of "go plug the box in and do stuff to it".
  - opendais 12y ago
    1) I stated it as a requirement. 2) The projects I work at on $DAY_JOB that I control all have 3 hardware nodes for HA in both staging & production. Staging is under artificial load that plays back the same series of events on dummy data. [e.g. Job A runs -> Completes -> Puts a mock job in Staging's Job Queue -> runs on Staging with mocked data -> I randomly restart a node to see what happens] Buying 3 1Us is $750 for a staging environment. If you have 1 DC availability, you can just put them on your Local LAN (Free). http://www.ebay.com/itm/Sun-Oracle-x4140-1U-Server-2x-Opteron-2431-Six-Core-32GB-SSD-SAS-RAID-2x-146GB-/141396271527?pt=COMP_EN_Servers&hash=item20ebdfe1a7 http://www.ebay.com/itm/Sun-Oracle-x4140-1U-Server-2x-Optero... Now, you suddenly have a rough approximation of a 3 Node / 1 DC environment. But what if you have multiple DCs, 3 Nodes per DC, Hot/Cold Loadbalancers and have DNS Failover Between Them?! Say, you run a LBx2+Nodex3/LBx2+Nodex3/LBx2+Nodex3 across 3 DCs. Assuming you automate the build, you could just pay by hour. But if not... Linode's Failover IPs let you do the Hot/Cold LB thing [$60 for 3 DCs worth]. 3 Nodes per DC [$40 Nodes for 4096 MB is probably more than enough to test most issues]. $360 So for $420/month, you could replicate a pretty big real world setup. What if you have 25 servers like stackoverflow and a hot/cold DC setup that is all Linux? http://highscalability.com/blog/2014/7/21/stackoverflow-update-560m-pageviews-a-month-25-servers-and-i.html http://highscalability.com/blog/2014/7/21/stackoverflow-upda... Linode's Failover IPs let you do the Hot/Cold LB thing [$40 for 2 DCs worth]. 13 Nodes per DC [$40 Nodes for 4096 MB is probably more than enough to test most issues]. $520 $560/month you can basically duplicate the 2 coast hot/cold setup with 26 servers. But its expensive! ...unless this is a one-man operation, not really. A fully loaded junior engineer is a minimum of $120k. A robust staging environment is very useful and isn't going to cost more than 1 engineer month to setup. Yes, you might not use the same hardware. However, it is close enough and honestly...the same hardware 1:1 with real servers [based on the ebay servers] and a rack at Fremont w/ Hurricane Electric is $600/month. 13 servers is $3,250 in hardware. Get someone else on the East Coast in like Buffalo, NY and you can get as close as your budget affords. I doubt you'd hit 1 Junior Engineer Year worth of $$. Not really Free. Power and A/C are things you'll pay for but its not a notable cost [less than $100/month].

deleted 12y ago

[deleted]

angersock 12y ago

4chan has better availability and higher traffic than healthcare.gov. Think about that.

elblanco 12y ago

I feel like there's lots of space for b2b disruption in banking. Banks really shouldn't be in the business of building secure and robust software. They don't manufacture their own vaults either.