6 ms·

Product Integration Testing at the Speed of Netflix

49 points by hepha1979 10y ago

bballer 10y ago
> "Secure Connection Failed" Link is broken.
- joshuacc 10y ago
  Looks fine here. Are you running HTTPS everywhere?
  - bballer 10y ago
    No. Doesn't appear to be an add-on problem. Firefox reports "Secure Connection Failed" and chrome reports "This site can’t be reached". Weird, just tested on Firefox for android (still connected to my local wifi) and it works fine. Debug time.
- turbohedgehog 10y ago
  I get the same issue on firefox/chrome/edge here.
joneholland 10y ago
Usually Netflix engineering impresses me. This does not. This sounds like all of the typical separate test org antipatterns that healthy companies eventually eliminate by eliminating the entire test team. Integrated tests are a scam. The only way to succeed in a micro services architecture is with a strong focus on quality inside each service and consumer driven contract testing between.
- existencebox 10y ago
  I have to disagree with your last statement (integrated tests are a scam) very strongly. Think about it as follows: We assert unit tests are useful, because although we may "trust" primitives (1+1=2) we can introduce semantic bugs in how we compose them. Does it not also follow that if our primitives are now "microservice functions" and our composition is the "integrated service", we should test the composition for validity? I truly believe in E2E testing, I could go on at length about esoteric bugs we found only in interactions between properly functioning microservices (Hell just today I watched a coworker work out a fun timezone doozie). One could make a feasible argument that with a more well defined set of constraints and capabilities, the microservice could have sufficiently covered unit test cases for all possible interactions from a stub perspective, but pragmatically, _this does not happen in the real world_, and E2E testing is a nice way of making sure your product is indeed continuing to not be on fire (mostly). One could feasibly (from my view of the world) say "Integration tests are a cheap fix", to which I'd say "yes, yes they are" and then continue to perform them, even if they are only in the form of a click through smoke test by hand after a prod deployment :)
  - deleted 10y ago
    [deleted]
  - joneholland 10y ago
    JB says it better than I can: http://blog.thecodewhisperer.com/permalink/integrated-tests-are-a-scam http://blog.thecodewhisperer.com/permalink/integrated-tests-... https://vimeo.com/80533536 https://vimeo.com/80533536
    - existencebox 10y ago
      He had me agreeing at first, but the blog at least kinda devolved into a highly numbered breakdown of the impossibility of total test coverage. As I said in my post, and the blog seemed to echo, I accept that an integration test is a sign that unit tests are not sufficient, but I didn't see a compelling argument in his blog as to how perfect unit test coverage (sufficient to not integration test the _happy path_, not all things ever) is obtained. I certainly can see the situation he's describing, but it feels like he goes on to expand it into a pathological strawman (ALL the unit tests EVERYWHERE testing EVERYTHING run at EVERY BUILD) more than a situation you manage a little more smartly. (As the oft-hammered quote, these things are all tools, the moment one stops not being useful/stops being worth the cost, we stop using it)
- deleted 10y ago
  [deleted]
- namidark 10y ago
  > The only way to succeed in a micro services architecture is with a strong focus on quality inside each service and consumer driven contract testing between. Do you have any examples from other large companies that do micro-services at Netflixs' scale?
was_boring 10y ago
Really disappointed everytime one of these types of articles comes out without any working code to go with it. It's great that high level information is being shared, but even better if the code was shared as well. It's not like Netflix is in the business of selling testing services.
- nitinreddy88 10y ago
  Exactly my thoughts. Its been same in many startups and companies. Whats something they are uniquely doing is not mentioned at all
- srean 10y ago
  That sounds a lot like entitlement to me.

49 points by hepha1979 10y ago

bballer 10y ago

> "Secure Connection Failed" Link is broken.

joshuacc 10y ago
Looks fine here. Are you running HTTPS everywhere?
- bballer 10y ago
  No. Doesn't appear to be an add-on problem. Firefox reports "Secure Connection Failed" and chrome reports "This site can’t be reached". Weird, just tested on Firefox for android (still connected to my local wifi) and it works fine. Debug time.
turbohedgehog 10y ago
I get the same issue on firefox/chrome/edge here.

joneholland 10y ago

Usually Netflix engineering impresses me. This does not. This sounds like all of the typical separate test org antipatterns that healthy companies eventually eliminate by eliminating the entire test team. Integrated tests are a scam. The only way to succeed in a micro services architecture is with a strong focus on quality inside each service and consumer driven contract testing between.

existencebox 10y ago
I have to disagree with your last statement (integrated tests are a scam) very strongly. Think about it as follows: We assert unit tests are useful, because although we may "trust" primitives (1+1=2) we can introduce semantic bugs in how we compose them. Does it not also follow that if our primitives are now "microservice functions" and our composition is the "integrated service", we should test the composition for validity? I truly believe in E2E testing, I could go on at length about esoteric bugs we found only in interactions between properly functioning microservices (Hell just today I watched a coworker work out a fun timezone doozie). One could make a feasible argument that with a more well defined set of constraints and capabilities, the microservice could have sufficiently covered unit test cases for all possible interactions from a stub perspective, but pragmatically, _this does not happen in the real world_, and E2E testing is a nice way of making sure your product is indeed continuing to not be on fire (mostly). One could feasibly (from my view of the world) say "Integration tests are a cheap fix", to which I'd say "yes, yes they are" and then continue to perform them, even if they are only in the form of a click through smoke test by hand after a prod deployment :)
- deleted 10y ago
  [deleted]
- joneholland 10y ago
  JB says it better than I can: http://blog.thecodewhisperer.com/permalink/integrated-tests-are-a-scam http://blog.thecodewhisperer.com/permalink/integrated-tests-... https://vimeo.com/80533536 https://vimeo.com/80533536
  - existencebox 10y ago
    He had me agreeing at first, but the blog at least kinda devolved into a highly numbered breakdown of the impossibility of total test coverage. As I said in my post, and the blog seemed to echo, I accept that an integration test is a sign that unit tests are not sufficient, but I didn't see a compelling argument in his blog as to how perfect unit test coverage (sufficient to not integration test the _happy path_, not all things ever) is obtained. I certainly can see the situation he's describing, but it feels like he goes on to expand it into a pathological strawman (ALL the unit tests EVERYWHERE testing EVERYTHING run at EVERY BUILD) more than a situation you manage a little more smartly. (As the oft-hammered quote, these things are all tools, the moment one stops not being useful/stops being worth the cost, we stop using it)
deleted 10y ago
[deleted]
namidark 10y ago
> The only way to succeed in a micro services architecture is with a strong focus on quality inside each service and consumer driven contract testing between. Do you have any examples from other large companies that do micro-services at Netflixs' scale?

was_boring 10y ago

Really disappointed everytime one of these types of articles comes out without any working code to go with it. It's great that high level information is being shared, but even better if the code was shared as well. It's not like Netflix is in the business of selling testing services.

nitinreddy88 10y ago
Exactly my thoughts. Its been same in many startups and companies. Whats something they are uniquely doing is not mentioned at all
srean 10y ago
That sounds a lot like entitlement to me.