17 ms·

A tool like this was at the core of the Twitter frontend rewrite done years ago ~2011 time frame. Diffed both HTML and JSON responses, at scale, to find any iss

A tool like this was at the core of the Twitter frontend rewrite done years ago ~2011 time frame. Diffed both HTML and JSON responses, at scale, to find any issues. Amusingly found lots of issues in the original where the new system was correct. edit: We did it via a front-end proxy that would send the request to the original system, get the response and send it, then send the request to the test system along with the response from the first system and diffed them. Had a bunch of heuristics for various element types where you could expect differences between them (timestamps mostly).

zackangelo 6y ago
open sourced as diffy i believe: https://github.com/opendiffy/diffy https://github.com/opendiffy/diffy
- spullara 6y ago
  At the time it was just hacked into our front-end traffic server. This is probably just inspired by it.
- sa46 6y ago
  I looked into diffy but the license was changed from Apache to AGPL about a year ago in what appeared to be a monetization scheme based on the links to the support company in the REAMDME. https://github.com/opendiffy/diffy/commit/0848f9888f59aded4d2635666bcb1750b6f82b77 https://github.com/opendiffy/diffy/commit/0848f9888f59aded4d...
homero 6y ago
Do you send all traffic? Wouldn't that require 3x the servers? Double API plus some proxy servers.
- spullara 6y ago
  Nah. At 500k rps there was no need, sampling wise and as you point out hugely expensive. Though the new system was 10x fewer servers at 10x lower latency :). At smaller scale, you could probably do all of them.
donavanm 6y ago
My recollection is that OpenDNS(?) did the same thing on an ongoing basis. They mirrored prod network traffic in to a gamma environment to test/compare responses. I seem to recall it come up at a DNS OARC workshop a few years back if anyone wants to look for more details. Another system I worked on essentially did passive packet level functional tests on production network traffic. Pretty good technique to pick up new variations in the protocol.

zackangelo 6y ago

open sourced as diffy i believe: https://github.com/opendiffy/diffy https://github.com/opendiffy/diffy

spullara 6y ago
At the time it was just hacked into our front-end traffic server. This is probably just inspired by it.
sa46 6y ago
I looked into diffy but the license was changed from Apache to AGPL about a year ago in what appeared to be a monetization scheme based on the links to the support company in the REAMDME. https://github.com/opendiffy/diffy/commit/0848f9888f59aded4d2635666bcb1750b6f82b77 https://github.com/opendiffy/diffy/commit/0848f9888f59aded4d...

homero 6y ago

Do you send all traffic? Wouldn't that require 3x the servers? Double API plus some proxy servers.

spullara 6y ago
Nah. At 500k rps there was no need, sampling wise and as you point out hugely expensive. Though the new system was 10x fewer servers at 10x lower latency :). At smaller scale, you could probably do all of them.

donavanm 6y ago

My recollection is that OpenDNS(?) did the same thing on an ongoing basis. They mirrored prod network traffic in to a gamma environment to test/compare responses. I seem to recall it come up at a DNS OARC workshop a few years back if anyone wants to look for more details. Another system I worked on essentially did passive packet level functional tests on production network traffic. Pretty good technique to pick up new variations in the protocol.