4 ms·

One of the worst things about dealing with SaaS is that you might have no practical way to generated your own audit records for reconciliation during a dispute.

by michael1999 3y ago

One of the worst things about dealing with SaaS is that you might have no practical way to generated your own audit records for reconciliation during a dispute. You have to ask the vendor for data to dispute the vendor! The organizational incentives are all against you. Just a cursed relationship.

ncallaway 3y ago
> You have to ask the vendor for data to dispute the vendor! Isn’t that literally the purpose of discovery in lawsuits (a practice that long predates digital records at all, much less SaaS)
- varispeed 3y ago
  If administrator can change entries and audit logs, there is no real way to prove the data has or has not been tampered with.
- donmcronald 3y ago
  There's nothing to discover if they simply don't collect information that proves your case. My favorite is "we have no evidence that XYZ was stolen" when a company gets compromised. Why would you want high quality logging at that point? Don't log access to sensitive information and then there's no evidence it was "stolen" (copied or cloned is the correct term). Retention policies are a huge issue too. There have been a lot of cover-ups discovered by people going through old boxes of papers. That's not going to happen in the future because tech companies are providing enterprise grade tooling to continually scrub history via retention policies. No one is going to find a 10 year old "smoking gun" email if you have a 7 year retention policy on all communications.
- michael1999 3y ago
  No! The problem here is the the Royal Mail had bad records. Discovery doesn't help at all! Discovery just gets you a copy of the bad data. You need good data, and nobody has it! This is why we invented double-entry bookkeeping 500 years ago. And even if, by fluke, discovery might help because there was actually a good copy somewhere, that means you've already lost by spending a fortune on lawyers in a dispute with a crown corporation! Look at these disputes - lives were ruined over $44k pounds. Suing the Post Office wouldn't be cheaper than that. In a traditional paper based flow, I have my own copies. I talk to a salesperson, get a price, and fax a PO. I keep a copy of that PO on file. The vendor delivers the order along with a waybill. My staff receive the order and count it, reconciling against my order. If the count is short, we check the waybill. Is the waybill short too? Then there was a problem in order flow. If the waybill matches my order, but the shipment is short, then the problem is in fulfillment. I call sales and hash it out. Honest errors happen, and reasonable people resolve them when given the facts. If the salesperson says their copy of the PO looks different, and calls me a liar when I send them my copy, then we have a real dispute. We might have to get a lawyer involved. But this is well beyond honest error. But in most SaaS environments (GB egress, bytes storage, etc), you are dependent on the vendor for the data. There is no practical way to disagree with AWS if they tell you your usage was 20TB and you expected 15. You are trusting the vendor not to lie (maybe reasonable), and also to never make an error (totally unreasonable!). We now have no defense against honest error. B2C is in a similar boat. In olden-times, I knew people who kept a log book and stop watch by the phone and logged all their calls. Every month, they reconciled their phone bill against their log. Just like they reconciled their bank statement against their cheque stubs and deposit slips. But who in this day keeps their own log of PayPal transactions safely outside Paypal? My main safety with PayPay is short latency - they report significant transactions right away so any errors can be corrected quickly.

ncallaway 3y ago

> You have to ask the vendor for data to dispute the vendor! Isn’t that literally the purpose of discovery in lawsuits (a practice that long predates digital records at all, much less SaaS)

varispeed 3y ago
If administrator can change entries and audit logs, there is no real way to prove the data has or has not been tampered with.
donmcronald 3y ago
There's nothing to discover if they simply don't collect information that proves your case. My favorite is "we have no evidence that XYZ was stolen" when a company gets compromised. Why would you want high quality logging at that point? Don't log access to sensitive information and then there's no evidence it was "stolen" (copied or cloned is the correct term). Retention policies are a huge issue too. There have been a lot of cover-ups discovered by people going through old boxes of papers. That's not going to happen in the future because tech companies are providing enterprise grade tooling to continually scrub history via retention policies. No one is going to find a 10 year old "smoking gun" email if you have a 7 year retention policy on all communications.
michael1999 3y ago
No! The problem here is the the Royal Mail had bad records. Discovery doesn't help at all! Discovery just gets you a copy of the bad data. You need good data, and nobody has it! This is why we invented double-entry bookkeeping 500 years ago. And even if, by fluke, discovery might help because there was actually a good copy somewhere, that means you've already lost by spending a fortune on lawyers in a dispute with a crown corporation! Look at these disputes - lives were ruined over $44k pounds. Suing the Post Office wouldn't be cheaper than that. In a traditional paper based flow, I have my own copies. I talk to a salesperson, get a price, and fax a PO. I keep a copy of that PO on file. The vendor delivers the order along with a waybill. My staff receive the order and count it, reconciling against my order. If the count is short, we check the waybill. Is the waybill short too? Then there was a problem in order flow. If the waybill matches my order, but the shipment is short, then the problem is in fulfillment. I call sales and hash it out. Honest errors happen, and reasonable people resolve them when given the facts. If the salesperson says their copy of the PO looks different, and calls me a liar when I send them my copy, then we have a real dispute. We might have to get a lawyer involved. But this is well beyond honest error. But in most SaaS environments (GB egress, bytes storage, etc), you are dependent on the vendor for the data. There is no practical way to disagree with AWS if they tell you your usage was 20TB and you expected 15. You are trusting the vendor not to lie (maybe reasonable), and also to never make an error (totally unreasonable!). We now have no defense against honest error. B2C is in a similar boat. In olden-times, I knew people who kept a log book and stop watch by the phone and logged all their calls. Every month, they reconciled their phone bill against their log. Just like they reconciled their bank statement against their cheque stubs and deposit slips. But who in this day keeps their own log of PayPal transactions safely outside Paypal? My main safety with PayPay is short latency - they report significant transactions right away so any errors can be corrected quickly.