4 ms·
> having a mechanism that provides detailed logs—including the POST bodies passed to the API—is invaluable. He includes a caveat later to avoid logging PII, bu
by muglug 4y ago
> having a mechanism that provides detailed logs—including the POST bodies passed to the API—is invaluable.
He includes a caveat later to avoid logging PII, but this should have a flashing red warning sign attached.
- adamisom 4y agoIs a logging service/pipeline that removes* PII the only real high-level solution? (* or 'does its best' to remove PII; I feel like there's an immovable object / unstoppable force thing if the answer is "0% PII _ever_!" because that conflicts with the need to log as many things as you can)
- deleted 4y ago[deleted]
- 0xbadcafebee 4y agoYou shouldn't log as many things as you can. You should only log useful things, when they're needed, along with context. Otherwise you have too much noise and not enough signal, and it's easier to leak PII. Applications should have a logging interface that can identify and conceal sensitive information, and data stored in the app/database/etc should have a type that can be denoted as sensitive. You will eventually also want to reduce the amount of logs you generate, so it's useful to have logging levels the same way operating systems do.
- bob1029 4y agoWe do explicit redaction of logs forwarded to us so that we don't have to see any end-customer information. The redaction process occurs automatically in the customer's secure environment, so theoretically it is clean from our perspective. This is indeed a very tricky process, but we have it close enough to make regulators happy. Most of our log information is stored in SQLite and XML, so we can do a lot of parsing magic to achieve determinism.
- rfrey 4y agoI wonder about this a bit. If I'm already storing all the PII in my database, probably on other people's computers via a managed database or at least a cloud server, why are logs different? Is it because they don't receive the same security scrutiny, e.g. any dev can view them versus having special permissions for the database? Or is there more there than authorization that I should be thinking about? (Obviously I'm not talking about logging cleartext passwords, which don't belong in the database either.)
- oandhjakk 4y agoI believe the main concern here, from dealing with clients that have mandated no PII in logs, is both authorisation as well as control. If it’s in your service logs, then it could be in your Splunk logs, it could be in a storage repository and it could be in your requests that you send to the service provider to troubleshoot some issue. Unless there is a valid use case for logging PII (and I can’t think of any which can’t be engineered around) then I think it’s best to avoid it in principle. I think of it as the same as logging passwords, keys or tokens.
- throwaway74829 4y agoPermissions is the big one. Second is potential for DDoS (either intentionally or unintentionally). Third is possibility of "oopsies" via me intentionally or unintentionally including my passwords/sensitive info/what have you in the POST body. Now you have to add branch to look for and scrub sensitive info in your logger -- otherwise my PII has now been logged (and if I were a massive asshole looking for a quick payday, I could throw up a fuss). It's fine in dev, but shouldn't be in prod. Too much liability.
- michaelmior 4y agoI believe this also plays into "right to be forgotten legislation" such as in the GDPR. Suppose you log PII and then a user requests all their PII be deleted. I believe you would then also be required to scrub those logs instead of just dropping records from a database.