4 ms·

The problem is a simple one. Most telemetry tools (Mixpanel, Sentry, etc.) don't give the developers who are adding them into their products the ability to quic

The problem is a simple one. Most telemetry tools (Mixpanel, Sentry, etc.) don't give the developers who are adding them into their products the ability to quickly add respectful consent flows. This really needs to be a feature of the telemetry tools in the first place. Because, ultimately, most telemetry is being implemented by startup engineers who are burning the midnight oil to complete the telemetry JIRA ticket before going back to the long list of other stuff they have to implement. I have experienced this from all three sides - as a software engineer implementing telemetry, as a product manager consuming telemetry, and now as a founder who is building a tool to collect telemetry in the most respectful manner possible.

TeMPOraL 5y ago
> now as a founder who is building a tool to collect telemetry in the most respectful manner possible. Thank you for taking being respectful to users seriously. I'd be very interested in learning how your consent flows look, and what other aspects of your product are driven by the goal to "collect telemetry in the most respectful manner possible". I couldn't see much on it on the landing page, so if you have a moment, could you provide additional information, either here or in private?
- zomglings 5y ago
  Happy to show you some code. You cannot set up Bugout telemetry in your codebase without first defining your consent flow. We have a library of consent mechanisms that you can chain together like lego blocks to build these flows. For example, our Python consent library is here: https://github.com/bugout-dev/humbug/blob/main/python/humbug/consent.py https://github.com/bugout-dev/humbug/blob/main/python/humbug... Consent is calculated at the time that each reports are sent back. This means that your users can grant and revoke their consent on a per-report basis, which is the only respectful way to do things. We are also building programs which will deidentify reports on the client side, before any data is even sent back to our servers. This work is still in the early stages, but here's v0.0.1 of the Python stack trace deidentifier: https://www.kaggle.com/simiotic/python-tracebacks-redactor/edit https://www.kaggle.com/simiotic/python-tracebacks-redactor/e... Besides Python, we also support Javascript, Go, and we added Java support last week. I would really love to hear any feedback you have.
  - TeMPOraL 5y ago
    Hey, thanks for replying! I like the design for your consent pipeline, and the code itself is very readable. I have some further questions: 1. You say: > You cannot set up Bugout telemetry in your codebase without first defining your consent flow How is it enforced? Is it just an API limitation that I could work around by defining my consent block as below? def much_consent_so_informed() -> ConsentMechanism: def mechanism() -> bool: return True return mechanism That is, are you relying entirely on trust and/or contractual obligations, or do you have some means of enforcing that the user of your SDK isn't cheating? > Consent is calculated at the time that each reports are sent back. This means that your users can grant and revoke their consent on a per-report basis, which is the only respectful way to do things. Correct. I like how you think about this. I assume the SDK user will be ultimately responsible for prompting the end-user for consent; I wonder if you have any "best practices" documents for the software authors, so that they don't have to reinvent respectful consent flow UX from scratch? > We are also building programs which will deidentify reports on the client side, before any data is even sent back to our servers. I don't see any code in that Kaggle notebook you linked (I'm not very familiar with Kaggle, I might be clicking wrong). Should I assume your approach is based on training a black-box ML model? Or do you use some heuristics to identify what data to cut?
    - zomglings 5y ago
      Thanks for looking at the code, and for your feedback! Here is a recipe for adding error reporting (reporting of all uncaught exceptions) in a Python project. The highlighted line shows that, when you instantiate a reporter, you have to pass a consent mechanism: https://github.com/bugout-dev/humbug/blob/main/python/recipes/error_reporting.py#L61 https://github.com/bugout-dev/humbug/blob/main/python/recipe... We allow you to create a consent mechanism that always returns true: > consent = HumbugConsent(True) But even with that mechanism, we ultimately respect BUGGER_OFF=true: https://github.com/bugout-dev/humbug/blob/main/python/humbug/consent.py#L37 https://github.com/bugout-dev/humbug/blob/main/python/humbug... Of course, someone can always create their own subclass of HumbugConsent which overrides that check. We don't have a good way to prevent this, nor would we want to restrict anyone's freedom to modify code. re: Kaggle and stack trace deidentification We started by crawling public GitHub issues for Python stack traces and built up a decent sized dataset of these: https://www.kaggle.com/simiotic/python-tracebacks https://www.kaggle.com/simiotic/python-tracebacks Our emphasis is on building simple programs that we can reasonably expect to run on any reasonable client without using an exorbitant amount of CPU or memory. For this reason, we aren't using black box ML models. Rather, we analyzed the data and came up with some simple regex based rules on how to deidentify stack traces for our v1 implementation. We are in the process of doing this for more languages and building this into a proper deidentification library that can be imported into any runtime - Python + Javascript + Go + etc. Apologies for the link not working. It seems I had to publish a version of the notebook. This link should work now: https://www.kaggle.com/simiotic/python-tracebacks-redactor https://www.kaggle.com/simiotic/python-tracebacks-redactor Actually, we started this work on a livestream if you're interested in watching: https://youtu.be/TFKe614Ml1M https://youtu.be/TFKe614Ml1M Again, really appreciate your engagement and feedback. Thank you!

TeMPOraL 5y ago

> now as a founder who is building a tool to collect telemetry in the most respectful manner possible. Thank you for taking being respectful to users seriously. I'd be very interested in learning how your consent flows look, and what other aspects of your product are driven by the goal to "collect telemetry in the most respectful manner possible". I couldn't see much on it on the landing page, so if you have a moment, could you provide additional information, either here or in private?

zomglings 5y ago
Happy to show you some code. You cannot set up Bugout telemetry in your codebase without first defining your consent flow. We have a library of consent mechanisms that you can chain together like lego blocks to build these flows. For example, our Python consent library is here: https://github.com/bugout-dev/humbug/blob/main/python/humbug/consent.py https://github.com/bugout-dev/humbug/blob/main/python/humbug... Consent is calculated at the time that each reports are sent back. This means that your users can grant and revoke their consent on a per-report basis, which is the only respectful way to do things. We are also building programs which will deidentify reports on the client side, before any data is even sent back to our servers. This work is still in the early stages, but here's v0.0.1 of the Python stack trace deidentifier: https://www.kaggle.com/simiotic/python-tracebacks-redactor/edit https://www.kaggle.com/simiotic/python-tracebacks-redactor/e... Besides Python, we also support Javascript, Go, and we added Java support last week. I would really love to hear any feedback you have.
- TeMPOraL 5y ago
  Hey, thanks for replying! I like the design for your consent pipeline, and the code itself is very readable. I have some further questions: 1. You say: > You cannot set up Bugout telemetry in your codebase without first defining your consent flow How is it enforced? Is it just an API limitation that I could work around by defining my consent block as below? def much_consent_so_informed() -> ConsentMechanism: def mechanism() -> bool: return True return mechanism That is, are you relying entirely on trust and/or contractual obligations, or do you have some means of enforcing that the user of your SDK isn't cheating? > Consent is calculated at the time that each reports are sent back. This means that your users can grant and revoke their consent on a per-report basis, which is the only respectful way to do things. Correct. I like how you think about this. I assume the SDK user will be ultimately responsible for prompting the end-user for consent; I wonder if you have any "best practices" documents for the software authors, so that they don't have to reinvent respectful consent flow UX from scratch? > We are also building programs which will deidentify reports on the client side, before any data is even sent back to our servers. I don't see any code in that Kaggle notebook you linked (I'm not very familiar with Kaggle, I might be clicking wrong). Should I assume your approach is based on training a black-box ML model? Or do you use some heuristics to identify what data to cut?
  - zomglings 5y ago
    Thanks for looking at the code, and for your feedback! Here is a recipe for adding error reporting (reporting of all uncaught exceptions) in a Python project. The highlighted line shows that, when you instantiate a reporter, you have to pass a consent mechanism: https://github.com/bugout-dev/humbug/blob/main/python/recipes/error_reporting.py#L61 https://github.com/bugout-dev/humbug/blob/main/python/recipe... We allow you to create a consent mechanism that always returns true: > consent = HumbugConsent(True) But even with that mechanism, we ultimately respect BUGGER_OFF=true: https://github.com/bugout-dev/humbug/blob/main/python/humbug/consent.py#L37 https://github.com/bugout-dev/humbug/blob/main/python/humbug... Of course, someone can always create their own subclass of HumbugConsent which overrides that check. We don't have a good way to prevent this, nor would we want to restrict anyone's freedom to modify code. re: Kaggle and stack trace deidentification We started by crawling public GitHub issues for Python stack traces and built up a decent sized dataset of these: https://www.kaggle.com/simiotic/python-tracebacks https://www.kaggle.com/simiotic/python-tracebacks Our emphasis is on building simple programs that we can reasonably expect to run on any reasonable client without using an exorbitant amount of CPU or memory. For this reason, we aren't using black box ML models. Rather, we analyzed the data and came up with some simple regex based rules on how to deidentify stack traces for our v1 implementation. We are in the process of doing this for more languages and building this into a proper deidentification library that can be imported into any runtime - Python + Javascript + Go + etc. Apologies for the link not working. It seems I had to publish a version of the notebook. This link should work now: https://www.kaggle.com/simiotic/python-tracebacks-redactor https://www.kaggle.com/simiotic/python-tracebacks-redactor Actually, we started this work on a livestream if you're interested in watching: https://youtu.be/TFKe614Ml1M https://youtu.be/TFKe614Ml1M Again, really appreciate your engagement and feedback. Thank you!