6 ms·

I once worked on a platform producing analytics using data that, at its source, was manually typed in by people. My product managers would insist we do distinc

I once worked on a platform producing analytics using data that, at its source, was manually typed in by people. My product managers would insist we do distinct counts on the aggregates instead of using probabilistic algorithms, because we "needed" the absolute 100% accurate output. No matter how many times I would explain the data was never 100% accurate to begin with and that the error rate using HyperLogLog wouldn't make an ounce of difference, we never were allowed to do that. As a consequence the performance of the system when doing distinct counts on interactive queries was about 100x worse. This didn't seem to bother anyone. I never understood why.

crazygringo 4y ago
100x is not that big of a difference, and I've been burned by "clever" algorithms before because they weren't coded exactly right. Because if they drift into a broken state, nobody can tell. While a simple sum is unlikely to fail in subtle ways. There are benefits to simplicity. If there isn't a reason to need 100x faster performance, then why complicate things? That would be my why.
- dgb23 4y ago
  That's a great point. But I want to add that anecdotally users do care about performance and they will often thank you for anything that is noticeable. Sometimes just because it feels nice, but it can also enable a fast feedback loop, which increases immersion and productivity. In a broader sense, we use computers not only because they can perform work autonomously but also because they are fast, correct and remember details almost perfectly.
dgb23 4y ago
I can imagine some reasons: - They don't care about performance and think their users don't care about performance. - They have sold something that doesn't make sense but sounds good. - They are afraid of their users. - Their users don't understand the problem you're describing or simply don't believe it. - The business domain of their users actually knows the about the issue, but they still want or need the exact counts.
bcherny 4y ago
Why would your PM dictate the technical implementation? This seems like something better left to engineers.
- riku_iki 4y ago
  distinct vs approximate user facing result seems like product question.
  - bcherny 4y ago
    The commenter seemed to indicate that even the exact result wasn’t very accurate. Perhaps a failure on their part to communicate this nuance well, or understand the business context.
    - gorbachev 4y ago
      No. Everyone involved had roughly a decade of experience with the data in question, the processes that touched it before it reached us and within the company, the various problems with the data and how they impacted the analytics. They also understood how probabilistic algorithms would impact the results. Additionally we had a team whose main responsibility was to continuously monitor and improve the quality of the data we were receiving. My team had frequent conversations with that team, because anything they couldn't detect would show up in our results. They just had a fixation on "accuracy" stemming mostly from "our clients wouldn't understand it". Which is interesting because we had frequent complaints from clients who had secondary sources and found differences between what we provided and what their other sources provided.
dataflow 4y ago
I could see myself being like your PM in some situations, depending on the nature of the data and how it was being used. To give an analogy that might perhaps help with understanding the sentiment, imagine how your fellow citizens would feel if the government used some {hot approximation algorithm} for counting votes in the next election, rather than a straightforward count.
- kqr 4y ago
  Vote counts are already approximate. That's why close results are re-counted using slower methods. (Additionally, I would argue more generally that vote counts should be approximate. If a vote is close to tied then we run the largest risk of making an incorrect decision. Better to continue debate and/or refine the proposal until it has wider acceptance. This does not work for selecting government with term limits obviously.)
  - dataflow 4y ago
    That's an incredibly misleading way to phrase it. The counting algorithm is exact. Errors creep in during execution, resulting in an approximate result. An approximation algorithm is a whole different beast, and that's the point of the discussion here.
    - kqr 4y ago
      Does really how the error comes into life matter more than the magnitude of the error?
      - dataflow 4y ago
        It's pretty easy to find out - go pitch some approximate vote counting algorithm (say, 50% random subsampling to get 2x speed? or whatever) to random people on the street, and ask them if they would care if the government decided the president that way.
- naijaboiler 4y ago

crazygringo 4y ago

100x is not that big of a difference, and I've been burned by "clever" algorithms before because they weren't coded exactly right. Because if they drift into a broken state, nobody can tell. While a simple sum is unlikely to fail in subtle ways. There are benefits to simplicity. If there isn't a reason to need 100x faster performance, then why complicate things? That would be my why.

dgb23 4y ago
That's a great point. But I want to add that anecdotally users do care about performance and they will often thank you for anything that is noticeable. Sometimes just because it feels nice, but it can also enable a fast feedback loop, which increases immersion and productivity. In a broader sense, we use computers not only because they can perform work autonomously but also because they are fast, correct and remember details almost perfectly.

dgb23 4y ago

I can imagine some reasons: - They don't care about performance and think their users don't care about performance. - They have sold something that doesn't make sense but sounds good. - They are afraid of their users. - Their users don't understand the problem you're describing or simply don't believe it. - The business domain of their users actually knows the about the issue, but they still want or need the exact counts.

bcherny 4y ago

Why would your PM dictate the technical implementation? This seems like something better left to engineers.

riku_iki 4y ago
distinct vs approximate user facing result seems like product question.
- bcherny 4y ago
  The commenter seemed to indicate that even the exact result wasn’t very accurate. Perhaps a failure on their part to communicate this nuance well, or understand the business context.
  - gorbachev 4y ago
    No. Everyone involved had roughly a decade of experience with the data in question, the processes that touched it before it reached us and within the company, the various problems with the data and how they impacted the analytics. They also understood how probabilistic algorithms would impact the results. Additionally we had a team whose main responsibility was to continuously monitor and improve the quality of the data we were receiving. My team had frequent conversations with that team, because anything they couldn't detect would show up in our results. They just had a fixation on "accuracy" stemming mostly from "our clients wouldn't understand it". Which is interesting because we had frequent complaints from clients who had secondary sources and found differences between what we provided and what their other sources provided.

dataflow 4y ago

I could see myself being like your PM in some situations, depending on the nature of the data and how it was being used. To give an analogy that might perhaps help with understanding the sentiment, imagine how your fellow citizens would feel if the government used some {hot approximation algorithm} for counting votes in the next election, rather than a straightforward count.

kqr 4y ago
Vote counts are already approximate. That's why close results are re-counted using slower methods. (Additionally, I would argue more generally that vote counts should be approximate. If a vote is close to tied then we run the largest risk of making an incorrect decision. Better to continue debate and/or refine the proposal until it has wider acceptance. This does not work for selecting government with term limits obviously.)
- dataflow 4y ago
  That's an incredibly misleading way to phrase it. The counting algorithm is exact. Errors creep in during execution, resulting in an approximate result. An approximation algorithm is a whole different beast, and that's the point of the discussion here.
  - kqr 4y ago
    Does really how the error comes into life matter more than the magnitude of the error?
    - dataflow 4y ago
      It's pretty easy to find out - go pitch some approximate vote counting algorithm (say, 50% random subsampling to get 2x speed? or whatever) to random people on the street, and ask them if they would care if the government decided the president that way.
naijaboiler 4y ago