3 ms·

Any time you can convert a problem from an `N x K` problem to an `N + K` problem, there's some asshole administrator trying to turn an `N + K` problem into an `

Any time you can convert a problem from an `N x K` problem to an `N + K` problem, there's some asshole administrator trying to turn an `N + K` problem into an `N x K` problem. It wouldn't surprise me if there's huge amounts of redundant information in there.

acchow 4y ago
> In the newly-released data, each "negotiated rate" (or simply "price") is associated with a lot of metadata, but it boils down to: who's paying, who's getting paid, what they're getting paid for, plus some extra fluff to keep track of versioning. The hundreds of billions of prices in the dataset (probably over a trillion) result from all the possible combinations of these things. They basically denormalized all the dimensions. Imagine you have a function which takes 5 arguments and returns one value. You could give me the source code and let me run this function. Or you could give me a mapping of every possible combination of the 5 inputs to the returned value. The former could be quite small, but the latter would be a massive number of rows.
- imoverclocked 4y ago
  > You could give me the source code and let me run this function If I understand correctly, in this case, that function's source is highly distributed in wetware. It's about as closed-source as it gets; nobody has anywhere near the full source. Each hospital is its own fiefdom!
  - acchow 4y ago
    Yeah this is part of the problem. But even if you had like 10M rows of pricing and then gave a 2% discount to entity A, 3% discount to entity B, 4% discount to C, etc. You could publish these discount rules. Or you could just multiply the 10M rows by the number of different entities giving 10*n M rows. And then let the consumer of the data try to figure out the rules from the output...?
    - ctchocula 4y ago
      It would be interesting to see whether it's possible to reconstruct the rules by comparing the negotiated prices to a baseline like Medicare price, and doing classification into discount buckets to recover the categories insurers negotiated.

acchow 4y ago

> In the newly-released data, each "negotiated rate" (or simply "price") is associated with a lot of metadata, but it boils down to: who's paying, who's getting paid, what they're getting paid for, plus some extra fluff to keep track of versioning. The hundreds of billions of prices in the dataset (probably over a trillion) result from all the possible combinations of these things. They basically denormalized all the dimensions. Imagine you have a function which takes 5 arguments and returns one value. You could give me the source code and let me run this function. Or you could give me a mapping of every possible combination of the 5 inputs to the returned value. The former could be quite small, but the latter would be a massive number of rows.

imoverclocked 4y ago
> You could give me the source code and let me run this function If I understand correctly, in this case, that function's source is highly distributed in wetware. It's about as closed-source as it gets; nobody has anywhere near the full source. Each hospital is its own fiefdom!
- acchow 4y ago
  Yeah this is part of the problem. But even if you had like 10M rows of pricing and then gave a 2% discount to entity A, 3% discount to entity B, 4% discount to C, etc. You could publish these discount rules. Or you could just multiply the 10M rows by the number of different entities giving 10*n M rows. And then let the consumer of the data try to figure out the rules from the output...?
  - ctchocula 4y ago
    It would be interesting to see whether it's possible to reconstruct the rules by comparing the negotiated prices to a baseline like Medicare price, and doing classification into discount buckets to recover the categories insurers negotiated.