4 ms·

This is basically the approach I'm doing now, with granularity being 1 hour. The problem is that when you're in the middle of an hour, the bucket for the live s

This is basically the approach I'm doing now, with granularity being 1 hour. The problem is that when you're in the middle of an hour, the bucket for the live sim only accounts for half of the hour, but the stored bucket from the full day's data represents the entire hour. I could assume the bucket accumulates its count linearly through the hour, and that'd be much better than my current approach, but it still feels a bit weak. 5 minute buckets would make this comparison problem easier, but it's too much data to store.

dabreegster 6y ago
https://github.com/a-b-street/abstreet/issues/85#issuecomment-794532989 https://github.com/a-b-street/abstreet/issues/85#issuecommen... Here's a video of the problem with the current approach, which is storing a count per hour. No sliding windows; 7:59am and 8:01am are different buckets. Green is "less than usual", red is "more than usual." Every time the hour resets, lots of green appears, and slowly fades.
- CaveTech 6y ago
  You'll have to store more information, you just need to decide what is more space efficient. A possible approach would be to store both the bucket total, as well as the components to an equation that can roughly model the rate of change within the histogram. If you find an equation that generally models observed rates dependent on time `t`, then you can interpolate an approximate value for any point in time. You can get sliding windows out of this as well, by using the equations for adjacent periods. You'll incur a runtime cost for this of course, but relatively small.
- contravariant 6y ago
  As it is you still have the problem that you're not comparing like with like, you can't compare the total for an hour with the total for part of an hour. So you'd basically need to restrict your visualization to only show the image at the moments when you do have all the data. I'd strongly suggest biting the bullet and just storing the data in 5 minute buckets if at all possible, then you can have both sides in the same format and compare them easily. 5 minute buckets is only 12 times the amount of data you currently need, which might be a reasonable trade-off. If you can't or don't want to do that then you'll need to pull some tricks. In essence you want to compare two 'rates' against each other right? Now technically the 5-minute bucket idea is just a crude approximation, what you actually want is the average rate at any point in time a histogram is just a crude method to determine this rate. This means you might get away with it if you try to calculate this rate using 2 different methods on both sides. For the presampled data you could: - Interpolate the hourly data (keep in mind that the hourly totals run 30 minutes behind) - Fit a function to the data rather than just sampling it (i.e. if its periodic you could just use the low frequency part of its Fourier series) Now I presume the data coming in is coming in online (i.e. you can't just wait for a full sample and compare it afterwards, otherwise just do the same thing as above). This means you'll need to do something like: - Use a sliding window (again keep in mind that this delays the signal) - Use an online low-pass filter. - Fit a function to the partial data (tricky and might be unstable) Hope this helps.

dabreegster 6y ago

https://github.com/a-b-street/abstreet/issues/85#issuecomment-794532989 https://github.com/a-b-street/abstreet/issues/85#issuecommen... Here's a video of the problem with the current approach, which is storing a count per hour. No sliding windows; 7:59am and 8:01am are different buckets. Green is "less than usual", red is "more than usual." Every time the hour resets, lots of green appears, and slowly fades.

CaveTech 6y ago
You'll have to store more information, you just need to decide what is more space efficient. A possible approach would be to store both the bucket total, as well as the components to an equation that can roughly model the rate of change within the histogram. If you find an equation that generally models observed rates dependent on time `t`, then you can interpolate an approximate value for any point in time. You can get sliding windows out of this as well, by using the equations for adjacent periods. You'll incur a runtime cost for this of course, but relatively small.
contravariant 6y ago
As it is you still have the problem that you're not comparing like with like, you can't compare the total for an hour with the total for part of an hour. So you'd basically need to restrict your visualization to only show the image at the moments when you do have all the data. I'd strongly suggest biting the bullet and just storing the data in 5 minute buckets if at all possible, then you can have both sides in the same format and compare them easily. 5 minute buckets is only 12 times the amount of data you currently need, which might be a reasonable trade-off. If you can't or don't want to do that then you'll need to pull some tricks. In essence you want to compare two 'rates' against each other right? Now technically the 5-minute bucket idea is just a crude approximation, what you actually want is the average rate at any point in time a histogram is just a crude method to determine this rate. This means you might get away with it if you try to calculate this rate using 2 different methods on both sides. For the presampled data you could: - Interpolate the hourly data (keep in mind that the hourly totals run 30 minutes behind) - Fit a function to the data rather than just sampling it (i.e. if its periodic you could just use the low frequency part of its Fourier series) Now I presume the data coming in is coming in online (i.e. you can't just wait for a full sample and compare it afterwards, otherwise just do the same thing as above). This means you'll need to do something like: - Use a sliding window (again keep in mind that this delays the signal) - Use an online low-pass filter. - Fit a function to the partial data (tricky and might be unstable) Hope this helps.