6 ms·

Author here. We had a need for a tool to graph IO activity at arbitrary sampling rates, so there was finally a solid excuse to do some datavis work :) The thi

Author here. We had a need for a tool to graph IO activity at arbitrary sampling rates, so there was finally a solid excuse to do some datavis work :) The thing I wanted to show is not the program itself, but the idea of factoring values and displaying factorization to cover several magnitude levels in one go. I suspect that this has been done before, but I couldn't find anything similar when I was researching this beforehand.

LukeShu 9y ago
I've never seen something similar before either, and it is a very useful visualization. I'm thinking about how to do something similar in my monitoring. How would you extend it to allow zooming on the X-axis? With a continuous log scale, or a simple linear scale, it's easy to know how to compress the data horizontally: take the min/max/avg of the datapoints that go in to one horizontal pixel. I'm not sure how to apply that to your discrete log scale; it seems to work on the assumption that points along the X-axis are discrete. Edit: I said Y-axis, I meant X-axis. I'm embarrassed.
- apankrat 9y ago
  > How would you extend it to allow zooming on the Y-axis? Well, the whole point is to NOT need to do any Y-scaling. What would be the case for zooming in?
  - LukeShu 9y ago
    I'm sorry, I meant X-axis. I'm embarrassed.
    - apankrat 9y ago
      Ha. Good question, actually. The simplest option would obviously be to just show a conventional candlestick when zoomed out. There might be an option to do something interesting with rendering the stick when min/max have the same higher-order factors... but it might also end up looking like a royal mess.
arunc 9y ago
Nice work. Do you plan to open source it?
- apankrat 9y ago
  I see no point in that. It's trivial to do in several hours from scratch.
  - arunc 9y ago
    Even if you think your work is trivial, it may not be so for others. :) Also, gives an opportunity for people to extend the current feature set. Entirely your choice though.
    - IMTDb 9y ago
      And a great way for anyone wanting to start some windows programming to see an example of a nice simple program with a GUI that interacts with WIN32 apis
  - boromi 9y ago
    How's the GUI setup?
    - apankrat 9y ago
      Basic window with stuff drawn on it with GDI and GDI+ in a double-buffered way. Edit: Scratch the GDI+ bit, it's just GDI.
  - defgeneric 9y ago
    I came to the comments to look for a link to the source. I'm a systems programmer but don't have much knowledge of how you guys do things in Win32. A ~70kb executable with nice graphs was enticing and I was really interested to see the source. But as it is, I can't afford to download closed freeware that can't be audited just for some nice graphs.
gg3 9y ago
How are you reading disk I/O rates? Is there a logs from Windows on this?
- apankrat 9y ago
  IOCTL_DISK_PERFORMANCE https://msdn.microsoft.com/en-us/library/windows/desktop/aa365183(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... This relies on IO performance counters enabled for the target drive, but that's typically the case. If they are off, they can be enabled with "diskperf.exe".
dracodoc 9y ago
Maybe you can use slightly different color/background color for the bands? That way it's more obvious that the 3 bands are separated instead of one continuous band.
- apankrat 9y ago
  It started this way, actually. But in the day-to-day use it ends up being an embellishment, because there would typically be a string of read/writes that fall on the first pixel of every band so they'll highlight the boundaries, for free.
  - dracodoc 9y ago
    Different color could be a little bit distracting for minimal design, maybe there could be gaps between bands. The bands are independent with each other, a 999M 999K 999B band look like a continuous band but it should be 3 separate bars. This is just pure nick picking, your current design probably suit your need already.
ComodoHacker 9y ago
>a tool to graph IO activity at arbitrary sampling rates Why sampling rate is fixed then?
- apankrat 9y ago
  Because public build is not the same as the one we use internally.
deleted 9y ago
[deleted]
arnioxux 9y ago
This visualization is really cool and innovative! It's useful for seeing the changes in the stream of values even if they are small relative to absolute magnitude (e.g., 1000001 -> 1000005 -> 1000003 etc). The tradition way to solve this is to rescale the y axis (for example google stock price chart ranges from 1000-1200). But this doesn't work for graphing IO because it spikes up/down too often in a huge range. Your visualization plots the base-1024 digits so whenever the IO is sustained at a certain range, the top few digits will be held constant and the next digit will show the variation within that scale. Unfortunately I don't think this solves the spiking problem completely. It's mentally hard to parse since two adjacent values are only comparable whenever their top digits match. For example it's only meaningful to compare (1gb+2mb+3kb) with (1gb+2mb+5kb) on the kb graph. If it spikes down to 0gb+0mb+0kb, you need to switch to reading the gb graph (since comparing the kb value to one before is pretty meaningless unless for some crazy reason you actually care about the mod 1024 value of two very different numbers). To fix this, you should color code the "runs" of values that are meaningful to compare! In other words, kb values have the same color only if their gb/mb values match. And mb values have the same color if their gb value matches.
- jdmichal 9y ago
  I like the idea of using a color signal to show a changed value, with the change cascading to lower "digits". So, for instance, if the GB change, then so do MB, KB, and B. If GB and MB remains the same, then only KB and B will change. Hard part would be how to do this over a long span of columns without the whole thing looking like a unicorn puked on the chart...
deleted 9y ago
[deleted]
tlarkworthy 9y ago
Its a standard option in Grafana https://community.grafana.com/t/advanced-graphing-part3-right-y-axis-log-scales/400 https://community.grafana.com/t/advanced-graphing-part3-righ...
kiki_jiki 9y ago
A Rainmeter version of this would be very cool.

LukeShu 9y ago

I've never seen something similar before either, and it is a very useful visualization. I'm thinking about how to do something similar in my monitoring. How would you extend it to allow zooming on the X-axis? With a continuous log scale, or a simple linear scale, it's easy to know how to compress the data horizontally: take the min/max/avg of the datapoints that go in to one horizontal pixel. I'm not sure how to apply that to your discrete log scale; it seems to work on the assumption that points along the X-axis are discrete. Edit: I said Y-axis, I meant X-axis. I'm embarrassed.

apankrat 9y ago
> How would you extend it to allow zooming on the Y-axis? Well, the whole point is to NOT need to do any Y-scaling. What would be the case for zooming in?
- LukeShu 9y ago
  I'm sorry, I meant X-axis. I'm embarrassed.
  - apankrat 9y ago
    Ha. Good question, actually. The simplest option would obviously be to just show a conventional candlestick when zoomed out. There might be an option to do something interesting with rendering the stick when min/max have the same higher-order factors... but it might also end up looking like a royal mess.

arunc 9y ago

Nice work. Do you plan to open source it?

apankrat 9y ago
I see no point in that. It's trivial to do in several hours from scratch.
- arunc 9y ago
  Even if you think your work is trivial, it may not be so for others. :) Also, gives an opportunity for people to extend the current feature set. Entirely your choice though.
  - IMTDb 9y ago
    And a great way for anyone wanting to start some windows programming to see an example of a nice simple program with a GUI that interacts with WIN32 apis
- boromi 9y ago
  How's the GUI setup?
  - apankrat 9y ago
    Basic window with stuff drawn on it with GDI and GDI+ in a double-buffered way. Edit: Scratch the GDI+ bit, it's just GDI.
- defgeneric 9y ago
  I came to the comments to look for a link to the source. I'm a systems programmer but don't have much knowledge of how you guys do things in Win32. A ~70kb executable with nice graphs was enticing and I was really interested to see the source. But as it is, I can't afford to download closed freeware that can't be audited just for some nice graphs.

gg3 9y ago

How are you reading disk I/O rates? Is there a logs from Windows on this?

apankrat 9y ago
IOCTL_DISK_PERFORMANCE https://msdn.microsoft.com/en-us/library/windows/desktop/aa365183(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... This relies on IO performance counters enabled for the target drive, but that's typically the case. If they are off, they can be enabled with "diskperf.exe".

dracodoc 9y ago

Maybe you can use slightly different color/background color for the bands? That way it's more obvious that the 3 bands are separated instead of one continuous band.

apankrat 9y ago
It started this way, actually. But in the day-to-day use it ends up being an embellishment, because there would typically be a string of read/writes that fall on the first pixel of every band so they'll highlight the boundaries, for free.
- dracodoc 9y ago
  Different color could be a little bit distracting for minimal design, maybe there could be gaps between bands. The bands are independent with each other, a 999M 999K 999B band look like a continuous band but it should be 3 separate bars. This is just pure nick picking, your current design probably suit your need already.

ComodoHacker 9y ago

>a tool to graph IO activity at arbitrary sampling rates Why sampling rate is fixed then?

apankrat 9y ago
Because public build is not the same as the one we use internally.

deleted 9y ago

[deleted]

arnioxux 9y ago

This visualization is really cool and innovative! It's useful for seeing the changes in the stream of values even if they are small relative to absolute magnitude (e.g., 1000001 -> 1000005 -> 1000003 etc). The tradition way to solve this is to rescale the y axis (for example google stock price chart ranges from 1000-1200). But this doesn't work for graphing IO because it spikes up/down too often in a huge range. Your visualization plots the base-1024 digits so whenever the IO is sustained at a certain range, the top few digits will be held constant and the next digit will show the variation within that scale. Unfortunately I don't think this solves the spiking problem completely. It's mentally hard to parse since two adjacent values are only comparable whenever their top digits match. For example it's only meaningful to compare (1gb+2mb+3kb) with (1gb+2mb+5kb) on the kb graph. If it spikes down to 0gb+0mb+0kb, you need to switch to reading the gb graph (since comparing the kb value to one before is pretty meaningless unless for some crazy reason you actually care about the mod 1024 value of two very different numbers). To fix this, you should color code the "runs" of values that are meaningful to compare! In other words, kb values have the same color only if their gb/mb values match. And mb values have the same color if their gb value matches.

jdmichal 9y ago
I like the idea of using a color signal to show a changed value, with the change cascading to lower "digits". So, for instance, if the GB change, then so do MB, KB, and B. If GB and MB remains the same, then only KB and B will change. Hard part would be how to do this over a long span of columns without the whole thing looking like a unicorn puked on the chart...

deleted 9y ago

[deleted]

tlarkworthy 9y ago

Its a standard option in Grafana https://community.grafana.com/t/advanced-graphing-part3-right-y-axis-log-scales/400 https://community.grafana.com/t/advanced-graphing-part3-righ...

kiki_jiki 9y ago

A Rainmeter version of this would be very cool.