4 ms·

Terabyte of RAM is my favorite because I choose to believe he had a perfect data model and was literally telling you exactly what he needed :D In my recent wor

Terabyte of RAM is my favorite because I choose to believe he had a perfect data model and was literally telling you exactly what he needed :D In my recent work doing financial modeling (options, cryptos, other nonsense), I pretty much just try to cache absolutely everything with some arbitrary 1 week TTL and then continue to cache new data every day. Cache hits cause a 10% bump in the current TTL time delta. Kinda heuristic but whatever, works in most cases. Any other funny stories? I always love these engineering horror stories. Have you looked at Julia at all? Would be interesting to hear if there's been an uptick in usage from the research community from a veteran research code surgeon!

chunkyks 5y ago
The mathematician that designed the model I referred to at the end ended up passing it on to me. I've now built something of an empire of people and projects using it, "just" by making better tools to manage its inputs, outputs, visualisation and automation. I was stupidly excited to recently do more work on that same model - I took one of its internal datastructures and turned it from a simple queue into a priority queue using a couple different queueing metrics. In both cases, I had run cases out to a scale where I was able to actually show going from O(1) to O(log(n)) performance - but got beautiful wins because the fastest work is work you never have to do. Not often I can empirically show traditional CS101 stuff in a real-world chart.
- dirkt 5y ago
  Your work sounds awesome. (And you probably mean O(n) and not O(1), otherwise O(log(n)) isn't really an improvement).
  - chunkyks 5y ago
    No, that's the point; the previous algorithm was O(1) [a simple queue] but that led to doing a huge amount of extra work. By using a smarter queue that "cost" an extra O(log(n)), we could skip work that didn't need doing in the first place.

chunkyks 5y ago

The mathematician that designed the model I referred to at the end ended up passing it on to me. I've now built something of an empire of people and projects using it, "just" by making better tools to manage its inputs, outputs, visualisation and automation. I was stupidly excited to recently do more work on that same model - I took one of its internal datastructures and turned it from a simple queue into a priority queue using a couple different queueing metrics. In both cases, I had run cases out to a scale where I was able to actually show going from O(1) to O(log(n)) performance - but got beautiful wins because the fastest work is work you never have to do. Not often I can empirically show traditional CS101 stuff in a real-world chart.

dirkt 5y ago
Your work sounds awesome. (And you probably mean O(n) and not O(1), otherwise O(log(n)) isn't really an improvement).
- chunkyks 5y ago
  No, that's the point; the previous algorithm was O(1) [a simple queue] but that led to doing a huge amount of extra work. By using a smarter queue that "cost" an extra O(log(n)), we could skip work that didn't need doing in the first place.