4 ms·

That's not actually true. When you move to the enterprise layer, suddenly you get the opposite problem, you have a low amount of "users" but you often need a l

That's not actually true. When you move to the enterprise layer, suddenly you get the opposite problem, you have a low amount of "users" but you often need a load of CPU intensive or DB intensive processing to happen quickly. One company I worked for had their system built by, ummmm, not the greatest engineers and were literally running out of time in the day to run their program. Every client was scheduled over 24 hours, and they'd got to running the program for 22 hours per day and were desperately trying to fix it before they ran out of "time". They couldn't run it in parallel because part of the selling point of the program was that it amalgamated data from all the clients.

bearjaws 6mo ago
Without seeing more this seems like it could be solved by not recomputing the entire history to add on data. Depends what kind of math you are doing however. Some sort of check point system could likely save significant IO. What am I missing that requires you to recompute all data every day?
- mattmanser 6mo ago
  I can't go into too much detail because of NDA. It was receiving huge volumes of data from each financial client, and matching it all up to try and find certain things. And match it with existing historical data. Not ads or online tracking. So the loop was adding this data and recalculating everything, It had to be done sequentially, I can't remember the exact reason, but it was a good one. I was only there a few months as they were so dysfunctional I jumped to another job offer I'd received. We were having endless sprint meetings to "plan" this work when all it needed was someone experienced like me refactoring it for a coupe of days. There was a lot of junior devs with senior developer titles as everyone invariably got promoted every year. The funny thing about the sprint cards was all the tasks I put up a 1 for, the other developers put up a 10, and all the ones I put up 10s for the other developers put up 1s. That's what happens when you let junior devs have a say, no comprehension of what's hard or not. Before I went I did point out the multiple pretty obvious O(n²) loops they had in the main calculation loop the results of which could easily be cached but I don't know if it went whoosh over their heads. I'm pretty certain if they'd just let me got on with it instead of holding up sprint cards in the meetings they'd have been down to doing the whole lot is a 1/2 hour run a day, even in the short time I was there. In my experience the first run of optimizing something like that usually doesn't take long and has huge benefits.

bearjaws 6mo ago

Without seeing more this seems like it could be solved by not recomputing the entire history to add on data. Depends what kind of math you are doing however. Some sort of check point system could likely save significant IO. What am I missing that requires you to recompute all data every day?

mattmanser 6mo ago
I can't go into too much detail because of NDA. It was receiving huge volumes of data from each financial client, and matching it all up to try and find certain things. And match it with existing historical data. Not ads or online tracking. So the loop was adding this data and recalculating everything, It had to be done sequentially, I can't remember the exact reason, but it was a good one. I was only there a few months as they were so dysfunctional I jumped to another job offer I'd received. We were having endless sprint meetings to "plan" this work when all it needed was someone experienced like me refactoring it for a coupe of days. There was a lot of junior devs with senior developer titles as everyone invariably got promoted every year. The funny thing about the sprint cards was all the tasks I put up a 1 for, the other developers put up a 10, and all the ones I put up 10s for the other developers put up 1s. That's what happens when you let junior devs have a say, no comprehension of what's hard or not. Before I went I did point out the multiple pretty obvious O(n²) loops they had in the main calculation loop the results of which could easily be cached but I don't know if it went whoosh over their heads. I'm pretty certain if they'd just let me got on with it instead of holding up sprint cards in the meetings they'd have been down to doing the whole lot is a 1/2 hour run a day, even in the short time I was there. In my experience the first run of optimizing something like that usually doesn't take long and has huge benefits.