4 ms·

> I have a heuristic that if you're doing temp tables and derived values and whatnot, you're edging into reporting, so denormalizing and aggregating makes a lot

by gregmac 10y ago

> I have a heuristic that if you're doing temp tables and derived values and whatnot, you're edging into reporting, so denormalizing and aggregating makes a lot of sense for that. Totally agree. > 10sec load times are similiarly the domain of the DMV and trivial hobby apps on a free Heroku. Unfortunately, the app I was talking about: 1) Allows customization of several views, including adding several columns that are expensive (and users can add all of them, if they want). This was not a big deal with hundreds of items, but is with hundreds of thousands. 2) Has a hierarchy above these objects, but doesn't enforce any context. So literally you can load this view for the entire system (which can be hundreds of thousands of objects, with some columns that summarize data from tens of millions of rows) 3) Some of the expensive calculated values are time-dependent, eg "past hour", "past day" etc, which also makes pre-calculating a very expensive operation (not impacting user time directly, but adding overall load to the system). 4) Was built on top of MS SQL server, deployed both as a "cloud" model and as a product on-premise at customer sites. This meant several customers were heavily invested in SQL licenses, and beefy hardware to run it all. Even if we overlooked the challenge of updating our own product (half a million lines? give or take) and data migration (dozens of ten to hundred-GB databases, and hundreds of smaller ones), changing to another DBMS and telling our customers that their tens-of-thousands of dollars investments in hardware/licenses were useless was unacceptable. So yes, in this case it was possible to get multiple-minute load times in the worst cases. Why would anyone want to see many of these columns for hundreds of thousands of rows at once? I have no idea, but they could, and it was slow, and they complained. To your point: this absolutely should have been considered 'reporting', but the app didn't make that distinction. This was the main view of data, essentially, and it was possible for users to make it really slow. We ended up redoing part of the UI at one point, and during that time, removed sorting from some of the columns. Users complained about this at the time, but got over it. We also put a hard limit on of 20,000 objects: more than that and it just says "please pick a group" first. Again, users complained about this, but got over it. Going through that was crappy, but the constant complains about being slow stopped. I should note, there is a separate reporting UI that does allow sorting by everything. I wanted to go much further than that, and completely re-think the UI and some of the design concepts (such as not allowing a single list to be viewed for the entire system at once, and instead either summarizing multiple groups, or displaying detail for one) but at the time the business side (including up to the CEO) was unwilling to do a change like that, and we also had no formal product management to really push the issue.

xixi77 10y ago
I am thinking of a pretty expensive product that we are currently using that is very similar to what you are describing. It has an old search/filter interface that allowed filters on arbitrary fields, and indeed filtered results would sometimes take 10-30 seconds to display. Now it is being phased out in favor of new, prettier and faster interface, that only allows filters on some fields, and on top of that has a much lower limit on filtered results displayed. It does load a page of results on screen faster, but leads to never-ending frustration when trying to do anything useful: now, instead of filtering on what I actually want, I have to iteratively do a zillion filters on irrelevant items, trying to fit within the row limit, combine the output, and then filter that again manually, sometimes taking hours to do what is possible to do within minutes in the old version. On top of that, for a longest time an absolutely essential filter was hardcoded to be on in the "new" version, making the whole thing useless entirely :( The whole discussion reminds me of discussions of "feature creep" in products like Office. Indeed, most people use only ~20% of the features -- the problem is that these 20% are not the same for everyone.
- gregmac 10y ago
  > Indeed, most people use only ~20% of the features -- the problem is that these 20% are not the same for everyone. Yep. The trick is walking the line: if you don't develop the features in the first place, you may not get some of those customers because you don't meet their needs. If you try to be everything to everyone, you end up with everyone being upset later when it's slow/buggy and you're inevitably forced to change it. There's also going to be some portion, call it 20% or 5% or whatever, of your features that almost nobody uses, and that's the stuff that really can bog you down. I think it's healthy for a company to really take a hard look at this from time to time: take into account the costs of those features to calculate the profitability of those customers, consider their future potential, the market reaction/perception of having them (or losing them), and really consider if those features (ad customers) are truly worth keeping.
xixi77 10y ago
Also, regarding "So yes, in this case it was possible to get multiple-minute load times in the worst cases. Why would anyone want to see many of these columns for hundreds of thousands of rows at once? I have no idea, but they could, and it was slow, and they complained." -- most likely, because they either need to compute statistics or do an analysis that your app does not provide, or in order to make custom queries based on that output.

by gregmac 10y ago

xixi77 10y ago

I am thinking of a pretty expensive product that we are currently using that is very similar to what you are describing. It has an old search/filter interface that allowed filters on arbitrary fields, and indeed filtered results would sometimes take 10-30 seconds to display. Now it is being phased out in favor of new, prettier and faster interface, that only allows filters on some fields, and on top of that has a much lower limit on filtered results displayed. It does load a page of results on screen faster, but leads to never-ending frustration when trying to do anything useful: now, instead of filtering on what I actually want, I have to iteratively do a zillion filters on irrelevant items, trying to fit within the row limit, combine the output, and then filter that again manually, sometimes taking hours to do what is possible to do within minutes in the old version. On top of that, for a longest time an absolutely essential filter was hardcoded to be on in the "new" version, making the whole thing useless entirely :( The whole discussion reminds me of discussions of "feature creep" in products like Office. Indeed, most people use only ~20% of the features -- the problem is that these 20% are not the same for everyone.

gregmac 10y ago
> Indeed, most people use only ~20% of the features -- the problem is that these 20% are not the same for everyone. Yep. The trick is walking the line: if you don't develop the features in the first place, you may not get some of those customers because you don't meet their needs. If you try to be everything to everyone, you end up with everyone being upset later when it's slow/buggy and you're inevitably forced to change it. There's also going to be some portion, call it 20% or 5% or whatever, of your features that almost nobody uses, and that's the stuff that really can bog you down. I think it's healthy for a company to really take a hard look at this from time to time: take into account the costs of those features to calculate the profitability of those customers, consider their future potential, the market reaction/perception of having them (or losing them), and really consider if those features (ad customers) are truly worth keeping.

xixi77 10y ago

Also, regarding "So yes, in this case it was possible to get multiple-minute load times in the worst cases. Why would anyone want to see many of these columns for hundreds of thousands of rows at once? I have no idea, but they could, and it was slow, and they complained." -- most likely, because they either need to compute statistics or do an analysis that your app does not provide, or in order to make custom queries based on that output.