4 ms·
Heck, many apps are even stupider and query the entire dataset from the DB, do paging in code I would only expect to see something like this at a company with
by rhizome 10y ago
Heck, many apps are even stupider and query the entire dataset from the DB, do paging in code
I would only expect to see something like this at a company with glacial bureaucracy because that's a technique that was already garbage in 2007 when I had to babysit an app whose devs decided to retrieve a user's entire object graph whenever doing a User#show.
10sec load times are similiarly the domain of the DMV and trivial hobby apps on a free Heroku.
At the risk of exposing myself as an FB user and thus older than 25, I think a better example for discussion is FB's wall sort: "Top Stories" vs. "Most Recent." If you're like me, you like "Most Recent," but Zuck has his own ideas about how I should live my life and will switch me back to "Top Stories" anywhere from 6 to 24 hours after my last visit. You cannot hard-set your wall to "Most Recent." Educated me would think that sorting by date descending is the O(nlogn)-iest of sorts, yet here we are, with Big Data Massage turning out to be a forced default.
Point being, I hope: it's not just a choice between 10s loads and the elimination of features. The difference is rarely that stark, and with simple attribute sorts (even multiple) should be nigh-on unnoticeable.
Except sometimes you can't simply add an index, because you're displaying a calculated value and only calculate it for the page you're displaying. In order to sort by that value, you have to calculate for the entire system, so now you're into O(n^2) or worse complexity. You can also denormalize or pre-cache calculated values in some way (even going as far as adding something like ElasticSearch on top), but that introduces another dimension of problems (expensive updates, more potential for bugs and/or stale data).
I have a heuristic that if you're doing temp tables and derived values and whatnot, you're edging into reporting, so denormalizing and aggregating makes a lot of sense for that.
Apologies if I've ranted entirely past your point!
- gregmac 10y ago> I have a heuristic that if you're doing temp tables and derived values and whatnot, you're edging into reporting, so denormalizing and aggregating makes a lot of sense for that. Totally agree. > 10sec load times are similiarly the domain of the DMV and trivial hobby apps on a free Heroku. Unfortunately, the app I was talking about: 1) Allows customization of several views, including adding several columns that are expensive (and users can add all of them, if they want). This was not a big deal with hundreds of items, but is with hundreds of thousands. 2) Has a hierarchy above these objects, but doesn't enforce any context. So literally you can load this view for the entire system (which can be hundreds of thousands of objects, with some columns that summarize data from tens of millions of rows) 3) Some of the expensive calculated values are time-dependent, eg "past hour", "past day" etc, which also makes pre-calculating a very expensive operation (not impacting user time directly, but adding overall load to the system). 4) Was built on top of MS SQL server, deployed both as a "cloud" model and as a product on-premise at customer sites. This meant several customers were heavily invested in SQL licenses, and beefy hardware to run it all. Even if we overlooked the challenge of updating our own product (half a million lines? give or take) and data migration (dozens of ten to hundred-GB databases, and hundreds of smaller ones), changing to another DBMS and telling our customers that their tens-of-thousands of dollars investments in hardware/licenses were useless was unacceptable. So yes, in this case it was possible to get multiple-minute load times in the worst cases. Why would anyone want to see many of these columns for hundreds of thousands of rows at once? I have no idea, but they could, and it was slow, and they complained. To your point: this absolutely should have been considered 'reporting', but the app didn't make that distinction. This was the main view of data, essentially, and it was possible for users to make it really slow. We ended up redoing part of the UI at one point, and during that time, removed sorting from some of the columns. Users complained about this at the time, but got over it. We also put a hard limit on of 20,000 objects: more than that and it just says "please pick a group" first. Again, users complained about this, but got over it. Going through that was crappy, but the constant complains about being slow stopped. I should note, there is a separate reporting UI that does allow sorting by everything. I wanted to go much further than that, and completely re-think the UI and some of the design concepts (such as not allowing a single list to be viewed for the entire system at once, and instead either summarizing multiple groups, or displaying detail for one) but at the time the business side (including up to the CEO) was unwilling to do a change like that, and we also had no formal product management to really push the issue.
- xixi77 10y agoI am thinking of a pretty expensive product that we are currently using that is very similar to what you are describing. It has an old search/filter interface that allowed filters on arbitrary fields, and indeed filtered results would sometimes take 10-30 seconds to display. Now it is being phased out in favor of new, prettier and faster interface, that only allows filters on some fields, and on top of that has a much lower limit on filtered results displayed. It does load a page of results on screen faster, but leads to never-ending frustration when trying to do anything useful: now, instead of filtering on what I actually want, I have to iteratively do a zillion filters on irrelevant items, trying to fit within the row limit, combine the output, and then filter that again manually, sometimes taking hours to do what is possible to do within minutes in the old version. On top of that, for a longest time an absolutely essential filter was hardcoded to be on in the "new" version, making the whole thing useless entirely :( The whole discussion reminds me of discussions of "feature creep" in products like Office. Indeed, most people use only ~20% of the features -- the problem is that these 20% are not the same for everyone.
- gregmac 10y ago> Indeed, most people use only ~20% of the features -- the problem is that these 20% are not the same for everyone. Yep. The trick is walking the line: if you don't develop the features in the first place, you may not get some of those customers because you don't meet their needs. If you try to be everything to everyone, you end up with everyone being upset later when it's slow/buggy and you're inevitably forced to change it. There's also going to be some portion, call it 20% or 5% or whatever, of your features that almost nobody uses, and that's the stuff that really can bog you down. I think it's healthy for a company to really take a hard look at this from time to time: take into account the costs of those features to calculate the profitability of those customers, consider their future potential, the market reaction/perception of having them (or losing them), and really consider if those features (ad customers) are truly worth keeping.
- xixi77 10y agoAlso, regarding "So yes, in this case it was possible to get multiple-minute load times in the worst cases. Why would anyone want to see many of these columns for hundreds of thousands of rows at once? I have no idea, but they could, and it was slow, and they complained." -- most likely, because they either need to compute statistics or do an analysis that your app does not provide, or in order to make custom queries based on that output.
- jessaustin 10y agoAt the risk of exposing myself as an FB user and thus older than 25... As an FB nonuser much older than that, I'm amused... The internet wins eventually.