3 ms·

Why do you go with "must"? For example I interact with a medical app. When you open a patient card, there's lots of data getting requested. It halts the new co

Why do you go with "must"? For example I interact with a medical app. When you open a patient card, there's lots of data getting requested. It halts the new context opening for a long time, even though you need only a fragment of this data. I'd love it if it didn't wait and instead opened the basic information with placeholder blocks that get loaded as the data comes in. You really don't need all the old visits loaded in a collapsed tree that you're not going to use. This would save significant amount of time ever day.

ndriscoll 2y ago
How much is lots? I would think for a single patient, you'd generally be requesting maybe a couple dozen kB unless there are images, but those should be loaded separately. Looking at the Crowdstrike thread from the other day on this site for example, it looks like 500 comments amounts to ~120 kB and loads in about as many ms (and that's with gzip. br or zstd should do better).
- viraptor 2y ago
  It's not the size. The main view contains bits from so many tables and in situations where the context decides how to calculate other results. Compression is cool, but it can't solve running a chain of queries.
  - wruza 2y ago
    If you can’t quickly collect data for a principal view, the data structure is incorrect. This doesn’t solve your problem cause it’s usually unlikely that one can refactor a data problem on sight in a shipped app, but doesn’t make this case normal too. Most apps don’t need such partial loading optimization and shouldn’t be designed as such. Normalizing it creates a web full of never-ending spinners, which already is a plague.
    - viraptor 2y ago
      It's medical software which existed for over a decade. The main view for the patient includes billing details, family connections, reminders, previous booking, documents, outstanding tests, pregnancy status, preventive care preferences, etc. There's lots to optimise in this app/db for sure, but you're underestimating the real complexity. Whatever you do, you're still querying 20+ tables (in non trivial ways) and calculations.
      - ndriscoll 2y ago
        Of course one does not simply rewrite an existing system, but for what it's worth, you could still make that fast to query by doing xml or json queries with subselects. Some testing I've done with that technique with a mock social-media use-case hit ~15k requests/second end-to-end on pretty weak hardware (4 cores) needing to query 8 tables to construct a profile with 1M profiles in the database. I don't have the exact results saved, but IIRC it was a ~10 ms response time under load.
      - wruza 2y ago
        Whatever you do I've managed a database of local rescuers veterans, including their full PII, illnesses, statuses, family, etc. Even the floor they lived on and how much they drink was in there. It was one form per person with all the data, opened in an instant in a DBF-based system. The "trick" was to store all data in a single table with a few trivially indexed subtables and packed-table-blobs for tabular data. Shocking, I know. It's not "whatever you do", it's a matter of being RDBMS practicioner vs SQL pedant, which is where all the real complexity comes from. If data cannot be collected in a reasonable time for a primary form, it's bad design by definition. It's more reasonable to spend a few minutes to collect a report required once a year than spend a minute on a form required often. Again, not criticizing you for this mess, but it's not something a generic ui framework should solve for.
tuyiown 2y ago
The problem you describe is more about alignment between implementation and use cases. e.g. if your main use case is to have the data displayed on screen to be loaded, then that should be loaded. If it's desirable to preload other data in the background for later screen updates then just do that. The idea is more about first intent, load what is displayed on screen, if it's better to have parts loaded incrementally, then just do that.
- viraptor 2y ago
  That's why I was asking about you stressing "must". In that case it's a mix of fetch before the screen, fetch to replace placeholders, fetch details on demand. That sounds like it breaks your "most fetch before rendering" rule.

ndriscoll 2y ago

How much is lots? I would think for a single patient, you'd generally be requesting maybe a couple dozen kB unless there are images, but those should be loaded separately. Looking at the Crowdstrike thread from the other day on this site for example, it looks like 500 comments amounts to ~120 kB and loads in about as many ms (and that's with gzip. br or zstd should do better).

viraptor 2y ago
It's not the size. The main view contains bits from so many tables and in situations where the context decides how to calculate other results. Compression is cool, but it can't solve running a chain of queries.
- wruza 2y ago
  If you can’t quickly collect data for a principal view, the data structure is incorrect. This doesn’t solve your problem cause it’s usually unlikely that one can refactor a data problem on sight in a shipped app, but doesn’t make this case normal too. Most apps don’t need such partial loading optimization and shouldn’t be designed as such. Normalizing it creates a web full of never-ending spinners, which already is a plague.
  - viraptor 2y ago
    It's medical software which existed for over a decade. The main view for the patient includes billing details, family connections, reminders, previous booking, documents, outstanding tests, pregnancy status, preventive care preferences, etc. There's lots to optimise in this app/db for sure, but you're underestimating the real complexity. Whatever you do, you're still querying 20+ tables (in non trivial ways) and calculations.
    - ndriscoll 2y ago
      Of course one does not simply rewrite an existing system, but for what it's worth, you could still make that fast to query by doing xml or json queries with subselects. Some testing I've done with that technique with a mock social-media use-case hit ~15k requests/second end-to-end on pretty weak hardware (4 cores) needing to query 8 tables to construct a profile with 1M profiles in the database. I don't have the exact results saved, but IIRC it was a ~10 ms response time under load.
    - wruza 2y ago
      Whatever you do I've managed a database of local rescuers veterans, including their full PII, illnesses, statuses, family, etc. Even the floor they lived on and how much they drink was in there. It was one form per person with all the data, opened in an instant in a DBF-based system. The "trick" was to store all data in a single table with a few trivially indexed subtables and packed-table-blobs for tabular data. Shocking, I know. It's not "whatever you do", it's a matter of being RDBMS practicioner vs SQL pedant, which is where all the real complexity comes from. If data cannot be collected in a reasonable time for a primary form, it's bad design by definition. It's more reasonable to spend a few minutes to collect a report required once a year than spend a minute on a form required often. Again, not criticizing you for this mess, but it's not something a generic ui framework should solve for.

tuyiown 2y ago

The problem you describe is more about alignment between implementation and use cases. e.g. if your main use case is to have the data displayed on screen to be loaded, then that should be loaded. If it's desirable to preload other data in the background for later screen updates then just do that. The idea is more about first intent, load what is displayed on screen, if it's better to have parts loaded incrementally, then just do that.

viraptor 2y ago
That's why I was asking about you stressing "must". In that case it's a mix of fetch before the screen, fetch to replace placeholders, fetch details on demand. That sounds like it breaks your "most fetch before rendering" rule.