2 ms·
"a big intermediate result that you have to process aside from the database because your ORM doesn't allow you to do that within the database." Processing in t
by ShabbyDoo 16y ago
"a big intermediate result that you have to process aside from the database because your ORM doesn't allow you to do that within the database."
Processing in the DB isn't always desirable or even possible. Let's say that you are writing the Mint.com killer, and you want to fetch a user's portfolio and query a bunch of 3rd party systems (Yahoo finance, whatever) for stuff related to individual holdings. Let's say you have a table of holdings (security symbol as business key) along with a related set of buy/sell records for each holding.
Let's say that, there a 5% chance that there's something interesting found from the 3rd party systems which requires the buy/sell history for evaluation (capital gains tax-related? I'm just contriving an example here). One strategy would be to do a big join fetch of the holdings and history table. This would be efficient from a DB perspective if you would eventually require a high enough percentage of the history records. But, if you only require a small fraction of them, then it would be better to lazily initialize the history set on a per-portfolio basis. The exact percentage at which you would be indifferent between strategies from a performance perspective would probably be dependent on so many things that you'd have to just experiment with both fetch plans.
Unless an ORM tool is smart enough to observe code execution paths over time, profile them, and dynamically adjust fetch plans accordingly (a cool idea?), the ORM client code has to be able to specify the strategy it thinks is optimal.
Granularity becomes important when doing batch-y stuff. Throughput is significantly higher when you select or insert records by the thousands instead of individually. However, you do take a big memory hit because you're keeping thousands of records around at once. So, if using table X for some batch activity running in only one thread of your application, it's probably best to use coarse grained fetching. However, if you're reading/writing as part of user transactions and are concerned about allocation/deallocation, record-by-record processing might be better. My knowledge of DB internals is too weak to discuss the trade-offs in the db of fetch granularity decisions.