3 ms·
That's cool but seems overkill for processing 100K-1 million rows I just feel like that is not really a lot of data for a single computer For example you coul
by superconduct123 2y ago
That's cool but seems overkill for processing 100K-1 million rows
I just feel like that is not really a lot of data for a single computer
For example you could write a python script that loops over an array of a million values and runs a calculation like its nothing.
Yet processing a column of 1M values in excel just falls apart and becomes really slow, even the UI becomes laggy
Something about the implementation just can't handle that
- sargstuff 2y agoIt's the difference between the way things are accessed/handled behind the scenes in a spreadsheet vs. database[3][1b]. Adding a programming / scripting language isn't going to change exel data access patterns. One possible approach: Using excel as front end and storing actual data in database would speed things up. This assumes one uses 'python' on the database to do behind the scenes analysis[1] vs. moving complete excell worksheet(s) into / out of database[2]). Relevant background note: Spent quite a few years supporting researchers who were comfortable with excel; but the basic data requirements far exceeded what excel was capabile of handling (pre-analysis, 1-2milion excel cells). aka local machine access only, no remote / cloud stuff. --------------------------- [1] : a) excel with access database : https://www.youtube.com/watch?v=-DbvcuE8ITA https://www.youtube.com/watch?v=-DbvcuE8ITA b) excel with postgres : https://rowzero.io/blog/connect-postgresql-to-microsoft-excel https://rowzero.io/blog/connect-postgresql-to-microsoft-exce... [2] input/export excel data : https://www.youtube.com/watch?v=9PxaTPZIYmc https://www.youtube.com/watch?v=9PxaTPZIYmc [3] why database instead of excel : https://baserow.io/blog/database-instead-of-excel https://baserow.io/blog/database-instead-of-excel