3 ms·

That's neat! In retrospect, did the database make the problem more tractable from an algorithmic or practical standpoint, or was it mostly just using the tools

That's neat! In retrospect, did the database make the problem more tractable from an algorithmic or practical standpoint, or was it mostly just using the tools you're familiar with? If I were to approach the same problem, I likely would have tried to keep all the data in RAM and serialize it out to a file for incremental runs.

qsort 4y ago
IME a database does not algorithmically improve anything, but it does make at your disposal some very efficient data structures that are definitely nontrivial to replicate. This is especially the case if you have to work with datasets that don't fit in memory! An added bonus is that you can use SQL.
- zerkten 4y ago
  A friend works for a financial company that gets applicants from .NET and Linux backgrounds. He observed that the .NET devs almost always thought about solving these problems with a database and SQL that the Linux people would solve with pipelines of commands. Both were valid in his environment since they were both a big Oracle and SQL Server shop while having high Linux usage as well. I must ask him if PowerShell has changed things at all.
theamk 4y ago
Definitely the latter. There are many of other products which implement "keep track of partial file contents to deduplicate", for example casync borg and restic. None of them use sqlite for metadata storage, which kinda makes sense - in author's schema, each block uses (fileid, block_index, hash) tuple, which is 2/3 wasted space, as fileids are mostly same (if files are big) and block_index is always increasing. A custom data structure would be both faster and more efficient.
nonethewiser 4y ago
I have used SQLite to deduplicate csvs when it couldn’t all fit into memory. I imagine Dask would be a better tool for this job but I wasn’t familiar with it at the time and it did allow me to deduplicate using Python and no external dependencies.

qsort 4y ago

IME a database does not algorithmically improve anything, but it does make at your disposal some very efficient data structures that are definitely nontrivial to replicate. This is especially the case if you have to work with datasets that don't fit in memory! An added bonus is that you can use SQL.

zerkten 4y ago
A friend works for a financial company that gets applicants from .NET and Linux backgrounds. He observed that the .NET devs almost always thought about solving these problems with a database and SQL that the Linux people would solve with pipelines of commands. Both were valid in his environment since they were both a big Oracle and SQL Server shop while having high Linux usage as well. I must ask him if PowerShell has changed things at all.

theamk 4y ago

Definitely the latter. There are many of other products which implement "keep track of partial file contents to deduplicate", for example casync borg and restic. None of them use sqlite for metadata storage, which kinda makes sense - in author's schema, each block uses (fileid, block_index, hash) tuple, which is 2/3 wasted space, as fileids are mostly same (if files are big) and block_index is always increasing. A custom data structure would be both faster and more efficient.

nonethewiser 4y ago

I have used SQLite to deduplicate csvs when it couldn’t all fit into memory. I imagine Dask would be a better tool for this job but I wasn’t familiar with it at the time and it did allow me to deduplicate using Python and no external dependencies.