3 ms·
A minor nit but the function to dedup the image ids and the corresponding comments seem to be off from a data structure/algorithmic POV. "Looking at our dedupl
by pstoll 9y ago
A minor nit but the function to dedup the image ids and the corresponding comments seem to be off from a data structure/algorithmic POV.
"Looking at our deduplication function, it’s functional and performant, but not very descriptive. Essentially it checks a running set called seen, which is checked for originality as the deduplication script progresses. As python Dicts are essentially a hash map, the in check compares element hashes instead of each individual component of the dictionary object. This massively speeding up the deduplication process."
Uhm..I don't think so.
It is just extra work to check if the objects are already in the set vs just stuffing them all in and let the set handle uniqueness.
Am I missing something?
https://gist.github.com/pstoll/ae73582763540051d321a4eb15304226 https://gist.github.com/pstoll/ae73582763540051d321a4eb15304...
Again, a minor thing. But seeing something like that makes my 'what else do I need to review' detectors go up.