An interesting question is how can you parallelize it? Suppose the data is huge, but you have many available threads, or machines.. Suppose there are N nodes,

An interesting question is how can you parallelize it? Suppose the data is huge, but you have many available threads, or machines.. Suppose there are N nodes, each of which owns part of the array. So one way is for each node to broadcast each entry to all of the other nodes. As each node receives a broadcast, it checks its array for duplicates, and deletes them.