3 ms·
in a couple of projects I worked on, we also stored visited urls in a set of bloom filters, also stored in flat files on disk. At some point querying the db to
by riffraff 13y ago
in a couple of projects I worked on, we also stored visited urls in a set of bloom filters, also stored in flat files on disk.
At some point querying the db to check what URLs you have can become quite heavy
- deleted 13y ago[deleted]