3 ms·
Hi HN, I'm the author of DeepSearch. I built this while dealing with large SMB shares over WiFi, where traditional directory scanning tools were either too sl
by dohuyhoangvn93 10mo ago
Hi HN,
I'm the author of DeepSearch.
I built this while dealing with large SMB shares over WiFi, where traditional
directory scanning tools were either too slow or caused unnecessary network load.
The main idea is a layered scanning strategy:
- Minimize SMB round-trips
- Use Rayon for controlled parallelism
- Avoid spawning excessive IO-bound tasks
- Keep memory usage predictable
This is not meant to be a fancy UI tool, but a fast and deterministic building
block that can be embedded into other systems.
The project is open-source (Apache 2.0).
I'm especially interested in feedback on:
- The concurrency model
- SMB traversal strategy
- Any edge cases you've seen in similar systems
Thanks for taking a look.
- holg 10mo agoHi thx for sharing, in general about concurrency Rayon excels on CPU-heavy tasks, but this is more IO for traversal avoid any extra round-trips, simple the metadata already given edge-cases would be infinite loops on symlinks, the Windows 255 char limits and on WIFI packet loss due to too many reqs
- dohuyhoangvn93 10mo agoThank you for your feedback. In the initial CLI test, I tried using Tokio and found that performance didn't improve and the earliest result return speed was quite poor. I skipped the root cause analysis and kept the original rayon design. I will try redoing the tests and taking more detailed measurements for evaluation. Thank you again.