4 ms·

> Each Lambda invocation executed a simple TruffleHog scan command with concurrency set to 1000. This setup allowed me to complete the scan of 5,600,000 reposit

by jsiepkes 11mo ago

> Each Lambda invocation executed a simple TruffleHog scan command with concurrency set to 1000. This setup allowed me to complete the scan of 5,600,000 repositories in just over 24 hours. Gitlab must have been thrilled about a bot cloning 5.6 million repo's in 24 hours. That doesn't really sound responsible to me.

treyd 11mo ago
That's 64 clones per second. That's quite a lot but it seems like something that a forge operating at the scale of GitHub can handle, especially if they were --depth=1 (which might have missed some secrets if someone was lazy about clearing their git history).
- nojs 11mo ago
  Gitlab*
- digi59404 11mo ago
  Provided someone told GitLab Support. This was likely fine. GitLab can handle this much load. The platform as a whole has increased and improved over the years as new customers are added. Think about this… every CI/CD Job runs a clone. That’s a lot..
47282847 11mo ago
If they don’t like, they will apply rate limiting? Assuming they were well behaved (user agent, IPs).
- pcdevils 11mo ago
  Assuming bog standard lambda they'd have to rate limit a whole Aws region lambda range which would risk affecting legit usage. Bit of an arse way to behave against a service
3eb7988a1663 11mo ago
I also thought the sleep(0.03) was cute. Some well deserved rest for the server to avoid hammering it.