2 ms·

Author here. I'm running a "show hn" on what is a complete backend project - but I'm well aware interest in the project will be a lot higher when a registration

Author here. I'm running a "show hn" on what is a complete backend project - but I'm well aware interest in the project will be a lot higher when a registration interface is implemented. Any feedback on the work thus far would be appreciated.

detaro 11y ago
I had a short look at CT recently and came away with the impression that you'd need serious resources to usefully process the data - certainly doable, but more than most private persons or even organizations would be willing to invest into watching after their domains, so it is great to see a project for a watcher-service. What kind of hardware/instance size do you actually need to run this and keep up?
- technion 11y ago
  Thank you for this feedback. I do believe actually retaining the data in a searchable format would require significant resources. I note Comodo wrote their own PostgreSQL extension for this purpose[0], something I'd imagine would not be done lightly. However, I've found it surprisingly efficient for the type of service I'm running. It's currently running on a single Linode instance with 2G RAM. I've got load averages on the box of 0.05, so I think it's easily handling things. I have various parts of the service running in spawned threads, and I've tested that I can configure the timing to be more aggressive than it currently is it needed. In terms of keeping up, I had the service shutdown for over 48 hours days whilst debugging a frustrating race condition in the PostgreSQL driver[1] and it managed to catch up in a few hours. What has held up online registration is the place where load is in fact a significant issue - in testing, I set myself up monitoring *.cloudflaressl.com, which is known to rotate regularly. I nearly hit the Amazon SES free tier limit and gave myself an Office 365 inbox I can't open without crashing IE. Obviously I need to ensure users can't do the same. [0] https://github.com/crtsh/libx509pq https://github.com/crtsh/libx509pq [1] https://github.com/epgsql/epgsql/issues/80 https://github.com/epgsql/epgsql/issues/80

detaro 11y ago

I had a short look at CT recently and came away with the impression that you'd need serious resources to usefully process the data - certainly doable, but more than most private persons or even organizations would be willing to invest into watching after their domains, so it is great to see a project for a watcher-service. What kind of hardware/instance size do you actually need to run this and keep up?

technion 11y ago
Thank you for this feedback. I do believe actually retaining the data in a searchable format would require significant resources. I note Comodo wrote their own PostgreSQL extension for this purpose[0], something I'd imagine would not be done lightly. However, I've found it surprisingly efficient for the type of service I'm running. It's currently running on a single Linode instance with 2G RAM. I've got load averages on the box of 0.05, so I think it's easily handling things. I have various parts of the service running in spawned threads, and I've tested that I can configure the timing to be more aggressive than it currently is it needed. In terms of keeping up, I had the service shutdown for over 48 hours days whilst debugging a frustrating race condition in the PostgreSQL driver[1] and it managed to catch up in a few hours. What has held up online registration is the place where load is in fact a significant issue - in testing, I set myself up monitoring *.cloudflaressl.com, which is known to rotate regularly. I nearly hit the Amazon SES free tier limit and gave myself an Office 365 inbox I can't open without crashing IE. Obviously I need to ensure users can't do the same. [0] https://github.com/crtsh/libx509pq https://github.com/crtsh/libx509pq [1] https://github.com/epgsql/epgsql/issues/80 https://github.com/epgsql/epgsql/issues/80