5 ms·
I have a syslog-ng -> MongoDB project that I've been working on at my university. github.com/ngokevin/netshed It is written in Python current parses out field
by ngokevin 15y ago
I have a syslog-ng -> MongoDB project that I've been working on at my university.
github.com/ngokevin/netshed
It is written in Python current parses out fields from several types of logs (such as dhcpd). It is initially set up to read from named pipes (it has a tail function as well). Each type of log is dumped to its own database, and each date has its own collection. I have it set up with a master/slave configuration to overcome the global write lock. It has functions to simulate capped collections by days. It is followed with a Django frontend for querying via PyMongo.
This version is several weeks old and I will push out a new one soon.
- alexchamberlain 15y agoHave you got more details about overcoming the global write lock?
- ngokevin 15y agoOh sorry, when I mean overcome the global write lock, I don't mean getting rid of it, but simply allowing me to querying a replicated slave database while the master database is getting hundreds of writes a second...so the writes don't block the reads.