3 ms·
I am doing exactly this in my current project - I put server logs of API services to column-based database with extensive SQL support, and it's feeling great. S
by jetter 6y ago
I am doing exactly this in my current project - I put server logs of API services to column-based database with extensive SQL support, and it's feeling great.
SQL allows me to build great analytics dashboard in a simple dialect that I know, and in terms of performance it runs circles around ELK stack (short animated GIF preview of UI is available on https://apiroad.net/ https://apiroad.net/ )
- ayewo 6y agoApiRoad is an interesting project, are you running this all by yourself? In any case, do you have a write-up somewhere of how you use a columnar DBMS to slurp your server logs?
- jetter 6y agoThank you! Yes this is a big solo project now. Didn't have time to do a write up yet. I am a big fan of Clickhouse, so I use it for log storage. To collect server logs, it is not exactly a simple file parsing setup - I have a gateway daemon that stays in front of the API server. The daemon receives all the requests from subscribers, authorizes that this specific connection has access to specific API, rate limits, and proxies all the connections to upstream API, dumping HTTP logs to Clickhouse once in a while. Then Vue.js & Laravel powered dashboard queries Clickhouse to generate various stats, which is later used for analytics and usage-based billing.