3 ms·
I feel this, working on a small team moving things to microservices. My primary problem is observability. It's become a huge chore figuring what, exactly, is go
by cfeduke 3y ago
I feel this, working on a small team moving things to microservices. My primary problem is observability. It's become a huge chore figuring what, exactly, is going wrong in production when something goes wrong. It's not enough to tail the logs of some distributed application, I need to tail the logs of several distributed applications where there messages are interspersed with one another. I suppose when we get some way to visualize these traces - tooling - it'll be okay. But, small team, limited human bandwidth, and we don't have this tooling in place yet.
The monolith, in contrast, had NewRelic integrated years ago. There were performance problems with this monolith which have been mostly solved through indexes and a couple of materialized views. Trivial to figure out what is going wrong. The code may be old and full of race conditions, but solving problems isn't difficult.
I dread dealing with multiple separate database instances each backing their own microservice when it comes time to upgrade those databases instances. I was hoping for a single database instance with multiple databases, but that particular architecture isn't on the menu. :\
- antonvs 3y agoAre you not using cloud? Because the cloud providers provide centralized logging, so all you need to do is pass a request id between services and include that I’d in log entries, and you can trace requests across services.
- jskrablin 3y agoTake a look at the OTEL (Open Telemetry) tooling and libraries. Or Grafana stack/offering with Prometheus, Tempo and Loki. Centralized logging and service calls/code execution tracing is not exactly new. It is often an afterthought.. and then you get yourself is this kinds of unpleasant situations. And since you didn't implement correct tooling from the start, your team is even smaller and more limited... because you have little to zero idea on what your services are up to. As per db instances... you upgrade them one by one. Unless there's some really bad bugs present (security or otherwise) there's no rush in upgrading stuff just because.