3 ms·

The system I work on as of now isn't really fit for a unified log design. But there are definitely some critical part of the system that can use a high performa

by tmd83 9y ago

The system I work on as of now isn't really fit for a unified log design. But there are definitely some critical part of the system that can use a high performance centralized event log and can be made simpler and better (faster/more reliable). The ability to decouple producer & consumer, fanning in/out in both side, batching all are very attractive properties. But I live in a transactional world and even though in quite a few cases I can make the consumer idempotent the publishing of an event becomes tricky. If you are not using the log as the only store but in addition to a traditional transactional data store (databases for e.g.) how can you ensure that events are published only and surely for a successful transaction? The staged log record idea is interesting but it makes the database bottleneck bigger specially if a lot of different transaction (spread across many tables) now is bottlenecked on the same stagedlog table. Maybe I'm missing something here?

bjt 9y ago
I don't see the single stagedlog table being a bottleneck if you're just doing inserts to it. Postgres is already doing appends to a single WAL internally.
- tmd83 9y ago
  I have seen it happen though in such contexts (though in Oracle). I assume the difference is that the database has much more freedom in how it writes to WAL (batching and such) but in the table itself it has to do some degree of coordination and locking. The other big issue is that even if the insert into the shared table is fast (simpler structure) the transaction hence the commit can be long due to the complexity of the application transaction. I will be happy to be corrected though if we are doing something wrong.
  - bjt 9y ago
    By my (admittedly limited) understanding of MVCC, one long running transaction that includes an insert will not block other inserts to the same table. The worst you'd see in that situation is if you were using a sequence for IDs, an ID might be grabbed from the sequence at the beginning of transaction A, then another ID grabbed and inserted into the table by transaction B, and then when transaction A finally commits, the order of the IDs won't match the chronological order of the commits.

by tmd83 9y ago

bjt 9y ago

I don't see the single stagedlog table being a bottleneck if you're just doing inserts to it. Postgres is already doing appends to a single WAL internally.

tmd83 9y ago
I have seen it happen though in such contexts (though in Oracle). I assume the difference is that the database has much more freedom in how it writes to WAL (batching and such) but in the table itself it has to do some degree of coordination and locking. The other big issue is that even if the insert into the shared table is fast (simpler structure) the transaction hence the commit can be long due to the complexity of the application transaction. I will be happy to be corrected though if we are doing something wrong.
- bjt 9y ago
  By my (admittedly limited) understanding of MVCC, one long running transaction that includes an insert will not block other inserts to the same table. The worst you'd see in that situation is if you were using a sequence for IDs, an ID might be grabbed from the sequence at the beginning of transaction A, then another ID grabbed and inserted into the table by transaction B, and then when transaction A finally commits, the order of the IDs won't match the chronological order of the commits.