5 ms·
We use a columnar database and we insert data to it periodically (30s to 90s) using Kinesis as middleware. Files are immutable and each insert creates [count of
by buremba 10y ago
We use a columnar database and we insert data to it periodically (30s to 90s) using Kinesis as middleware. Files are immutable and each insert creates [count of columns] * [count of tables] files. The database also periodically merges the batch files in background.
Since EFS was not available, I was trying to develop a FUSE filesystem that writes data to both S3 and local filesystem (EBS) for durability and read only from local filesystem for performance. Even though it's cheaper than EFS, it's slow since each file needs to be sent to S3 individually. Our requirements are not that much because the files are immutable, there are not many files (I heard that EFS doesn't work well with many small files.), no need for concurrent access. I think I will give it a shot since it also fits our use-case.
- takeda 10y agoIf files are immutable, isn't s3 much better fit for it? s3 main weaknesses is that you can't modify part of a file, which makes it unsuitable for a filesystem, but looks like it satisfies your use case.
- buremba 10y agoMaybe but unfortunately the database doesn't support "backup store" that we can use directly S3, it just writes the data to specified directory in a filesystem. We should either fork the database or implement "S3 backed" filesystem and we picked the latter one. Unfortunately the performance FUSE is not that good for read and writing data to S3 is somehow expensive.
- ak4g 10y agoIf you don't mind me asking, what DB is this?
- buremba 10y agoWe're implementing Clickhouse (http://clickhouse.yandex http://clickhouse.yandex) to our open-source analytics platform Rakam. (https://github.com/rakam-io/rakam https://github.com/rakam-io/rakam)
- mej10 10y agoWhy not have a service watch for filesystem changes and then upload to S3?
- buremba 10y agoAFAIK, there is no synchronous way to do it and if I do it asynchronously, partial data loss is possible.
- Someone 10y agoThey use S3. Their problem likely is that they write many small files to S3, and repeat it every minute or so. That's a combination S3 isn't designed/optimized for.