3 ms·

Does this use the FS or is it purely in memory? I've got GBs of CSV data that I'm managing manually and due to response times, need solutions that reduce wire

Does this use the FS or is it purely in memory? I've got GBs of CSV data that I'm managing manually and due to response times, need solutions that reduce wire trips. This means that traditional databases have proven difficult due to the distance between process and DB. Something designed for local usage (SQLite/etc) is an area I want to research, so ObjectBox sounds interesting on that front as well. Though, it looks like its goals are mainly IoT, so I'm not too hopeful on this front.

greenrobot 8y ago
It's ACID compliant, and thus not not in-memory. https://golang.objectbox.io/faq https://golang.objectbox.io/faq covers that in detail. In the FAQ, there is a section on comparing to JSON; I guess the same would hold true for CSV. PS.: Wouldn't worry about a label saying "IoT" if the technology fits your need.
IanCal 8y ago
I've been using clickhouse (https://clickhouse.yandex https://clickhouse.yandex) a lot recently. One thing I found useful was that the underlying tables are just folders on disk that you can easily copy around. The data will be automatically compressed too. I'm sure it'll depend on some of your other constraints, but may be worth looking at. I've been extremely impressed with its performance.
- Keyframe 8y ago
  How did you find clickhouse vs drill/dremio? Have you tried maybe both and compared them?
  - IanCal 8y ago
    I'm afraid I've not tried those, I played around only a bit and mostly spent a while trying to get cassandra setup but failed. Then I hit on https://tech.marksblogg.com/benchmarks.html https://tech.marksblogg.com/benchmarks.html and saw clickhouse sitting at a very impressive point given the hardware it's on there. It was one of the few data things I saw that looked like it scaled down to a level relevant for me (cassandra I think talks about a dev setup of 5 servers). It's been fast enough with zero tuning or work so far that I've not done that much. I'm sure I could get closer to its speed by carefully setting some things in postgres, or clever work somewhere but I've got other things to do and that's not my speciality. It has a few issues with ingesting CSV (I end up quoting everything, though I think it's less hassle if you use TSV instead). Frankly, it's been incredible for my analysis work. apt-get install setup, fast to ingest data, easy to query and output large amounts of things. Happily works on a single box, though apparently scales up as well. More than happy to waffle on about this but I'm aware I've gone on very long already for a "I don't know" answer.
module0000 8y ago
SQLite is pretty bulletproof and tested in this area. Lots of examples for your project to take advantage of, and very high performance for in-memory data sets.
- greenrobot 8y ago
  SQLite is a very nice embedded DB indeed. If you like SQL. ObjectBox enables you to work with Go structs and gives you additional performance.
sitkack 8y ago
I have used SQLite databases sharded at 2GB due to mobile fs limitations, storing 10-14GB in total, total lookup times across whole query was still on the order of 100ms untuned, unoptimized. SQLite has your back.

greenrobot 8y ago

It's ACID compliant, and thus not not in-memory. https://golang.objectbox.io/faq https://golang.objectbox.io/faq covers that in detail. In the FAQ, there is a section on comparing to JSON; I guess the same would hold true for CSV. PS.: Wouldn't worry about a label saying "IoT" if the technology fits your need.

IanCal 8y ago

I've been using clickhouse (https://clickhouse.yandex https://clickhouse.yandex) a lot recently. One thing I found useful was that the underlying tables are just folders on disk that you can easily copy around. The data will be automatically compressed too. I'm sure it'll depend on some of your other constraints, but may be worth looking at. I've been extremely impressed with its performance.

Keyframe 8y ago
How did you find clickhouse vs drill/dremio? Have you tried maybe both and compared them?
- IanCal 8y ago
  I'm afraid I've not tried those, I played around only a bit and mostly spent a while trying to get cassandra setup but failed. Then I hit on https://tech.marksblogg.com/benchmarks.html https://tech.marksblogg.com/benchmarks.html and saw clickhouse sitting at a very impressive point given the hardware it's on there. It was one of the few data things I saw that looked like it scaled down to a level relevant for me (cassandra I think talks about a dev setup of 5 servers). It's been fast enough with zero tuning or work so far that I've not done that much. I'm sure I could get closer to its speed by carefully setting some things in postgres, or clever work somewhere but I've got other things to do and that's not my speciality. It has a few issues with ingesting CSV (I end up quoting everything, though I think it's less hassle if you use TSV instead). Frankly, it's been incredible for my analysis work. apt-get install setup, fast to ingest data, easy to query and output large amounts of things. Happily works on a single box, though apparently scales up as well. More than happy to waffle on about this but I'm aware I've gone on very long already for a "I don't know" answer.

module0000 8y ago

SQLite is pretty bulletproof and tested in this area. Lots of examples for your project to take advantage of, and very high performance for in-memory data sets.

greenrobot 8y ago
SQLite is a very nice embedded DB indeed. If you like SQL. ObjectBox enables you to work with Go structs and gives you additional performance.

sitkack 8y ago

I have used SQLite databases sharded at 2GB due to mobile fs limitations, storing 10-14GB in total, total lookup times across whole query was still on the order of 100ms untuned, unoptimized. SQLite has your back.