monstrado

8 ms·

31.
by monstrado 6y ago
I noticed there is "Clustering" mentioned under enterprise features, but I can't seem to find any references to it in the documentation. Is this something that will be strictly closed source?
32.
by monstrado 6y ago
Any plans on integration with Apache Arrow?
33.
by monstrado 6y ago
Thanks for firing up the time machine! I've been using FDB for a little over a year now and can't recommend it enough. Such a solid piece of meticulous engineering.
34.
by monstrado 6y ago
Huge fan of your work! I was curious if you've ever attempted to run your (or part of) Mongo test suite against FoundationDB using their DocumentLayer since it's supposed to be Mongo API compatible.
35.
by monstrado 6y ago
Not sure what you mean. PrestoDB is an analytics data store. The only correlation between NiFi and Presto would be if you wrote data to Presto using NiFi.
36.
by monstrado 6y ago
It was built from scratch at the NSA and open sourced a few years ago.
37.
by monstrado 6y ago
NiFi is also very useful for visually chaining together scripts (e.g. foo | bar | baz). In NiFi you can use ExecuteProcess and ExecuteStreamCommand to basically do what pipes (|) does. Further, the intermediate data between each process is
38.
by monstrado 6y ago
I fine that explaining NiFi tends to be difficult, especially if you're an advanced user. This is because NiFi is extremely general purpose. I've used it for many different types of projects. That being said, here's a few use
39.
by monstrado 6y ago
Incredible piece of software. I've used it in production at my last two jobs. You can build almost anything in NiFi once you get into the mindset of how it works. A good way to get started with NiFi is to use it as a highly available q
40.
by monstrado 7y ago
On a related note, one of their engineers wrote a POC that uses FoundationDB instead of their custom storage engine. https://github.com/richardartoul/tsdb-layer The README does a really good job explaining the internal
41.
by monstrado 7y ago
I've had alot of success using Apache NiFi as a distributed scheduler / general purpose workflow tool.
42.
by monstrado 7y ago
Related, but ClickHouse utilizes this for their regex parsing.
43.
by monstrado 7y ago
Cool project! I've had success with ClickHouses's local utility which is extremely fast. It helps that its basically a "local" version of an already insanely efficient columnar database. https://www.altinity.c
44.
by monstrado 8y ago
Interesting enough, FoundationDB currently uses sqllite's storage engine for persistence.
45.
by monstrado 8y ago
What differentiates this from Apache NiFi?
46.
by monstrado 8y ago
That's where we disagree. You're taking some pretty radical jumps to get to the conclusion you want. Let's assume for a second that you're right, and EMR somehow kills an entire company (who sells much more than just a C
47.
by monstrado 8y ago
Curious where you get "Hadoop is gone" by two companies (who employ hundreds of open source contributors and committers) merging. I think it's in fact the complete opposite. I get the whole buy-in into the FUD that Cloudera s
48.
by monstrado 8y ago
Awesome, thanks!
49.
by monstrado 8y ago
Is there a way to send logs to Loki without using the tail agent? For example, a REST API for storing messages?
50.
by monstrado 8y ago
Are there any plans to extend the API compatibility of the document layer to be compatible with other APIs, such as, DynamoDB?
51.
by monstrado 8y ago
This is massive for adoption. Can't wait to give it a try.
52.
by monstrado 8y ago
Agreed 100% and as someone who has had to use DDB before, nothing would make me happier than seeing this built.
53.
by monstrado 8y ago
There's not really a concept of "Database" in FDB. There is however a concept of key spaces, and "directories", which are basically the same, and these all support transactions. i.e. /database1/key1 = foo
54.
by monstrado 8y ago
FoundationDB https://www.foundationdb.org/ not only supports transactions, they are mandatory. They also go one step further and support atomic operations, which are especially killer.
55.
by monstrado 8y ago
Also, better insight into partition sizes / what's causing hot spotting. The DB abstracts a lot from the user, which isn't necessarily great, because it's still subject to the normal pitfalls of a NoSQL database.
56.
by monstrado 8y ago
Hey, I've written some comments regarding the use case in the past you can see here: https://news.ycombinator.com/item?id=18305446
57.
by monstrado 8y ago
Exactly, and if you /really/ wanted to, you could run a one node Kafka "cluster" using data replication of one instead of three for topics. I totally get the fun of creating something from scratch, but you introduce far
58.
by monstrado 8y ago
Have you given https://apple.github.io/foundationdb/time-series.html a read? Since FoundationDB is lexicographically sorted it's pretty straight forward to keep things sorted for reading data in chronological orde
59.
by monstrado 8y ago
Congrats to the team on the release. Using FoundationDB has been one of the most rock solid NoSQL experiences I've ever had, and I've used a lot. After having a few months to hammer my cluster with fairly low level atomic operatio
60.
by monstrado 8y ago
Turns out we are doing a shuffle :) - We're using Apache Flink for the aggregation step (5 second window) which performs a merge on key before writing the value out. So at the end of the day, we would only read/deserialize/me

More ›