skyde

46 ms·

481.
by skyde 6y ago
interesting without me having to read both paper what would you say is the main differences? Pros and cons of each architectur?
482.
by skyde 6y ago
it is but I think you can do more complex query and « the current state » is a table built from all event ever received not a small time window
483.
by skyde 6y ago
Can someone explain in simple english what is the difference between this and Automatic Materialized as implemented in SqlServer and Oracle
484.
by skyde 6y ago
What about language designed for HPC like Chapel and Fortress. Or any nested data-parallel languages. All we need is An easy way to do safe stateful parallel algorithm using Distributed transaction or purely functional data parallelism over
485.
by skyde 6y ago
so is that thread-safe ?
486.
by skyde 6y ago
did the linux kernel system call changes or is it only a problem with libc?
487.
by skyde 6y ago
consider people in Montreal, same time zone good university with computer science but much lower salary and cost of living
488.
by skyde 6y ago
I don't know your use-case exactly but every distributed system engineer should know that 2PC is extremely dangerous. It is a blocking protocol, This mean if the coordinator crash , some participants will never resolve their transactio
489.
by skyde 6y ago
I would be very curious to know how someone could design an airline Computer reservation system this way. I agree that when using a SQL database that use locks instead of snapshot isolation you need to be careful about the order you scan ro
490.
by skyde 6y ago
yes it can but it’s a real pain compared to hashicorp version. I really hope they clean up the library to have a simple API. This way it could become the standard for GO and we can stop wasting effort on multiple raft implementation.
491.
by skyde 6y ago
No this was a misunderstanding I said if you need (concurrency control/transaction and index) use a real database not a filesystem, since it’s the whole reason database exist. But if you don’t need it there are plenty of good distribut
492.
by skyde 6y ago
if you care about lost updates you should use transaction or at least an API that support optimistic concurrency control. Posix File system API don’t offer much for concurrency control other than lock. In my line of work we use HDFS or AWS
493.
by skyde 6y ago
are you hiring :-) I would love to work on this. I consider myself a RAFT expert and worked in kafka for the past 5 years.
494.
by skyde 6y ago
i am curious to know why people expect a raft library to be more reliable if embedded inside the kafka controller versus running inside a service like Etcd. in the end broker will do RPC to a service (kafka controller/ etcd) and this s
495.
by skyde 6y ago
this pain come from paxos raft or Zab algorithm used by concensus system. I don’t think it can be avoided.
496.
by skyde 6y ago
this lib u is great and 1000 time cleaner better tested and easier to use compared to trying to reuse etcd raft implementation :-)
497.
by skyde 6y ago
100 % agree we need good (etch/zookeeper) as a library. I wrote something similar but that provide only (lock/lease) using paxos. The problem with raft ETCD and zookeeper replicates state machine design is that : - machine leavi
498.
by skyde 6y ago
are you using Jocko in production?
499.
by skyde 6y ago
so this use raft for topic data not just metadata ?
500.
by skyde 6y ago
interesting idea: I think one issue is that the write throughout of a single master Instance is very limited. But if working set fit in memory on the instances processing the writes, you could use PostgreSQL logical replication to update m
501.
by skyde 6y ago
While i agree with all you said materialized index are not enough you also need transaction that span more than one entity. You can’t simply put all entity you might need to update atomically in the same key/document.
502.
by skyde 6y ago
100% agree and that’s why google Spanner, Cockroachdb and all the NeeSqL exist. People found that they miss the abstraction that SQL and transaction offer as it simplify your code a lot. They simply needed something that scale better than d
503.
by skyde 6y ago
it seem funny to compare kafka to sql for a OLTP use case. The main benefit of SQL are index and Transactions. If you don’t need them you might as well use a file system directly but if you need it, there is not much alternative :-)
504.
by skyde 6y ago
thanks a lot. This seem to be a very simple but smart design.
505.
by skyde 6y ago
if your use case is ok with ack=1 then kafka is perfect for you. because this mean you don’t care about losing some of the messages for which the broker confirmed to the client library that the write was successful. if you care more about m
506.
by skyde 6y ago
Yes I agree if using extremely small partition size ~100mb and using Cruise Control it should be able to auto heal quickly. But you will still have broker with like 100% cpu usage and high network usage while other are mostly idle. Cruise c
507.
by skyde 6y ago
not true for HDFS, Cassandra ,pulsar and most distributed file system. As soon as a segment is under-replicated it”s replication factor is restored under less than 2 minutes by selecting new machine as replica. Kafka try to do it with “kafk
508.
by skyde 6y ago
Assuming replication factor of 3 i’m not talking about 3 machine going down at exact same time, but in sequence. like 5 minutes apart. All other distributed system handle this with no issue. Ex: Cassandra, Hbase/hdfs, CockroachDB
509.
by skyde 6y ago
because the only sane write mode in kafka is “ack=-1” which mean wait for confirmation from all in sync replica. The more replica you have the worse is your write latency. In the case of pulsar ack quorum can be much smaller than replicatio
510.
by skyde 6y ago
if replication factor is 3 and 3 server go down in the span of 1 or 2 hours no alert will save you

More ›