6 ms·

Confluent, a company for Apache Kafka and realtime data

108 points by sbilstein 12y ago

capkutay 12y ago
Good luck to these guys. Its certainly an up and coming area with some overlap with an existing, mature market. Tibco (the information bus company) is a $4b company with relatively dated technology...Informatica is another legacy solution in this area with tons of revenues. Either way, the market is certainly poised for some disruption
mintplant 12y ago
Does anyone have a mirror for those of us without LinkedIn accounts?
- joshmn 12y ago
  http://pastebin.com/raw.php?i=Z6bRDA7V http://pastebin.com/raw.php?i=Z6bRDA7V
- postreader 12y ago
  You don't need an account to read the post.
  - mintplant 12y ago
    Are you sure? This is what I'm being redirected to when I click on the link: http://i.imgur.com/w9YxLvA.png http://i.imgur.com/w9YxLvA.png
    - sbilstein 12y ago
      Weird, that shouldn't be there. I double checked in an incognito and things seemed fine. LinkedIn blog posts are supposed to be visible with or without an account, so this must be a bug of some kind.
      - mintplant 12y ago
        It doesn't seem to be doing it anymore. Strange! Maybe whatever it was got fixed.
- bcantoni 12y ago
  Same article is on the new company blog: http://blog.confluent.io/2014/11/06/announcing-confluent-a-company-for-apache-kafka-and-real-time-data/ http://blog.confluent.io/2014/11/06/announcing-confluent-a-c...
  - mintplant 12y ago
    Thank you!
joshmn 12y ago
In case anyone else is wondering why the article is linked to LinkedIn, Jay Kreps, co-founder/CEO was previously with LinkedIn. LinkedIn is also an investor of Confluence.
- lern_too_spel 12y ago
  I doubt anybody is wondering that. The very first section is titled "Origin at LinkedIn."
gfodor 12y ago
Congratulations guys and good luck on this next adventure! How does Samza fit into this new venture?
- mountaineer 12y ago
  Samza's role in this is a question I had as well. But, certainly Hadoop needs some polish first.
marknadal 12y ago
It is surprising how long it has taken for the common practice of databases to be distributed. I'm glad that people are starting to move in that direction, and open source their work. This is a win for our industry overall. We're working on a database/cache/messaging system too, http://github.com/amark/gun http://github.com/amark/gun it is dedicated to removing the pain I and other Javascript/NodeJS developers had when it came to managing/debugging databases (devops and sysadmin work is frustrating).
bkirwi 12y ago
Kafka is a Big Deal, and it's great to see it get first-class support. In the same way that Hadoop is starting to feel outdated, but HDFS doesn't seem to be going anywhere -- I think we'll see a lot of innovation in stream processing frameworks in the next few years, but Kafka will just keep on going.
davidjgraph 12y ago
Sounds somewhat like Confluence, this will get confusing. Already I can see joshmn has made the mistake in his comment.
mountaineer 12y ago
I love the true engineers' launch, with no logo (twitter account is just the egg).
beagle3 12y ago
Good luck to the Confluent guys! There's a missing piece in the realtime puzzle, at least one that I haven't been able to find, for which Kafka is an overkill - perhaps someone here knows of a solution: I have tens of system endpoints connected through unreliable (Line-of-sometimes-occluded-sight, 2G and 3G WWAN, some are in vehicles so connections are intermittent). I just want to consistently tail their logs in a bandwidth-efficient, connection-drop resistant way; and I can't find any standard thing that does this. Kafka would fit the bill in general, but would require a lot of work (reading textual logs into kafka, querying kafka for new stuff across connection, reading from kafka and writing to text files) - and I'm not sure how well it deals with dropped connections. My existing solution is to rsync the log directories (--append, --inplace) as infrequently as I can from an operational view, which is 1 minute. It is relatively bandwidth efficient (although could be much better), robust with respect to connection issues, and generally works. However, it is less efficient than it could be: if directories have a lot of files, like /var/log often does, there's a lot of sync overhead. The delay is 1 minute instead of a couple of seconds (which is what you would get with a simple "tail -f" through a TCP connection), and it doesn't play well with common log rotation schemes (though that's relatively easy to work around). Anyone has a better solution, kafkaesque or otherwise?
- easytiger 12y ago
  How about a tiny forwarder on each device. It fopens the file and reads to the end, when the file is appended to it will be able to continue reading. Packet up each line and forward to you server using zeromq PUSH/PULL. If you are disconnected they queue up and send when zeromq automagically reconnects.
  - imperialWicket 12y ago
    Since logstash was already mentioned, it's worth putting https://github.com/elasticsearch/logstash-forwarder https://github.com/elasticsearch/logstash-forwarder here, as the logstash team built it as a solution for logstash-type needs on systems that may not be able to support logstash itself.
- lrm242 12y ago
mlhamel 12y ago
I'm a bit confused. The more i read the doc, the more i feel it look Just Another Queue system. I don't really see the difference between Kafka and let's say RabbitMQ, Celery or also statsd...
- possibilistic 12y ago
  There are differences, the biggest of which is throughput. Kafka can handle incredible load. The messaging semantics are also a bit different. Here's a pretty good comparison: http://www.quora.com/RabbitMQ-vs-Kafka-which-one-for-durable-messaging-with-good-query-features http://www.quora.com/RabbitMQ-vs-Kafka-which-one-for-durable...

108 points by sbilstein 12y ago

Good luck to these guys. Its certainly an up and coming area with some overlap with an existing, mature market. Tibco (the information bus company) is a $4b company with relatively dated technology...Informatica is another legacy solution in this area with tons of revenues. Either way, the market is certainly poised for some disruption

mintplant 12y ago

Does anyone have a mirror for those of us without LinkedIn accounts?

joshmn 12y ago
http://pastebin.com/raw.php?i=Z6bRDA7V http://pastebin.com/raw.php?i=Z6bRDA7V
postreader 12y ago
You don't need an account to read the post.
- mintplant 12y ago
  Are you sure? This is what I'm being redirected to when I click on the link: http://i.imgur.com/w9YxLvA.png http://i.imgur.com/w9YxLvA.png
  - sbilstein 12y ago
    Weird, that shouldn't be there. I double checked in an incognito and things seemed fine. LinkedIn blog posts are supposed to be visible with or without an account, so this must be a bug of some kind.
    - mintplant 12y ago
      It doesn't seem to be doing it anymore. Strange! Maybe whatever it was got fixed.
bcantoni 12y ago
Same article is on the new company blog: http://blog.confluent.io/2014/11/06/announcing-confluent-a-company-for-apache-kafka-and-real-time-data/ http://blog.confluent.io/2014/11/06/announcing-confluent-a-c...
- mintplant 12y ago
  Thank you!

joshmn 12y ago

In case anyone else is wondering why the article is linked to LinkedIn, Jay Kreps, co-founder/CEO was previously with LinkedIn. LinkedIn is also an investor of Confluence.

lern_too_spel 12y ago
I doubt anybody is wondering that. The very first section is titled "Origin at LinkedIn."

gfodor 12y ago

Congratulations guys and good luck on this next adventure! How does Samza fit into this new venture?

mountaineer 12y ago
Samza's role in this is a question I had as well. But, certainly Hadoop needs some polish first.

marknadal 12y ago

It is surprising how long it has taken for the common practice of databases to be distributed. I'm glad that people are starting to move in that direction, and open source their work. This is a win for our industry overall. We're working on a database/cache/messaging system too, http://github.com/amark/gun http://github.com/amark/gun it is dedicated to removing the pain I and other Javascript/NodeJS developers had when it came to managing/debugging databases (devops and sysadmin work is frustrating).

bkirwi 12y ago

Kafka is a Big Deal, and it's great to see it get first-class support. In the same way that Hadoop is starting to feel outdated, but HDFS doesn't seem to be going anywhere -- I think we'll see a lot of innovation in stream processing frameworks in the next few years, but Kafka will just keep on going.

davidjgraph 12y ago

Sounds somewhat like Confluence, this will get confusing. Already I can see joshmn has made the mistake in his comment.

mountaineer 12y ago

I love the true engineers' launch, with no logo (twitter account is just the egg).

beagle3 12y ago

Good luck to the Confluent guys! There's a missing piece in the realtime puzzle, at least one that I haven't been able to find, for which Kafka is an overkill - perhaps someone here knows of a solution: I have tens of system endpoints connected through unreliable (Line-of-sometimes-occluded-sight, 2G and 3G WWAN, some are in vehicles so connections are intermittent). I just want to consistently tail their logs in a bandwidth-efficient, connection-drop resistant way; and I can't find any standard thing that does this. Kafka would fit the bill in general, but would require a lot of work (reading textual logs into kafka, querying kafka for new stuff across connection, reading from kafka and writing to text files) - and I'm not sure how well it deals with dropped connections. My existing solution is to rsync the log directories (--append, --inplace) as infrequently as I can from an operational view, which is 1 minute. It is relatively bandwidth efficient (although could be much better), robust with respect to connection issues, and generally works. However, it is less efficient than it could be: if directories have a lot of files, like /var/log often does, there's a lot of sync overhead. The delay is 1 minute instead of a couple of seconds (which is what you would get with a simple "tail -f" through a TCP connection), and it doesn't play well with common log rotation schemes (though that's relatively easy to work around). Anyone has a better solution, kafkaesque or otherwise?

easytiger 12y ago
How about a tiny forwarder on each device. It fopens the file and reads to the end, when the file is appended to it will be able to continue reading. Packet up each line and forward to you server using zeromq PUSH/PULL. If you are disconnected they queue up and send when zeromq automagically reconnects.
- imperialWicket 12y ago
  Since logstash was already mentioned, it's worth putting https://github.com/elasticsearch/logstash-forwarder https://github.com/elasticsearch/logstash-forwarder here, as the logstash team built it as a solution for logstash-type needs on systems that may not be able to support logstash itself.
lrm242 12y ago

mlhamel 12y ago

I'm a bit confused. The more i read the doc, the more i feel it look Just Another Queue system. I don't really see the difference between Kafka and let's say RabbitMQ, Celery or also statsd...

possibilistic 12y ago
There are differences, the biggest of which is throughput. Kafka can handle incredible load. The messaging semantics are also a bit different. Here's a pretty good comparison: http://www.quora.com/RabbitMQ-vs-Kafka-which-one-for-durable-messaging-with-good-query-features http://www.quora.com/RabbitMQ-vs-Kafka-which-one-for-durable...