addisonj

12 ms·

61.
by addisonj 6y ago
StreamNative | Remote, Full-Time | https://streamnative.io StreamNative are the main developers behind Apache Pulsar. We are looking for multiple different roles to join our team to help us build and maintain our rapidly growing
62.
by addisonj 6y ago
Hey, I work on Pulsar, will try and answer this :) Topics (actually bundles of topics, called bundles) are what is assigned to Brokers. Topic assignment is dynamic, so when a new broker is added, the system will try and shed load from the b
63.
by addisonj 6y ago
This is definitely an area where Pulasr is trying to improve, getting started is not easy. That said, the progress I have seen in the last year since I have been involved with it is really promising.
64.
by addisonj 6y ago
Pulsar offers a standalone mode, which runs everything inside a single JVM and you can just backup the data files. Pulsar also can be made to run with multiple components, but not distributed.
65.
by addisonj 6y ago
As someone who has ran a number of messaging systems in production, this is what my current take is in general: If you are moving to a more "event-sourced" architecture, usually two main concerns (beyond basic operational stuff of
66.
by addisonj 6y ago
Agreed. You aren't going to get this to work with Wikipedia. But I could see some utility on the small scale (if it weren't easy to break with conflicts). A Discourse install for local discussion in an oppressive regime run across
67.
by addisonj 6y ago
Hrm.. interesting idea! The tl:dr; appears to be "we built a blockchain based query logger" where essentially the consensus appears to just be that it is valid SQL, and defines an ordering of the statements. Then once everyone agr
68.
by addisonj 7y ago
Pulsar docs indicate "millions" of topics but IDK what 50 million would look like but from what I know I would be a bit nervous about it :) It might be worth chatting with Pulsar devs on their slack community ( https://a
69.
by addisonj 7y ago
In this case, we use just straight VPC peering with a full mesh of all our regions. We may eventually migrate to being built on our VPN based mesh (we do that in other places) Bandwidth is certainly a concern and that is one of the nice bit
70.
by addisonj 7y ago
I am not familiar enough with either Yedis or your use case to make a recommendation, but I can say that Pulsar has a great set of features, particularly if you need long term retention, that make it very attractive. I also been impressed w
71.
by addisonj 7y ago
1. On a short time horizon, not as sure, back of the napkin, it took ~12 dev months (5 months with 2.5 people average on it). However, our cost per 1000 msgs/sec is much lower (like 1/4 the cost of Kinesis) so we fully expect th
72.
by addisonj 7y ago
I think the other reply captures most of it for core NATS, but we also looked at NATS Streaming a bit, but it seems to be pretty immature (though promising) and doesn't check all the boxes around integrations into the streaming ecosyst
73.
by addisonj 7y ago
We have a number of systems, some use SQS, some use Kinesis. Part of the draw of Pulsar is having one piece of tech that we can unify everything over and offer more baseline features, like infinite retention via storage offloading or Pulsar
74.
by addisonj 7y ago
Biggest pain points with Kinesis: - ordering is really hard, you don't get guaranted ordering unless you write one message at a time or do a lot of complexity on writes (see https://brandur.org/kinesis-order ) and the s
75.
by addisonj 7y ago
I have used (and deployed) rabbitmq in the past and really love it for pub/sub, but for our needs, we keep needing retention, particularly long retention that we process with Flink for computing views. Having one system to do both is g
76.
by addisonj 7y ago
Oh forgot one aspect about DR: for critical data, we can easily turn on geo-replication (with a single API call) and have that data now in another region purely for DR purposes (or for cross region use cases)
77.
by addisonj 7y ago
Biggest thing is that we find ourselves needing to retain this data for more than initial delivery and also for use cases where we want to use the data more like a log and need ordering guarantees. It isn't just our current SQS use cas
78.
by addisonj 7y ago
The main driver for Pulsar is that we have a number of different messaging use cases, some more "pub/sub" like and some that are more "log" like. Pulsar really does unify those two worlds while also being a ton more
79.
by addisonj 7y ago
In Pulsar, offsets are tracked by the service as part of the bookkeeper data (unless you use the reader API which is only really needed for advanced use cases like Flink), that means we just need to do DR for bookkeepers, which I touch on i
80.
by addisonj 7y ago
This is one of the main benefits of Pulsar is that because state is split between brokers and bookkeeper and bookkeeper doesn't need re-balanced (due to it's segment based architecture where you choose new bookies with each new se
81.
by addisonj 7y ago
for the k8s definitions, we started with the helm chart, rendered the template, and then moved it into kustomize, as that is our tool of choice ATM, IDK if I would recommend that approach for everyone (we expect we might move to helm v3 at
82.
by addisonj 7y ago
I just finished rolling out Pulsar to 8 AWS regions with geo-replication. Messages rates are currently at about 50k msgs/sec but still in the process of migrating many more applications. We run on top of kubernetes (EKS). It took about
83.
by addisonj 7y ago
Disclaimer: I work at Instructure, the company that makes the LMS Princeton is moving too. Opinions are my own, etc First things first, Blackboard is some of the worst of enterprise software and a huge part of why they have lost so much mar
84.
by addisonj 8y ago
Nor do I think you should, my only point is to say that people's FUD about Wayland often has roots in FOMO about not being able to use the new and shiny for their workflow and no clear path to when/how that will be fixed, regardle
85.
by addisonj 8y ago
Your work on sway and wlroots is phenomenal, but I feel like there is quite a gap here in terms of why people are concerned about Nvidia support. Yes, Nvidia are not great when it comes to OSS, but like it or not, for doing real work in ML,
86.
by addisonj 8y ago
Nice to see, this has felt like a gap in cloud offerings for a while... and the open source options have difficulties. From the little that was said, going to guess this uses something like Beringei ( https://code.fb.com/core
87.
by addisonj 8y ago
See this link for context: https://jaxenter.com/end-life-comes-early-jdk-8-140824.html Tl;Dr: with Oracle changing the release versioning of Java, LTS releases are now essentially a paid product. If you run a large business
88.
by addisonj 8y ago
I highly recommend https://github.com/codekitchen/dinghy for MacOS. It predates docker for Mac but offers most of the same features while mostly being "just docker".
89.
by addisonj 9y ago
It will be interesting to see what the subset is. I have been writing scala for 2 years, mostly in spark and flink and as much as I love parts of it, overall, the language is just not the most approachable. Some of the features are super
90.
by addisonj 9y ago
I live and work in Utah, which, for not completely clear to me reasons, is home to a lot of MLMs. Nu Skin, Usana, Xango, doTerra, Young Living, Jamberry (and those are the ones I can just name off the top of my head) are all headquartered h

More ›