7 ms·

Show HN: Minibatch – Python stream processing

50 points by miraculixx 7y ago

miraculixx 7y ago
Scenario: you really have a streaming use case, but you don't want to deal with the complexity and overhead of some of the other streaming frameworks like Apache Spark or Flink. If so minibatch is for you
- tudelo 7y ago
  do you have a list of features? For example, implemented windowing type? Triggering?
  - miraculixx 7y ago
    good feedback, updated the README
ewhauser421 7y ago
Why not Apache Beam? It doesn’t require Spark or Flink
- ewhauser421 7y ago
  (Beam has a DirectRunner which is just in memory)
- miraculixx 7y ago
  Thanks for the question. Beam has a more complex execution model and AFAIK also needs some executor environment like Spark to really parallelize workloads. Given a mongodb that all producers and consumers can attach to, minibatch runs anywhere.
kstrauser 7y ago
It lost me at "for humans". I'm not even kidding: it's trite, demeaning to every similar project (with the implication that everything else is obviously for weirdos, not like you and me, amirite?), and shows a kind of naivete (everything else is overly complicated, lol, and I have no idea why that would be!). It was momentarily cute the first time I saw it but that faded quickly. By the 100th "for humans" project, it had become a distinct code smell.
- thiagomgd 7y ago
  Agree! I just assume everything posted here is "for humans". So... what's the differential here?
- throwaway55554 7y ago
  I agree! Anything that uses "Modern" does the same for me as well.
- proc0 7y ago
  People with depth of knowledge that can use advanced tools are also human!
- pryelluw 7y ago
  There's another thread right now on HN's front end asking about online toxicity. Since you've decided to openly mock this project and it's maintainers, I've decided it's fair to remind you about your humanity. Why do projects insist on labeling themselves as "for humans"? That makes no sense! Libraries have historically been terse and hard to understand. It's why stackoverflow exists. It allows those with more experience help others. Labeling a library as "for humans" means that special consideration was made to make it easier to grasp. That might hide some of the complexity. It could also limit the scope of the library's abilities. But it puts people and the UX first. Is this the correct approach? We should have simple and complex libraries. For every "for humans" library there should be a more complex one. And vice versa. Be human. Emphatic. Quick to praise. Slow to condem. Imagine it was you announcing a similar library. How would you feel to be openly ridiculed in such forum?
  - kstrauser 7y ago
    I didn't ridicule anyone. I did criticize what I see as an obnoxious trend in software labeling and I stand by that. But if you re-read my post, I didn't say one thing bad about the project itself, and didn't mention the author at all. My comment was entirely scoped to the "for humans" description. I emphatically insist that it's absolutely OK to lambast industry trends, and that's what I did here. I'm not sure how you inferred ridicule from that.
AndrewKemendo 7y ago
Kafka-python does python native streaming already. Why would I use this instead?
kissgyorgy 7y ago
For God's sake, forget the "for Humans" already! It is very agressive communication. EVERY library should be for HUMANS and the goal should be simple, easy-to-use APIs.
woile 7y ago
Interesting I've started a project with a friend to build a simple toolkit for stream processing https://github.com/python-streaming/dam https://github.com/python-streaming/dam I'm happy to see the streaming ecosystem is getting bigger in python, Java has a lot of market share
dennisy 7y ago
Do you have any scale or performance metrics?
- miraculixx 7y ago
  no benchmarks yet unfortunately. While it should scale O(n) in the number of streams, there will be limitations to scale a single stream because the stream processing functions by default are executed synchronously (pending enhancement).

50 points by miraculixx 7y ago

Scenario: you really have a streaming use case, but you don't want to deal with the complexity and overhead of some of the other streaming frameworks like Apache Spark or Flink. If so minibatch is for you

tudelo 7y ago
do you have a list of features? For example, implemented windowing type? Triggering?
- miraculixx 7y ago
  good feedback, updated the README

ewhauser421 7y ago

Why not Apache Beam? It doesn’t require Spark or Flink

ewhauser421 7y ago
(Beam has a DirectRunner which is just in memory)
miraculixx 7y ago
Thanks for the question. Beam has a more complex execution model and AFAIK also needs some executor environment like Spark to really parallelize workloads. Given a mongodb that all producers and consumers can attach to, minibatch runs anywhere.

kstrauser 7y ago

It lost me at "for humans". I'm not even kidding: it's trite, demeaning to every similar project (with the implication that everything else is obviously for weirdos, not like you and me, amirite?), and shows a kind of naivete (everything else is overly complicated, lol, and I have no idea why that would be!). It was momentarily cute the first time I saw it but that faded quickly. By the 100th "for humans" project, it had become a distinct code smell.

thiagomgd 7y ago
Agree! I just assume everything posted here is "for humans". So... what's the differential here?
throwaway55554 7y ago
I agree! Anything that uses "Modern" does the same for me as well.
proc0 7y ago
People with depth of knowledge that can use advanced tools are also human!
pryelluw 7y ago
There's another thread right now on HN's front end asking about online toxicity. Since you've decided to openly mock this project and it's maintainers, I've decided it's fair to remind you about your humanity. Why do projects insist on labeling themselves as "for humans"? That makes no sense! Libraries have historically been terse and hard to understand. It's why stackoverflow exists. It allows those with more experience help others. Labeling a library as "for humans" means that special consideration was made to make it easier to grasp. That might hide some of the complexity. It could also limit the scope of the library's abilities. But it puts people and the UX first. Is this the correct approach? We should have simple and complex libraries. For every "for humans" library there should be a more complex one. And vice versa. Be human. Emphatic. Quick to praise. Slow to condem. Imagine it was you announcing a similar library. How would you feel to be openly ridiculed in such forum?
- kstrauser 7y ago
  I didn't ridicule anyone. I did criticize what I see as an obnoxious trend in software labeling and I stand by that. But if you re-read my post, I didn't say one thing bad about the project itself, and didn't mention the author at all. My comment was entirely scoped to the "for humans" description. I emphatically insist that it's absolutely OK to lambast industry trends, and that's what I did here. I'm not sure how you inferred ridicule from that.

AndrewKemendo 7y ago

Kafka-python does python native streaming already. Why would I use this instead?

kissgyorgy 7y ago

For God's sake, forget the "for Humans" already! It is very agressive communication. EVERY library should be for HUMANS and the goal should be simple, easy-to-use APIs.

woile 7y ago

Interesting I've started a project with a friend to build a simple toolkit for stream processing https://github.com/python-streaming/dam https://github.com/python-streaming/dam I'm happy to see the streaming ecosystem is getting bigger in python, Java has a lot of market share

dennisy 7y ago

Do you have any scale or performance metrics?

miraculixx 7y ago
no benchmarks yet unfortunately. While it should scale O(n) in the number of streams, there will be limitations to scale a single stream because the stream processing functions by default are executed synchronously (pending enhancement).