15 ms·

Apache Flink

115 points by 0xedb 8y ago

ckdarby 8y ago
Does anyone have experience running in production? If you have an article about using it or slides please send links.
- wenc 8y ago
  Not sure why the Flink github site got linked. Here's a list of companies using it in production, with links to details. https://flink.apache.org/poweredby.html https://flink.apache.org/poweredby.html
- zbentley 8y ago
  > Does anyone have experience running in production? We do: https://klaviyo.tech/scaling-klaviyos-real-time-analytics-system-with-stream-processing-4b3bb87cd6b5?gi=da76fad9e49f https://klaviyo.tech/scaling-klaviyos-real-time-analytics-sy...
- rehevkor5 8y ago
  Yes. No public material though, sorry.
- deleted 8y ago
  [deleted]
- withhighprod 8y ago
  Alibaba, earth scale https://medium.com/@alitech_2017/alibaba-blink-real-time-computing-for-big-time-gains-707fdd583c26 https://medium.com/@alitech_2017/alibaba-blink-real-time-com...
meritt 8y ago
Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about the non-Apache stream-processing frameworks out there. * Apache Flink is an open source stream processing framework * Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data. * Apache Storm is a distributed stream processing computation framework * Apache Samza is an open-source near-realtime, asynchronous computational framework for stream processing * Apache Spark is an open-source distributed general-purpose cluster-computing framework. * Apache Apex is a YARN-native platform that unifies stream and batch processing. * Apache Kafka is an open-source stream-processing software platform
- virgilp 8y ago
  > Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. Well, no, you went too far. Kafka is primarily used for communication & data transport, by most people (can be used in other ways, and it has the KafkaStreams library that enables you do to some computation on said data - but it is, primarily, a transport & communication mechanism; also maybe storage, if you squint right) Spark and Flink might be similar on first sight, but if you look a bit closer you realize Spark is primarily geared towards batch workloads, and Flink towards realtime. Sure, you can do micro-batch in Spark and pretend that's realtime stream processing, but the focus of it is fairly clear - as is the focus of Flink. So both have legitimate rights to exist. I'm not sure about the others, haven't used them. There may be indeed consistent overlap - but I'm sure they are different approaches. What's wrong with that?
  - meritt 8y ago
    On the contrary, I didn't go far enough. I didn't talk about Apache Gearpump, NiFi, Beam, Ignite, or Trident. I know there are subtle differences in each specific technology, and that's probably the same justifications used to support the thesis when building yet another very similar framework. > What's wrong with that? I believe it drastically reduces adoption of these tools because many of us avoid what appears to be bandwagon technologies as we don't want to consciously add layers of future technical debt when a majority of these projects will be abandoned.
    - virgilp 8y ago
      I said "you went to far" to claim Kafka, Spark and Flink do basically the same thing. It's reasonable to use all 3 of them in the same team - so they clearly don't do the same thing. > I believe it drastically reduces adoption of these tools As does any competition. Tons of smartphone makers = less adoption for any one of them - and many will close down. Still not a bad thing.
      - rakoo 8y ago
        > As does any competition. Those are open-source frameworks, which is antithetical to competition. The whole point of open-source is that the contributions of one entity can benefit everyone, while in this situation, having so many frameworks "dilutes" the efforts because the same problems need to be solved in each framework. Smartphone makers aren't here to share their technical advances with everyone, they're here to make money. I'm not saying we should purposefully agree on killing all but one framework, but the ideal situation is if one or two come out at the top as being just the best in their category so that everyone just use them, and all efforts converge towards those.
_ZeD_ 8y ago
"Prerequisites for building Flink: Unix-like environment (we use Linux, Mac OS X, Cygwin) Java 8 (Java 9 and 10 are not yet supported)" Sigh is too much to ask for proper crossplatform support? And when the hell they will add support for recent versions of Java????
- skrebbel 8y ago
  Make a PR
  - IshKebab 8y ago
    I wish people would stop making this predictable and tedious comment that adds nothing to the conversation.
    - 11sdmb8m 8y ago
      "Make a PR" actually means "This seems not to be relevant enough for the current maintainers to do it, and if it feels important to you, you should write it yourself and post a patch", which is a good answer.
      - humanbeinc 8y ago
        No, it's not and you know it
      - vageli 8y ago
        > "Make a PR" actually means "This seems not to be relevant enough for the current maintainers to do it, and if it feels important to you, you should write it yourself and post a patch", which is a good answer. So we just assume that a desire to do something translates to an _ability_ to do it?
        lmm 8y ago
        If you want it done and don't have the skills you can hire someone who does.
CSDude 8y ago
AWS lets you uplod Flink programs to process to Kinesis streams, and Google Cloud also has support for Apache Beam
- rehevkor5 8y ago
  Yeah, or any other input. I don't think it's tied explicitly to Kinesis. This is definitely easier than other ways to deploy! EMR also has Flink as an option.
deleted 8y ago
[deleted]
withhighprod 8y ago
It’s actually a huge hype in China already as Alibaba widely use it in production https://medium.com/@alitech_2017/alibaba-blink-real-time-computing-for-big-time-gains-707fdd583c26 https://medium.com/@alitech_2017/alibaba-blink-real-time-com... Downside:Alibaba already acquired the company behind it
lenticular 8y ago
Controversial opinion here, but all of these distributed streaming architectures are massively overused. They certainly have their place, but you probably don't need them. I see it all the time with ML work. You wind up using a cluster to overcome the memory inefficiency of Spark, when you could have just used a single machine. For example, I've done huge graph clustering models on a single machine just by being smart about memory consumption. It would have taken an enormous and expensive Spark cluster.
- manigandham 8y ago
  We have also seen this. The huge memory capacities now offered in the cloud makes single node processing very capable with entire TB datasets fitting into RAM, and running quickly enough to offset the hourly cost. Spark clusters are more efficient though for longer-running or continuous background processes.
  - riku_iki 8y ago
    And spark also has some other goodies: integration with storages, monitoring, fault tolerance, common algorithms implemented.
- bunderbunder 8y ago
  This has been my experience, too. I worked at just one place that had a really good handle on high-volume, high-velocity streaming data, and they didn't use Flink or Storm or Kafka or anything like that. They mostly just used the KISS principle and a protobuf-style wire format.[1] There is definitely a point where these sorts of scale-out-centric solutions are unavoidable. Short of that point, though, they're probably best avoided. [1]: (It's truly amazing how many CPU cycles you can reclaim just by removing branch instructions from your message deserialization code.)
  - lenticular 8y ago
    That's a smart way of doing it. In much ML work, I've found that the hash trick can be used to train many more things in a constant memory, out-of-core fashion than most people think.
meatyapp 8y ago
What's the easiest way to get started with trying Apache Kafka/Spark/Flink on the cloud? If I want to try out Redis there's RedisLabs, CloudAMQP for RabbitMQ, Compose for Postgres/Redis/RabbitMQ, offerings like Google Cloud SQL/MemoryStore and AWS RDS/ElastiCache, etc. Where do I go for some easy Apache deployments?
- deepsun 8y ago
  Google cloud has "one-click" installation of integrated 3rd-party solutions. Never tried that though.

Apache Flink

115 points by 0xedb 8y ago

ckdarby 8y ago

Does anyone have experience running in production? If you have an article about using it or slides please send links.

wenc 8y ago
Not sure why the Flink github site got linked. Here's a list of companies using it in production, with links to details. https://flink.apache.org/poweredby.html https://flink.apache.org/poweredby.html
zbentley 8y ago
> Does anyone have experience running in production? We do: https://klaviyo.tech/scaling-klaviyos-real-time-analytics-system-with-stream-processing-4b3bb87cd6b5?gi=da76fad9e49f https://klaviyo.tech/scaling-klaviyos-real-time-analytics-sy...
rehevkor5 8y ago
Yes. No public material though, sorry.
deleted 8y ago
[deleted]
withhighprod 8y ago
Alibaba, earth scale https://medium.com/@alitech_2017/alibaba-blink-real-time-computing-for-big-time-gains-707fdd583c26 https://medium.com/@alitech_2017/alibaba-blink-real-time-com...

meritt 8y ago

Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about the non-Apache stream-processing frameworks out there. * Apache Flink is an open source stream processing framework * Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data. * Apache Storm is a distributed stream processing computation framework * Apache Samza is an open-source near-realtime, asynchronous computational framework for stream processing * Apache Spark is an open-source distributed general-purpose cluster-computing framework. * Apache Apex is a YARN-native platform that unifies stream and batch processing. * Apache Kafka is an open-source stream-processing software platform

virgilp 8y ago
> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. Well, no, you went too far. Kafka is primarily used for communication & data transport, by most people (can be used in other ways, and it has the KafkaStreams library that enables you do to some computation on said data - but it is, primarily, a transport & communication mechanism; also maybe storage, if you squint right) Spark and Flink might be similar on first sight, but if you look a bit closer you realize Spark is primarily geared towards batch workloads, and Flink towards realtime. Sure, you can do micro-batch in Spark and pretend that's realtime stream processing, but the focus of it is fairly clear - as is the focus of Flink. So both have legitimate rights to exist. I'm not sure about the others, haven't used them. There may be indeed consistent overlap - but I'm sure they are different approaches. What's wrong with that?
- meritt 8y ago
  On the contrary, I didn't go far enough. I didn't talk about Apache Gearpump, NiFi, Beam, Ignite, or Trident. I know there are subtle differences in each specific technology, and that's probably the same justifications used to support the thesis when building yet another very similar framework. > What's wrong with that? I believe it drastically reduces adoption of these tools because many of us avoid what appears to be bandwagon technologies as we don't want to consciously add layers of future technical debt when a majority of these projects will be abandoned.
  - virgilp 8y ago
    I said "you went to far" to claim Kafka, Spark and Flink do basically the same thing. It's reasonable to use all 3 of them in the same team - so they clearly don't do the same thing. > I believe it drastically reduces adoption of these tools As does any competition. Tons of smartphone makers = less adoption for any one of them - and many will close down. Still not a bad thing.
    - rakoo 8y ago
      > As does any competition. Those are open-source frameworks, which is antithetical to competition. The whole point of open-source is that the contributions of one entity can benefit everyone, while in this situation, having so many frameworks "dilutes" the efforts because the same problems need to be solved in each framework. Smartphone makers aren't here to share their technical advances with everyone, they're here to make money. I'm not saying we should purposefully agree on killing all but one framework, but the ideal situation is if one or two come out at the top as being just the best in their category so that everyone just use them, and all efforts converge towards those.

_ZeD_ 8y ago

"Prerequisites for building Flink: Unix-like environment (we use Linux, Mac OS X, Cygwin) Java 8 (Java 9 and 10 are not yet supported)" Sigh is too much to ask for proper crossplatform support? And when the hell they will add support for recent versions of Java????

skrebbel 8y ago
Make a PR
- IshKebab 8y ago
  I wish people would stop making this predictable and tedious comment that adds nothing to the conversation.
  - 11sdmb8m 8y ago
    "Make a PR" actually means "This seems not to be relevant enough for the current maintainers to do it, and if it feels important to you, you should write it yourself and post a patch", which is a good answer.
    - humanbeinc 8y ago
      No, it's not and you know it
    - vageli 8y ago
      > "Make a PR" actually means "This seems not to be relevant enough for the current maintainers to do it, and if it feels important to you, you should write it yourself and post a patch", which is a good answer. So we just assume that a desire to do something translates to an _ability_ to do it?
      - lmm 8y ago
        If you want it done and don't have the skills you can hire someone who does.

CSDude 8y ago

AWS lets you uplod Flink programs to process to Kinesis streams, and Google Cloud also has support for Apache Beam

rehevkor5 8y ago
Yeah, or any other input. I don't think it's tied explicitly to Kinesis. This is definitely easier than other ways to deploy! EMR also has Flink as an option.

deleted 8y ago

[deleted]

withhighprod 8y ago

It’s actually a huge hype in China already as Alibaba widely use it in production https://medium.com/@alitech_2017/alibaba-blink-real-time-computing-for-big-time-gains-707fdd583c26 https://medium.com/@alitech_2017/alibaba-blink-real-time-com... Downside:Alibaba already acquired the company behind it

lenticular 8y ago

Controversial opinion here, but all of these distributed streaming architectures are massively overused. They certainly have their place, but you probably don't need them. I see it all the time with ML work. You wind up using a cluster to overcome the memory inefficiency of Spark, when you could have just used a single machine. For example, I've done huge graph clustering models on a single machine just by being smart about memory consumption. It would have taken an enormous and expensive Spark cluster.

manigandham 8y ago
We have also seen this. The huge memory capacities now offered in the cloud makes single node processing very capable with entire TB datasets fitting into RAM, and running quickly enough to offset the hourly cost. Spark clusters are more efficient though for longer-running or continuous background processes.
- riku_iki 8y ago
  And spark also has some other goodies: integration with storages, monitoring, fault tolerance, common algorithms implemented.
bunderbunder 8y ago
This has been my experience, too. I worked at just one place that had a really good handle on high-volume, high-velocity streaming data, and they didn't use Flink or Storm or Kafka or anything like that. They mostly just used the KISS principle and a protobuf-style wire format.[1] There is definitely a point where these sorts of scale-out-centric solutions are unavoidable. Short of that point, though, they're probably best avoided. [1]: (It's truly amazing how many CPU cycles you can reclaim just by removing branch instructions from your message deserialization code.)
- lenticular 8y ago
  That's a smart way of doing it. In much ML work, I've found that the hash trick can be used to train many more things in a constant memory, out-of-core fashion than most people think.

meatyapp 8y ago

What's the easiest way to get started with trying Apache Kafka/Spark/Flink on the cloud? If I want to try out Redis there's RedisLabs, CloudAMQP for RabbitMQ, Compose for Postgres/Redis/RabbitMQ, offerings like Google Cloud SQL/MemoryStore and AWS RDS/ElastiCache, etc. Where do I go for some easy Apache deployments?

deepsun 8y ago
Google cloud has "one-click" installation of integrated 3rd-party solutions. Never tried that though.