5 ms·
Apache Hop 2.0
- rektide 4y agoOoh Drools plugins, for rule based event-processing. Neat. Hope I can find some examples! I havent used Airflow, but my impression is this fits a similar role. That itcs built atop good tech like Apache Beam & can use things like Flink is, in my book, a nice win.
- cmcconomy 4y ago"What is HOP?" https://hop.apache.org/manual/latest/getting-started/hop-what-is-hop.html https://hop.apache.org/manual/latest/getting-started/hop-wha...
- chrisweekly 4y ago> "VISUAL DESIGN AND METADATA > Apache Hop, short for Hop Orchestration Platform, is a data orchestration and data engineering platform that aims to facillitate all aspects of data and metadata orchestration. Hop lets you focus on the problem you’re trying to solve without technology getting in the way. Simple tasks should be easy, complex tasks need to be possible. > Hop allows data professionals to work visually, using metadata to describe how data should be processed. Visual design enables data developers to focus on what they want to do instead of how that task needs to be done. This focus on the task at hand lets Hop developers be more productive than they would be when writing code."
- Simon_O_Rourke 4y agoThanks for expanding that, it reads like it's some Airflow competitor. Would be curious how it handles all the authentication management for the various pipeline elements.
- jjtheblunt 4y agoI'm misunderstanding how so many Apache hosted projects P let someone focus on X without Y getting in the way, totally ignoring the complexity of introducing P and altering everything to align with P, thereby forbidding focus on X. Are these really often useful?
- ATsch 4y agoAs far as I can tell, the ASF is just where many companies send their failed Java projects for palliative care.
- waynesonfire 4y ago"Hop initially (late 2019) started as a fork of the Kettle (Pentaho Data Integration)."
- mason55 4y agoWow. PDI is one of the worst pieces of software I've ever used. Possibly only second to Pentaho Report Designer. From looking at the Apache HOP docs, it doesn't look like they have changed the UI much (if at all). I wonder if they at least made it less buggy.
- waynesonfire 4y agoI'd be curious how this contrasts with apache nifi.
- mring33621 4y agopotato, potahto
- deleted 4y ago[deleted]
- arthurcolle 4y agoWow I was expecting this to be the 463rd distributed computing streaming framework since its Apache. Shocked that its not
- mi_lk 4y agotangent - what is it about Apache or big data that the associated softwares are mostly written in Java?
- zekrioca 4y agoGood question, but maybe due to Java's stability and portability.
- oaiey 4y agoJava, like .NET, are just solid application platforms which are statically typed and their performance is good. Java has a history in big systems for soon 30 years. Rust, Python and Go are just not there yet. Rust is too low level, Python is not statically typed and will always suffer performance wise and Go ... I is a youngster :). And .NET is always not everyone's free choice. And Apache, well they just liked Java for their applications. They started with some C/C++ code but then quickly aggregated a lot of Java tech.
- ojhughes 4y agoMapReduce and HDFS were written in Java and they paved the way for a lot of the other big data tools
- erik_seaberg 4y agoI get the impression that Google used C++ (their dialect) for the first MapReduce system, while open source Hadoop and HDFS came later in Java.
- frellus 4y agoAside from this platform, which I've never heard about until now, I'm wondering what others are using in the workflow orchestration space? I'd assume Airflow is the most prevalent, but there's also Argo getting quite a bit of momentum lately.
- yamrzou 4y agoIn my previous job, I used Dagster. It has served us well. See my comment about it here: https://news.ycombinator.com/item?id=28803117 https://news.ycombinator.com/item?id=28803117
- kodablah 4y agohttps://temporal.io/ https://temporal.io/ (disclaimer: I work for Temporal on the Go SDK and upcoming Python SDK)
- wharfjumper 4y agoOooh I hadn't realised there was a Python SDK under development - will take a look.
- ricklamers 4y agoWe’re building Orchest (https://github.com/orchest/orchest https://github.com/orchest/orchest) tldr; a GUI for Argo
- adamzegelin 4y agoCadence: https://cadenceworkflow.io https://cadenceworkflow.io
- kaelin 4y ago
- abrazensunset 4y ago"I want to write my orchestration in Python and I'm comfortable hosting my own compute" -> Prefect (lightweight) or Dagster (heavier but featureful) "My team already knows Airflow and/or I want to pay Astronomer a lot of money" -> Airflow "I love YAML and everything is on k8s anyway" -> Argo "I just want something that works out of the box and don't want to host my own compute" -> Shipyard, maybe Orchest "I want a more flexible, generic workflow engine and don't care about writing orchestration in Python" -> Temporal/Cadence "I am very nostalgic" -> Azkaban, Oozie, Luigi "I love clunky Java solutions to data problems" -> Nifi et al "I like to pay for half-managed solutions and late upgrades to a first-generation technology" -> AWS/GCP hosted Airflow options "I am on AWS and it doesn't need to be complicated" -> AWS Step Functions
- tiffanyh 4y agoSlightly OT: what library do people use as the frontend web GUI for workflow orchestration? E.g. a drag’n drop web GUI workflow library like pictured below? https://customer.io/wp-content/uploads/2021/06/use-case-onboarding.svg https://customer.io/wp-content/uploads/2021/06/use-case-onbo...
- ricklamers 4y agoWe’ve rolled our own but https://reactflow.dev https://reactflow.dev is pretty neat and OSS.
- tiffanyh 4y agoThis is perfect. Thanks so much. Anyone know of others?
- Tostino 4y agoHow does this compare with Apache DolphinScheduler? That seems to fit the orchistration / workflow scheduling role pretty well, and seems like the next iteration of Airflow... Not quite getting how this compares, and not finding anyone directly comparing them on Google (though that's been less reliable lately).
- mbrodersen 4y agoWorkflow applications gets reinvented all the time (there are hundreds of them out there plus many “standards”). However I have never seen a successful usage of workflow applications in industry. So my question is if anybody has any good examples of workflow applications being more successful than “normal” business applications?
- nojito 4y agopower automate, airflow, etc.
- PeterStuer 4y agoBuilt an automated credit decision support system for a major financial services institution that used workflows. We needed masses of legacy data systems integration, rule based descisions and human task coordination. This system was built with Microsoft's Biztalk around 2006. It performed very well in production, but Biztalk had quite a few gotchas that needed to be creatively worked around in development.
- lovelearning 4y ago(Just a feedback for the project's tech writer) I always include a "What problem does this solve?" and screenshots in my docs. I think they help people understand a project better. Here too, I understood Hop's purpose only after seeing the screenshots on secondary pages like https://hop.apache.org/manual/latest/getting-started/hop-gui-workflows.html https://hop.apache.org/manual/latest/getting-started/hop-gui.... Abstract statements like "aims to facilitate all aspects of data and metadata orchestration" in the front page, or even in the "What is Hop?" doc, didn't help.
- sdoering 4y agoThanks for sharing the page. I was wondering what hop was and if it could potentially be a solution for a current problem I am facing. Now at least I know a bit more.
- zasdffaa 4y agoI too am getting tired of this shite. This disease afflicts corporate product descriptions to the point I just use wikipedia to get information on WTF £££expensive product does, and it's now metastasizing to free software. "Apache Hop, short for Hop Orchestration Platform, is a data orchestration and data engineering platform that aims to facillitate all aspects of data and metadata orchestration. Hop lets you focus on the problem you’re trying to solve without technology getting in the way" What is 'data orchestration'? Ditto 'data engineering platform'? 'facillitate all aspects of data and metadata orchestration' What the hell does this even mean. 'Hop lets you focus on the problem you’re trying to solve' so what problem do you think I'm trying to solve? It's just so bizarre, it's like language meaning has separated from language itself like layers of plywood left in the rain. And there is no wikipedia page to help out.
- bradknowles 4y agoSo: 1 Corinthians 13: "Faith, Hop and Charity, and the greatest of these is Hop." Does that sound about right?