13 ms·
Pipes: A spiritual successor to Yahoo Pipes
- deleted 2y ago[deleted]
- dang 2y agoRelated: The architecture and software behind Pipes - https://news.ycombinator.com/item?id=22959013 https://news.ycombinator.com/item?id=22959013 - April 2020 (12 comments) Show HN: Early-stage Yahoo Pipes spiritual successor - https://news.ycombinator.com/item?id=14440993 https://news.ycombinator.com/item?id=14440993 - May 2017 (126 comments)
- DonHopkins 2y ago"Put down the Yahoo Pipe and step away from the keyboard!" ;)
- jimbobthrowawy 2y agoAlso: Yahoo Pipes End-Of-life Announcement - https://news.ycombinator.com/item?id=9660682 https://news.ycombinator.com/item?id=9660682 - June 2015 (178 comments)
- ReadCarlBarks 2y agoIt is on GitHub: https://github.com/pipes-digital/pipes https://github.com/pipes-digital/pipes.
- _gtly 2y agoI want to support this idea, though If you use their service it's free to play around with 3 feeds, but then it gets more pricey. Maybe worth it for some?https://www.pipes.digital/pricing https://www.pipes.digital/pricing
- yreg 2y agoThree free pipes sound reasonable for me. If you have more than three use cases in your life, then it's perhaps worth it to pay a few bucks.
- ninetyninenine 2y agoevery form of visual programming I've ever seen is a singular style of some of a box that does processing and a line that connects that box to another box.
- wild_egg 2y agoAnyone know of an example of any other style? Would love to see any half-baked experiments even. Seems like a space that could really use some innovation
- __MatrixMan__ 2y agoI don't have any code for this, but I always thought it would be better if the boxes represented data types and the arrows represented functions. Then you could use the arrow as a progress bar, and you'd have something that's a bit more focused on the inputs and outputs. Also, the data types could show up as a stack of boxes (like a deck of cards) so you can see when you've calculated a novel dataset (highlight new card on top the deck) or when you've recalculated a previous dataset (highlight the nth card). Such a view would resemble a commutative diagram (from category theory) and if you let the cards pile up for a while you could use their sizes to reason about which functions are bijective, which are good candidates for memorization, etc. When you want to know why a certain datum is the way it is, you can use the same view to trace it back to it's inputs (in the boxes-as-tasks mode, you end up zooming in on some imperative artifact... logs usually... And discerning the inputs is left as an exercise to the reader). I think this goes by the buzzword "data lineage" and it's usually this extra thing but I think it should be the main thing. Which is why I want to make the nodes into edges and the edges into nodes.
- deleted 2y ago[deleted]
- andyferris 2y agoThis is interesting. How do you visually identify a function with multiple inputs and outputs? I tend to imagine it as a bi-graph - with boxes for values and for functions, and arrows in between. The stack idea is interesting and can work here too. I could get behind saying "all functions have exactly one input value and one output value" but you really need to be able to construct and deconstruct structs/records/tuples to be able to e.g. pass two values to one function.
- jzemeocala 2y agoI wonder if we could pull off some wackiness with custom RSS feeds. Might have some utility in keeping an LLM constantly up to date as well.
- deleted 2y ago[deleted]
- akshayrajp 2y agoCan Zapier and similar tools be considered a spiritual successor to Yahoo Pipes too?
- ozten 2y agoYahoo pipes was amazing! My vague memory of how this ended last time was processing cost and lack of a business model.
- nhggfu 2y agothe twitter example link goes nowhere.
- yalogin 2y agoI think this has some value potentially now, but they should not use “pipes” to describe it. Yahoo pipes is dead and there is no point associating one self with that brand anymore. The concept is easy enough to explain without using that word.
- scq 2y agoWhat's wrong with the word "pipes"?
- ReadCarlBarks 2y agoPipes was created back when people were looking up alternatives for the Yahoo service. It isn't a new project.
- Towaway69 2y agoUnix pipes came before yahoo pipes (in fact they took their name from unix pipes). Unix pipes are alive and well, or are they also an association with a dead product?
- jumploops 2y ago> The default output format of a pipe is RSS. Do people still use RSS feeds? I work on a similar product[0], conceptually at least, to Pipes, and many of our users have asked for RSS feeds as output. Until now I’ve rationalized that as low priority, but maybe I’m wrong? [0] https://magicloops.dev https://magicloops.dev
- rtpg 2y agopeople totally use RSS feeds still. So many systems just make it by default, and I (and many other people) just use RSS readers for following blogs and whatnot. It's not a complicated thing to get setup! There's a bunch of details you can flub but at the end of the day it's a big thing of markup. The "good enough" thing is easy.
- onion2k 2y agomany of our users have asked for RSS feeds as output If your users are asking for it then it's locally popular regardless of how much use it gets on the general internet, and therefore it should be important to you. Local popularity (eg the market for your app) is significantly more important to product design than general popularity.
- JoshTriplett 2y agoExactly. The user base for "people who want Pipes" is likely to disproportionately want RSS as a way of generating data they can consume.
- gumby 2y agoRSS is my main interface to the web
- InsideOutSanta 2y agoIs there anyone who doesn't use RSS, whether it is as infrastructure behind the scenes (e.g. for a podcast or some kind of notification system) or explicitly in a feed reader?
- rcarmo 2y ago
- theshrike79 2y agon8n is a similar tool and can be self-hosted: https://n8n.io https://n8n.io And NodeRED is big in the home automation space: https://nodered.org https://nodered.org
- rapnie 2y agon8n "AI-native workflow automation". Wtf.
- yakito 2y agoI totally agree. What are they doing over there? It's funny because some of these companies are just jumping into trends, in this case, AI, probably to lure investors and users. They might be achieving the complete opposite result by doing so. If I didn't know about n8n from before and read the new headline on their homepage, I would have thought the tool was not useful for what I need.
- theshrike79 2y agoDunno what they mean about that :D Nothing in the system says "AI" to me, and I've self-hosted one for a while now
- ozarker 2y agoI love NodeRED, been running it in my homelab for a few years
- theshrike79 2y agoIt's one of the few situations where "visual programming" actually works really well. I can't grok the Home Assistant UI for creating any kind of complex automations (and I've been writing software for a living for 25 years), but NodeRED works perfectly for complex branching operations.
- dbacar 2y agoYou know, there is also Apache Nifi.
- onli 2y agoThis is my project :) Nice to see this here again! If there are questions, I'm ready to answer (I have a rather young baby here though that wanted attention while writing this, but she sleeps now, so I think we are good). Pipes saw some internal updates recently. It was fighting with some instability. My first measure was an internal re-architeture, on a small scale. Before, the data was transported as text from block to block. (Almost) Every block parsed the input, created an RSS feed object and then worked with that on the data, only to then output the created RSS feed as string. This was changed to just work with the RSS object directly and move that from block to block. That had some consequences for some existing pipes, mainly because it also changed when the input feeds gets normalized. As that did not help, next step was a server upgrade. I'm not sure whether it was directly the additional processing capacity or the dependency upgrades, but that one helped so far. Knock on wood. Also reconfigured the amount of threads and puma workers, that might have solved a bottleneck somewhere. I sadly never pinpointed exactly what made the server process stop before. I had played with a proper split between web frontend and pipes processing, but at least my approach was not viable. Worked nicely while developing it, immediately crashed with the production workload. And Pipes is not all that huge... I'll have to try again some time.
- Gys 2y agoWhat I am missing is a block to manipulate contents. For example, I might want to add an AI generated summary to certain feeds. Or if there is no image, add an alternative image. For those cases it could be useful to have a block that can post a feed item (or one selecteble field) to my custom url and add or replace that item with the result. Then I can make my own feed item manipulators for certain tasks. Thinking bigger, you might offer a (third party?) store of custom manipulators: 'add image', 'add summary', etc.
- onli 2y agoI like that. I think it would have to be all items that get POSTed one by one to a url to work with the current flow, but that should be equally useful, no? Pipes does have some building blocks to prepare that operation or work with the result, there is an extract block and an insert block (that works by xpath). But I was always missing a model for the request block that I remember from Yahoo Pipes, that's why there is only a feed and a generic download block so far. Post somewhere, replace the item (or the item's content?) with the answer, that might be it. Edit: Thinking a bit more about this, don't forget that you can already chain pipes. A service you can POST to can probably also consume an RSS feed, download it by itself. You can then have one pipe that creates the feed up to the point where you would want to have it be treated by something else, let that thing output its result, and in a second pipe download the output and create the final feed. I still think the request block to POST items is a good idea, and there are probably services that can't store the output for the second pipe as they'd just react directly to a POST, but that chainability might be worth to keep in mind.
- harinijan 2y agoInteresting to see the evolution of Visual programming across many tools. We are working on something similar[0] for low-code/no-code devs focusing on creation of APIs, backend tasks, and AI workflows. [0] https://buildship.com https://buildship.com
- freeqaz 2y agoThis is the code for a YC startup I ran in this space. Very similar. Build on AWS Lambda: https://github.com/refinery-labs/refinery https://github.com/refinery-labs/refinery I need to fix the docs because they are dead: https://web.archive.org/web/20200819221527/https://docs.refinery.io/getting-started/ https://web.archive.org/web/20200819221527/https://docs.refi... Yahoo Pipes was an inspiration though. It was similar to Heroku in many ways.
- smetj 2y agoI have no experience with this particular implementation but I do have experience with some important business process developed in Azure Logic Apps, which is kind of the same in spirit I believe. From what I have seen, these kind of frameworks facilitates the worst possible combination of factors: A solution designed, created and implemented by non-programmers, not using a programming language ending up in a production environment. One could argue this is a success in its own but I have only seen these kind of things hit their limits almost immediately after the initial POC and evolve into terrible tech debt. Just learn to program or script already. That being said, I do not want to disrespect the effort into building this and perhaps there is a place for these kind of solutions I yet have to experience.
- blowski 2y agoIf the PoC can be built by those with strong domain knowledge, providing the basis for a detailed spec, that seems like a win to me.
- smetj 2y agoYes, I agree with that.
- robertlagrant 2y agoThat would be a win, but then the hideous code becomes the spec, and it becomes a project ten times the size to reverse engineer the "spec" into an actual spec.
- hermitcrab 2y agoBut if you let them do it in Excel, it would be 10 times worse again.
- onli 2y agoFair enough I'd say :) The concern is understandable, but I honestly think Pipes is different. The operations Pipes can do are at the same time very generic and very targeted. It's stuff like "Download this url" and "Filter this feed for that word", and it gives you a selection of blocks like extract and insert with which you can manipulate the data in the pipe (or the feed, depending how you look at it) a bit more freely. There is surprisingly a lot that can be done with that, especially with a replace block that supports regex, but it's not a deep abstraction layer above things that are not mappable to rather small operations.
- nsonha 2y agokinda related but does anyone know anything similar to yql, in the sense that you can query everything using the same syntax?
- cranberryturkey 2y agoYahoo pipes was awesome. Way ahead of its time. I don’t know why yahoo shut it down.
- klaussilveira 2y agoFinally! Yahoo Pipes was so ahead of its time. The whole concept of microdata, RSS and wrangling data around in pipes was amazing and it is sad that it died. The web was so promising at that time.
- BaudouinVH 2y agotiny bit of feedback : I tried to sign-in with a non gmail address and did not receive the confirmation email. Switched to a gmail address and everything was smooth in the sign-in process.
- RyLuke 2y agoI see a lot of folks here asking about what happened to the original Yahoo Pipes. We had the same question, so we went and talked with a lot of the original team and wrote up the story[0]. We also made a fun mini-site that contains a lot of easter eggs (e.g. if you click on the "Memory Pipes" folder on the desktop, you'll see a bunch of candid photos of the original team circa 2007) [0] https://retool.com/pipes https://retool.com/pipes
- jjordan 2y agoThose shiny gumdrop buttons hit hard with the nostalgia. <3 Everything is so flat and boring now...
- paledot 2y agoBy the way, that page is entirely unusable without JavaScript enabled, apparently in service of a modal preloader? Retro vibes are nice, but let's leave Flash-based websites in the past.
- anothername12 2y agoBack in the day I had hundreds of pipes, sub-pipes for all kinds of stuff from porn to price gathering, comparisons, to banking. When Yahoo Pipes died so did my will to port all that to something else. If I had to do it all again in 2024, what’s a robust self hosted project like this?
- Towaway69 2y agoTry out nodered.org - it's more generalistic, so you can do a lot more but it can also handle XML and JSON parsing.
- gabigrin 2y agoVery cool project. A great demonstration that when you're useful you don't need a fancy interface! If anyone is interested in a more generic visual-programming language, check out Flyde - https://www.flyde.dev https://www.flyde.dev - an open-source visual-programming language I'm working on
- lolive 2y agoAt work, we use extensively Palantir Foundry and its [excellent] PipelineBuilder. I was reluctant at first with this low-code no-code stuff. But after porting custom python code, I must admit it covers 99% of the transformation needs we have. Plus it keeps a semantic all along, that is much more difficult to understand/maintain by code [especially in Python].
- shouyatf 2y agorss-funnel[0] is a self-hosted feed filtering pipeline with similar goal. [0]: https://github.com/shouya/rss-funnel https://github.com/shouya/rss-funnel disclaimer: rss-funnel is my own project.