20 ms·
Show HN: Early-stage Yahoo Pipes spiritual successor
- rjeli 9y agoDead (could not connect to server)
- onli 9y agoPlease try again. It should be up (works for me and http://downforeveryoneorjustme.com/www.pipes.digital http://downforeveryoneorjustme.com/www.pipes.digital supports that). Though I forgot scaling up the droplet it is running on before posting here...
- rjeli 9y agohuh, it fails immediately for me about 5/6 times on iOS Safari. weird
- onli 9y agoI'd be interested to know which blocks and functionality is missing to support the use cases you used Yahoo Pipes for! And of course general feedback is always welcome.
- insomniacity 9y agoNice. It's not very clear that you need to link the pipeline to the right-hand side. (I can't remember if you had to do that on Yahoo, it's been so long!). It's particularly a problem on a wide screen because the eye starts on the left, and I built my pipeline on the left and it took me a while to figure that out. But well done on scratching an itch that lots of people have had!
- onli 9y ago> I can't remember if you had to do that on Yahoo, it's been so long! I have a screenshot on my disk: https://imgur.com/a/TExPX https://imgur.com/a/TExPX - it had a bigger pipe output block at the bottom. I can see how the red circle alone is not enough, thanks!
- RRRA 9y agoLibrarians are gonna go wild over this...
- arjie 9y agoNice. Used to love Yahoo! Pipes What's your data model look like?
- onli 9y agoEvery pipe is a recursive list of blocks, like a tree. Basically, a pipe has a starting block that produces the pipe output. To do that it calls the process function of its input block, which again run their input blocks, ..., the feed blocks serve as tree leaves.
- pbiggar 9y agoCool product, and great to see this get traction! My new company is very much inspired by Yahoo pipes and making something similar. Would love to chat about your approach!
- onli 9y agoAbsolutely, I'll send you a mail
- michaelbuckbee 9y agoThis is cool, but I feel like Zapier, IFTTT and their open source equivalents have moved far past Yahoo Pipes. How do you see this stacking up?
- onli 9y agoI'm interested in the answer to this questions myself :) For me, this is about the ease of use of the visual programming interface, and the focus on feeds to enable the pipe concept. I do not see this in the same space as Zapier and IFTTT, but I might be wrong!
- wocram 9y agoAs far as I Kno, IFTTT only supports simple if a: then b triggers, you cannot have multiple inputs or outputs. Do zapier or others improve on this?
- deleted 9y ago[deleted]
- bryanh 9y agoYeah, Zapier has multi-step workflows https://zapier.com/learn/getting-started-guide/multi-step-zaps/ https://zapier.com/learn/getting-started-guide/multi-step-za....
- Muanh 9y agoThis doesn't include if true do A else do B logic right? I see they have filters, maybe this can be abused somehow?
- kilroy123 9y agoAgree Zapier is pretty awesome and pretty powerful. (I have no affiliation, just a happy customer)
- LoSboccacc 9y agoThey do a fraction of the data manipulation options available from pipes and they require connectors being custom developed, plus they limit you on result delivery, which cannot be pulled.
- bergie 9y agoInteresting, have to take a proper look tomorrow. We've done similar things with NoFlo (https://noflojs.org https://noflojs.org)
- zeptomu 9y agoI've some experience in visual programming languages (both professional and in my studies) and I advise against the graph-form where you connect nodes and edges. This does not mean that text is the only way to implement programs (and even if you connect nodes, you're still programming), but maybe a good local optimum is in-between, e.g. interactive shells like Jupyter's Notebook and the Mathematica interface. I know there is LabView, the Blender-Editor and AFAIK some Unreal-Engine tool that uses this model, but bigger programs seem really incomprehensible to me. It's always easy to show-case 5 programs in these node-&-edge editors, but I do not think it is the best approach for visual languages, as one should not under-estimate the layout problems and how to represent information in boxes. So, I am all for new ideas in visual programming, but I am not sure if the "free canvas" approach works.
- phreeza 9y agoI wonder if there is a way to get the best of both worlds? What do you make of Blockly for example? https://developers.google.com/blockly/ https://developers.google.com/blockly/
- zeptomu 9y ago> I wonder if there is a way to get the best of both worlds? [...] I believe so. Furthermore high-level languages with syntax-highlighting (think Haskell, Python, Elm, etc.) in state-of-the-art text editors are in a better position than the "free-canvas" node-edge editors to provide the base for such a widely used language. Blockly goes into the right direction, but has a childish touch and I do not think the keywords should have a background with padding.
- fiatjaf 9y agoBlockly (and all the similar visual programming languages) are so awful! They're basically writing, but much harder. "set", "repeat", "do"? What?
- phreeza 9y agoI think the advantage is that they make syntax errors impossible and are more discoverable, because the available blocks can be seen in the sidebar. Of course once you have mastered a language, it is more efficient to write text, but for a beginner it can make the entry less daunting.
- leguan 9y agoSimilar IoT oriented http://www.zenodys.com http://www.zenodys.com
- kod3r 9y agoHello there, never heard of zenodys, what is actually? Any experience with it?
- sx6 9y agoWe do for our rapid projects. Pretty nice for connectivity, haven't tried for production yet.
- speedoo 9y agoI tried zenodys awhile ago. Looks advanced, still bit tight on documentation imho
- tlautar123 9y agolooks great, but i cannot find any documentation
- spacialz 9y agothis looks very handy, especially with options to expand it with your code
- mgkimsal 9y agowould like to suggest a couple of "starter pipes" - samples to more easily illustrate some possibilities.
- anonfunction 9y agoI used to use yahoo pipes for a few twitter bots I had. There was an rss to twitter webapp that I hooked up and it was fully automated for years before it stopped working. I didn't see the thin red circle on the right, which I now understand to be the output. I almost gave up before realizing my mistake.
- hehheh 9y agoNeat. Any chance of adding JSON processing where you can pick a key (ala `jq`[0]) and then combine it with RSS or other JSON or what have you? [0] https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
- onli 9y agoI plan on enabling filtering via xpath, and in parts jq is like xpath for json. But I'm not sure whether you mean something more with "then combine it with RSS or other JSON or what have you"? Merging generic input into a feed at a chosen element? Side remark: Pipes uses https://github.com/feedparser/feedparser https://github.com/feedparser/feedparser to normalize atom and rss feeds, and that gem just got support for the jsonfeed format. Untested, but worth a shot if you are looking for json input.
- arjie 9y agoShould you feel like, there's also JMESPath for JSON http://jmespath.org http://jmespath.org
- fiatjaf 9y agoI've made something[1] that used "jq for everything" in the past. RequestHub was initially a way to connect webhooks from one service to API calls to another services, using jq scripting for all the customization. Later I realized it could be used for processing anything (even text!), not only webhooks, but it was too late. Maybe someday I'll do a Yahoo Pipes revival that will just use jq for scripting inside the boxes. [1]: http://archive.is/nGyH3 http://archive.is/nGyH3, https://github.com/fiatjaf/requesthub.xyz https://github.com/fiatjaf/requesthub.xyz
- gingerlime 9y agoWhy did you discontinue requesthub?
- fiatjaf 9y ago
- Toast_ 9y agoDo you plan on supporting scraping content via css selectors/xpath/regex?
- RandomBookmarks 9y agoOr maybe integrating visual selection and OCR like Kantu: https://a9t9.com/kantu/scraping#ocr https://a9t9.com/kantu/scraping#ocr
- Toast_ 9y agoLooks cool, reminds me of portia. https://scrapinghub.com/portia/ https://scrapinghub.com/portia/
- onli 9y agoI'm not sure. There are a couple of existing services that do that already, and which give you a rss feed you could then import here. But if it gets requested I would look whether it is doable in the constraints of this system - at least I remember yahoo pipes supported it, that's a plus.
- Toast_ 9y ago>I remember yahoo pipes supported it, that's a plus. Yup, pretty much why I used pipes to begin with. I think being able to scrape/manipulate/output data, while being able to keep it private, would be a fantastic service. Looks good so far!
- onli 9y agoI added a block to download a page (instead of a feed), another block for extracting content via css selector or xpath, and a feed builder block to later combine those (but the extract block already creates a feed one could use as pipe output). If more is needed please say so, I am now convinced this fits well to this page.
- 9y ago
- mortadelegle 9y agoVery interesting, I do think that visual programming tends to work better when constrained to a small niche of computing, that's the hypothesis behind https://github.com/AlvarBer/persimmon https://github.com/AlvarBer/persimmon
- onli 9y agoWish I had that when I was still doing machine learning. Looks nice!
- unityByFreedom 9y agoNice. Anyone else here ever use Ab Initio? It's an ETL tool for transferring data between databases and manipulating in the process. Seems very similar. I've always wanted to recreate something with those ideas because it was a very efficient cross between visualizing code and writing it. This looks pretty similar although for different data sources.
- yownie 9y agocouple of code examples would be nice
- infectoid 9y agoWhere I work we have a custom made system that was inspired by Yahoo Pipes. We are in the slow process of rewriting it. Also this reminds me of and IoT solution I was shown recently. https://nodered.org/ https://nodered.org/
- nthcolumn 9y agoPipes was awesome. Would it have killed them to just let it go free instead? Yahoo engineering did a lot cool of stuff.
- scottmcleod 9y agoI really loved Yahoo Pipes - Was ahead of its time if you ask me..Built some very cool automations on top of this. Basically was IFTTT and Zapier well before them.
- beardicus 9y agoI'm surprised nobody has mentioned Node-RED yet: http://nodered.org/ http://nodered.org/ Node.js-based with a Pipes-like visual wiring interface. It's quite popular with the Raspberry Pi and Arduino crowd. Lot's of input and output plugins, and you can drop into Node scripting when necessary. I quite like it.
- perryprog 9y agoNode Red is very cool, especially for IOT type apps.
- dundercoder 9y agoI thought RSS had died a rather unfortunate and inconvenient death... am I wrong and it's still been kicking all this time?
- Viper007Bond 9y agoStill very much alive, just not quite as popular due to the death of Google Reader.
- j45 9y agoPipes was a handy tool, hope this project keeps up!
- Mister_Snuggles 9y agoI'd love something like this that I can run locally. From looking at the comments, it looks like Node-RED and NoFlo are two possibilities. I was intrigued by Yahoo Pipes a while back, but didn't want to invest much in it in case it was shut down. Sadly, that worry was well founded.
- Xeoncross 9y agoUsing yahoo! pipes I created a lifestream application[0] some years ago that won an honorable mention for the 10k javascript challenge. It was nice to have a dynamic site that required no server work on my part. Updating it for this would probably be a neat way to test out the RSS feed usage. [0]:https://github.com/Xeoncross/MicroStream https://github.com/Xeoncross/MicroStream
- fareesh 9y agoUnrelated to pipes specifically but along the same lines, I feel like app developers ought to conform to a pre-defined standard like schema.org for their respective content so that everything can be inter-operable in theory. That way if I'm using Microsoft Todo or Google Keep or whatever, the potential for Google Assistant or Siri or Cortana to add to whichever Todo App I'm using is already there. What are the drawbacks behind something like this?
- bsandert 9y agoVendor lock-in.
- comboy 9y agoNo incentive. If your service is used through something else then you are easily replaceable and users don't spend time in your app. edit: Just to be clear, I'd love that too. Hopefully it will start happening when everything is driven more by micropayments rather than ads. I'd imagine you'll have "gui companies" and "service companies".
- onli 9y agoIt's also really hard to do. I mean, imagine how one would achieve interoperability for a project like this. Placing blocks via an API from another service? The language/spec supporting that and everything else would be too complex. But thankfully we have RSS for the data representation, which is a related idea, just the output side. The core of that idea is what enables this site in the first place.
- houshuang 9y agoThis discussion is very interesting to us, as we are building a workflow system for collaborative learning (students might do an activity individually, the output of that activity is aggregated and transformed, students are grouped based on an algorithm that processes data from the previous activity, output from one activity is redistributed to another activity, etc). (quick demo: https://www.youtube.com/watch?v=HQ9AyzLOn3Q https://www.youtube.com/watch?v=HQ9AyzLOn3Q) So this discussion about visual languages, data transformation etc, is very relevant to us. One thing we're working on is how to make data transformation more intuitive... Right now we are using JSONPath to enable selection of one field to aggregate on (ie. you have a form where people input ideas, and another activity that takes a list of ideas, so you can input a JSONPath for the field to get aggregated). However, looking at JMESPath (http://jmespath.org/examples.html http://jmespath.org/examples.html), it looks much more powerful. Has anyone seen any examples of graphical interfaces for going from one data representation to another, with preview, selecting fields, aggregation etc?
- technologia 9y agoIts interesting and I do applaud your work, but I'll probably just stick to Apache Nifi with its flexibility.
- JPKab 9y agoHow do you like NiFi? I work at Alteryx (we have a closed source visual data integration tool targeted at business analysts and data scientists), but was actually investigating NiFi quite heavily before I joined my current company. It didn't appear to me to have much adoption, but that was over a year ago. It certainly looked extremely compelling as a tool targeted at advanced users.
- technologia 9y agoSo I would agree that its not intended to be a tool that you could use like yahoo pipes. To the extent that it requires programming, I guess I would agree it requires an advanced user. Putting that aside, I would say the Nifi is easy to work with, performant with large datasets, and I personally love the notifications I get when tasks get screwed up, etc. I would not suggest using this to expose data flow to your user as its UI is not really geared towards folks who are programming averse (to a certain extent).
- laretluval 9y agoDoesn't seem to work on mobile Safari.
- jonincanada 9y agoI archived all the public yahoo pipes as Python modules -- before it was shut down #oldhardrives
- timvdalen 9y agoLooks interesting but the editor didn't give me any feedback/output and the login email never arrived.
- onli 9y agoSo far you need to log in to save the pipe and then see the output. Shoot me a mail to admin@pipes.digital (or to the one in my HN profile) if you want to try to sort out the non-arriving email. Or if you have a gmail address, log in with that, portier (the login system) supports the OIDC flow for gmail.
- hehejo 9y agoWhat about Talend ETL? You connect and configure various compontents (Input, Output and Manipulation) to achieve you desired dataflow. It's compiled to Java and really Vera versatile.