8 ms·

This is my project :) Nice to see this here again! If there are questions, I'm ready to answer (I have a rather young baby here though that wanted attention whi

by onli 2y ago

This is my project :) Nice to see this here again! If there are questions, I'm ready to answer (I have a rather young baby here though that wanted attention while writing this, but she sleeps now, so I think we are good). Pipes saw some internal updates recently. It was fighting with some instability. My first measure was an internal re-architeture, on a small scale. Before, the data was transported as text from block to block. (Almost) Every block parsed the input, created an RSS feed object and then worked with that on the data, only to then output the created RSS feed as string. This was changed to just work with the RSS object directly and move that from block to block. That had some consequences for some existing pipes, mainly because it also changed when the input feeds gets normalized. As that did not help, next step was a server upgrade. I'm not sure whether it was directly the additional processing capacity or the dependency upgrades, but that one helped so far. Knock on wood. Also reconfigured the amount of threads and puma workers, that might have solved a bottleneck somewhere. I sadly never pinpointed exactly what made the server process stop before. I had played with a proper split between web frontend and pipes processing, but at least my approach was not viable. Worked nicely while developing it, immediately crashed with the production workload. And Pipes is not all that huge... I'll have to try again some time.

Gys 2y ago
What I am missing is a block to manipulate contents. For example, I might want to add an AI generated summary to certain feeds. Or if there is no image, add an alternative image. For those cases it could be useful to have a block that can post a feed item (or one selecteble field) to my custom url and add or replace that item with the result. Then I can make my own feed item manipulators for certain tasks. Thinking bigger, you might offer a (third party?) store of custom manipulators: 'add image', 'add summary', etc.
- onli 2y ago
  I like that. I think it would have to be all items that get POSTed one by one to a url to work with the current flow, but that should be equally useful, no? Pipes does have some building blocks to prepare that operation or work with the result, there is an extract block and an insert block (that works by xpath). But I was always missing a model for the request block that I remember from Yahoo Pipes, that's why there is only a feed and a generic download block so far. Post somewhere, replace the item (or the item's content?) with the answer, that might be it. Edit: Thinking a bit more about this, don't forget that you can already chain pipes. A service you can POST to can probably also consume an RSS feed, download it by itself. You can then have one pipe that creates the feed up to the point where you would want to have it be treated by something else, let that thing output its result, and in a second pipe download the output and create the final feed. I still think the request block to POST items is a good idea, and there are probably services that can't store the output for the second pipe as they'd just react directly to a POST, but that chainability might be worth to keep in mind.
  - Gys 2y ago
    Ok, I understand the concept of connecting pipes and that will work as well. Still, POSTing items is more separation of concerns. The remote service only has to process one item (no need to read feeds, worry about formatting, etc).

by onli 2y ago

Gys 2y ago

What I am missing is a block to manipulate contents. For example, I might want to add an AI generated summary to certain feeds. Or if there is no image, add an alternative image. For those cases it could be useful to have a block that can post a feed item (or one selecteble field) to my custom url and add or replace that item with the result. Then I can make my own feed item manipulators for certain tasks. Thinking bigger, you might offer a (third party?) store of custom manipulators: 'add image', 'add summary', etc.

onli 2y ago
I like that. I think it would have to be all items that get POSTed one by one to a url to work with the current flow, but that should be equally useful, no? Pipes does have some building blocks to prepare that operation or work with the result, there is an extract block and an insert block (that works by xpath). But I was always missing a model for the request block that I remember from Yahoo Pipes, that's why there is only a feed and a generic download block so far. Post somewhere, replace the item (or the item's content?) with the answer, that might be it. Edit: Thinking a bit more about this, don't forget that you can already chain pipes. A service you can POST to can probably also consume an RSS feed, download it by itself. You can then have one pipe that creates the feed up to the point where you would want to have it be treated by something else, let that thing output its result, and in a second pipe download the output and create the final feed. I still think the request block to POST items is a good idea, and there are probably services that can't store the output for the second pipe as they'd just react directly to a POST, but that chainability might be worth to keep in mind.
- Gys 2y ago
  Ok, I understand the concept of connecting pipes and that will work as well. Still, POSTing items is more separation of concerns. The remote service only has to process one item (no need to read feeds, worry about formatting, etc).