6 ms·

Show HN: Cptn.io – open-source integration platform

139 points by kc10 3y ago

Hi, I am Krishna Thota. I am building an open source integration and data platform(https://cptn.io https://cptn.io). The product is MIT licensed and the repo is at https://github.com/cptn-io/el-cptn https://github.com/cptn-io/el-cptn. I have started on my startup journey an year ago and launched a monitoring platform called DevRaven. Unfortunately the product did not takeoff as expected. That story is for another day. But during the course of building the product, I have built several integrations leveraging MQs and Cloud Functions. While building and deploying Cloud Functions for happy paths is easy, I had to monitor logs for failures, build retry mechanisms or manually process failed events, keep instances running to prevent cold start timeouts. It can also get expensive with charges for MQs, server time for running cloud functions etc and costs can be unpredictable. I thought of building a platform where I can build integrations quickly, have the ability to look at incoming/outgoing events, look at logs, retry any failed events etc. And finally, predictable costs for running the infrastructure. cptn.io provides all these capabilities and more. You can build pipelines to integrate with any cloud services, send data from your backend to data warehouses, listen to web hook events etc. The platform can be integrated into any stack by sending events to HTTP end points. Instead of trying to build a business first or launch an open source product under restrictive licenses, the platform will be available under MIT license so any user or customer can use it. There is no ee folder or complex dual licensing and I am also committing to releasing SSO under MIT. The plan is to offer a managed service in the cloud at a later time, accept sponsors for prioritizing features for enterprise customers and charge for enterprise support. It should take less than 5 minutes to get the platform running on your machine. Welcome any feedback, feature requests, PRs and bug reports.

quickthrower2 3y ago
Another "is it like $thing" question. This time $thing="https://nodered.org/ https://nodered.org/"
dgwight 3y ago
Is this an open source Segment Analytics competitor? I liked the idea of segment, but the pricing jumped way up after the free tier and I had to remove it
- kc10 3y ago
  Technically the product can be used as CDP platform if a client library is capable of sending tracking events to the Source URL and you just want to pipe them to your buckets/data warehouses.
lagniappe 3y ago
This is pretty cool! The UI example reminds me of Yahoo Pipes (rip).
codegeek 3y ago
So is this like an Open Source Zapier or at least with similar ambitions if not quite there yet ?
- kc10 3y ago
  Yes, over time with contributions from community the app library will grow and pipelines can be built with ready to use apps. For complex cases or specialized integrations, you can use any npm library to build your flow. You can integrate into your stack with HTTP API and we can build SDKs for popular languages so it's just a function call to send the events.
- MilStdJunkie 3y ago
  I wonder how cptn compares with ActivePieces. AP's been declared the "Open Source Zapier" and I have been having a blast with it. https://www.activepieces.com/pieces https://www.activepieces.com/pieces https://github.com/activepieces/activepieces https://github.com/activepieces/activepieces
- tootie 3y ago
  Seems more like open source fivetran. For loading data into a warehouse.
robertlagrant 3y ago
I worked with Software AG webMethods for quite a while and always wondered about building something like this. Nice job.
badrequest 3y ago
How does this compare to, say, RudderStack?
- kc10 3y ago
  It can work similar to RudderStack if we build a JS library that can send tracking events into the platform. But I think Rudderstack has more features around ETL and Reverse ETL which do not exist today in cptn. RudderStack is AGPL while cptn is MIT.
nkmnz 3y ago
What differentiates this from n8n?
- kc10 3y ago
  I haven't used n8n before, but n8n looks much more mature, feature rich and no code capabilities. You can build any workflow in cptn as well, but most of the workflow is built via scripting. Licensing "seems" to be restrictive if you want to use for commercial purposes (I am not quite sure about Sustainable Use License).
- sneak 3y ago
  n8n is not open source.
BionicPimp 3y ago
I like the UI. seems pretty straight forward. Best of luck.
henricourdent 3y ago
Hi! In what does this differentiate from Windmill.dev? The license seems to be more permissive (AGPL vs MIT) but there isn't any Python support, and less integrations (for now ?) :) Wishing you the best of luck! https://github.com/windmill-labs/windmill https://github.com/windmill-labs/windmill
- kc10 3y ago
  Thanks. I think there are a number of platforms for building workflows and integrations today. Some of the platforms serving specific niche (e.g. RudderStack chose to serve CDP, airplane for building internal dev tools). But in the end all of them act on a trigger to transform inbound event data and send it to a destination end point. Windmill looks cool with support for multiple languages, no-code editor and all the OOTB integrations. cptn is still evolving - it needs to have more integrations, UX flows need to be improved, it has to get better from Ops perspective.
born-jre 3y ago
Hmm.. Gives me early vibes. Which reminds me, maybe I should start promoting my thing[0] too best of luck [0]: https://github.com/temphia/temphia https://github.com/temphia/temphia
- kc10 3y ago
  Thanks. It's not v1 yet. It's about 3 months in the making and needs improvements around ops, tests. I am hoping to make progress with adding tests in the coming days and provide monitoring capabilities.
- jamesmcintyre 3y ago
  Your project looks really interesting. I've been looking into many different "plugin" architecture options (mainly with the goal to run untrusted js/ts code) and your extended readme sounds interesting. But i can tell it's early so i'm not fully seeing how this would be used (like an example folder). - I too have tried to think of how plugins could be "full-stack" with both sandboxing on BE & FE via iframe but i'm not seeing in your repo the code that handles the FE sandboxing? (I could be totally looking in the wrong place) - I looked into using wasm "containers" as well- does your architecture have a scalable way to optimize "cold boot" of a container for near-instant run/render? Again, cool project and good luck! I think there's so much room for innovation in this area of making micro-functionality truly portable & composable!
  - born-jre 3y ago
    thanks yeah making core framework in fast low level language and making scripting/embed language on top is such underexplored area i think. i have removed suborigin bashed frontend sandbox for now, inline iframe is just faster to iterate, i will bring back suborigin later. iframe related code would here. https://github.com/temphia/temphia/blob/main/code/frontend/lib/engine/template.ts https://github.com/temphia/temphia/blob/main/code/frontend/l... https://github.com/temphia/temphia/blob/main/code/frontend/entries/portal/launcher/hostproxy/iframe/iframe_execute.svelte#L38 https://github.com/temphia/temphia/blob/main/code/frontend/e... suborigin code is also same but template part would be in go template and you will need to setup (<wildcard>.myapp.com) domain name. no i donot do use any wasm runtime snapshoting, i do cache the instance once they are run though so yeah first one will have latency. i will try to post some demo/examples of app running later on, maybe. ¯\_(ツ)_/¯
gigatexal 3y ago
is this a sort of heroku? this gives me pause because i am dubious about building on a platform's PaaS as it is now I have this as an option but they (AWS, GCP, etc) are likely to be around, but you're just you. So is this something we can self host or... is there a hostable solution you want to push folks to?
- kc10 3y ago
  It is self-hostable today. Just clone the repository, update the .env file and run "docker compose up" to get the instance running.
  - gigatexal 3y ago
    Really cool!
nico 3y ago
Can you buffer requests to Twilio dynamically based on feedback? ie. Can the system be fed back number of undelivered messages and adjust throttle accordingly?
- kc10 3y ago
  There is no throttling today based on outbound failures. But I think it is a pretty common use case for handling destination failures. Something like - if x or x% events are failing in a pipeline, pause the pipeline for y minutes or until some one unpauses it. I can look at adding it.
  - nico 3y ago
    That would be amazing
replwoacause 3y ago
No PowerShell support?
- kc10 3y ago
  Only javascript is supported for building transformations and destinations. But you can use any npm library in your script, platform will automatically download it at runtime and executes your script.
culopatin 3y ago
We use a platform called Actions Express that I’m not most fond of. Any ideas of integrators with SeviceNow plugins? I’m basically doing web calls and parsing data, but the ServiceNow plugin is nice to have.
- kc10 3y ago
  Not sure I understood the use case. Are you looking into integrate into SN platform? You can add a pipeline to integrate with SN platform using OOTB rest api or add scripted rest end points for custom integrations. A ServiceNow integration is on my radar. But if you can provide more details I can definitely try to address it when I build the app. Please feel free to log an issue in github if you can provide any details or send an email to me (email in my profile).
deevus 3y ago
Can this work for building pipelines for specific users? E.g. our customers can connect their CRM to our platform.
- kc10 3y ago
  There are a couple of ways to address this: 1. Create a Source for each customer and send customer specific data to each source. Configured pipelines for each source will receive the event data and processes it. 2. Create one source with multiple pipelines connected to it. In each pipeline, you can drop processing the data based on any conditions. #1 is preferred so you are sure each customer's data is only reaching the pipelines configured for that customer data.
  - deevus 3y ago
    Can this be done by API?
    - kc10 3y ago
      The entire backend is available via REST interface. I am looking to support finer API permissions via auth tokens. But the APIs are accessible today with basic auth.
esafak 3y ago
Krishna: From the comments, you can tell everybody is trying to position your product, so I suggest explicitly naming your competitors in your landing page, and explaining how you are better or different. Don't worry about driving prospective customers away; everybody already knows about the competitors.
- kc10 3y ago
  Thank you. Fair feedback. I definitely need to anchor myself to specific niche instead of saying it can work for anything and everything.
- joshxyz 3y ago
  yes pls i would love to know how it compares with n8n.io, segment.io, products like those
SeriousM 3y ago
How does it compare to https://n8n.io https://n8n.io?
dayeye2006 3y ago
How does compare to airbyte?
novoreorx 3y ago
Cptn.io is ideal for quickly prototyping a custom backend for specific tasks. While n8n also offers this capability through its webhooks, Cptn is more lightweight and developer-oriented. To illustrate, suppose I want to create an API that saves a Pinboard link and sends it to a Telegram channel when passed notes. In such cases, Cptn is a better choice than n8n.

139 points by kc10 3y ago

quickthrower2 3y ago

Another "is it like $thing" question. This time $thing="https://nodered.org/ https://nodered.org/"

dgwight 3y ago

Is this an open source Segment Analytics competitor? I liked the idea of segment, but the pricing jumped way up after the free tier and I had to remove it

kc10 3y ago
Technically the product can be used as CDP platform if a client library is capable of sending tracking events to the Source URL and you just want to pipe them to your buckets/data warehouses.

lagniappe 3y ago

This is pretty cool! The UI example reminds me of Yahoo Pipes (rip).

codegeek 3y ago

So is this like an Open Source Zapier or at least with similar ambitions if not quite there yet ?

kc10 3y ago
Yes, over time with contributions from community the app library will grow and pipelines can be built with ready to use apps. For complex cases or specialized integrations, you can use any npm library to build your flow. You can integrate into your stack with HTTP API and we can build SDKs for popular languages so it's just a function call to send the events.
MilStdJunkie 3y ago
I wonder how cptn compares with ActivePieces. AP's been declared the "Open Source Zapier" and I have been having a blast with it. https://www.activepieces.com/pieces https://www.activepieces.com/pieces https://github.com/activepieces/activepieces https://github.com/activepieces/activepieces
tootie 3y ago
Seems more like open source fivetran. For loading data into a warehouse.

robertlagrant 3y ago

I worked with Software AG webMethods for quite a while and always wondered about building something like this. Nice job.

badrequest 3y ago

How does this compare to, say, RudderStack?

kc10 3y ago
It can work similar to RudderStack if we build a JS library that can send tracking events into the platform. But I think Rudderstack has more features around ETL and Reverse ETL which do not exist today in cptn. RudderStack is AGPL while cptn is MIT.

nkmnz 3y ago

What differentiates this from n8n?

kc10 3y ago
I haven't used n8n before, but n8n looks much more mature, feature rich and no code capabilities. You can build any workflow in cptn as well, but most of the workflow is built via scripting. Licensing "seems" to be restrictive if you want to use for commercial purposes (I am not quite sure about Sustainable Use License).
sneak 3y ago
n8n is not open source.

BionicPimp 3y ago

I like the UI. seems pretty straight forward. Best of luck.

henricourdent 3y ago

Hi! In what does this differentiate from Windmill.dev? The license seems to be more permissive (AGPL vs MIT) but there isn't any Python support, and less integrations (for now ?) :) Wishing you the best of luck! https://github.com/windmill-labs/windmill https://github.com/windmill-labs/windmill

kc10 3y ago
Thanks. I think there are a number of platforms for building workflows and integrations today. Some of the platforms serving specific niche (e.g. RudderStack chose to serve CDP, airplane for building internal dev tools). But in the end all of them act on a trigger to transform inbound event data and send it to a destination end point. Windmill looks cool with support for multiple languages, no-code editor and all the OOTB integrations. cptn is still evolving - it needs to have more integrations, UX flows need to be improved, it has to get better from Ops perspective.

born-jre 3y ago

Hmm.. Gives me early vibes. Which reminds me, maybe I should start promoting my thing[0] too best of luck [0]: https://github.com/temphia/temphia https://github.com/temphia/temphia

kc10 3y ago
Thanks. It's not v1 yet. It's about 3 months in the making and needs improvements around ops, tests. I am hoping to make progress with adding tests in the coming days and provide monitoring capabilities.
jamesmcintyre 3y ago
Your project looks really interesting. I've been looking into many different "plugin" architecture options (mainly with the goal to run untrusted js/ts code) and your extended readme sounds interesting. But i can tell it's early so i'm not fully seeing how this would be used (like an example folder). - I too have tried to think of how plugins could be "full-stack" with both sandboxing on BE & FE via iframe but i'm not seeing in your repo the code that handles the FE sandboxing? (I could be totally looking in the wrong place) - I looked into using wasm "containers" as well- does your architecture have a scalable way to optimize "cold boot" of a container for near-instant run/render? Again, cool project and good luck! I think there's so much room for innovation in this area of making micro-functionality truly portable & composable!
- born-jre 3y ago
  thanks yeah making core framework in fast low level language and making scripting/embed language on top is such underexplored area i think. i have removed suborigin bashed frontend sandbox for now, inline iframe is just faster to iterate, i will bring back suborigin later. iframe related code would here. https://github.com/temphia/temphia/blob/main/code/frontend/lib/engine/template.ts https://github.com/temphia/temphia/blob/main/code/frontend/l... https://github.com/temphia/temphia/blob/main/code/frontend/entries/portal/launcher/hostproxy/iframe/iframe_execute.svelte#L38 https://github.com/temphia/temphia/blob/main/code/frontend/e... suborigin code is also same but template part would be in go template and you will need to setup (<wildcard>.myapp.com) domain name. no i donot do use any wasm runtime snapshoting, i do cache the instance once they are run though so yeah first one will have latency. i will try to post some demo/examples of app running later on, maybe. ¯\_(ツ)_/¯

gigatexal 3y ago

is this a sort of heroku? this gives me pause because i am dubious about building on a platform's PaaS as it is now I have this as an option but they (AWS, GCP, etc) are likely to be around, but you're just you. So is this something we can self host or... is there a hostable solution you want to push folks to?

kc10 3y ago
It is self-hostable today. Just clone the repository, update the .env file and run "docker compose up" to get the instance running.
- gigatexal 3y ago
  Really cool!

nico 3y ago

Can you buffer requests to Twilio dynamically based on feedback? ie. Can the system be fed back number of undelivered messages and adjust throttle accordingly?

kc10 3y ago
There is no throttling today based on outbound failures. But I think it is a pretty common use case for handling destination failures. Something like - if x or x% events are failing in a pipeline, pause the pipeline for y minutes or until some one unpauses it. I can look at adding it.
- nico 3y ago
  That would be amazing

replwoacause 3y ago

No PowerShell support?

kc10 3y ago
Only javascript is supported for building transformations and destinations. But you can use any npm library in your script, platform will automatically download it at runtime and executes your script.

culopatin 3y ago

We use a platform called Actions Express that I’m not most fond of. Any ideas of integrators with SeviceNow plugins? I’m basically doing web calls and parsing data, but the ServiceNow plugin is nice to have.

kc10 3y ago
Not sure I understood the use case. Are you looking into integrate into SN platform? You can add a pipeline to integrate with SN platform using OOTB rest api or add scripted rest end points for custom integrations. A ServiceNow integration is on my radar. But if you can provide more details I can definitely try to address it when I build the app. Please feel free to log an issue in github if you can provide any details or send an email to me (email in my profile).

deevus 3y ago

Can this work for building pipelines for specific users? E.g. our customers can connect their CRM to our platform.

kc10 3y ago
There are a couple of ways to address this: 1. Create a Source for each customer and send customer specific data to each source. Configured pipelines for each source will receive the event data and processes it. 2. Create one source with multiple pipelines connected to it. In each pipeline, you can drop processing the data based on any conditions. #1 is preferred so you are sure each customer's data is only reaching the pipelines configured for that customer data.
- deevus 3y ago
  Can this be done by API?
  - kc10 3y ago
    The entire backend is available via REST interface. I am looking to support finer API permissions via auth tokens. But the APIs are accessible today with basic auth.

esafak 3y ago

Krishna: From the comments, you can tell everybody is trying to position your product, so I suggest explicitly naming your competitors in your landing page, and explaining how you are better or different. Don't worry about driving prospective customers away; everybody already knows about the competitors.

kc10 3y ago
Thank you. Fair feedback. I definitely need to anchor myself to specific niche instead of saying it can work for anything and everything.
joshxyz 3y ago
yes pls i would love to know how it compares with n8n.io, segment.io, products like those

SeriousM 3y ago

How does it compare to https://n8n.io https://n8n.io?

dayeye2006 3y ago

How does compare to airbyte?

novoreorx 3y ago

Cptn.io is ideal for quickly prototyping a custom backend for specific tasks. While n8n also offers this capability through its webhooks, Cptn is more lightweight and developer-oriented. To illustrate, suppose I want to create an API that saves a Pinboard link and sends it to a Telegram channel when passed notes. In such cases, Cptn is a better choice than n8n.