9 ms·
Rudder, an open source Segment alternative
- sails 7y agoHow does this compare to Snowplow? https://github.com/snowplow/snowplow https://github.com/snowplow/snowplow
- namanyayg 7y agoI've been using snowplow for 2M+ events/mo and can still remember the pains of setting up. Plus, there's a fixed schema required. I haven't looked into rudder but I'll switch if it offers easier setup and schema.
- soumyadeb 7y agoWould love to work with you while you are evaluating this. Please email soumyadeb@rudderlabs.com or join our slack https://rudderlabs.herokuapp.com https://rudderlabs.herokuapp.com if you need any help
- namanyayg 7y agoThanks for making a great product with an open source. I'll definitely be trying it out soon for client web and mobile apps. I know self hosting snowplow is already dirt cheap; but is there any price comparison by any chance?
- soumyadeb 7y agoHonestly, we haven't figured out our pricing model yet - it will likely be a combination of support+enterprise features (don't know what those features would be). At this point we want to make our pilots/customers successful. Would love to understand your use case first and we can discuss pricing (e.g. if you need us to manage it etc)? Follow-up with you on your HN email?
- sumanthpur 7y agoThanks for asking, I am one of the authors of Rudder. Snowplow is a great analytics tool, especially used for internal analytics. It is open-source and on-prem and keeps your data privacy. It is centered around event enriching and storing to a data warehouse. We are aiming at routing the events reliably to destinations, transforming events real-time, storing them into your data warehouse with a dynamic schema and eventually build a data platform with help from the community.
- sails 7y agoOk that is interesting. I've been thinking about a Snowplow implementation for a while but seems like a big task. Could you expand the differences? > Snowplow is a great analytics tool, especially used for internal analytics\ > centered around event enriching and storing to a data warehouse Is rudder not aimed at these use cases? I seem to gather that you are building a more full stack platform, with a core feature set similar/better to/than Snowplow? Edit: Can you expand on the types of destinations you plan on supporting? I see Sources [Android, iOS, JS] and Destinations [Amplitute, GA, Hubspot etc] but no data warehouses. Can I send raw events to a Snowflake dwh for instance?
- yodi 7y agoThis is awesome! Never know if anyone build things like Segment. I will try this alternative solution to our company and will keep update here about the result.
- soumyadeb 7y agoAuthor here. Happy to help you out with this, in case you need it. Please email me at soumyadeb@rudderlabs.com. Or please join our slack - https://rudderlabs.herokuapp.com/ https://rudderlabs.herokuapp.com/
- mushufasa 7y agoalso, segment was the O.G. segment alternative https://github.com/segmentio/analytics.js https://github.com/segmentio/analytics.js
- soumyadeb 7y agoAnalytics.js was only a client/browser side utility. They developed an entire backend stack later - which is needed for data-warehouse dump, event replay etc etc. We are also developing a complete backend stack with all those features. This is still a work in progress
- amelius 7y agoIf every successful business is eventually copied by open source, can we really blame big companies for protective practices such as customer lock-in?
- dv_dt 7y agoIf there is no competition, markets cannot work as intended.
- jarfil 7y agoYes. Customer services, as in both support and feature development, should be enough to keep paying customers paying. If a bunch of "use at your own risk" software can replace it, what's the actual value of a business?
- HillRat 7y agoIt’s licensed under SSPL, so the definition of “open source” may vary. (Specifically, they have an enterprise business model, so this is more properly a proprietary, source-available on-prem Segment competitor.)
- soumyadeb 7y agoFounder here. Yes, that is an appropriate statement. We want to be a business on the likes of Mattermost, Kafka, ElasticSearch and so on. We are committed to building an open-source community. Honestly, we were not prepared for this post. We don't even have a license attorney. We just picked SSPL because that's what everyone seemed to suggest to prevent the likes of AWS cloning it. By no means we are an expert.
- HillRat 7y agoI think you’ve got a strong value proposition, especially in a GDPR/CCPA world. Have you figured out a channel and integration partner strategy yet? That’s probably the hill that you’re going to need to fight on.
- 7y ago
- soumyadeb 7y agoOne of the authors here. This post was a pleasant surprise!! Happy to answer any questions. Or please feel to reach out to me at soumyadeb@rudderlabs.com.
- AndrewKemendo 7y agoThis is awesome, I've been hoping someone would make this! And being enterprise focused is even better. Thanks!
- soumyadeb 7y agoThank you :)
- tyri_kai_psomi 7y agoYou put a spotlight by saying "privacy and security focused alternative to segment" Are there things you believe make segment not privacy and security focused? As a long time user of segment, I find their protocols feature and new data and privacy features world class for this. Having also just left their Synapse conference, privacy and security was the #1 topic of discussion throughout the conference. I would say they are very much privacy and security focused. Not trying to shill, but it comes off as maybe you are misrepresenting segment a little bit. "Open source segment alternative" would have probably been just fine.
- soumyadeb 7y agoWe started this before segment launched the privacy tool so yes that was a good validation of the pain point. However, unlike segment we don't require you to send all your events to us. You are welcome to download and run the software yourself without ever talking to us. We believe, having to send everything to a 3rd party (segment) goes against the fundamental goal of guaranteeing user privacy.
- colordrops 7y agoThe author responded to you, but I think this deserves emphasis. True privacy is not about additional features or promises made by a cloud provider. It's not private unless you handle the data yourself, period point blank. This should be obvious.
- tankster 7y agoSegment has done an awesome job of building a great product but it is impossible to be provably secure and private given that they host all the data. Rudder can avoid situations such as the Segment Security incident that happened recently. https://segment.com/security/bulletins/incident090519/ https://segment.com/security/bulletins/incident090519/ Very excited about this project!
- soumyadeb 7y agoThanks :)
- dacompton 7y agoThe majority of risk lies outside of Segment and they are damned serious about the risk they own
- soumyadeb 7y agoI am sure they do. We have great respect for segment. However, at some point/scale a company needs to take ownership of their their user-data. Yes, some events may go to 3rd party vendors (analytics, advertising) but not everything goes and those often have PII removed. We want Rudder to be the point where you decide and enforce what goes where.
- vollmarj 7y agoAs a longtime segment customer, I am excited to see this exist. Segment's product is fairly good but their pricing model is really bad. We recently ran into an issue where they were billing us for almost 10x the number of users we were actually tracking. It was a nightmare that required a few months of back and forth with their support trying to get it fixed. In the end, they gave us a partial refund for the overages but we had to do a lot of technical work to resolve the issue ourselves. It would be nice to have an open source alternative that doesn't get you locked into an unpredictable pricing model that you have very little control over.
- soumyadeb 7y agoSome of our initial pilots had similar problems. Segment's pricing also doesn't work when you have a lot of free users. That(and privacy) are the two pain points we hope to address with this. Happy to help you try this out if you want (please email soumyadeb@rudderlabs.com). We are behind on number of integrations (vs segment) and features but we will catch-up. And hoping to get community support on that.
- dinrat 7y agothis is so awesome! i have been waiting for it.
- soumyadeb 7y agoThank you :)
- indianCoder 7y agoOne issue I had with Segment, I couldn't run real-time transformation of the event to join data from our data tables. We eventually got over it with AWS Lambda and sending it back to Segment. Segment recently announced functions to help on this, still could not get my hands on it. Any plans on this?
- soumyadeb 7y agoYes, that is exactly the use case of our "user defined" transformation functions. You can define any javascript function (right now by modifying the code but will be available from the UI in the release coming next week). Inside that function you can filter/transform/enhance the event in any way you like. You can lookup your DB, call external APIs etc. You can also combine multiple events into one. Since this whole thing runs inside your VPC, you don't have to open up your production database to 3rd party as you have to do with segment. Happy to work with you on your use case. Please email soumyadeb@rudderlabs.com We wrote a couple of blog posts on that. Case Study: https://rudderlabs.com/customer-case-study-casino-game/ https://rudderlabs.com/customer-case-study-casino-game/ Transformation Details: https://rudderlabs.com/transformations-in-rudder-part-1/ https://rudderlabs.com/transformations-in-rudder-part-1/ https://rudderlabs.com/transformations-in-rudder-part-2/ https://rudderlabs.com/transformations-in-rudder-part-2/
- indianCoder 7y agoThis would be so awesome. Looking forward to it.
- NuclearTide 7y agoHey there! I work at Segment, and I'm one of the engineers working on Segment Functions. If you let me know your Segment workspace's name (get in touch through jason.tu@segment.com), I can grant you beta access.
- indianCoder 7y agoThanks. Will drop an email.
- platform 7y agovery interesting and approacheable solution. With regards to: >" Rudder runs as a single go binary with Postgres. It also needs the destination (e.g. GA, Amplitude) specific transformation code which are node scripts. " what would be a recommended approach, if I would like to keep the data internally, and not use external Analytics engine?
- soumyadeb 7y agoWe also have a S3 destination so you can just add S3. Support for other data-warehouses (Redshift, Bigquery etc) is coming soon.
- platform 7y agoThank you, my understand was that S3 is also 'external' (that's AWS service). What I was hoping/looking for is something where another tool lightweight tool that can be installed alongside Rudder, that can use PG instance (may be just different db within Rudder's instance or just different schema). For a light weight, end-to-end solution (including analytics) that can 'grow' with needs. For many small self-funded startups that care about privacy a combination of following challenges exist: a) need self-hosted solutions that require minimal footprint (because data, even anonymized, is hosted on self-paid VPS instances ) b) need to use solutions that do not require initial purchase/investment, but yet can scale out (both technically, and in terms of support) when/if commercial model of the startup becomes more successful c) need to use solutions that are not heavy VC backed, because those typically do not have stable engagement model (as VCs rates of return demand, typically, changes in initial service/open source model, while self-funded startups are typically much slower growing (and slower in committing to buying expenses)) d) need self-hosted solutions to cover lot of ground in one (so that the learning curve, and time needed to integrate are minimized).
- sumanthpur 7y agoIn next 2 weeks, we are releasing Redshift as destination. After that, we have PostgreSQL destination in our pipeline. You can configure it and capture everything there. If you prefer that to be in files, you could setup a minio server on your VPS instance. It's coming in next weeks too. https://github.com/minio/minio https://github.com/minio/minio We would like to understand your preference on this so that we could align our next set of destinations. Please drop an email at sumanth@rudderlabs.com or join our Slack/Discord channels.
- beager 7y agoSo what's the pricing model? Your site lists "pricing" and shows no info. - Are you charging for support? - Do you/will you have a paid enterprise tier that will increasingly be the only tier with a viable feature set? - What's keeping you from dumping on Segment's market until you hit traction then ratcheting up to Segment's pricing? - Who are you? Who are your investors?
- soumyadeb 7y agoPricing: Honestly, we haven't figured out the business model yet. Like other open-source products, it will likely be a combination of support + enterprise features (like HA, auto-scale etc) but again we don't know what those enterprise features would be. Ratch up Pricing: Good question and not sure how to answer. Our vision is to be like other open-source companies like Mattermost, ES. Our base version (which would work for 90% of users) would be free and under open-source license. But I do understand your concern - maybe there is a way to put that in the license (that the base version will be perpetually open-source) - Here is our company page (https://www.linkedin.com/company/rudderlabs https://www.linkedin.com/company/rudderlabs). Our lead investor is S28 capital (Partner: Shvet) - they have also invested in Mattermost (an open-source slack competitor)
- orasis 7y agoGood luck, but this is a huge amount of work to keep something like this running. I hope it’s successful!
- soumyadeb 7y agoIndeed. That's why we have raised a seed round and doing this full time as a serious business. We are passionate about privacy but understand that we cannot have a good shot while doing this part-time.
- cyberferret 7y agoI am a happy Segment user, however their handling of a recent data breach did leave a bit of a sour taste. They took several days to get back to me with a definitive answer as to whether our customer data that we collected was compromised by the internal breach. I am keen to look at competitive product where we may have more control over the data collected and can manage the risk ourselves.
- soumyadeb 7y agoWould love if you try this :) If you need help, please email(soumyadeb@rudderlabs.com) or join our slack community - https://rudderlabs.herokuapp.com/ https://rudderlabs.herokuapp.com/
- deleted 7y ago[deleted]
- gauravagarwal 7y agoThis looks exciting
- drixta 7y agoWe're lucky to find and deploy this project early at our startup. Being a cybersecurity company, we cannot have out customers data leave our aws account.
- rixed 7y ago> we cannot have out customers data leave our aws account. Unintentionally funny?
- sundbry 7y agoIs it really so difficult for engineers to create a task to process a Kafka topic? It takes one day to write a program to consume from a topic of events and push to an API like Amplitude, and you have total flexibility in how you push to those integrations. Why would you use postgres for an event professing system? This seems like an inefficient architecture.
- soumyadeb 7y agoGreat question. Complication arises because of failures. One or more destinations may be down for any length of time, individual payloads may be bad etc. To handle all these you need to retry with timeouts while not blocking other events to other destinations. Also, not all events may be going to all destinations. We build our own streaming abstraction on top of Postgres. Think layman's leveled compaction. We will write a blog on that soon. The code (jobsdb/jobsdb.go) should have some comments too in case you want to check out. Segment had a similar architecture and had a blog post on it but can't seem to find it. Also, eventually we will replace postgres with something low-level like RocksDB or even native files. Yes, in theory you can use Kafka's streaming abstraction and create create a topic per destination. Two reason's we didn't go that route 1) We were told Kafka is not easy to support in an on-prem environment. We are not Kafka expert but paid heed to people who have designed and shipped such on prem softwares. 2) More importantly, for a given destination, we have dozens of writers all reading from same stream. Only ordering requirement is events from a given device (end consumer) are in order. So we assign same user to same writer. However, the writers themselves are independent. If a payload fails, we just block events from that user while other users continue. Blocking the whole stream for that one bad payload (retried 4-5 times) will slow things down quite a bit. If we had to achieve the same abstraction on Kafka, we would have had to create dozens of topics per destination.
- tomnipotent 7y agoReferring to this Segment blog post? Was my first read on the space, found it pretty informative. https://segment.com/blog/exactly-once-delivery/ https://segment.com/blog/exactly-once-delivery/
- tomnipotent 7y agoI want to see this married with the Meltano project from Gitlab - it would create an unprecedented end-to-end data environment. https://meltano.com/ https://meltano.com/
- torpedolaser 7y agoWe are in the mobile game industry. Due to the incredible high volume events data per user generates per day, we need to join certain events together to reduce the events amount send to our analytics platform. There is no other segment tool can do this for us. We have been working with Rudder Labs to solve this problem. They have been really helpful and act super fast to our requests and suggestions. With Rudder Labs SDK, we are able to join event data on the platform(which we choose to host internally) + all other segment tool features. Besides that, since we run a freemium game so that most of our users are free users/non-payers, cost is another thing we have to consider, current segment market pricing apparently is way too high for us. Rudder labs solves this problem for us as well. Great deal!
- soumyadeb 7y agoThanks for being our first pilot and for all your patience while we were fixing/improving the product.
- dehrmann 7y ago> Due to the incredible high volume events data per user generates per day, we need to join certain events together to reduce the events amount send to our analytics platform. Don't gloss over the fact that telemetry over cell networks can be costly for users (more and more plans are unlimited, so this doesn't worry me as much) and draining on batteries. However you do it, data that's not latency-critical should be buffered and batched.
- soumyadeb 7y agoYes, the SDK buffers and batches events before sending to Rudder BE. In the BE, the events are combined (value summed up, not batched) before it is sent to the analytics platform. We wrote a blog for this interesting use case https://rudderlabs.com/customer-case-study-casino-game/ https://rudderlabs.com/customer-case-study-casino-game/
- Roritharr 7y agoInteresting, we've built our in-house solution that is also Go based, also writing to a postgres db besides forwarding events, but much simpler, without a UI and comes with backend sdks already. What I found interesting is that you wrote 3k/events per second on a rather beefy 2xlarge machine. Our version is MUCH less demanding, I wonder if there isn't a lot of performance left on the table here. I'll keep this in mind once we've grown out of our solution, though.
- soumyadeb 7y agoThe bottleneck for us (on that instance) is not postgres but transformations. Transformations are tiny snippets of javascripts which convert the event from Rudder JSON to whatever structure (JSON, keyval etc) that the destinations expect. We also support user defined transformations - functions defined by the user to transform/enhance the event. Currently, transformations are run in nodeJS. So, for every batch of events there is a call into nodeJS from go and that is slow. We do batching/parallel-calls but still. I think, postgres gets us > 15K/sec throughput.
- LittlePeter 7y agoWhat happens if I pass a 64 bit integer, and the Rudder pipeline being in JavaScript silently down-casts it to a 53 bit integer? Segment's pipeline involves JS at some point. We had an issue where our 64 bit integers were down-casted silently. We found out the hard way. We use strings now (perhaps should have used strings right away, I am not necessarily the sharpest tool in the shed).
- soumyadeb 7y agoHmm, never thought about that. Need to think how to handle it - great point!!
- qurt 7y agoI tought this was https://www.rudder.io/en/ https://www.rudder.io/en/
- soumyadeb 7y agoHaha, it is https://www.rudderlabs.com https://www.rudderlabs.com Unfortunately, we haven't done anything re: SEO/SEM. I think we have no-index on our website (default with wp-engine) - would fix it.
- martingxx 7y agoThis looks great, and I look forward to finding time to try it soon. However, I believe it's misleading to call it "open source". The SPPL license is not generally considered to be an open source licence by any meaningful definition, and in particular does not meet the OSI definition and is incompatible with most licences that do. I understand the need these days to protect against aggressive cloud providers, but there are other ways to achieve that without becoming completely non open source, such as the BSL (See license.https://opensourceforu.com/2019/06/cockroach-labs-changes-its-licensing-strategy-adopts-permissive-version-of-business-source-license/ https://opensourceforu.com/2019/06/cockroach-labs-changes-it... )
- soumyadeb 7y agoThanks for the pointer - will check it out. We are totally novice on this - we don't even have a license attorney. We just picked SSPL because that's what everyone seemed to suggest to prevent the likes of AWS cloning it. Now that we got some visibility, we will carefully take a look at this issue. But at heart we want to build an open-source community while still being a viable business on the likes of MatterMost, Elastic etc.
- abdullahkhalids 7y agoThis is, what I think is, Mattermost's answer to the current debate around the ethics of FAANG (and others) using open source software to make lots of money without substantially contributing back to the OS projects financially or in code. https://github.com/mattermost/mattermost-server/blob/master/LICENSE.txt https://github.com/mattermost/mattermost-server/blob/master/... My understanding is that, Mattermost is okay with others making money from their software if they don't modify it - which will practically work for some, but not all, small companies, and will be very difficult for the big companies to use. If the big companies want to modify and use Mattermost-server for free they are forced to contribute back the changes to the OS project, and then can make as much money as they want. Or use option 2, pay Mattermost a bunch of money for the privilege of not contributing back code to the OS project. In other words FAANG and co can either contribute to Mattermost financially or in code - their pick.
- fuzzyfroghunter 7y agoThis looks great, nice work. Is there an easy way for someone to set it up on their cloud infrastructure?
- soumyadeb 7y agoYes, this is exactly the use case we want to target. What's your cloud infra? We can easily run inside your AWS VPC. If you have your own private cloud, we can run there too - just need to disable the S3 dump. The only dependency we have is Postgres. Happy to help you setup - Please email soumyadeb@rudderlabs.com OR join our slack https://rudderlabs.herokuapp.com https://rudderlabs.herokuapp.com
- fuzzyfroghunter 7y agoActually asking because we are building data science automation software which we deploy onto our customers' cloud infrastructure. Very similar value proposition to Rudder re: privacy and security, but different use case. Always interested in how other people achieve similar effects. Right now, for AWS, we provide customers with a zipped up Packer+Terraform configuration with source and a README which lets them: 1. (terraform+sh) Create an IAM user responsible for the deployment with the right policy to deploy and export environment variables for the user. 2. (packer) Create an AMI for an instance that can host our APIs 3. (terraform) Spin up an EC2 instance on the desired VPC to host our APIs (with optional key pair for debugging, etc.)
- mychael 7y agoSegment IO is currently being blocked on browsers with adblock. Does this mean it will work even with adblock?
- soumyadeb 7y agoYes, if you host the JS SDK yourself. However, certain destinations require loading their javascript anyway (segment calls them client-side integration) and those will still be blocked. However, a vast majority of others (e.g. analytics) products would work
- davej 7y agoBy the way, you can get around this by using their custom domain support and proxying the segment script.