8 ms·
Drasi: Microsoft's open source data processing platform for event-driven systems
- deleted 2y ago[deleted]
- gigatexal 2y agoOh this very much reminds me of [feldera](https://feldera.com https://feldera.com) — they do incremental loads and computations using some novel approaches (most of which i am too dumb to follow). Really nice folks too.
- woozyolliew 2y agoOr the related Materialize stuff https://materialize.com/ https://materialize.com/
- hobofan 2y agoI took a brief look into Drasi and it looks like it doesn't do any of the differential/timely dataflow stuff (like Materialize does), or any other sophisticated incremental view maintenance methods that are rooted in Microsoft Research.
- smarx007 2y agohttps://azure.microsoft.com/en-us/blog/drasi-microsofts-newest-open-source-project-simplifies-change-detection-and-reaction-in-complex-systems/ https://azure.microsoft.com/en-us/blog/drasi-microsofts-newe...
- deleted 2y ago[deleted]
- CharlieDigital 2y agoVery interesting choice of using Cypher[0] In 2014, we built a similar type event-driven system (but specifically for document distribution (a document can be distributed to a target set of entities; if a new entity is added, we need to resolve which distributions match)) and also ended up using Cypher via Neo4j (because of the complex taxonomical structure of how we mapped entities). It is a super underrated query language and while most of the queries could also be translated to relational SQL, Cypher's linear construction using WITH clauses is far, far easier to reason about, IMO. EDIT: feel like the devs went overboard with the mix of languages. Shoehorned in C# Blazor? Using JS and Jest for e2e testing? [0] https://drasi.io/reference/query-language/ https://drasi.io/reference/query-language/
- JanSt 2y agoI too have great memories of cypher. Such an elegant way to write queries.
- CharlieDigital 2y agoIf you haven't been following it, I recently found out that it is now supported in a limited capacity by Google Spanner[0]. The openCypher initiative started a few years back and it looks like it's evolved into the (unfortunate moniker) GQL[1]. So it may be the case that we'll see more Cypher out in the wild. [0] https://cloud.google.com/spanner/docs/graph/opencypher-reference https://cloud.google.com/spanner/docs/graph/opencypher-refer... [1] https://neo4j.com/blog/cypher-gql-world/ https://neo4j.com/blog/cypher-gql-world/
- leeoniya 2y ago> while most of the queries could also be translated to relational SQL, Cypher's linear construction using WITH clauses is far, far easier to reason about, IMO. https://prql-lang.org/ https://prql-lang.org/
- CharlieDigital 2y agoDidn't look too deeply, but one of the keys with Cypher (at least in the context of graph databases) is that it has a nice way of representing `JOIN` operations as graph traversals. MATCH (p:Person)-[r]-(c:Company) RETURN p.Name, c.Name Where `r` can represent any relationship (AKA `JOIN`) between the two collections `Person` and `Company` such as `WORKS_AT`, `EMPLOYED_BY`, `CONTRACTOR_FOR`, etc. So I'd say that linear queries are one of the things I like about Cypher, but the clean abstraction of complex `JOIN` operations is another huge one.
- UltraSane 2y agoThe neat thing about Neo4j is that the [r] isn't a join, it is an actual relationship stored on disk.
- f4c39012 2y agoPurple!
- computronus 2y agoGreen!
- JaimeThompson 2y agoJust my luck. I get stuck with a race that speaks only in macros.
- fatliverfreddy 2y agoI wish I could use Cypher for everything
- hggigg 2y ago[flagged]
- _f9cu 2y agoWhat does it process it from and what does it process it to? Is it programmable or you have a concrete concept theorised? What is it useful for? How it helps business in saving cost or increasing profit? Is it a hobby project?
- agentofreality 2y agoThis post may help answer some of your questions: https://opensource.microsoft.com/blog/2024/10/22/detect-and-react-intelligently-to-changes-in-data-with-drasi/ https://opensource.microsoft.com/blog/2024/10/22/detect-and-... And the project docs may help answer others: https://drasi.io/ https://drasi.io/ Also, the Drasi team are most active over on our discord channel (https://aka.ms/drasidiscord https://aka.ms/drasidiscord) where we would be happy to answer questions and help you get started using Drasi.
- otterley 2y agoLooks very Azure-centric. Both installation guides (https://drasi.io/how-to-guides/install-sample-applications/building-comfort/ https://drasi.io/how-to-guides/install-sample-applications/b... and https://drasi.io/how-to-guides/install-sample-applications/curbside-pickup/ https://drasi.io/how-to-guides/install-sample-applications/c...) require Azure to work. And then there's this: > Installing Drasi in an EKS cluster can be significantly more complex than a standard installation on other platforms. Instead of downloading a CLI binary using the provided installation scripts, this approach requires modifying the source code of the Drasi CLI and building a local version of the CLI. Is this an actual requirement or just the current easy path?
- dtquad 2y agoThat is usual for new Microsoft open source projects. It takes 1-2 months for the Azure dependencies to go away.
- 3abiton 2y agoI'm curious about the other examples? I get it though, as many of these projects are built fulfilling a specific need within MS infrastructure.
- pjmlp 2y agoAzure is the new Windows, as timesharing OS, thus yeah that is to be expected.
- jameslevy 2y agoDoes it require Azure to work? Or could the Azure steps be relatively easily be swapped out for AWS/GCP/etc?
- agentofreality 2y agoDrasi does not require Azure to work. You install Drasi on a Kubernetes cluster, configure a Drasi Source to connect to a supported database, write Continuous Queries in Cypher Query Language, and configure a Drasi Reaction to do something with the output of the Continuous Query. For K8S, you can currently use k3s, kind, AKS (docs coming soon), or EKS (requires some work arounds). Drasi docs are here - https://drasi.io/ https://drasi.io/ The Drasi Team is most active over on discord, where we are happy to answer questions and help get people started using Drasi (https://aka.ms/drasidiscord https://aka.ms/drasidiscord)
- sitkack 2y agoBut at what cost?
- stefanos82 2y agoDrasi...React...well played Microsoft, well played :D Assuming they choose this name from the Greek δράση which means action, React of course is the exact opposite to action, thus the React-ion; an action expects a reaction, somewhere somehow!
- benbristow 2y agoNot like Microsoft to name things well...
- j-a-a-p 2y agoVMS++ = Windows NT?
- TeMPOraL 2y agoHere I thought they were accidentally or intentionally referring to: https://babylon5.fandom.com/wiki/Drazi https://babylon5.fandom.com/wiki/Drazi But now I noticed the spelling difference :/.
- resters 2y agoThis is a very solid pattern. Many systems that are built using traditional relational database systems would lend themselves to far simpler designs using this paradigm. It is not necessarily immediately obvious but nonetheless quite true.
- unit149 2y agoBeginning with Boolean operators: and / or - this relational service model can distribute queries. Curious why Cypher [0] abandons this syntax.
- SiddanthEmani 2y agoCypher is so cool. I included a graph database in my RAG patient chatbot https://github.com/SiddanthEmani/patient_chatbot https://github.com/SiddanthEmani/patient_chatbot
- akmittal 2y agoGo seem to be good choice for data processing systems.
- dxxvi 2y agoIs this what can be done with Apache Kafka Connect (to get data from another source to a Kafka cluster), Kafka (including Kafka Streams)? This image (https://github.com/drasi-project/community/raw/main/images/drasi_components.png https://github.com/drasi-project/community/raw/main/images/d...) is like Kafka Streams with a single topic. This image (https://github.com/drasi-project/community/raw/main/images/curbside_pickup_drasi.png https://github.com/drasi-project/community/raw/main/images/c...) is like joining 2 streams in Kafka Streams.
- ultrafez 2y agoIt also seems reminiscent of KSQL - consuming multiple input topics, and producing output to a topic defined using a query written in a SQL-like language that defines how the inputs are combined and filtered.
- fatliverfreddy 2y agoI see more Cypher fans out here - check out https://cyphernet.es https://cyphernet.es if you work with Kubernetes!
- jeremycarter 2y agoBrilliant
- UltraSane 2y ago[dead]
- mnsc 2y agoI finished reading Kleppman's Designing Data-Intensive Applications last night and this looks like it's straight out of the last chapter that talk about the future. They don't use the term "dataflow" though. https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/ https://www.oreilly.com/library/view/designing-data-intensiv...
- 9dev 2y agoThat one’s also on my reading list. Was it worth the read?
- yas_hmaheshwari 2y agoThis book is definitely worth the read. Or maybe worth 10 reads. Its really that awesome!
- xnorswap 2y agoI read it over the summer and I'd say it's essential reading for any developer who deals with data. Perhaps most importantly, the book empowered me to talk confidently about the trade-offs involved with different choices of handling data, and gave me a language framework to talk accurately about those choices. Previously even the parts I did understand was from experience, and not an academic background, so my explanations were hand-wavy or sloppy, but now I can state my case for different solutions much more clearly.
- ourmandave 2y ago[flagged]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- iamstan23 2y agoWeird thing about this project is that neither the website (https://drasi.io https://drasi.io) or the repo (https://github.com/drasi-project/drasi-platform https://github.com/drasi-project/drasi-platform) mention that it's a Microsoft project. Also the only cloud provider it has installation instructions for is AWS's EKS platform. Yet it has integration instructions for Azure CosmosDb Gremlin API. That one customer out there using EKS and Gremlin on CosmosDb is probably over the moon right now.
- deleted 2y ago[deleted]
- vladsanchez 2y agohttps://azure.microsoft.com/en-us/blog/drasi-microsofts-newest-open-source-project-simplifies-change-detection-and-reaction-in-complex-systems/ https://azure.microsoft.com/en-us/blog/drasi-microsofts-newe... > "The Microsoft Azure Incubations team is excited to announce that Drasi is now available as an open-source project."
- deleted 2y ago[deleted]
- agentofreality 2y agoI am the Drasi engineering manager, and I can assure you that any weirdness you sense is purely accidental. The project was announced through standard Microsoft channels by Mark Russinovich and is part of the Microsoft Collaboration on GitHub. But we are predominantly an open source project from the Azure Incubations team which has a history of releasing open source projects. So we don't feel the need to constantly remind everybody that we are a Microsoft project and team. The documentation site is missing some content that wasn't ready in time for the release but it includes AKS install instructions as well as additional Source and Reaction docs. These will be out soon. If you know that customer that uses EKS and Cosmos Gremlin, please let us know, we would also be over the moon. In any case, the Drasi Team is most active over on our discord channel(https://aka.ms/drasidiscord https://aka.ms/drasidiscord) where we would love to answer any questions you have about Drasi and help you get up and running.
- cheaprentalyeti 2y ago[flagged]
- unixhero 2y agoI would really enjoy using it. But as a novice data intensive application developer, why would I not query the table 30 seconds and look for changes with a Python program (or another regular programming language)?
- bobnamob 2y agoOne of the best resources to understand "why would I not ... ?" in a data intensive context is Kleppmann's Designing Data-Intensive Applications[1] (mentioned elsewhere in the comments). There's a lot of nuance to why event streaming wins out over periodically "polling" a database, mostly about maintaining correctness while being able to scale horizontally. Taking a look at the Kafka docs [2] is also enlightening. [1] https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321 https://www.amazon.com/Designing-Data-Intensive-Applications... [2] https://kafka.apache.org/documentation/#gettingStarted https://kafka.apache.org/documentation/#gettingStarted
- agentofreality 2y agoHere is a new Drasi blog post that might help you understand why Drasi is preferable to writing code that periodically polls a database: https://opensource.microsoft.com/blog/2024/10/22/detect-and-react-intelligently-to-changes-in-data-with-drasi/ https://opensource.microsoft.com/blog/2024/10/22/detect-and-...
- Malidir 2y ago[flagged]
- deleted 2y ago[deleted]
- zombot 2y ago[flagged]
- yanis_t 2y ago[flagged]
- hellojason 2y ago[flagged]
- deleted 2y ago[deleted]
- lasermike026 2y agoI suppose students need to prepare to defend what they are writing. Also, teachers may need a bit of a demotion when making accusations of plagiarism or generated papers. Teachers at the very least should be able to reasonably prove their accusations. There is a greater problem with tutors writing students papers. If teachers and students worked more closely this wouldn't be an issue.
- dijksterhuis 2y agoi feel like this comment was maybe supposed to be posted under this article? https://news.ycombinator.com/item?id=41896973 https://news.ycombinator.com/item?id=41896973
- emmanueloga_ 2y agoThey don't mention "CDC" (Change Data Capture) directly anywhere, but I think that's what Drasi is? (they call it "Data Change Processing platform"). "Debezium", an alternative CDC system, is mentioned in the documentation and sources [1]. I'm not sure if Drasi uses Debezium, or aims to be compatible with it. Maybe someone here can shine more light on the relationship between these two? -- 1: https://github.com/drasi-project/drasi-platform/tree/main/reactions/debezium/debezium-reaction https://github.com/drasi-project/drasi-platform/tree/main/re...
- agentofreality 2y agoHi, I am the Drasi engineering manager, maybe I can clarify. Drasi is not intended to be another CDC alternative. It doesn't compete with Debezium, in fact we already have some integration with Debezium and hope to do more in the future. People often use CDC to replicate, consolidate, filter, and transform data. And sometimes they use it as a source of change events to build components/services that look for specific data changes and do something when they detect those changes. This kind of components/services tend to be more complex to build, operate, and maintain than expected. Especially if they bring together data from multiple sources, have complex criteria, need to react in near real-time, be secure, and resilient. Most people that have had to build these can probably agree that they would like it to be less complex. Drasi was created so people don't have to build this kind of component/service. They just write a Continuous Query (in Cypher), and then configure it to connect to supported Data Sources and Reactions (which do something when changes are detected). Drasi manages the connection to the Source systems to get the low-level changes when they occur (sometimes using the Debezium library), maintains a perpetually accurate result set for the Continuous Query, and every time a source change results in a change to the Continuous Query result, Drasi generates a diff and sends it to the set of subscribed Reactions. The Reactions do something with those diffs depending on their purpose e.g. update a database, post an event, send an email, send a text message. All of this with no code in a platform that can scale to support many such queries. There is more to it, but a good starting point is if you ever think to yourself that you want to query a database and then compare the results to a previous query result, and you want to do this periodically, you might consider Drasi as an alternative. The Drasi Team is most active over on our discord channel(https://aka.ms/drasidiscord https://aka.ms/drasidiscord) and we would be happy to answer questions and help you evaluate whether Drasi is something that might be useful to you.
- purpleidea 2y agoThis feels like a specialized version of https://github.com/purpleidea/mgmt/ https://github.com/purpleidea/mgmt/ but Microsoft only.