8 ms·
Apache Geode: Distributed, in-memory database
- orless 11y agoI'm starting to get lost in the new Apache animals. How does Apache Goede compares to Apache Ignite (advertised as "in-memory data fabric"?
- GregChase 11y agoApache Geode and Apache Ignite are more similar than they are different. Apache Ignite, based off the commercial distribution Grid Gain is newer to market. Apache Geode, based off the commercial distribution GemFire, has a long history in the market.
- orless 11y agoI wonder why Apache would need to have two "more similar than different" products.
- osi 11y agobecause people are interested in maintaining each
- wmf 11y agoApache accepts anything that companies donate. (They say they don't, but it's hard to find anything they've rejected.)
- rectang 11y agoThat's not quite the right emphasis. Apache is happy to provide a home for any community that is willing to adhere to our governance rules and traditions. Competing projects are OK. Projects are almost never rejected because preparing a proposal for incubation is rigorous and many projects who would be a poor fit self-select out. Source: former VP Apache Incubator, who has both helped prepare successful proposals and privately counseled projects who decided not to come to Apache.
- hkuser 11y agoIt looks better. Since 2002 in dev.
- espeed 11y agoIgnite vs Pivotal Gemfilre (Apache Geode) https://ignite.apache.org/use-cases/compare/gemfire.html https://ignite.apache.org/use-cases/compare/gemfire.html This is cool: "Ignite allows for most of the data structures from java.util.concurrent framework to be used in a distributed fashion." https://ignite.apache.org/features/datastructures.html https://ignite.apache.org/features/datastructures.html
- fiatjaf 11y agoLet me guess: it's written in Java?
- fiatjaf 11y agoYes. Why are all Apache projects are written in Java?
- nkozyra 11y agoThere are a few good reasons, but more importantly, why does it matter?
- timClicks 11y agoCan't speak for others, but I find deploying the JVM alongside non-JVM software to create a full product pretty frustrating.
- zidad 11y agoApache isn't written in Java :) https://en.wikipedia.org/wiki/Apache_HTTP_Server https://en.wikipedia.org/wiki/Apache_HTTP_Server
- rylee 11y agoThat's Apache httpd. Apache is a software organization.
- dang 11y agoAll: this is a major project (with a long history: http://geode.incubator.apache.org/about http://geode.incubator.apache.org/about) and a major release. It deserves a substantive thread, so please let's not get sidetracked by a language troll.
- Brainix 11y agoFeedback on the documentation: It's not immediately obvious to me how Geode is different from Redis. When would I want to use Geode over Redis, and vice versa?
- espeed 11y agoHere's an old post showing how to create a custom indexing schemes in Geode (QuadTreeIndex for spatial data): http://blogs.vmware.com/vfabric/2012/12/gemfire-patternspart-1-the-value-architecture-code-for-building-geography-based-apps.html http://blogs.vmware.com/vfabric/2012/12/gemfire-patternspart...
- GregChase 11y agoRedis and in-memory data grids are pretty different animals. I would characterize IMDG's like Geode to be concurrent write intensive, and have flexible data models. It also scales out better than Redis in a more automated fashion. Redis is a great read-intensive cache. It also has a powerful data model, but you have to use their data models. Example: If you want to run calculations on lists or sets, they have powerful operations you can call. IMDG's such as Geode were built with the rise of automated trading in the finance industry.
- deleted 11y ago[deleted]
- glogla 11y agoThis > Data is persisted in write-optimized disk storage. Consistency checking is configurable between highest performance caching and ACID transactions. seems pretty different to redis. I don't think Redis does ACID transactions.
- krenoten 11y agoRedis is single-threaded, so everything it does on a per-node basis is implicitly atomic. You can also force the single thread to handle a block of commands from a single socket at once by using MULTI (otherwise commands can be interleaved with other commands from other sockets).
- LoSboccacc 11y ago> Gemcached > Geode servers can be configured to talk memcached protocol. hey this is very interesting, it could work as a persistent acid memcached drop-in replacement!
- metatype 11y agoYep. Also some work underway to talk redis protocol at https://cwiki.apache.org/confluence/display/GEODE/Geode+Redis+Adapter https://cwiki.apache.org/confluence/display/GEODE/Geode+Redi....
- hyc_symas 11y agoThere's already memcacheDB for that https://github.com/LMDB/memcachedb https://github.com/LMDB/memcachedb
- LoSboccacc 11y agoThat one doesn't seem distributed. But I just glanced at the readme.
- hyc_symas 11y agoDistribution is usually handled by the memcache client libraries.
- mclarenfan 11y agoSome advantages of gemcached over memcached are here: https://cwiki.apache.org/confluence/display/GEODE/Moving+from+memcached+to+gemcached https://cwiki.apache.org/confluence/display/GEODE/Moving+fro...
- sandstrom 11y agoDidn't understand much from the linked page, but I found this website (from Pivotal, the commercial entity behind Geode) quite informative. Perhaps it's useful to others. https://pivotal.io/big-data/pivotal-gemfire https://pivotal.io/big-data/pivotal-gemfire === I found this interesting deployment: China National Railways use Geode to run railway ticketing for the entire country with a 10 node cluster, managing 2 TB of "hot data" in memory, and 10 backup nodes for high availability and elastic scale. Holiday travel periods [Chinese New Year's] create peaks of 15,000 tickets sold per minute, 1.4 billion page views per day and 40,000 visits per second. http://pivotal.io/big-data/case-study/scaling-online-sales-for-the-largest-railway-in-the-world-china-railway-corporation http://pivotal.io/big-data/case-study/scaling-online-sales-f...
- mclarenfan 11y agoAlso Indian railways: 200,000 Concurrent ticket Purchases https://pivotal.io/big-data/case-study/distributed-in-memory-data-management-solution https://pivotal.io/big-data/case-study/distributed-in-memory...
- trhway 11y agoto understand where it comes historically and ideologically/architecturally from - the keyword here is Smalltalk https://en.wikipedia.org/wiki/Gemstone_%28database%29 https://en.wikipedia.org/wiki/Gemstone_%28database%29 The guys have been for 3 decades sitting on the forested Oregon river bank (the place is called Beavertown for reason :) thinking straight and clear in Smalltalk... :)
- GregChase 11y agoGeode and its predecessor GemFire are pure Java. The original engineering expertise came from the team that built the Gemstone OODB. Beaverton isn't so forested anymore, but lots of indigenous object oriented expertise for sure.
- trhway 11y ago>Geode and its predecessor GemFire are pure Java. i know (was talking at some point about joining, already here in SV), and this is why i wrote only "3 decades" as i don't know precise dates of GemFire only suppose it to be beginning of 200x. My simple arithmetics mistake though - 2002 - 1982 is 2 decades, sorry, i see where confusion comes from :)
- tuyguntn 11y agoLanding page and documentation should be reconsidered. lots of unanswered questions, - distributed, in-memory database - how does it compare to SAP HANA? or Redis? if "... database" where is the querying language? if you have own querying language other than SQL please show us. - Performance is key - tell us about some benchmarks - Consistency is a must - CA or CP in CAP theorem? - Can I fully drop RDBMS in favor of Geode? - web site built as Landing page/PR for product, then all of a sudden starts Community/Contributors/Getting started columns, with loooooong list of mailing lists, smaller content for contributors and very important part(getting started) presented as tiny small link :(, please put there some useful info about product, you already have Community/Contribute menus.
- metatype 11y agoComplex objects can be stored by key and automatically partitioned and replicated. Query the data with OQL (an ODMG standard). For example: select * from /person p where p.name = 'foo' and p.age > 42
- tuyguntn 11y agofrom docs at[1] they only support SELECT statements in OQL, calling object method in query reminds me Esper complex event processing engine. [1] - http://geode-docs.cfapps.io/docs/getting_started/querying_quick_reference.html http://geode-docs.cfapps.io/docs/getting_started/querying_qu...
- finnh 11y ago> - Consistency is a must - CA or CP in CAP theorem? There is no CA. So, they must mean CP. Remember, kids: if they promise CA, run away.
- fweespeech 11y agoCA is possible if its read only, essentially, with all updates synchronized when a partition is not present [which is the majority of the time in the real world].
- GregChase 11y agoIf you are interested to join the developer community around Apache Geode, subscribe via: dev-subscribe@geode.incubator.apache.org
- avodonosov 11y agoWould you choose it over Datomic?
- espeed 11y agoDatomic's immutable storage and time-travel query capabilities are awesome, and I often miss them in other DBs. But Datomic currently isn't designed for write-intensive workloads. And while you can shard Datomic's transactor and then combine multiple DBs in a query (http://nosql.mypopescu.com/post/19310504456/thoughts-about-datomic#comment-465580077 http://nosql.mypopescu.com/post/19310504456/thoughts-about-d...), that's only going to get you so far. However, Apache Geode lets you add custom indexes so it might not be too hard to add Clojure's persistent data structures as a custom index scheme and hook in Apache Geode as a backend to Clojure Datalog: Clojure Datalog: https://github.com/fogus/bacwn https://github.com/fogus/bacwn Datscript: https://github.com/tonsky/datascript https://github.com/tonsky/datascript Clojure's Persistent Data Structures for Java: https://github.com/grignaak/clj-ds https://github.com/grignaak/clj-ds
- dberg 11y agoI believe this competes with another well known open-source In-Memory Data Grid, Hazelcast. worth checking out.
- threeseed 11y agoLast time I checked Hazelcast couldn't be run standalone. Also I wouldn't use it as a database for anything more than a few MB of data. This is probably closer to Apache Ignite aka Gridgain.
- dberg 11y agoCan you explain why ? I remember reading a ton about off-heap memory work they were doing
- rdtsc 11y agoSource here: https://github.com/apache/incubator-geode https://github.com/apache/incubator-geode
- Thaxll 11y agoIt looks similar to Hazelcast / Oracle Coherence?
- wener 11y agoYes, both Geocode, Coherence, Ignite and Hazelcast are quit similar.Grid computation.
- deleted 11y ago[deleted]
- espeed 11y agoTwo Videos: 1. "Open Sourced GemFire In-Memory Distributed Database and Apache Contributors" (https://www.youtube.com/playlist?list=PL62pIycqXx-TTMXsq09BEGfacLjeHZM28 https://www.youtube.com/playlist?list=PL62pIycqXx-TTMXsq09BE...) 2. "Creating a Highly Scalable Stock Prediction System with R, Geode & Spring XD" (https://www.youtube.com/playlist?list=PL62pIycqXx-Rzd_HcjU7YXq1kIOa28M8X https://www.youtube.com/playlist?list=PL62pIycqXx-Rzd_HcjU7Y...)
- jacques_chester 11y agoFWIW, Pivotal is hiring in our Big Data team, largely based in Palo Alto. Geode (incubating), HAWQ (incubating), Greenplum, Pivotal HD, MADlib etc are all mostly developed with engineering effort that we donate. Hit me up with an email (jchester@pivotal.io) or visit pivotal.io/careers if you're interested.
- lobster_johnson 11y agoCan anyone comment on Geode's non-Java support? I'm asking because a lot of Java "big data" stuff tend to prioritize Java clients (ZooKeeper, Kafka, Hadoop HDFS, Storm, VoltDB and HBase come to mind), and while there are sometimes clients in other languages, they tend to be second-class citizens that take years to reach feature/performance parity with the Java stuff. For example, last I checked there still wasn't a mature, feature-complete Kafka client (consumer and producer with built in offset management) for Go.
- mclarenfan 11y agoGemfire (on which geode is based) has a fully featured c, c++, c# client which has feature parity with the Java client. I don't know if pivotal is going to open source these clients too. There however is a REST api and a python client https://github.com/gemfire/py-gemfire-rest https://github.com/gemfire/py-gemfire-rest
- wiradikusuma 11y agoSorry for the stupid question: how do in-memory DBs deal with power failures? e.g. someone walking in the server room and trip power cable. I can understand for read-only in-memory, but what about writes?
- markito 11y agoIn the case of Geode, you can make a "Region" (think table) persistent on disk and using the concept of shared nothing architecture [1] to avoid SPOFs. What's also interesting is that we offer a very efficient way to recover data from disk as well[2] in the case of a crash of a single node or the entire cluster. [1] https://en.wikipedia.org/wiki/Shared_nothing_architecture https://en.wikipedia.org/wiki/Shared_nothing_architecture [2] http://gemfire.docs.pivotal.io/docs-gemfire/latest/managing/disk_storage/how_startup_works_in_system_with_disk_stores.html http://gemfire.docs.pivotal.io/docs-gemfire/latest/managing/...
- metatype 11y agoAlso, geode supports redundancy zones so data is replicated such that a cluster can survive a rack failure without data loss.
- markito 11y agoGeode FAQ Wiki page https://cwiki.apache.org/confluence/display/GEODE/Technology+FAQ https://cwiki.apache.org/confluence/display/GEODE/Technology...