8 ms·
Amazon Neptune – Fast, reliable graph database built for the cloud
- bdcravens 9y agoInteresting that it doesn’t support GraphQL, but rather Gremlin and SparQL. Surely that will impact adoption.
- chamakits 9y agoI believed that GraphQL is more of an protocol for interacting with an api, and not for graph databases, which is what Gremlin and SparQL are used for.
- jahewson 9y agoGraphQL is not a graph database query language. It's an alternative to REST.
- exogen 9y agoI love GraphQL and use it quite a bit, but the "graph" part of it is a bit of a misnomer given all the existing graph database and query technologies. It doesn't really offer anything in terms of interacting with RDF triples or making complex graph queries. It has no relational algebra semantics or ability to query relationships between arbitrary nodes, which is what folks using graph databases typically want. (I didn't downvote you though, it's a common misconception.)
- lolive 9y agoI second all the comments about GraphQL not being a graph query language. But we must agree that 90% of the quries we run on a graph database could be modelized with GraphQL. (retrieve nodes of a given type plus some of their properties). For the 10% other percent, Gremlin or SPARQL are then the way to go.
- lolive 9y agoThe lack of support of OpenCypher is also a bit intriguing.
- obi1kenobi 9y agoGraphQL can in fact be a graph query language -- my company has built and open-sourced a tool that makes that possible. Here's a blog post that describes how it works: https://blog.kensho.com/compiled-graphql-as-a-database-query-language-72e106844282 https://blog.kensho.com/compiled-graphql-as-a-database-query... If you want to try it out: pip install graphql-compiler
- nicklasss 9y agoSuper excited about this!!!!! BUT The preview link (https://pages.aws.com/NeptunePreview.html https://pages.aws.com/NeptunePreview.html) is broken, can anyone at AWS team help us with that?
- appwiz 9y agoIt’s fixed now: https://pages.awscloud.com/NeptunePreview.html https://pages.awscloud.com/NeptunePreview.html
- beebs_aws 9y agohttps://pages.awscloud.com/NeptunePreview.html https://pages.awscloud.com/NeptunePreview.html
- xvf22 9y ago"Your storage cost will be $0.10 per GB-mon0h," on https://aws.amazon.com/neptune/pricing/ https://aws.amazon.com/neptune/pricing/
- deleted 9y ago[deleted]
- Dryken 9y agoSadly they use Gremlin that is so often said to have poor performances
- chamakits 9y agoI believe Gremlin is just the query language. There is an original backend that implemented it, which might be what you are thinking has performance issues. But the query language intrinsically doesn’t have issues I don’t thinks.
- makmanalp 9y agoAFAIK gremlin is just a query language - it shouldn't have much to do with performance.
- rajman187 9y agoGremlin is indeed the query language but requires a gremlin engine. This is generally passing strings to the DB (which gives you advantages like pushdown-predecate, essentially DB-side filtering) but there is associated overhead with something like Cypher that is now serialised and very fast with the Bolt protocol
- Dryken 9y agothat was my point but my Rhetoric was not as good as yours :)
- lolive 9y agoCurrently I work on a project with Neo4J and Cypher. And I miss some of Gremlin tricks to optimize some graph traversals (for example to stop some sub-traversal when a given limit of matches have been reached).
- InverseFalcon 9y agoWhile this isn't natively supported yet, there are some tricks to achieve something like this, either with using APOC Procedures for subqueries, or, if your expansion-stop case is based on labels, APOC's path expansion procedures. https://neo4j.com/developer/kb/limiting-match-results-per-row/ https://neo4j.com/developer/kb/limiting-match-results-per-ro...
- Graphguy 9y agoIs this JanusGraph under the covers? Guessing since Neptune is a nod to Janus.
- igravious 9y agohttp://janusgraph.org/ http://janusgraph.org/ Support for various storage backends: - Apache Cassandra® - Apache HBase® - Google Cloud Bigtable - Oracle BerkeleyDB I don't understand how a database doesn't have its own native store. What exactly does a graph database actually do if it doesn't manage the data fed to it? Same is true for CayleyGraph† https://github.com/cayleygraph/cayley https://github.com/cayleygraph/cayley and proabably others. †Plays well with multiple backend stores: - KVs: Bolt, LevelDB - NoSQL: MongoDB - SQL: PostgreSQL, CockroachDB, MySQL - In-memory, ephemeral
- manigandham 9y agoJanusGraph provides a graph data model on top of an existing storage layer. In this case it's using wide-column key/value systems. It works well, letting each layer do what it's good at while limiting the amount of separate systems needing to be maintained.
- barakm 9y agoAs per the other commenter, Cayley provides a graph data model on top of an existing storage layer. However, when we get to manage the storage layer (Bolt, Level -- that's being generalized into local-KVs in the next release) we get to build our own indexes for better data management and performance. But there's no reason we can't hand that job off either -- hence supporting multiple (remote) backends. For the local stuff, though, at some point, Bolt is just a very good BTree implementation.
- riku_iki 9y ago> What exactly does a graph database actually do if it doesn't manage the data fed to it? It provides tools to run complex queries on graphs, and manages data models and indices to execute them fast.
- 9y ago
- joak 9y agoAre they using X1 ? https://aws.amazon.com/ec2/instance-types/x1/ https://aws.amazon.com/ec2/instance-types/x1/ For efficient graph DBs it's better to have a lot of ram and cores ...
- lolive 9y agoOr they choose a horizontally-scalable architecture, a la TitanDB. Btw, anyone knows how such solutions handle cross machine traversals? Are they schema-based? So the DB knows how to manage data locality and efficient joins/traversals?
- anonetal 9y agoI don't know about Neptune -- curious to hear what it is based on -- but TitanDB never really supported cross-machine traversals for the execution engine. The data was stored in a distributed fashion (across say a Cassandra cluster), but any instance of the execution engine was single-machine, with no easy way to talk between multiple instances of the execution engine.
- luisdbosquez 9y agoOne database service that supports horizontally scaled graphs is Azure CosmosDB Graph API: https://docs.microsoft.com/en-us/azure/cosmos-db/graph-introduction https://docs.microsoft.com/en-us/azure/cosmos-db/graph-intro... Worth to take a look if you need a managed Gremlin solution with some degree of global distribution.
- randomor 9y agoOnly had experience with Cypher, really liked it. It will be interesting to see how Neo4j responds to this. Regardless of tech specs, the fully-managed Neptune vs a community version on AWS Marketplace seems to give Neptune unfair advantage.
- mcphage 9y ago> seems to give Neptune unfair advantage What do you mean by "unfair" here?
- chatmasta 9y agoIt seems a lot of Amazon services are managed instances of open source applications. For example, commenters are suggesting this may be based on Janus. Elastic load balancers, at least originally, were likely based on haproxy. Etc etc. Has anyone ever considered the licensing implications of this? How is amazon able to convert an open source product into a proprietary one and then charge for access to it? Of course you can argue they’re charging for the infrastructure management, not the software itself. But that argument quickly breaks down as Amazon introduces new software, under new names, with a proprietary management interface over an open source core. Try to find the source code; you can’t. And if you accept the premise that they’re just charging for hosting, then it leads to the question of why an open source project doesn’t reap any benefits from that hosting, or at the very least, from the management interface on top of it. It seems like a better solution would be something akin to AWS marketplace, where open source projects are available to be hosted, and the maintainers can see some revenue from them. It seems like unfair rent seeking behavior that amazon is able to slap a management interface on open source software and then charge for it under the guise of “hosting.”
- eitland 9y ago> How is amazon able to convert an open source product into a proprietary one and then charge for access to it? Totally no problem with liberal licensed open source software. This is also the intended behaviour of such licenses. Also many of those big bad commercial companies contribute back big time to a number of projects. Why? I guess sometimes because devs want to and also because it makes sense business wise so they don’t have to maintain the code themselves.
- chatmasta 9y agoDepends on the license, doesn’t it? I’m not a licensing expert, but my understanding is GPLv2 / copy left licensing means that if you create a derivative product, you need to open source the new code along with the dependencies. Seems like a management interface is a clear cut derivative product. Where’s the source code? Or perhaps amazon does consider licensing and only builds on top of, eg Apache licensed projects?
- alexchamberlain 9y agoFor wider context here, is this leading the pack or do other public clouds have competing products already?
- arthursilva 9y agoIt could be a modified JanusGraph frontend backed by DynamoDB.
- kendallgclark 9y agoIf you actually read the docs, it's not Janus-based, it's based on BlazeGraph, which Amazon reportedly acquihired last year.
- jnwatson 9y agoIs that public information? I don't see any press releases about it.
- kendallgclark 9y agoThere was no PR. But there are traces, like Amazon acquired the domains, etc. Many former Blazegraph engineers are now Amazon Neptune engineers according to LinkedIn, etc. It was rumored widely in the graph db world fwiw.
- lolive 9y agoExcellent news! And congratulations to all the Blazegraph people. Afaik, Wikidata SPARQL endpoint runs on Blazegraph. Am I right? Then will Wikidata have to migrate to AWS to maintain the endpoint?
- nicolastorzec 9y agoTo add on Kendall's comment. Amazon owns the BLAZEGRAPH trademark: https://www.trademarkia.com/blazegraph-86498414.html https://www.trademarkia.com/blazegraph-86498414.html Blazegraph's CEO is currently at Amazon as Principal Product Manager: https://www.linkedin.com/in/bradley-bebee-a15764b/ https://www.linkedin.com/in/bradley-bebee-a15764b/
- brianbreslin 9y agoCan someone explain to me in lamens terms what a graph database is?
- mcphage 9y agoIt's a database that's designed to store relationships between objects instead of just facts. It has efficient methods of following long chains of associations. So think of how you store tree structures in a relational database—there are a lot of different ways of doing it, and they're all frustrating. Storing trees is something graph databases do naturally.
- kchoudhu 9y agoI've been playing around with graph databases for a while (I am writing one that turns Postgres into one of my own for kicks[1], [2]) and one of the things that became obvious after using the project in production was that it promotes functional reactive programming in a way that most other database paradigms don't. Even propagation and node invalidation are awesome for rapid what-if style experimentation, and I am so psyched to see more and more attention being paid to graph computing in general. [1] https://www.github.com/kchoudhu/openarc https://www.github.com/kchoudhu/openarc [2] https://www.anserinae.net/whats-cooking-openarc-edition.html#whats-cooking-openarc-edition https://www.anserinae.net/whats-cooking-openarc-edition.html...
- michaelbuckbee 9y agoTrying to get this straight in my mind here. Is it fair to say that traditional RDBMS/SQL are for storing different "sets" of related information (tables for products, users, orders). Graph databases are for storing data about the _same_ set of data as it interrelates to itself. - a User and and all their Friends (who are also users) - a Keyword and all associated Terms (which are also keywords) Is that right?
- InverseFalcon 9y agoI think you're concentrating on the wrong thing, here. Just as RDBMS can have tables about different things (Products, Users, Orders), graph databases can use labels on nodes for different things (so you can have :Product nodes, :User nodes, :Order nodes). Though with graph databases, there is often less rigidity in the associated data than in RDBMS, as there is no requirement for explicit schema for properties on nodes of different types in a graph db (plus you can multi-label nodes). The real differentiator is how relationships are modeled, and how they're traversed in queries. With RDMBS/SQL you're going to be working with data in tables, and use join tables as the relationships between them. You're likely going to need to be explicit about what is being joined together, so the relationship chain is likely to be very rigid. With graph databases, relationships and relationship traversal is used in place of join tables and table joins, which gives much more flexibility over how to traverse. You can certainly do friend-of-friend-of-friend queries much more easily, but you can also perform variable-length traversals using custom logic for which nodes are in the path and which relationships are traversed (type, direction, and count), and that can be very well-defined, or very loosely defined, or a mix, as needed. I don't believe there are good ways to do that kind of ad-hoc table joining in RDBMS. As an example of very loosely defined traversals in queries, you can ask for a shortest path between two nodes, knowing nothing about the nodes or relationships that could be between them, and get a path back showing the connecting nodes, with the relationships between the nodes providing context.
- Varcht 9y agoWhy "Neptune"? Having a hard time riddling that name out.
- alexbilbie 9y agoTwo other well known graph databases are "Janus" and "Titan" both of which are named after ancient gods
- lmeyerov 9y agoAwesome surprise to see the embargo lifted -- sounds like I can now say the Graphistry team will be doing a follow-up talk at Amazon Re:Invent tomorrow (Thursday) on Amazon Neptune + Graphistry. We've been incorporating this into visual investigation workflows for security, fraud, health records, etc. They've been doing cool bits on the managed graph layer, and were early to graph GPU tech (Blazegraph team members), and our side starts bringing that kind of thinking to visual GPU analytics & workflow automation tech. If you're in town and into this stuff, ping me at leo [at] graphistry, and would love to catch up Th/F for coffee+drinks. Also here + email, of course!
- rdslw 9y agoYet another amazon service to lock you in. And then after two years, when you're no longer startup with 100usd bill, but bigger company, you're completly tied to a jungle of amazon products, and your exit strategy is very very costly. clever amazon, clever.
- derefr 9y agoThere is some truth to this, but in a larger sense (on an ecosystem level, rather than from the perspective of an individual company), I can only be happy when AWS enters a new space. It makes that component into table-stakes in the IaaS game, which means every other big player is about to step up with their own offering as well, and the third-party SaaS and open-source self-hosted offerings in the same space all are going to heat up as well. Consider the evolution of container hosting services: first we had PaaSes like Heroku with proprietary container formats; then we got Docker, but Docker Swarm was nascent and there was no serious Docker Swarm IaaS-cloud offering. But then, very quickly, AWS built ECS; Google responded with Kubernetes; and then Kubernetes became the open standard, made everyone forget about Docker Swarm, and took over (and is even replacing ECS now.) That's what happens when AWS enters a space. And it's great.
- chiefalchemist 9y agoYes and no. If you factor in the cost of not taking "the easy way" you'd likely never get past the 100usd phase. You're point is valid. I'm just suggesting the lens / context isn't as one-sided as you've presented. Put another way, plenty of startups and VCs would love to have "getting out from under AWS" at the top of their good problems to have list.
- deleted 9y ago[deleted]
- abalone 9y agoSo I get that this offers simpler paradigm for graph data, but how should we interpret the "fast & scalable" claim? Is it... a) Slower than RDBMS/NoSQL but still pretty respectable, so it's a good choice for things like offline analysis. b) About the same at RDBMS/NoSQL, so you could use it to handle production traffic if you want. c) Faster, so you should definitely prefer it in production, e.g. for fetching upvotes and comments on posts.
- lolive 9y agoJust an off-topic comment: i am the maintainer of a visual query builder for SPARQL queries. cf http://datao.net http://datao.net This tool proposes to design query patterns from a graph data model, via drag n drops. The tool can then compile the patterns as SPARQL, run them on an endpoint and format the results as map/forms/tables/graphs/HTML (via templating)/... Another service of Datao (http://search.datao.net http://search.datao.net) proposes a search-engine view of those queries so you can type the textual representation of an object in any public SPARQL endpoint, and the service will list the queries currently available in Datao that can be applied upon this object. You can then run these queries with a click, and get the HTML templating of the query results. Feel free to have a look at the website, if you find any interest in this tool. ANy feedback is welcome. PS: Sorry for the poor quality of the videos. I manage this project on my spare time :)
- lolive 9y ago1 point by lolive 14 hours ago [-] I really hope Amazon will propose a facility to retrieve the RDFS data model of an endpoint in a uniform way.
- danburbridge 9y agoAnyone know if it supports Jena?
- hmm_really 9y agoWhat inferencing does it offer to RDF? How would I bolt on an inference engine to this if none is offered, i.e. to provide OWL:RL?