5 ms·
Dragon: A distributed graph query engine
- deforciant 11y agolooks interesting, where can I download Dragon? :D
- lynxaegon 11y agoI don't think it's open source yet. Maybe in the near future
- rubyfan 11y agoSlight tangent here but is it me or has the experience in social networks taken a hit? Feels like it was better with low complexity hack technology than with technology solutions focused on scaling the experience. I sometimes wonder if we are changing experience too much for technology optimization. - LinkedIn has been terrible for about 2-3 years since things stopped updating real time and the timeline started getting random - Facebook randomly suggesting things from your graph based on what it thinks you like - Twitter I heard soon will be no longer in chronological order?!? I think there is something to be said for low complexity MySQL and memcached.
- chrisseaton 11y agoI don't think any of those things are caused by optimisations or needing to scale. Saying that Facebook is recommending things simultaneously at random, based on a reccomendation algorithm, and also as an optimisation which wouldn't be needed if they used MySQL, is obviously nonsense.
- rubyfan 11y ago@chrisseaton - big fan of your work, eagerly awaiting Truffle JRuby production ready release. I'm not saying MySQL > some specialized solution for scale... I'm guess I'm drawing the correlation that when companies have to make investments like this they are also trading off experience, or maybe to your point screwing up the experience using recommenders and algorithms instead of letting my feed flow.
- chrisseaton 11y agoYeah I see your clarification in your other comment where you say that when they start to scale it opens the door for tons of other complexity. That makes more sense.
- bananaoomarang 11y agoI don't think that's to do with stack complexity, more like: "Crap how do we make money" "Crap how do we retain users/keep them here longer" The stack is to do with providing a fast performance to vast numbers + making it easier to work on for engineers than it might be at that complexity.
- rubyfan 11y agoyeah I hear that and to be clear I'm not blaming all their experience issues with the stack. But there are definitely stack complexity trade offs when you get to the scale of these guys. Also could be more correlation not causation that when you get to a certain scale and more stack complexity you also start hiring specialized staff that end up diluting the culture and spirit of a startup.
- eranation 11y agoNice! What are the benefits over TitanDB, Apache Spark/GraphX, GraphLab, Giraph, Neo4j, openCypher Tinkerpop etc?
- mark_l_watson 11y agoThe query examples looked like Clojure code; is that correct? The technology of Facebook and Google (where I worked as a contractor) impresses! That said, I like a simpler Internet with smaller scale services like Gnu Social, and simply using email to stay in touch with family and friends. I have this preference both for privacy reasons and also prefer more one on one communication.
- hellofunk 11y agoIt looks a lot like Clojure but I'm not sure it is. For example, this expression: (filter (> age 20)) That second form to filter should be a function, but in this case it is an expression that returns a boolean and is not itself a function.
- j_m_b 11y ago>That second form to filter should be a function, but in this case it is an expression that returns a boolean and is not itself a function. I'm pretty sure this is Clojure or something very close to it. ->> is the thread-last macro. What it does is take the first expression, ($alice) and inserts it as the last expr in the next form so that the second step (assoc $friends) would eval to (assoc $friends $alice). It continues this by inserting this expr into the last part of the next form. So the example would reduce to: (filter (> age 20) (assoc $friends (assoc $friends $alice)))
- hellofunk 11y agoBut as I mentioned, the second argument to Clojure's filter is a function, and (> age 20) is not a function; therefore, this is not normal Clojure (or Clojure or at all). You'd have to replace the standard library's filter function to get the syntax you see in this article, which I doubt they are doing. therefore: not Clojure.
- emidln 11y agoYou could also walk the ast and do something special with unknown symbols (and functions that contain them). i.e. while walking, we notice a function (> age 20) and age isn't visible in scope. We thus rewrite that into something like (fn [age] (> age 20)) or (fn [x] (> (:age x) 20)). I've taken this approach before with DSLs. It makes for extremely fast to write business logic where business users don't have to care as much about where their data comes from (they just need field names).
- jimbokun 11y agoThis sounds really cool. For distributed relational or graph databases, seems like the key trick for making queries efficient is to get related data on the same host, whenever possible. So would be cool to dig into the specifics of the algorithms they are using, to see exactly how they are optimizing where to store the data. With a graph database, it's impossible to guarantee having all related data together (eventually, friend of a friend of a friend...will be on a different host). So needs to be heuristics based. For tree shaped data, on the other hand, it is possible to have the root of each tree and all of its related data on the same host (assuming each tree is "reasonable" size). Google's F1 project took this approach.
- adsharma 11y agoHello, I wrote the blog post. There is a link to a video in the blog and we published something at NSDI last week: https://www.usenix.org/conference/nsdi16/technical-sessions/presentation/shalita https://www.usenix.org/conference/nsdi16/technical-sessions/...
- jimbokun 11y agoThanks for the link! Looks like an interesting read.