9 ms·
A New Approach to Databases — Simulating 10Ks of Spaceships on my laptop
- lootsauce 14y agoI'm no database expert but won't the limitations of scaling across machines largely limit the benefits of this system?
- pron 14y agoProbably not -- if you're using the right data structure, that is. I've written about the theoretical performance here, http://highscalability.com/blog/2012/8/20/the-performance-of-distributed-data-structures-running-on-a.html http://highscalability.com/blog/2012/8/20/the-performance-of..., and pretty soon we'll publish some empirical results.
- papsosouid 14y ago>First, this means using a programming language built for concurrency. At present, there are two such languages in wide(ish) use: Erlang[2] and Clojure. How exactly does clojure qualify while go and haskell do not?
- pron 14y agoOh, Haskell qualifies, it's just not in wide(ish) use yet. Also, I don't know whether it supports mutable concurrent constructs (as Clojure does with refs and STM). Without those you don't have concurrent writes. As for Go, it allows sharing of mutable state, and so does not prevent races and does not make it clear when modifications to said mutable state are visible. It supports concurrency well but is not built around it.
- papsosouid 14y ago>Oh, Haskell qualifies, it's just not in wide(ish) use yet. Seems to be more widely used than clojure. >Also, I don't know whether it supports mutable concurrent constructs Yeah, Control.Concurrent.STM has vars, queues, channels, etc.
- mtrimpe 14y agoClojure runs on the JVM which means that it's 'politically usable' in many more places than Haskell
- papsosouid 14y agoYes, that is certainly the theory. Have you seen that actually happen in practice? Java shops almost exclusively continue to use java, and refuse to allow the dozens of other jvm languages. Every measure I can find shows haskell is more widely used than either clojure or erlang.
- tlack 14y agoCan you share your Haskell vs. Erlang data?
- papsosouid 14y agoJust the usual suspects (redmonk, langpop, tiobe).
- deleted 14y ago[deleted]
- banachtarski 14y agoAs a professional practictioner of all 3, haskell is emphatically not used more widely than clojure or erlang.
- papsosouid 14y agoHow does being a "professional practictioner of all 3" qualify one to make that assertion? And how do you reconcile that with the fact that every objective measure available shows otherwise?
- BrokenEnso 14y agoThe RedMonk ranking from September 2012 [1] seems to contradict your statement about Haskell prevalence. Note, it edges out Erlang on both metrics. [1] http://redmonk.com/sogrady/2012/09/12/language-rankings-9-12/ http://redmonk.com/sogrady/2012/09/12/language-rankings-9-12...
- deleted 14y ago[deleted]
- fghh45sdfhr3 14y agoWhat is RedMonk, and how is their ranking less B.S. than rankings according to Netcraft?
- deleted 14y ago[deleted]
- jvanenk 14y agoHaskell has marvelous concurrent constructs. It's actually one of the few places where I end up thinking "wow, I'm glad I get to use threads here" and I'm not thinking it sarcastically. http://hackage.haskell.org/packages/archive/base/4.6.0.1/doc/html/Control-Concurrent.html http://hackage.haskell.org/packages/archive/base/4.6.0.1/doc... http://hackage.haskell.org/packages/archive/stm/2.4.2/doc/html/Control-Concurrent-STM.html http://hackage.haskell.org/packages/archive/stm/2.4.2/doc/ht...
- sbov 14y agoWith OTP it's pretty easy to make an Erlang program with race conditions.
- lucian1900 14y agoIndeed. All 4 are reasonably mature, have excellent concurrency and parallelism stories and have been used practically for this purpose. The author is right about there being few languages that do this, though.
- carterschonwald 14y agoHaskell BEAUTFIFULLY qualifies, its just that STM and the other abstractions have been in GHC since around 2005-2007, so they're hardly "novel" in that community.
- eitan101 14y ago.
- robertfw 14y agoWhat are the pricing options for spacebase? I can't seem to find them
- dafnap 14y agoemail us at info@paralleluniverse.co and we'd be happy to give you the details.
- mikecx 14y agoUgh, maybe i'm the only one that hates this but if I have to spend time/effort just to get a price for something it's already off the table as a choice. This makes it seem like it's either too high priced and you know it or the pricing scheme is too complicated and should be simplified.
- pron 14y agoNo, we actually kinda hate it, too :) It's just that as a young startup, it's very important for us to learn about our customers. At this stage, it might be beneficial for us to offer you a fantastic deal if we think we there is much to gain by the partnership. Also, learning about the problems our customers face is more important to us than sheer sales volume at this stage. That's why we want you to talk to us. But we're sorry for the inconvenience.
- sethrin 14y agoI work freelance. When someone wants a price from me, they want a number. It might not be the number they want to hear. It might not be an accurate number. A lot of the time, it won't be the actual number used, and it will probably not have any relation to the final invoice. There are an infinity of reasons why you might not give someone a number when they ask for a price, and they're all irrelevant. It's okay to qualify the number. "Ten million dollars, but order now and..." If you'll note, your customer just said the same thing I did. Listen to him.
- riobard 14y agoCorrect me if I'm wrong, but doubling the performance of the simulation from a dual-core to a quad-core processor doesn't necessarily mean you have ”linear scalability” unless you can demonstrate the trend continues as more cores are used (e.g. 8-, 16-, and 32-cores). I think the ideal way to demonstrate this kind of claims while eliminating the effect of other factors such as memory performance is to use a beefy server with many cores (let's say N >= 16), and adjust your benchmark to use 1, 2, 3, ..., N cores, then plot the line to see if it's really linear. Two dots make a line in geometry, but you need more dots to project the trend. Time to spawn an EC2 Cluster Compute Eight Extra instance!
- cpressey 14y agoOT, but would you like some constructive criticism on usability of the article? The lack of contrast made it difficult to read for me; clicking on a footnote scrolls in such a way that the footnote text is hidden by the header; and having a link that says "Discuss on Hacker News" and also a comment section under the article sends a mixed message.
- SCdF 14y agoOT: your overlay that appears at the top of the screen if you scroll down far enough means that when you click a footnote you can't actually see the footnote without scrolling up a little.
- qznc 14y agoIt seems like the author considers 10K spaceships a lot. That is something like 1MB of of data. Not that much, even for a single core. Also, there should be multiple phases like, first everybody shoots, then everybody is hit, then blast force is applied. Otherwise there might be an unfair advantage for some spaceships, which can destroy their opponent, before it had the chance to return fire.
- jpollock 14y agoIt doesn't sound like a lot if it is processed linearly. However, 10,000 reader/writers on a single shared resource is a lot. Naive implementations would have a single global lock on the resource, resulting in lots of contention. They appear to be saying that by converting to callbacks, they are able to determine which transactions will interfere with each other and limit the contention to just those portions. Deadlock would be a serious possibility here, same as in regular databases. The size of the data is largely unimportant for this problem, it's the contention that's the issue. If the spaceships didn't have to interact through the global state then you could have sharding. You can change the problem by adding "planetary systems", and saying that ships in one system can't interact with another. Instant shard.
- pron 14y agoRight. That's the whole point: shards are a solution that imposes constraints on the application, meaning it's not really a solution. Also, SpaceBase guarantees no deadlocks.
- jpollock 14y agoAccording to the documentation, it does this by preventing callbacks from initiating transactions on the same SpaceBase store? :) :) :) Isn't that a constraint on the application? :) :) :) I think I'm going to have to look at the demo and see how a shot which triggers an explosion then does the query to perform the push of the surrounding ships. I would expect something like: Ship A: Query(origin+direction+range, callback Shot()) Shot() if Exploding Delete Query(origin, range, callback Boom()) Boom() Update location Except that's not allowed? An AOE which causes cascades of death would be cool to see too. Nifty idea.
- notdonspaulding 14y agoI'm confused. Is the author trying to convince me to give up the expressivity, portability, and readability of my Python app-layer codebase for the performance gains to be had by writing my logic in stored procedures? Having moved one codebase OUT of T-SQL stored functions and onto Django's ORM, I can say that this article does not have me convinced the way forward is to move the code back INTO the DB. If my data model is going to be dictating my development toolset, it better be giving me some amazing performance gains and be as good as the toolset I've come to enjoy in a post-ruby-on-rails-world. What am I missing?
- pron 14y agoYou don't have to give up anything. These "stored procedures" are not what you think they are, and neither is the database. The idea is that the db and the application become one - not that you have to program in plsql or something. You could continue to use python or ruby. And yes, the performance gains are amazing.
- nobodysfool 14y agoYou're missing ten thousand spaceships. He's gained performance by not returning a resultset for something that is just going to get written back to the database anyway. Normally, if you are writing a program, you get all the data from the database, then do some logic on it to determine what's going to happen. Say for example you store everything it the database regarding a million objects, and you want to have the ability for them to collide with each other. You'd get those million objects, then in your app layer, check if they are close enough to collide. What he's doing is using the database to determine if they are close enough, and then calculating the collisions in the database itself. I think if you just used the database to determine how close things are, and then switched to the app layer to do the appropriate collision calculations, you'd have a more configurable app. 10fps for 10k spaceships interacting with each other is not too shabby for a database. I'd rather not have so many updates in my database though, and I'd be okay with the fact that if my app crashed, the battles would have to start over again about 5 minutes in the past. However, you aren't writing games in Django's ORM that need realtime updates, so in your use case, it wouldn't make sense.
- capkutay 14y agoIs "Real time spatial database" just a fancy word for another distributed, in-memory datastore? There are many implementations of distributed data structures, and many are used for in-memory analytics. I realize spacebase also minimizes remote lookups/puts in the average case. I just want to understand the "spatial" aspect they try to integrate into their product.
- hendzen 14y agoThe spatial aspect comes from the ability to efficiently answer queries such as "what is set of objects within the following cube?" , etc. Look up R-Trees if you're curious.
- pc86 14y agoThis is more a UI/theme issue than anything else, but clicking a footnote places that citation behind the sticky header/search bar in FF17
- davidmr 14y agoThe problem sounds not dissimilar to hydrodynamic simulations, for which people have been using MPI for >20 years. I'm curious why you didn't use it. Can you speak to this?
- pron 14y agoBecause our purpose was not to show how fast a simulation can run. We haven't tried to come up with a new approach to simulations. We've tried to present a new approach to databases, and show an example where a naive implementation with the right database can have decent performance. Also, the very same code would work if actions were triggered by a network request rather than by a main loop. We don't assume anything about how the spaceships are coordinated. If you coordinate them to run in lockstep, you could get better performance. But, again, we've tried to show the viability of the naive implementation, where nothing is assumed.
- belorn 14y agoI wonder if this type of approach might push programming language developers to integrate databases into the language itself beyond just giving the programmers a interface for the database.
- threepipeproblm 14y ago"But if the new Moore’s law gives us exponential core-number growth, to exploit those cores we need to grow the number of shards exponentially as well, meaning we’re placing and exponential number of constraints on an application that probably only needs to accommodate a linearly increasing number of users, and that’s bad." I'm no expert on parallel programming, but isn't this sloppy thinking? Like the implication is that we must have an exponential increase in cores in order to accommodate a linear number of users... More generally is anything about this approach really novel? Databases can already run transactions and queries asynchronously. Any language that supports concurrent programming can take advantage of this, even if the database isn't issuing a callback. And ruling out concurrent languages that are capable of implementing this approach, but include some language features the authors don't like seems like a side issue. Not just trying to snark... if someone can explain what I'm missing I'm open to it!
- pron 14y ago> Like the implication is that we must have an exponential increase in cores in order to accommodate a linear number of users... As the article says, you can just ignore those extra cores. But, if you want to do cooler stuff with your data you'd better use them, and if so, sharding is a dead-end. > More generally is anything about this approach really novel? I said it's a repurposing of an old idea, that of the old RDMSs where the application was the database. The idea is not about concurrent transactions but about uniting the application and the database and letting the database run and parallelize the application's logic, and be the engine of scaling. I think most people see the database simply as a data storage component, and try to make it fast and scalable enough. We say, let the database run the application. > And ruling out concurrent languages that are capable of implementing this approach... I haven't ruled out any language. In fact, the demo is in Java, which certainly doesn't qualify. Any language can do this. I only said that in order to go forward, in order for the common programmer to take advantage of modern hardware, we must use languages built specifically for modern hardware. I've named two that I think fit the bill, but if other languages end up becoming the language for the new age - it's all the same.
- threepipeproblm 14y ago
- deleted 14y ago[deleted]