5 ms·

"We store a lot of data... 2TB" Given that a pair of 2TB drives is less than $250 on ebuyer right now, 2TB of data is not 'big data'. You could comfortably stu

"We store a lot of data... 2TB" Given that a pair of 2TB drives is less than $250 on ebuyer right now, 2TB of data is not 'big data'. You could comfortably stuff that in any decent database (SQL Server for example, I'm sure PostgreSQL would work too). Just because a tiny machine on slicehost isn't big enough doesn't mean that your data won't fit in a normal database.

defroost 16y ago
> You could comfortably stuff that in any decent database (SQL Server for example, I'm sure PostgreSQL would work too). You may have noticed that the OP's requirements were: * a REST interface * sharding The costs of SQL Server are unbelievably high, and which you can do horizontal partitioning and sharding, Riak is designed to be used in this manner. On the other hand, re: SQL Server according to this article the "horizontal partitioning feature requires Enterprise or Datacenter licenses which have a retail price of 27,495 and 54,990 per processor." I think the OP made a wise choice. http://www.infoq.com/news/2011/02/SQL-Sharding http://www.infoq.com/news/2011/02/SQL-Sharding
- JoachimSchipper 16y ago
  You don't necessarily need to shard if the data fits comfortably on a standard disk... and the parent did mention PostgreSQL.
  - grncdr 16y ago
    Sharding (for free) on PostgreSQL is surprisingly painful
    - JoachimSchipper 16y ago
      I thought people just built sharding into the application level when using MySQL stuff - I'd imagine the same thing works just fine for PostgreSQL. Or were you talking about something else?
      - prodigal_erik 16y ago
        Anyone who volunteers for app level sharding is going to find themselves deep in Special Hell. Step one is to rewrite all your queries so they no longer expect really exotic use cases like "SELECT ... WHERE" to actually work until you somehow figure out which nodes to run them on (and if it was supposed to be a join between entities crossing shards, good luck with that). Step two is to find working XA-aware drivers (haha). For step three you need a knife and a goat.... Apparently really high-end clustered databases solve this problem correctly, i.e., the schema is extended to specify where any record can be found, the cluster uses that to work out a minimally-stupid query plan, and record->shard mapping becomes merely a tuning decision. I've never had the opportunity to work with one. But I'm not bitter.
        JoachimSchipper 16y ago
        Actually, if you use MySQL as a "great hashtable in the sky (cloud)", it's not all that much more painful to shard things. That said, if you do that you're already in a state of sin/pain...
- philwise 16y ago
  'Sharding' and 'a rest interface' are not really business requirements. Sharding is only a solution to handling certain types of high load, and REST is but one solution for IPC.
erikpukinskis 16y ago
Just because your data will fit in a normal database doesn't mean it will be fast enough.

defroost 16y ago

> You could comfortably stuff that in any decent database (SQL Server for example, I'm sure PostgreSQL would work too). You may have noticed that the OP's requirements were: * a REST interface * sharding The costs of SQL Server are unbelievably high, and which you can do horizontal partitioning and sharding, Riak is designed to be used in this manner. On the other hand, re: SQL Server according to this article the "horizontal partitioning feature requires Enterprise or Datacenter licenses which have a retail price of 27,495 and 54,990 per processor." I think the OP made a wise choice. http://www.infoq.com/news/2011/02/SQL-Sharding http://www.infoq.com/news/2011/02/SQL-Sharding

JoachimSchipper 16y ago
You don't necessarily need to shard if the data fits comfortably on a standard disk... and the parent did mention PostgreSQL.
- grncdr 16y ago
  Sharding (for free) on PostgreSQL is surprisingly painful
  - JoachimSchipper 16y ago
    I thought people just built sharding into the application level when using MySQL stuff - I'd imagine the same thing works just fine for PostgreSQL. Or were you talking about something else?
    - prodigal_erik 16y ago
      Anyone who volunteers for app level sharding is going to find themselves deep in Special Hell. Step one is to rewrite all your queries so they no longer expect really exotic use cases like "SELECT ... WHERE" to actually work until you somehow figure out which nodes to run them on (and if it was supposed to be a join between entities crossing shards, good luck with that). Step two is to find working XA-aware drivers (haha). For step three you need a knife and a goat.... Apparently really high-end clustered databases solve this problem correctly, i.e., the schema is extended to specify where any record can be found, the cluster uses that to work out a minimally-stupid query plan, and record->shard mapping becomes merely a tuning decision. I've never had the opportunity to work with one. But I'm not bitter.
      - JoachimSchipper 16y ago
        Actually, if you use MySQL as a "great hashtable in the sky (cloud)", it's not all that much more painful to shard things. That said, if you do that you're already in a state of sin/pain...
philwise 16y ago
'Sharding' and 'a rest interface' are not really business requirements. Sharding is only a solution to handling certain types of high load, and REST is but one solution for IPC.

erikpukinskis 16y ago

Just because your data will fit in a normal database doesn't mean it will be fast enough.