3 ms·

My first question whenever I read about a new database system: Does it scale horizontally, by throwing more machines at it? That's the one basic requirement fo

by moe 18y ago

My first question whenever I read about a new database system: Does it scale horizontally, by throwing more machines at it? That's the one basic requirement for use in a website backend these days.

patio11 18y ago
That's the one basic requirement for use in a website backend these days. There are plenty of quite profitable websites which do not have this requirement. It is almost peculiar to sites which are trying to show display advertising to groups of users larger than many nation-states. You can make an awful lot of money with one commodity server if your business model supports it. I used to have an effective CPM of $80 and I know one which has in excess of $500. No, that is not a typo. (That is on six digits of pageviews per month.) You know how much scaling you need when essentially get 50 cents a pageview? Not much at all. FogCreek has, if I recall correctly, one database server. I haven't read how many total machines they're using recently but its a "count on your fingers and toes" number rather than a "zomg we need a colo facility to ourselves" number.
- moe 18y ago
  Well, a business model that pays 50 cent a page view sounds nice for sure - I fear most of us don't share that luxury. I figured that most sites who would be interested in such a database either fall into the retail (recommendation) or the community (social graph) category. Both operate mostly on volume and the last thing you want is a hard bottleneck just when you're at the verge of becoming successful. But well, if your business model doesn't require scalability on the web-tier then yes, these concerns ofcourse don't apply.
  - aaronblohowiak 18y ago
    I don't believe you'd put all of your information in this kind of custom db. The author remarks that they are running with memory-mapped disk back-end, which means a single machine should be able to take you pretty darn far.
gnaritas 18y ago
No it isn't, it's a nice to have but in no way is that a requirement. Most websites still run on standard RDBMs that don't scale well across many machines and frankly, most websites won't ever need to scale beyond one big beefy box. Stop drinking the cool-aid, it's killing your perspective.
- moe 18y ago
  Actually it's well understood how to scale a RDBMS horizontally (sharding and partitioning). I simply asked whether there is a comparable strategy for this graph db. Also most websites probably don't need a graph database. But the few who do will likely also need the scalability - at least I cannot imagine many interesting web-applications where your one beefy box could possibly scale to a significant userbase (unless you're talking the z-series category of beefy). Ofcourse there are still many interesting applications outside the public interweb.
  - aaronblohowiak 18y ago
    So, assuming by "scaling" in the web space, you mean "be able to quickly respond to many reads and writes as my app generates" and not "handle large volumes of data", then you can do what LinkedIn does, which is to just copy the exact same graph to multiple machines, and rely on the fact that eventual consistency is good enough. You make things immediate for the user most concerned with immediate feedback (make them sticky to the graph their write updated) and for everybody else, a couple minutes of lag is no biggie. This is even more true in retail and other applications.
    - moe 18y ago
      Scaling in the web space usually means both; many reads/writes and large volumes of data. Yes, mirroring may work to a point but falls down eventually in write-heavy applications. Ideally you want something that you can just add machines to and it will scale near lineary. I'm not sure if that's entirely achievable for a graph search, but that's where my question was heading.
wheels 18y ago
At this point no, but that's a hurdle we'll cross when we get there. In theory it would be fairly easy to split things using one machine to do the hashing from name to indexes and then segmenting items across multiple machines.
anamax 18y ago
What are the odds that you'll become bigger than plentyoffish without having an influx of money that will let you rebuild?

by moe 18y ago

My first question whenever I read about a new database system: Does it scale horizontally, by throwing more machines at it? That's the one basic requirement for use in a website backend these days.

patio11 18y ago

That's the one basic requirement for use in a website backend these days. There are plenty of quite profitable websites which do not have this requirement. It is almost peculiar to sites which are trying to show display advertising to groups of users larger than many nation-states. You can make an awful lot of money with one commodity server if your business model supports it. I used to have an effective CPM of $80 and I know one which has in excess of $500. No, that is not a typo. (That is on six digits of pageviews per month.) You know how much scaling you need when essentially get 50 cents a pageview? Not much at all. FogCreek has, if I recall correctly, one database server. I haven't read how many total machines they're using recently but its a "count on your fingers and toes" number rather than a "zomg we need a colo facility to ourselves" number.

moe 18y ago
Well, a business model that pays 50 cent a page view sounds nice for sure - I fear most of us don't share that luxury. I figured that most sites who would be interested in such a database either fall into the retail (recommendation) or the community (social graph) category. Both operate mostly on volume and the last thing you want is a hard bottleneck just when you're at the verge of becoming successful. But well, if your business model doesn't require scalability on the web-tier then yes, these concerns ofcourse don't apply.
- aaronblohowiak 18y ago
  I don't believe you'd put all of your information in this kind of custom db. The author remarks that they are running with memory-mapped disk back-end, which means a single machine should be able to take you pretty darn far.

gnaritas 18y ago

No it isn't, it's a nice to have but in no way is that a requirement. Most websites still run on standard RDBMs that don't scale well across many machines and frankly, most websites won't ever need to scale beyond one big beefy box. Stop drinking the cool-aid, it's killing your perspective.

moe 18y ago
Actually it's well understood how to scale a RDBMS horizontally (sharding and partitioning). I simply asked whether there is a comparable strategy for this graph db. Also most websites probably don't need a graph database. But the few who do will likely also need the scalability - at least I cannot imagine many interesting web-applications where your one beefy box could possibly scale to a significant userbase (unless you're talking the z-series category of beefy). Ofcourse there are still many interesting applications outside the public interweb.
- aaronblohowiak 18y ago
  So, assuming by "scaling" in the web space, you mean "be able to quickly respond to many reads and writes as my app generates" and not "handle large volumes of data", then you can do what LinkedIn does, which is to just copy the exact same graph to multiple machines, and rely on the fact that eventual consistency is good enough. You make things immediate for the user most concerned with immediate feedback (make them sticky to the graph their write updated) and for everybody else, a couple minutes of lag is no biggie. This is even more true in retail and other applications.
  - moe 18y ago
    Scaling in the web space usually means both; many reads/writes and large volumes of data. Yes, mirroring may work to a point but falls down eventually in write-heavy applications. Ideally you want something that you can just add machines to and it will scale near lineary. I'm not sure if that's entirely achievable for a graph search, but that's where my question was heading.

wheels 18y ago

At this point no, but that's a hurdle we'll cross when we get there. In theory it would be fairly easy to split things using one machine to do the hashing from name to indexes and then segmenting items across multiple machines.

anamax 18y ago

What are the odds that you'll become bigger than plentyoffish without having an influx of money that will let you rebuild?