4 ms·

Scaling Early

18 points by nreece 18y ago

strlen 18y ago
Great read :-) Hooray for another Perl fan as well as for the simplicity and power of BerkeleyDB/other similar dbs. Have you look at Tokyo Cabinet / Tokyo Tyrant ( http://tokyocabinet.sourceforge.net/index.html http://tokyocabinet.sourceforge.net/index.html ) Also, what is the URL of the project/site/start-up you're scaling. I am interested very much in seeing the seeds of this effort.
- streety 18y ago
  The first slide says that the presentation is by Mark Maunder with an email address at http://feedjit.com http://feedjit.com so I'm guessing that's the site being optimized. The content in the presentation does make sense for what feedjit does.
- gcv 18y ago
  Tokyo Cabinet looks intriguing. One major limitation of Berkeley DB is that it cannot reliably store its files on a network filesystem (NFS or CIFS or what have you) because of unreliable POSIX semantics. It's a tough problem, but any idea if Tokyo Cabinet tackles it?
  - strlen 18y ago
    Using NFS doesn't seem like a good idea for this situation - from the operations perspective, NFS in production is going to be hell anyway. A better way would be (since Tokyo Cabinet is thread safe) having a separate thread/process which receives messages (using a message queuing system) that update the database files (ala MySQL replications). (Or you can separate out the db layer as a separate network service and partition the writes/reads to go to a specific several servers based on the key. Tokyo Cabinet even includes a memcached API compatible network layer for this - the Tokyo Tyrant).
JustAGeek 18y ago
Pretty nice. Hate to be a nitpicker but it's one of my pet peeves: What the slides describe isn't scaling but performance tweaking. Scalability != performance. A lot of people get that wrong, still annoys me, silly me... :D
- lallysingh 18y ago
  Well, there's two ways to look at scalability: - The ability to handle lots of load at all, which comes down to a stability and fault-tolerance issue. - The performance of the system under load. Both are pretty valid.
  - JustAGeek 18y ago
    No, only the first is valid. :) Some people _believe_ it's #2 but it isn't, sorry. :D "performance of the system under load" is just that, performance. See the Wikipedia article: http://en.wikipedia.org/wiki/Scalability http://en.wikipedia.org/wiki/Scalability
    - lallysingh 18y ago
      No, the latter does matter. If a system's performance significantly degrades under load, it doesn't scale. Quoting the wikipedia article: 'its ability to either handle growing amounts of work in a graceful manner, or to be readily enlarged.' The first clause is performing well under load, hence the performance requirement.

18 points by nreece 18y ago

Great read :-) Hooray for another Perl fan as well as for the simplicity and power of BerkeleyDB/other similar dbs. Have you look at Tokyo Cabinet / Tokyo Tyrant ( http://tokyocabinet.sourceforge.net/index.html http://tokyocabinet.sourceforge.net/index.html ) Also, what is the URL of the project/site/start-up you're scaling. I am interested very much in seeing the seeds of this effort.

streety 18y ago
The first slide says that the presentation is by Mark Maunder with an email address at http://feedjit.com http://feedjit.com so I'm guessing that's the site being optimized. The content in the presentation does make sense for what feedjit does.
gcv 18y ago
Tokyo Cabinet looks intriguing. One major limitation of Berkeley DB is that it cannot reliably store its files on a network filesystem (NFS or CIFS or what have you) because of unreliable POSIX semantics. It's a tough problem, but any idea if Tokyo Cabinet tackles it?
- strlen 18y ago
  Using NFS doesn't seem like a good idea for this situation - from the operations perspective, NFS in production is going to be hell anyway. A better way would be (since Tokyo Cabinet is thread safe) having a separate thread/process which receives messages (using a message queuing system) that update the database files (ala MySQL replications). (Or you can separate out the db layer as a separate network service and partition the writes/reads to go to a specific several servers based on the key. Tokyo Cabinet even includes a memcached API compatible network layer for this - the Tokyo Tyrant).

JustAGeek 18y ago

Pretty nice. Hate to be a nitpicker but it's one of my pet peeves: What the slides describe isn't scaling but performance tweaking. Scalability != performance. A lot of people get that wrong, still annoys me, silly me... :D

lallysingh 18y ago
Well, there's two ways to look at scalability: - The ability to handle lots of load at all, which comes down to a stability and fault-tolerance issue. - The performance of the system under load. Both are pretty valid.
- JustAGeek 18y ago
  No, only the first is valid. :) Some people _believe_ it's #2 but it isn't, sorry. :D "performance of the system under load" is just that, performance. See the Wikipedia article: http://en.wikipedia.org/wiki/Scalability http://en.wikipedia.org/wiki/Scalability
  - lallysingh 18y ago
    No, the latter does matter. If a system's performance significantly degrades under load, it doesn't scale. Quoting the wikipedia article: 'its ability to either handle growing amounts of work in a graceful manner, or to be readily enlarged.' The first clause is performing well under load, hence the performance requirement.