5 ms·

Just playing around a bit, even with soft durability, insertion speeds seem to be pretty slow (<100 inserts/s from ruby using bulk inserts on a i7 Macbook Pro w

by rb2k_ 13y ago

Just playing around a bit, even with soft durability, insertion speeds seem to be pretty slow (<100 inserts/s from ruby using bulk inserts on a i7 Macbook Pro with an SSD). Also: All the ruby driver doc seems wrong. 1.6 changed a lot of the APIs?

coffeemug 13y ago
< 100 inserts/s with soft durability is far too low. Shoot us an email to the google group (http://groups.google.com/group/rethinkdb http://groups.google.com/group/rethinkdb) and we'll see if we can work this out!
- rb2k_ 13y ago
  Seems the speed it really declines with the size of the inserted documents. My production data is a hash with 14-15 keys and string/int values of maybe 10 characters each When I loop and keep generating these documents: to_insert = {} 10.times {|i| to_insert["key#{i}"] = rand(33333).to_s * rand(6) } I get around 140/s. Without the insert() call, I reach 50.000+, so it doesn't seem to be the overhead. With simple documents (3 keys, int values) I reach 350-400. p.s. script @ https://gist.github.com/rb2k/5777997 https://gist.github.com/rb2k/5777997
  - coffeemug 13y ago
    Ah, I see. This is a known problem with the performance of the protobuf serialization library we're using in the Ruby driver (see https://github.com/rethinkdb/rethinkdb/issues/897 https://github.com/rethinkdb/rethinkdb/issues/897). It bottlenecks the CPU and should be fixed in the next release. In the meantime, you could try running a multithreaded/multiprocess script -- that would significantly increase throughput. Sorry you ran into this -- it'll be fixed ASAP.
    - rb2k_ 13y ago
      the driver doesn't seem to be threadsafe and just locks once I start using it in threads :(
      - coffeemug 13y ago
        Sorry -- the drivers are meant to be used in a way where you create a new connection in each thread. If you do it this way, things will work.
mglukhovsky 13y ago
Which parts of the ruby driver docs seem inconsistent / wrong? If there's a documentation issue I'd love to fix this ASAP.
- rb2k_ 13y ago
  It's fixed by now, but everything on http://www.rethinkdb.com/docs/guides/drivers/ http://www.rethinkdb.com/docs/guides/drivers/ was wrong. The old driver e.g. took several arguments, the 1.6 one takes a hash. The old version basically looked exactly like the python code.
molf 13y ago
Having the exact same problem, even with small documents. Using the NodeJS driver.
- mglukhovsky 13y ago
  If you're working with small documents, it's possible you're experiencing a performance bottleneck other than protobuf serialization. Could you provide some more info on the queries you're running and an example document on our Google Group so the rest of the team can weigh in? (http://groups.google.com/group/rethinkdb http://groups.google.com/group/rethinkdb)

by rb2k_ 13y ago

coffeemug 13y ago

< 100 inserts/s with soft durability is far too low. Shoot us an email to the google group (http://groups.google.com/group/rethinkdb http://groups.google.com/group/rethinkdb) and we'll see if we can work this out!

rb2k_ 13y ago
Seems the speed it really declines with the size of the inserted documents. My production data is a hash with 14-15 keys and string/int values of maybe 10 characters each When I loop and keep generating these documents: to_insert = {} 10.times {|i| to_insert["key#{i}"] = rand(33333).to_s * rand(6) } I get around 140/s. Without the insert() call, I reach 50.000+, so it doesn't seem to be the overhead. With simple documents (3 keys, int values) I reach 350-400. p.s. script @ https://gist.github.com/rb2k/5777997 https://gist.github.com/rb2k/5777997
- coffeemug 13y ago
  Ah, I see. This is a known problem with the performance of the protobuf serialization library we're using in the Ruby driver (see https://github.com/rethinkdb/rethinkdb/issues/897 https://github.com/rethinkdb/rethinkdb/issues/897). It bottlenecks the CPU and should be fixed in the next release. In the meantime, you could try running a multithreaded/multiprocess script -- that would significantly increase throughput. Sorry you ran into this -- it'll be fixed ASAP.
  - rb2k_ 13y ago
    the driver doesn't seem to be threadsafe and just locks once I start using it in threads :(
    - coffeemug 13y ago
      Sorry -- the drivers are meant to be used in a way where you create a new connection in each thread. If you do it this way, things will work.

mglukhovsky 13y ago

Which parts of the ruby driver docs seem inconsistent / wrong? If there's a documentation issue I'd love to fix this ASAP.

rb2k_ 13y ago
It's fixed by now, but everything on http://www.rethinkdb.com/docs/guides/drivers/ http://www.rethinkdb.com/docs/guides/drivers/ was wrong. The old driver e.g. took several arguments, the 1.6 one takes a hash. The old version basically looked exactly like the python code.

molf 13y ago

Having the exact same problem, even with small documents. Using the NodeJS driver.

mglukhovsky 13y ago
If you're working with small documents, it's possible you're experiencing a performance bottleneck other than protobuf serialization. Could you provide some more info on the queries you're running and an example document on our Google Group so the rest of the team can weigh in? (http://groups.google.com/group/rethinkdb http://groups.google.com/group/rethinkdb)