4 ms·

Its great to see two interesting languages used together. I wonder if an out of process solution was considered. For instance, Redis supports sorted sets: http

• jhgg 7y ago
  On a given node, we maintain a few million sorted sets that see a query and insertion velocity in the orders of millions of operations per second. When we are meticulously measuring operation time in the microsecond range to meet a performance target, a networked solution is out of the question.
  • jensneuse 7y ago
    What about a local redis instance using unix domain sockets?
    • jhgg 7y ago
      The cost of making a system call and network serialization + deserialization still makes this infeasible.
  • elcritch 7y ago
    Also, rustler handles compiling the rust code, and configuring the nif dynamic library details. I’d wager it’d be a ton easier than Redis infrastructure even if redis worked for the use case. The sheer amount of flexibility in the BEAM is one reason I love working in Elixir. Really it seems like a “write once and forget” type of nif. Have you had to make a lot of updates after the initial tuning with Rustler? Though I was curious if you tried ETS or mnesia? They both offer sorted sets. Not sure if they just use ordsets behind the scene. Edit: nvm, I see you answered in another thread.
    • jhgg 7y ago
      This data structure has been operating in production for almost a year now, we have not touched it for any modification after writing and deploying it. We had 0 issues deploying it, and thanks to the comprehensive testing suite and benchmarks, we were confident it would meet the performance objectives. It was actually a drop-in replacement for the pure elixir data structure we had written. Deploying it was literally just a find + replace of OrderedSet to SortedSet in our code, running our test suite and then gradually deploying the new code to production.
      • elcritch 7y ago
        Impressive work!