6 ms·

I see that MVCC is a planned feature. Why would you need MVCC for an embedded database? It seems like unnecessary overhead that conflicts with the performance g

I see that MVCC is a planned feature. Why would you need MVCC for an embedded database? It seems like unnecessary overhead that conflicts with the performance goals.

krenoten 8y ago
It's not needed for the single-key atomic record store, which is the sled bwtree index that is the current highest level module. MVCC is implemented in most popular embedded DBs because it is an effective way to manage mixed workloads that seek to read snapshots of the entire database at a single point in time as well as not blocking writes as this happens. This functionality is desirable for transactions that support mixed workloads. That's why I'm building it for a higher level module. This is a collection of modules that let you choose the abstraction and associated complexity that you want. There's also a modular pagecache that is totally decoupled and reusable for your own database experiments.
- eternalban 8y ago
  MVCC is a viable general purpose approach to deal with concurrency. It is the mother of all DAGs.
catwell 8y ago
I think they mean "embedded" as in "embedded in a program", not "targeting embedded hardware". In particular the README states they intend to use it in https://github.com/spacejam/rasputin https://github.com/spacejam/rasputin. This is a use case similar to LMDB, for instance, which has MVCC.
EugeneOZ 8y ago
In Rust abstractions are free.
- fooker 8y ago
  If that was so, there wouldn't have been a need for the unsafe mode.
  - PudgePacket 8y ago
    You're argument doesn't really make sense. Not all abstractions are workarounds for "unsafe" things. For example, unsafe is needed for dealing with the OS, no amount of abstraction will help you there. At some point you must talk to and accept memory that another process (OS) tells you is correct. I'm not so knowledgeable in Rust so I can't say definitely, but I think you should back up your statement with some (any) examples.
  - zacmps 8y ago
    Unsafe does not exist for performance. Quoting from the rust book: > Unsafe Rust exists because, by nature, static analysis is conservative. When the compiler is trying to determine if code upholds the guarantees or not, it’s better for it to reject some programs that are valid than accept some programs that are invalid. That inevitably means there are some times when your code might be okay, but Rust thinks it’s not! In these cases, you can use unsafe code to tell the compiler, “trust me, I know what I’m doing.” The downside is that you’re on your own; if you get unsafe code wrong, problems due to memory unsafety, like null pointer dereferencing, can occur. > There’s another reason Rust has an unsafe alter ego: the underlying hardware of computers is inherently not safe. If Rust didn’t let you do unsafe operations, there would be some tasks that you simply could not do. Rust needs to allow you to do low-level systems programming like directly interacting with your operating system, or even writing your own operating system! That’s one of the goals of the language. Let’s see what you can do with unsafe Rust, and how to do it.
    - steveklabnik 8y ago
      As the person who wrote those paragraphs, yes. The fundamental reason unsafe exists is not for performance reasons. That being said, the parent is also true that sometimes, unsafe is used to enhance performance. But that's not the usual case, and in fact, can even hurt performance in ways. For example, &mut T can't alias, but *mut T can, and so &mut T can be optimized more aggressively. (We recently turned these optimizations back on: https://github.com/rust-lang/rust/pull/50744 https://github.com/rust-lang/rust/pull/50744 )
- p0nce 8y ago
  Do you realize "zero-cost" abstractions are not specific to Rust?
  - steveklabnik 8y ago
    Your parent didn't say that. Speaking about Rust in a thread about a Rust project seems to be very on-topic? (I would also disagree with that statement, but in a different way: Rust enables zero-cost abstractions, but nothing inherently means they have to be. I can also make quite costly ones, and they'll compile.)
    - EugeneOZ 8y ago
      I would answer "you can write trash in any language, in Rust you need to find ways how to do it, when in other languages - how to avoid it". Looks like ycombinator now is the place when people just looking for negativity and what else ti disagree with.
    - p0nce 8y ago
      > Your parent didn't say that. Yes and it's more disturbing. It says "In Rust abstractions are free" when an algorithm was discussed. Rust having "free" abstractions does not change the performance characteristics of an algorithm: this is what the grand-parent was talking about.
      - steveklabnik 8y ago
        Yes, I certainly agree with that distinction. If you'd have said that in the first place, I wouldn't have said anything :)
      - EugeneOZ 8y ago
        What is results disturbing is that you are arguing just for the sake of arguing. "Algorithm" is a too wide term - everything is an algorithm, especially in programming. Talk was about abstractions, now you're trying to say it was about algorithms. We need "ignore" button here.
- nixpulvis 8y ago
  Some are (more or less) free, others are not. Rust simply gives you some great tools to make the choices yourself. The biggest win tends to be that you can get away without a GC, but still keep the guarantee that you're not leaking memory by forgetting to collect it.
  - steveklabnik 8y ago
    Rust doesn't guarantee freedom of leaks, as leaking is memory-safe.
    - saghm 8y ago
      To be fair, I think the way GP worded it might still be accurate; they said that you wouldn't leak by forgetting to collect it. I took that as a statement about not needing to manually free things than an assertion that Rust guarantees no memory leaks (which can still occur from things like reference cycles).
      - steveklabnik 8y ago
        You’re right, I might be overly sensitive to people incorrectly making this claim directly.
        nixpulvis 8y ago
        I too try (and sometimes fail) to be careful when talking about this. People get all hot and bothered when you go overboard :P It can be tempting to start explaining what an afine type system is, but that can hurt more than help.
        steveklabnik 8y ago
        <3

I see that MVCC is a planned feature. Why would you need MVCC for an embedded database? It seems like unnecessary overhead that conflicts with the performance goals.

krenoten 8y ago

It's not needed for the single-key atomic record store, which is the sled bwtree index that is the current highest level module. MVCC is implemented in most popular embedded DBs because it is an effective way to manage mixed workloads that seek to read snapshots of the entire database at a single point in time as well as not blocking writes as this happens. This functionality is desirable for transactions that support mixed workloads. That's why I'm building it for a higher level module. This is a collection of modules that let you choose the abstraction and associated complexity that you want. There's also a modular pagecache that is totally decoupled and reusable for your own database experiments.

eternalban 8y ago
MVCC is a viable general purpose approach to deal with concurrency. It is the mother of all DAGs.

catwell 8y ago

I think they mean "embedded" as in "embedded in a program", not "targeting embedded hardware". In particular the README states they intend to use it in https://github.com/spacejam/rasputin https://github.com/spacejam/rasputin. This is a use case similar to LMDB, for instance, which has MVCC.

EugeneOZ 8y ago

In Rust abstractions are free.

fooker 8y ago
If that was so, there wouldn't have been a need for the unsafe mode.
- PudgePacket 8y ago
  You're argument doesn't really make sense. Not all abstractions are workarounds for "unsafe" things. For example, unsafe is needed for dealing with the OS, no amount of abstraction will help you there. At some point you must talk to and accept memory that another process (OS) tells you is correct. I'm not so knowledgeable in Rust so I can't say definitely, but I think you should back up your statement with some (any) examples.
- zacmps 8y ago
  Unsafe does not exist for performance. Quoting from the rust book: > Unsafe Rust exists because, by nature, static analysis is conservative. When the compiler is trying to determine if code upholds the guarantees or not, it’s better for it to reject some programs that are valid than accept some programs that are invalid. That inevitably means there are some times when your code might be okay, but Rust thinks it’s not! In these cases, you can use unsafe code to tell the compiler, “trust me, I know what I’m doing.” The downside is that you’re on your own; if you get unsafe code wrong, problems due to memory unsafety, like null pointer dereferencing, can occur. > There’s another reason Rust has an unsafe alter ego: the underlying hardware of computers is inherently not safe. If Rust didn’t let you do unsafe operations, there would be some tasks that you simply could not do. Rust needs to allow you to do low-level systems programming like directly interacting with your operating system, or even writing your own operating system! That’s one of the goals of the language. Let’s see what you can do with unsafe Rust, and how to do it.
  - steveklabnik 8y ago
    As the person who wrote those paragraphs, yes. The fundamental reason unsafe exists is not for performance reasons. That being said, the parent is also true that sometimes, unsafe is used to enhance performance. But that's not the usual case, and in fact, can even hurt performance in ways. For example, &mut T can't alias, but *mut T can, and so &mut T can be optimized more aggressively. (We recently turned these optimizations back on: https://github.com/rust-lang/rust/pull/50744 https://github.com/rust-lang/rust/pull/50744 )
p0nce 8y ago
Do you realize "zero-cost" abstractions are not specific to Rust?
- steveklabnik 8y ago
  Your parent didn't say that. Speaking about Rust in a thread about a Rust project seems to be very on-topic? (I would also disagree with that statement, but in a different way: Rust enables zero-cost abstractions, but nothing inherently means they have to be. I can also make quite costly ones, and they'll compile.)
  - EugeneOZ 8y ago
    I would answer "you can write trash in any language, in Rust you need to find ways how to do it, when in other languages - how to avoid it". Looks like ycombinator now is the place when people just looking for negativity and what else ti disagree with.
  - p0nce 8y ago
    > Your parent didn't say that. Yes and it's more disturbing. It says "In Rust abstractions are free" when an algorithm was discussed. Rust having "free" abstractions does not change the performance characteristics of an algorithm: this is what the grand-parent was talking about.
    - steveklabnik 8y ago
      Yes, I certainly agree with that distinction. If you'd have said that in the first place, I wouldn't have said anything :)
    - EugeneOZ 8y ago
      What is results disturbing is that you are arguing just for the sake of arguing. "Algorithm" is a too wide term - everything is an algorithm, especially in programming. Talk was about abstractions, now you're trying to say it was about algorithms. We need "ignore" button here.
nixpulvis 8y ago
Some are (more or less) free, others are not. Rust simply gives you some great tools to make the choices yourself. The biggest win tends to be that you can get away without a GC, but still keep the guarantee that you're not leaking memory by forgetting to collect it.
- steveklabnik 8y ago
  Rust doesn't guarantee freedom of leaks, as leaking is memory-safe.
  - saghm 8y ago
    To be fair, I think the way GP worded it might still be accurate; they said that you wouldn't leak by forgetting to collect it. I took that as a statement about not needing to manually free things than an assertion that Rust guarantees no memory leaks (which can still occur from things like reference cycles).
    - steveklabnik 8y ago
      You’re right, I might be overly sensitive to people incorrectly making this claim directly.
      - nixpulvis 8y ago
        I too try (and sometimes fail) to be careful when talking about this. People get all hot and bothered when you go overboard :P It can be tempting to start explaining what an afine type system is, but that can hurt more than help.
        steveklabnik 8y ago
        <3