3 ms·

Thanks for this -- this is what I'm missing. I was starting to essentially suspect that NVMe is really more of a benefit for "hyperscaler"/IaaS providers... Thi

Thanks for this -- this is what I'm missing. I was starting to essentially suspect that NVMe is really more of a benefit for "hyperscaler"/IaaS providers... This lays it out well for me, and the how too. I rent my hardware so I don't have access to the hardware RAID stuff but this really filled out holes for me. > If you've got higher concurrency, you'll want several NVMe SSDs behind a hardware RAID controller, in RAID0. Hmnn why would you want RAID0 here? faster speed but absolutely no redundancy? what about RAID1+0/0+1? > That's the point when you're starting to reach those "basically identical to memory" figures that are the use-case for Intel's Optane. I need to do more reading on this but thanks for pointing this out

derefr 5y ago
> Hmnn why would you want RAID0 here? faster speed but absolutely no redundancy? what about RAID1+0/0+1? I can't speak to others†, but for our use-case, we're not using NVMe as "persistent storage but faster and smaller" but rather as "main memory but larger and slower". Our data isn't canonically on our nodes' NVMe pools, any more than it's canonically in the the nodes' local memory. The canonical representation of our (PGSQL data warehouse) data is its WAL log segments, shipped to/from object storage. The NVMe pool can be thought of as holding a "working" representation of the data — sort of like how, if you have a business layer that fetches a JSON document from a document store, then the business-layer node's main memory then holds a deserialized working representation of the data. You wouldn't back up/protect the data on your NVMe pool, any more than you'd back up the memory backing the deserialized JSON. If you wanted to modify the canonical data, you'd modify it in the canonical store. (For JSON, the document database; for our use-case, the WAL segments + base-backups in the object store.) † I would note that "large ephemeral working-set backing store" is seemingly largely considered by IaaS providers to be the main use-case for having NVMe attached to compute. The IaaS providers that support NVMe (AWS, GCP) only support it as instanced or scratch (i.e. ephemeral) storage, that doesn't survive node shutdown. These are strictly "L4 memory", not stores-of-record.
- hardwaresofton 5y ago
  > I can't speak to others†, but for our use-case, we're not using NVMe as "persistent storage but faster and smaller" but rather as "main memory but larger and slower". > Our data isn't canonically on our nodes' NVMe pools, any more than it's canonically in the the nodes' local memory. The canonical representation of our (PGSQL data warehouse) data is its WAL log segments, shipped to/from object storage. I appreciate this insight -- one of the features that drew me to KeyDB[0] was their FLASH feature, with drives being as fast as they are these days it feels like having tiered cache with DRAM + NVMe disk isn't so bad! Great to see others think this way as well. Redis is one thing but treating pg this way is somewhat interesting -- Do you also have the usual hot standby setup with synchronous replication just in case of a machine going down before the latest WAL segment is shipped out? > † I would note that "large ephemeral working-set backing store" is seemingly largely considered by IaaS providers to be the main use-case for having NVMe attached to compute. The IaaS providers that support NVMe (AWS, GCP) only support it as instanced or scratch (i.e. ephemeral) storage, that doesn't survive node shutdown. These are strictly "L4 memory", not stores-of-record. Reasonable, thanks for explicitly noting this. I think the write characteristics (and technically the read as this article shows) of NVMe do make it attractive but don't preclude using it as the store of record medium but what you said definitely syncs up with the other configurations I've seen and the offerings at AWS (I haven't really looked at GCP node types). [0]: https://github.com/EQ-Alpha/KeyDB/wiki/Legacy-FLASH-Storage https://github.com/EQ-Alpha/KeyDB/wiki/Legacy-FLASH-Storage
  - derefr 5y ago
    > Do you also have the usual hot standby setup with synchronous replication just in case of a machine going down before the latest WAL segment is shipped out? We don't have to — the data in our data warehouse is a rearrangement/normalization/collation of origin data from public sources. If our DB cluster dies, we stand up a new one from a backup, let it catch up from WAL segments, and then our ETL pipeline will automatically re-bind to it, discover the newest data it has available inside it, and begin ETLing to it. From our ETL ingestion-agent's perspective, there's no difference between "loading to a DB freshly restored from backup because it lost 10 minutes of updates" and "loading to a DB that is/was up-to-date because we just received some new origin data." (Our situation of our primary-source data being easily and reliably re-fetched is pretty unique to our vertical; but you can put pretty much any system into the same shape by first writing your primary-source data to an event store / durable unbounded-size MQ [e.g. Kafka] as you receive it; and then ETLing it into your DB from there. Forget DB disaster recovery; just keep the MQ data safe, and you'll always be able to replay the process that translates an MQ event-stream into a DB state.) We do have hot standbys, but they're async (using wal-g — master writes WAL segments to object store; replicas immediately fetch WAL segments from object store, as quickly as they can manage.) They're for query load scaling, not disaster recovery.

derefr 5y ago

> Hmnn why would you want RAID0 here? faster speed but absolutely no redundancy? what about RAID1+0/0+1? I can't speak to others†, but for our use-case, we're not using NVMe as "persistent storage but faster and smaller" but rather as "main memory but larger and slower". Our data isn't canonically on our nodes' NVMe pools, any more than it's canonically in the the nodes' local memory. The canonical representation of our (PGSQL data warehouse) data is its WAL log segments, shipped to/from object storage. The NVMe pool can be thought of as holding a "working" representation of the data — sort of like how, if you have a business layer that fetches a JSON document from a document store, then the business-layer node's main memory then holds a deserialized working representation of the data. You wouldn't back up/protect the data on your NVMe pool, any more than you'd back up the memory backing the deserialized JSON. If you wanted to modify the canonical data, you'd modify it in the canonical store. (For JSON, the document database; for our use-case, the WAL segments + base-backups in the object store.) † I would note that "large ephemeral working-set backing store" is seemingly largely considered by IaaS providers to be the main use-case for having NVMe attached to compute. The IaaS providers that support NVMe (AWS, GCP) only support it as instanced or scratch (i.e. ephemeral) storage, that doesn't survive node shutdown. These are strictly "L4 memory", not stores-of-record.

hardwaresofton 5y ago
> I can't speak to others†, but for our use-case, we're not using NVMe as "persistent storage but faster and smaller" but rather as "main memory but larger and slower". > Our data isn't canonically on our nodes' NVMe pools, any more than it's canonically in the the nodes' local memory. The canonical representation of our (PGSQL data warehouse) data is its WAL log segments, shipped to/from object storage. I appreciate this insight -- one of the features that drew me to KeyDB[0] was their FLASH feature, with drives being as fast as they are these days it feels like having tiered cache with DRAM + NVMe disk isn't so bad! Great to see others think this way as well. Redis is one thing but treating pg this way is somewhat interesting -- Do you also have the usual hot standby setup with synchronous replication just in case of a machine going down before the latest WAL segment is shipped out? > † I would note that "large ephemeral working-set backing store" is seemingly largely considered by IaaS providers to be the main use-case for having NVMe attached to compute. The IaaS providers that support NVMe (AWS, GCP) only support it as instanced or scratch (i.e. ephemeral) storage, that doesn't survive node shutdown. These are strictly "L4 memory", not stores-of-record. Reasonable, thanks for explicitly noting this. I think the write characteristics (and technically the read as this article shows) of NVMe do make it attractive but don't preclude using it as the store of record medium but what you said definitely syncs up with the other configurations I've seen and the offerings at AWS (I haven't really looked at GCP node types). [0]: https://github.com/EQ-Alpha/KeyDB/wiki/Legacy-FLASH-Storage https://github.com/EQ-Alpha/KeyDB/wiki/Legacy-FLASH-Storage
- derefr 5y ago
  > Do you also have the usual hot standby setup with synchronous replication just in case of a machine going down before the latest WAL segment is shipped out? We don't have to — the data in our data warehouse is a rearrangement/normalization/collation of origin data from public sources. If our DB cluster dies, we stand up a new one from a backup, let it catch up from WAL segments, and then our ETL pipeline will automatically re-bind to it, discover the newest data it has available inside it, and begin ETLing to it. From our ETL ingestion-agent's perspective, there's no difference between "loading to a DB freshly restored from backup because it lost 10 minutes of updates" and "loading to a DB that is/was up-to-date because we just received some new origin data." (Our situation of our primary-source data being easily and reliably re-fetched is pretty unique to our vertical; but you can put pretty much any system into the same shape by first writing your primary-source data to an event store / durable unbounded-size MQ [e.g. Kafka] as you receive it; and then ETLing it into your DB from there. Forget DB disaster recovery; just keep the MQ data safe, and you'll always be able to replay the process that translates an MQ event-stream into a DB state.) We do have hot standbys, but they're async (using wal-g — master writes WAL segments to object store; replicas immediately fetch WAL segments from object store, as quickly as they can manage.) They're for query load scaling, not disaster recovery.