benlwalker

9 ms·

31.
by benlwalker 5y ago
A flush command only guarantees, upon completion, that all writes COMPLETED prior to submission of the flush are non-volatile. Not all previously sent writes. NVMe base specification 2.0b section 7.1. That's a very important distinctio
32.
by benlwalker 5y ago
On very recent Linux kernels you can open the raw NVMe device and use the NVMe pass thru ioctl to directly send NVMe commands (or you can use SPDK on essentially any Linux kernel) and bypass whatever the fsync implementation is doing. That
33.
by benlwalker 5y ago
NVMe is certainly not more complex than either AHCI or SAS from a driver or hardware interface standpoint. And I've written drivers for all three that run in production today. Now maybe the controller design is more difficult - I would
34.
by benlwalker 5y ago
To expand on the batching details, I haven't seen exactly what he's doing here, but historically the numbers quoted are from benchmarks that submit something like 128 total queue depth in alternating batches of 64. The disks hit m
35.
by benlwalker 5y ago
There's one more aspect that I don't think gets enough consideration - you can batch many operations on different file descriptors into a single syscall. For example, if epoll is tracking N sockets and tells you M need to be rea
36.
by benlwalker 5y ago
I don't have any formal comparisons handy but there's lots of NVMe-oF benchmarks here: https://spdk.io/doc/performance_reports.html NVMe-oF can do both TCP (akin to iSCSI) and RDMA (akin to iSER). There are l
37.
by benlwalker 5y ago
There's more cool stuff coming in this area too. For a long time there's been the virtio family of protocols for shuttling IO to something outside QEMU to handle. Originally that was always KVM and the implementation is called vho
38.
by benlwalker 6y ago
eBPF the bytecode is not particularly limited. You can parse complex formats like Arrow or Parquet even. The Linux kernel overlays a verifier on top which adds all sorts of draconian restrictions (for good reason). When people talk of eBPF
39.
by benlwalker 6y ago
They don't actually say that the lambda executes any closer to the data than any other instance. It would make sense for these to operate on the S3 nodes of course, but the description doesn't say one way or the other.
40.
by benlwalker 6y ago
I think, as with anything, it can be used in bad faith. By taking advantage of the fact that many of the stakeholders are only able to give an idea some basic consideration due to time constraints, it's often possible to build consensu
41.
by benlwalker 6y ago
Since the project I work on ( https://spdk.io ) largely produces a set of executables as output, it was most natural to write the tests in bash. There's one top level bash script that kicks off the full suite of tests and tho
42.
SPDK on Windows, including NVMe-oF initiator and target(spdk.io)
1 points by benlwalker 6y ago|0 comments
43.
by benlwalker 6y ago
Whether you get more IOPs with smaller I/Os depends on a number of things. Most drives these days are natively 4KiB blocks and are emulating 512B sectors for backward compatibility. This emulation means that 512B writes are often quite
44.
by benlwalker 6y ago
The system that was tested there was PCIe bandwidth constrained because this was a few years ago. With your system, it'll get a bigger number - probably 14 or 15 million 4KiB IO per second per core. But while SPDK does have an fio plug
45.
by benlwalker 6y ago
Plug for a post I wrote a few years ago demonstrating nearly the same result but using only a single CPU core: https://spdk.io/news/2019/05/06/nvme/ This is using SPDK to eliminate all of the overhe
46.
by benlwalker 6y ago
I tried to switch to Signal the other day. In the US on Android, most people use the default messaging app and that means a lot of MMS/RCS messages. All the messages between myself and my wife are RCS, for example. Signal failed to imp
47.
by benlwalker 6y ago
Much of this is written in the context of the new vfio-user proposal he linked at the end. This is a new, more flexible mechanism for implementing device emulation in a separate process. That emulation code can certainly be written in rust,
48.
by benlwalker 6y ago
I think QUIC-the-transport is potentially an interesting transport for storage, but the data there is often already encrypted, so the double encryption is a waste. Is it possible to only encrypt the QUIC headers but leave the data unmodifie
49.
by benlwalker 6y ago
SPDK's RocksDB integration really hasn't gotten a lot of love. There's really two main challenges we hit and then never revisited. First, the IO threads in RocksDB are a thread pool that assume they perform blocking operation
50.
by benlwalker 6y ago
io_uring is a fantastic development for the kernel, and I really can't praise it enough. However, there's still lots of reasons to use SPDK. Performance is still significantly better[0], and you can directly access all the of the
51.
by benlwalker 7y ago
DPDK runs on Linux and FreeBSD officially. All of the momentum is on Linux though* *I was recently part of an effort to add FreeBSD testing to DPDK because SPDK's FreeBSD test agent kept failing when we updated DPDK. I'm a core ma
52.
by benlwalker 7y ago
I assumed so, but then why all the comparisons to file systems that are designed to run on top of a flash translation layer? I would have expected something in big bold letters at the top saying this is designed to run without an FTL, where
53.
by benlwalker 7y ago
All of the flash SSDs I am familiar with have no fixed relationship between a logical block and the actual NAND media backing it. In other words, the device automatically wear levels internally and the only control the user has over wear is
54.
10M Storage I/O per Second from One Thread(spdk.io)
1 points by benlwalker 7y ago|0 comments
55.
by benlwalker 7y ago
Jens' benchmark for SPDK quoted there is far off from the numbers we (the SPDK community) measure. We are able to replicate his io_uring numbers though, so we agree that the new interface is a large improvement. We're working to m
56.
Welcome NVMe/TCP to the NVMe-oF Family of Transports(nvmexpress.org)
2 points by benlwalker 8y ago|0 comments
57.
SPDK and Intel Optane SSD DC P4800X Benchmarks(spdk.io)
1 points by benlwalker 9y ago|0 comments
58.
by benlwalker 9y ago
I'm the technical lead for the Storage Performance Development Kit ( http://spdk.io ). I have one of these in my development system and we're hoping to post benchmarks fairly soon. SPDK further reduces the latency at QD
59.
by benlwalker 10y ago
I'm one of the authors of SPDK (which includes an NVMe driver and an IOAT driver). If the community wants to add rust bindings to those two components I'd be very supportive.
60.
by benlwalker 10y ago
Submitting and completing a 4k I/O using SPDK is about 7 times more CPU efficient than the equivalent operation with libaio, which is opening a raw block device with O_DIRECT. Said another way, on a recent Xeon CPU you can expect to dr

More ›