4 ms·

You can read on the background and challenges here in Salman Niazi's PhD - http://kth.diva-portal.org/smash/get/diva2:1260852/FULLTEXT02.pdf http://kth.diva-por

You can read on the background and challenges here in Salman Niazi's PhD - http://kth.diva-portal.org/smash/get/diva2:1260852/FULLTEXT02.pdf http://kth.diva-portal.org/smash/get/diva2:1260852/FULLTEXT0.... This is not a new topic - WinFS (part of the failed Longhorn project) was the best known example, which tried to use SQL Server as the metadata store for a new windows FS. Performance killed them. The reason why we managed to get better performance compared to all previous attempts it three-fold: 1. we used an in-memory high performance DB (NDB) and we did not de-normalize our schemas (no caching!) 2. with denormalized tables, we needed cross-partition transactions at high performance 3. we did a reasonable job at re-implementing HDFS' POSIX-like model using read-isolated transactions and row-level locking and new protocol for sub-tree operations.

lioeters 7y ago
Thank you for raising these relevant points. They led me down a fascinating rabbit hole on this topic of "database as file system". This is way over my head, but to pique others' curiosity, here's the abstract: Recent improvements in both the performance and scalability of shared-nothing, transactional, in-memory NewSQL databases have reopened the research question of whether distributed metadata for hierarchical file systems can be managed using commodity databases. In this paper, we introduce HopsFS, a next generation distribution of the Hadoop Distributed File System (HDFS) that replaces HDFS’ single node in-memory metadata service, with a distributed metadata service built on a NewSQL database. By removing the metadata bottleneck, HopsFS enables an order of magnitude larger and higher throughput clusters compared to HDFS. Metadata capacity has been increased to at least 37 times HDFS’ capacity, and in experiments based on a workload trace from Spotify, we show that HopsFS supports 16 to 37 times the throughput of Apache HDFS. HopsFS also has lower latency for many concurrent clients, and no downtime during failover. Finally, as metadata is now stored in a commodity database, it can be safely extended and easily exported to external systems for online analysis and free-text search.
- daviesliu 7y ago
  Have you publish the workload tracing from Spotify? We are very interesting on the results you got, looking for a way to replicate that using our setup.
  - jamesblonde 7y ago
    The workload isn't public, but if you follow the same ratio of FS operations (60+% getBlockLocations (file stat), etc), you will get similar results. You can even assume that operations follow that distribution uniformly, although there will be bursts of different op types - in the paper, we added microbenchmarks for each FS operation to show their load on the system. That distribution is in the paper and Salman's thesis.
smacktoward 7y ago
> Performance killed them. I've often wondered if there's a point at which something like WinFS would become practical simply through the perpetual advance of hardware. MS started working on the concept back in the early '90s (as part of Cairo: https://en.wikipedia.org/wiki/Cairo_(operating_system) https://en.wikipedia.org/wiki/Cairo_(operating_system)), and eventually gave up on it in 2006, which was nearly 15 years ago now. Devices today have significantly more memory and compute power on tap than they did in 2006, and if you roll the comparison all the way back to 1991 we're on a completely different planet today. And there are entirely new types of hardware that could be leveraged now too, such GPUs, solid-state disks, etc. I have no idea if MS ever published their performance data, but if they did, it'd be interesting to use it to figure out what a theoretical machine that could run WinFS, as it was when MS gave up on it, fast enough to be useable would look like, and how close to that theoretical machine the machines of today are.
- ktpsns 7y ago
  I guess when the next Windows would include a novel FS, people would not care. After decades of experimenting (does anybody remember the fears of users when it was introduced that Longhorn deprecates the traditional filesystem for query-based dynamic folders and thelike... In the end it remained only as another "addition", not a replacement), people expect a desktop filesystem to do what it always did. Even more, the way how Android and iOS hides away the filesystem from the user in everyday tasks shows that for end users, the technical fundament isn't in the focus anymore. Modern filesystems is something which will change how servers ("backends") work but not how customers do. That's why I think MS burrows WinFS and will probably never dig it out again...
  - teh_klev 7y ago
    I doubt MS "burrows" (I think you mean "buried") WinFS. There were likely many good lessons learned from that project about file systems, and there were a few useful outcomes that are present in the current NTFS. You don't know, what you don't know until you try stuff. It's called research.
  - nightski 7y ago
jamesblonde 7y ago
The OP video describes a FS based on Innodb and MySQL. It gets relatively poor performance (7-9X slower for common FS ops). However, the killer FS ops for a DB are subtree operations - 'rm -rf', 'chown -R', 'chmod -R'. In research in this field, these are the operations that get less limelight but are most important for a working system. I hazard a guess that the above implementation could still be made many times faster (maybe even 10X) by using InnoDB's native API, instead of the much slower MySQL Server SQL API. Here's an example of what you can achieve by going native - 1m ops/sec on InnoDB 8 years ago. This is the main guy (sorry Mark!) behind RocksDB implementation: http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as-nosql-story-for.html http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-a...
fnord123 7y ago
4. HDFS is slow.
- jamesblonde 7y ago
  Slow == throughput or latency? HDFS is not nearly as 'slow' as object stores, like S3 (in latency and throughput for a single namespace).
  - fnord123 7y ago
    Slow == performance left on the table. The namenode is a threaded model with a lot of locks for each operation. This will improve with moving to Ozone and RocksDB under the hood (because Rocks doesn't leave much performance on the table). But it runs a lot of threads which can cause a lot of context switches. The RPC uses protobuf which is cpu intensive so that's stealing from your RPS (which matters if you want to process multiple tens of thousands of requests per second). Finally it's using Java and when you want to scale your namenode beyond 16GB heap then, afaik, you run into GC issues. If it were written in a non GC language then the namenode could scale a lot further. The memory size isn't helped by having bloated types passed around. One exmaple is UUIDs as strings all over the place because Java doesn't have a u128 native type. So you have 36 bytes for the UUID string and ~32 bytes of Java object overhead (though the JVM can unbox - it's not really something you should rely on magically happening). Aside from all this (which are things that are baked in and will never change) Hops demonstrates there there's a lot of performance available in HDFS. But comparing XFS on MySQL to HDFS on NDB isn't really the same thing.
    - jamesblonde 7y ago
      That's a lot of technologies and issues that you touch on. Your points have validity (apart from heap sizes - several HDFS teams manage 300+GB heaps), but you are missing the fundamental challenge - HDFS' semantics (POSIX-like) and its implications on the concurrency model. You can weaken the semantics of HDFS (remove atomic rename) and scale. If you don't want to do that you have to deal with the single-writer, multiple-reader concurrency semantics for the namespace. Speed up the RPC, separate namespace from blockspace (Ozone - btw, how do they handle global states like safe-mode without consensus between both parts?) - you will still bottleneck on mutations of the namespace. We provided a multi-writer, multi-reader concurrency solution (see paper) based on cross-partition transactions, normalized schemas, row-level locking, and an application-level protocol for subtree ops. That is the fundamental (hard) problem for scaling hierarchical FS'.
j88439h84 7y ago
What project are you referring to that gets better performance?
- fnord123 7y ago
  HopFS https://www.logicalclocks.com/introducing-hops-hadoop/ https://www.logicalclocks.com/introducing-hops-hadoop/ https://github.com/hopshadoop/hops https://github.com/hopshadoop/hops
hnlmorg 7y ago
Nice work there. I've been fascinated with the idea of a relational database as a file system backend since the days of Longhorn / WinFS. I even had an attempt at creating one myself as a hobby project[1] in FUSE. Performance was never an end goal for me (I was writing it in FUSE + Go after all). Whilst I did get some read operations working (and surprisingly well too considering the design choices I had made) unfortunately I never managed to get write operations working before I got distracted with another project. So it's really interesting to see other people tackle this idea and I'll take great pleasure in reading the thesis you've linked to. Thank you [1] https://github.com/lmorg/myfs https://github.com/lmorg/myfs if anyone is curious. Be warned though, code isn't release quality.
ken 7y ago
> Performance killed them. I wonder how much was due to legacy databases living in files. For example, let's say you need to read /etc/foo.conf a lot. That file has internal structure, too, so even if you need just one value from it, you have to ask the filesystem (database) for the bytes that make up that file, then parse the whole thing (maybe), just to get the one value you want. Keeping it in a database just adds overhead. In contrast, if your filesystem/database had actual structure, and stored the values of /etc/foo.conf in a schema, then every program that needed one value from it could do one query for just that value they need. That could be even faster than a dumb filesystem. This seems like an area where you need to change everything all at once, or face a long slog of updating every piece of data in your system.

lioeters 7y ago

Thank you for raising these relevant points. They led me down a fascinating rabbit hole on this topic of "database as file system". This is way over my head, but to pique others' curiosity, here's the abstract: Recent improvements in both the performance and scalability of shared-nothing, transactional, in-memory NewSQL databases have reopened the research question of whether distributed metadata for hierarchical file systems can be managed using commodity databases. In this paper, we introduce HopsFS, a next generation distribution of the Hadoop Distributed File System (HDFS) that replaces HDFS’ single node in-memory metadata service, with a distributed metadata service built on a NewSQL database. By removing the metadata bottleneck, HopsFS enables an order of magnitude larger and higher throughput clusters compared to HDFS. Metadata capacity has been increased to at least 37 times HDFS’ capacity, and in experiments based on a workload trace from Spotify, we show that HopsFS supports 16 to 37 times the throughput of Apache HDFS. HopsFS also has lower latency for many concurrent clients, and no downtime during failover. Finally, as metadata is now stored in a commodity database, it can be safely extended and easily exported to external systems for online analysis and free-text search.

daviesliu 7y ago
Have you publish the workload tracing from Spotify? We are very interesting on the results you got, looking for a way to replicate that using our setup.
- jamesblonde 7y ago
  The workload isn't public, but if you follow the same ratio of FS operations (60+% getBlockLocations (file stat), etc), you will get similar results. You can even assume that operations follow that distribution uniformly, although there will be bursts of different op types - in the paper, we added microbenchmarks for each FS operation to show their load on the system. That distribution is in the paper and Salman's thesis.

smacktoward 7y ago

> Performance killed them. I've often wondered if there's a point at which something like WinFS would become practical simply through the perpetual advance of hardware. MS started working on the concept back in the early '90s (as part of Cairo: https://en.wikipedia.org/wiki/Cairo_(operating_system) https://en.wikipedia.org/wiki/Cairo_(operating_system)), and eventually gave up on it in 2006, which was nearly 15 years ago now. Devices today have significantly more memory and compute power on tap than they did in 2006, and if you roll the comparison all the way back to 1991 we're on a completely different planet today. And there are entirely new types of hardware that could be leveraged now too, such GPUs, solid-state disks, etc. I have no idea if MS ever published their performance data, but if they did, it'd be interesting to use it to figure out what a theoretical machine that could run WinFS, as it was when MS gave up on it, fast enough to be useable would look like, and how close to that theoretical machine the machines of today are.

ktpsns 7y ago
I guess when the next Windows would include a novel FS, people would not care. After decades of experimenting (does anybody remember the fears of users when it was introduced that Longhorn deprecates the traditional filesystem for query-based dynamic folders and thelike... In the end it remained only as another "addition", not a replacement), people expect a desktop filesystem to do what it always did. Even more, the way how Android and iOS hides away the filesystem from the user in everyday tasks shows that for end users, the technical fundament isn't in the focus anymore. Modern filesystems is something which will change how servers ("backends") work but not how customers do. That's why I think MS burrows WinFS and will probably never dig it out again...
- teh_klev 7y ago
  I doubt MS "burrows" (I think you mean "buried") WinFS. There were likely many good lessons learned from that project about file systems, and there were a few useful outcomes that are present in the current NTFS. You don't know, what you don't know until you try stuff. It's called research.
- nightski 7y ago

jamesblonde 7y ago

The OP video describes a FS based on Innodb and MySQL. It gets relatively poor performance (7-9X slower for common FS ops). However, the killer FS ops for a DB are subtree operations - 'rm -rf', 'chown -R', 'chmod -R'. In research in this field, these are the operations that get less limelight but are most important for a working system. I hazard a guess that the above implementation could still be made many times faster (maybe even 10X) by using InnoDB's native API, instead of the much slower MySQL Server SQL API. Here's an example of what you can achieve by going native - 1m ops/sec on InnoDB 8 years ago. This is the main guy (sorry Mark!) behind RocksDB implementation: http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-as-nosql-story-for.html http://yoshinorimatsunobu.blogspot.com/2010/10/using-mysql-a...

fnord123 7y ago

4. HDFS is slow.

jamesblonde 7y ago
Slow == throughput or latency? HDFS is not nearly as 'slow' as object stores, like S3 (in latency and throughput for a single namespace).
- fnord123 7y ago
  Slow == performance left on the table. The namenode is a threaded model with a lot of locks for each operation. This will improve with moving to Ozone and RocksDB under the hood (because Rocks doesn't leave much performance on the table). But it runs a lot of threads which can cause a lot of context switches. The RPC uses protobuf which is cpu intensive so that's stealing from your RPS (which matters if you want to process multiple tens of thousands of requests per second). Finally it's using Java and when you want to scale your namenode beyond 16GB heap then, afaik, you run into GC issues. If it were written in a non GC language then the namenode could scale a lot further. The memory size isn't helped by having bloated types passed around. One exmaple is UUIDs as strings all over the place because Java doesn't have a u128 native type. So you have 36 bytes for the UUID string and ~32 bytes of Java object overhead (though the JVM can unbox - it's not really something you should rely on magically happening). Aside from all this (which are things that are baked in and will never change) Hops demonstrates there there's a lot of performance available in HDFS. But comparing XFS on MySQL to HDFS on NDB isn't really the same thing.
  - jamesblonde 7y ago
    That's a lot of technologies and issues that you touch on. Your points have validity (apart from heap sizes - several HDFS teams manage 300+GB heaps), but you are missing the fundamental challenge - HDFS' semantics (POSIX-like) and its implications on the concurrency model. You can weaken the semantics of HDFS (remove atomic rename) and scale. If you don't want to do that you have to deal with the single-writer, multiple-reader concurrency semantics for the namespace. Speed up the RPC, separate namespace from blockspace (Ozone - btw, how do they handle global states like safe-mode without consensus between both parts?) - you will still bottleneck on mutations of the namespace. We provided a multi-writer, multi-reader concurrency solution (see paper) based on cross-partition transactions, normalized schemas, row-level locking, and an application-level protocol for subtree ops. That is the fundamental (hard) problem for scaling hierarchical FS'.

j88439h84 7y ago

What project are you referring to that gets better performance?

fnord123 7y ago
HopFS https://www.logicalclocks.com/introducing-hops-hadoop/ https://www.logicalclocks.com/introducing-hops-hadoop/ https://github.com/hopshadoop/hops https://github.com/hopshadoop/hops

hnlmorg 7y ago

Nice work there. I've been fascinated with the idea of a relational database as a file system backend since the days of Longhorn / WinFS. I even had an attempt at creating one myself as a hobby project[1] in FUSE. Performance was never an end goal for me (I was writing it in FUSE + Go after all). Whilst I did get some read operations working (and surprisingly well too considering the design choices I had made) unfortunately I never managed to get write operations working before I got distracted with another project. So it's really interesting to see other people tackle this idea and I'll take great pleasure in reading the thesis you've linked to. Thank you [1] https://github.com/lmorg/myfs https://github.com/lmorg/myfs if anyone is curious. Be warned though, code isn't release quality.

ken 7y ago

> Performance killed them. I wonder how much was due to legacy databases living in files. For example, let's say you need to read /etc/foo.conf a lot. That file has internal structure, too, so even if you need just one value from it, you have to ask the filesystem (database) for the bytes that make up that file, then parse the whole thing (maybe), just to get the one value you want. Keeping it in a database just adds overhead. In contrast, if your filesystem/database had actual structure, and stored the values of /etc/foo.conf in a schema, then every program that needed one value from it could do one query for just that value they need. That could be even faster than a dumb filesystem. This seems like an area where you need to change everything all at once, or face a long slog of updating every piece of data in your system.