13 ms·

WinFS, Integrated/Unified Storage, and Microsoft

87 points by wolfgke 13y ago

twoodfin 13y ago
Great stuff. I don't know how I missed this when it was first posted. I hope Hal gets around to Part 5 at some point. From this history, ISTM that WinFS was doomed as soon as they decided to build it on top of the "Mighty Mouse" embedded SQL Server. SQL Server is, by all accounts, a well-designed piece of software, but something doesn't become "platform layer" just because you say it is. I don't blame Microsoft's other BU's for not wanting to put what is essentially another application layer between their code and its persistent data. On another note, the proposed client use cases 10-15 years later read as almost quaint. The vast majority of photo management these days happens on the web and in HTTP-using client applications, where sharing, not editing, is the killer feature. The right client bet was not to help users manage data on their own PCs, but to take over that responsibility for them. Through SkyDrive and other "cloud" capabilities, Microsoft can do that today, but imagine if they'd started 12 years ago. Facebook only exists because they got photo sharing "good enough"; there's no reason Microsoft couldn't have beaten them to it.
- csmuk 13y ago
  >The vast majority of photo management these days happens on the web and in HTTP-using client applications, where sharing, not editing, is the killer feature. You'd be surprised. There are many of us who don't do social photography and use desktop photographic software and Windows' built in image management stuff. And Microsoft make more money out of us with Windows licenses than Facebook do with social photography. The right client bet was the choice of doing it where you please, which they offer(ed). However, they're pushing everything onto SkyDrive now in an "everything everywhere" approach which is a bad move. Surprisingly Apple and DIY (Linux or own software) are the only people offering choice still, or at least not nagging you about moving to the cloud.
  - twoodfin 13y ago
    I know there are a lot of you, and there always will be. But how many more Windows Phone licenses would Microsoft be selling if they were the online photo sharing platform instead of a distant also-ran? WinFS was the wrong strategic move because cross-device availability and network effects dwarf whatever value you can get out of a local store. It's exactly the reason web mail won for consumers.
    - csmuk 13y ago
      Note: I own a windows phone. It syncs photos with SkyDrive, allows you to post to facebook etc. It allows choice. Suprisingly, Windows has always allowed choice. Web mail won for consumers for a bit. Now it's heading towards apps and sync which are omnipresent on all platforms now. WinFS was replaced with metadata indexing which didn't incur the system-level performance hit. Metadata indexing is there on iOS, Windows at least. Edit: I don't actually take photos with my phone unless it's a quick shot of my car parked to make sure I don't get a dodgy ticket.
  - aaronbrethorst 13y ago
    There are many of us who don't do social photography and use desktop photographic software and Windows' built in image management stuff. I fall into a different category. I do some 'social photography' as a means of personal promotion, but almost never with my iPhone's camera. Virtually 100% of the photos I upload to, say, Facebook and Instagram came out of one of my dedicated cameras, like my DSLR or digital rangefinder. All of my photos are shot in camera RAW, and I manage them in Lightroom. It wouldn't be possible for me to use built-in image management tools on OS X or Windows, or manage the photos through Finder or Explorer, respectively, since I need non-destructive photo editing. Lightroom provides non-destructive photo editing, and basically everything else I need, and provides it through their .lrcat catalog files, which, interestingly, are actually just sqlite3 databases.
    - csmuk 13y ago
      I use a DSLR, shoot raw and use Lightroom as well. In fact I just filled up 12Gb of Harry Potter studio tour today... However we're a pretty niche "prosumer" market. The really big sector is "cheapy digital compact" and use windows' tools because it's what they have and it's the least effort. The tooling is shit but it's good enough for most things. I mentioned the studio tour becuase there is a good distribution of "normal" camera owners there. The breakdown was pretty much 10% DSLR, 70% compact digital, 20% smartphone.
- ams6110 13y ago
  Facebook only exists because they got photo sharing "good enough" I think that's an oversimplification. There was photo-sharing before Facebook, and there were social networks with photo-sharing before Facebook. Facebook got the whole package to be "good enough" along with some luck at being in the right place at the right time.
  - twoodfin 13y ago
    Necessary, but not sufficient. I agree.
kalleboo 13y ago
It's interesting to contrast this to what Apple ended up doing in OS X - HFS on the bottom with some indexed, extensible metadata bolted on, then the Core Data API for structured storage backed by SQLite, but still represented in "files". I wonder how far Core Data could be stretched - could they replace all of HFS/a traditional file hierarchy if they were willing to kill (or hack in backcompat) support for legacy formats?
- csmuk 13y ago
  Actually they did the same on Windows Vista and above.
amaks 13y ago
Few data points from someone who actually worked in the latest iteration of unified storage, WinFS (2004-2006). The first problem was the lack of clear vision, contradictory requirements (AKA solving all world problems). For example, the data model had to be changed very late in the game. The previous data model was incredibly complex. It had entities of different types, links between them (of different types IIRC), it allowed multiple entities to be a parent of another entity, and from security stand point this was not solvable in any meaningful way. Shame that hundreds of people written tons of code for the flawed model without even realizing that it won't work, despite all high level architects involved into the project. At the end they realized that they couldn't possibly ship that, and decided to do a micro reset of WinFS' data model (around mid of 2005). Second problem was SQL Server (actually a fork of what became SQL Server 2005) which was optimized to work as a dedicated service on a server machine and required lots of effort to massage it to work relatively well with other resource consumers on an average consumer grade machine. Third, WinFS APIs were all managed and Longhorn Explorer was written in managed code as well. The whole thing was slow as hell and extremely unstable. By the end the teams working on WinFS and related projects were fairly motivated though, we could see the light and the decision to kill WinFS (and do Longhorn reset) was simply a matter of time. Ironically, the news of cancelling the project came the next day after the team declared WinFS Beta 2 ready. Now I think it was the right decision.
- frik 13y ago
  WinFS teaser video was so cool: http://www.youtube.com/watch?v=UhWCH8ebShY http://www.youtube.com/watch?v=UhWCH8ebShY Watching various Channel9 videos with the cool demo apps, and then trying WinFS beta 2 on WinXP, I was shocked how alpha quality it still was. It was clear for me that it wouldn't ship with Whistler/Vista. Sadly, all cool WinFS demo apps like the "Life Journal" timeline app (where you could zoom in and see all photos and videos on a vertical timeline) were missing in WinFS beta 2. All related videos got removed from Channel9 :/ Life Journal Timeline app screenshot: http://siliconvalleysleuth.co.uk/photos/uncategorized/img_6110.jpg http://siliconvalleysleuth.co.uk/photos/uncategorized/img_61... The WinFS "store" was actually a sub-folder of the "System Volume Information" directory (a hidden dir on C: partition). You could access the WinFS using an UNC path and some .Net based utilities.
yuhong 13y ago
Personally my favorite is the Jet 4.0 story and how the Access team ended up having to fork it for Access 2007. I think for a year or so, any development on Jet 4.0 was stopped until it was eventually resumed due to security.
- i386 13y ago
  Do you have a link to that story?
  - yuhong 13y ago
    I don't have a single source, but it is not hard to figure out. Some of it is in MichKa's posts on Google Groups, and notice for example the >1 year long gap between Jet 4.0 SP5 and SP6.
kazagistar 13y ago
I was hoping for more analysis for why WinFS failed from a technical perspective, but unfortunately it ended with a bunch of boring internal political maneuvering in microsoft.
alkonaut 13y ago
Sad to see how close we were to a windows release with the CLR being a first class citizen and a revolutionary storage system as well. Fast forward a decade, a couple of major windows releases, and we are still doing C++, NTFS and COM.
yajoe 13y ago
I love this read, and I would love to read entire books of people from the trenches making software. However, most of the specific details Hal cites are... well... dated from someone who learned the craft in the 1980. This line struck out to me as especially myopic: > Most of the world of commerce we are used to was made possible by the creation and growth of the concept of Structured Storage. The modern world of Credit Cards and ATMs is 100% predicated on this work. Amazon.com was in the realm of science fiction in the 1940s. By the 1970s the conceptual basis for everything you needed to create it was in place. It took until the 1990s for those concepts to mature sufficiently to let Amazon happen. I can't put my finger on why this seems wrong, and it could be such a strong contrarian view I need a moment to accept it. I feel like the 1990s .com boom AND the modern social boom may depend on structured storage, but structured storage certainly isn't the sufficient condition. Reading through Hal's prose, it struck me that HTTP is the very thing Hal set out to build, and it seems that since he was so focused on file systems he missed another obvious technology choice from which to draw. Would it have been dog slow for every application to make local HTTP requests to read files? Heck Yeah. Is it insane? Yeah. Would it have been slower than what they built? I don't know... And HTTP was so well understood and supported it would have allowed applications to start to mix content from local and remote sources, what we effectively have with pure JavaScript apps today. But HTTP was a standard, and Microsoft from those days was allergic to standards. Alas, someone will eventually build a JavaScript-based OS that treats all files as HTTP endpoints. And then we'll get photos to sync with metadata. Just saying.
- MichaelGG 13y ago
  Can you explain a bit of how HTTP would actually help here? HTTP doesn't magically fix the issues of mixing content local and remote (latency, availability, etc.) This sounds like "throw more XML at it" except with HTTP instead.
  - geocar 13y ago
    No it doesn't. And those are important problems, but another very important problem that needs solving is getting everyone to put all of their content into WinFS and make all of their services available from WinFS. Using an existing system bypasses a huge part of that.
andor 13y ago
“I know I saw a spreadsheet a couple of weeks ago; when I want to find it again do I look in my file system or do I look in my email?”. Bill was trying to make multiple points with this simple example, but the primary one was not that there should be a way to search across disparate stores. (from part 1 of the series) My first thought was: put your mail in a Maildir instead of that pst file and you have "unified" storage ;-) It seems they wanted to build something like the opposite of Unix: "everything is a (structured) database" instead of "everything is a file".
- viraptor 13y ago
  Why would Maildir help here? Unless the file itself was saved separately and linked from the email in an app-specific way you still wouldn't be able to open it directly in your spreadsheet app. (excluding fusefs-style magic) Unless you have a well-defined structures stored somewhere, you just get extensions on top of extensions... see things as simple as 'ar' files for example. Also Maildirs have lots of different formats and ways of indexing - each app treating the files as "just files" and doing it in a very specific way :( Sometimes I really appreciate the "here's an API, you will never know where/how it's stored and you shouldn't care" approach.
  - andor 13y ago
    Unless the file itself was saved separately and linked from the email in an app-specific way you still wouldn't be able to open it directly in your spreadsheet app. That's true. You might be able to find the right mail file using text-based tools, if the mail isn't MIME encoded, but then you still don't have the file. I just found it fun to think of it as the opposite of Unix. The Nepomuk framework in KDE seems to have similar goals by the way. They kept their database separate from the filesystem, but integrated it in quite a few KDE applications. Nepomuk uses a graph database, so the schema is easily extendable. But apparently they don't even use all features of the current schema, only rating, tagging and comments appear to be consistently implemented in applications. http://userbase.kde.org/Nepomuk http://userbase.kde.org/Nepomuk
philsnow 13y ago
Reading these articles, I'm struck by the refusal to admit influence from other systems (or even point out how Windows' features have inspired the same in other systems). > For example, Windows had implemented the TransmitFile function for optimizing transmission of files from a web server by doing all the work in kernel mode. Isn't that just sendfile(2) from BSD ? > An application could attach another stream to the file to hold whatever it wanted. The file system guys really didn’t care, and they didn’t interpret the stream. So this was a very natural extension. Isn't that just forks from HFS/+ ? I actually don't know the history behind all of these things, so the influence could have been the other way around (or the features could have been developed completely independently and contemporaneously). Edit: just saw the nod to DEC, so it's not that they're completely absent, just (IMHO conspicuously) missing in some places.
pak6 13y ago
is this helpful for wedding photography like this http://www.shakkyastudio.com/ http://www.shakkyastudio.com/

87 points by wolfgke 13y ago

Great stuff. I don't know how I missed this when it was first posted. I hope Hal gets around to Part 5 at some point. From this history, ISTM that WinFS was doomed as soon as they decided to build it on top of the "Mighty Mouse" embedded SQL Server. SQL Server is, by all accounts, a well-designed piece of software, but something doesn't become "platform layer" just because you say it is. I don't blame Microsoft's other BU's for not wanting to put what is essentially another application layer between their code and its persistent data. On another note, the proposed client use cases 10-15 years later read as almost quaint. The vast majority of photo management these days happens on the web and in HTTP-using client applications, where sharing, not editing, is the killer feature. The right client bet was not to help users manage data on their own PCs, but to take over that responsibility for them. Through SkyDrive and other "cloud" capabilities, Microsoft can do that today, but imagine if they'd started 12 years ago. Facebook only exists because they got photo sharing "good enough"; there's no reason Microsoft couldn't have beaten them to it.

csmuk 13y ago
>The vast majority of photo management these days happens on the web and in HTTP-using client applications, where sharing, not editing, is the killer feature. You'd be surprised. There are many of us who don't do social photography and use desktop photographic software and Windows' built in image management stuff. And Microsoft make more money out of us with Windows licenses than Facebook do with social photography. The right client bet was the choice of doing it where you please, which they offer(ed). However, they're pushing everything onto SkyDrive now in an "everything everywhere" approach which is a bad move. Surprisingly Apple and DIY (Linux or own software) are the only people offering choice still, or at least not nagging you about moving to the cloud.
- twoodfin 13y ago
  I know there are a lot of you, and there always will be. But how many more Windows Phone licenses would Microsoft be selling if they were the online photo sharing platform instead of a distant also-ran? WinFS was the wrong strategic move because cross-device availability and network effects dwarf whatever value you can get out of a local store. It's exactly the reason web mail won for consumers.
  - csmuk 13y ago
    Note: I own a windows phone. It syncs photos with SkyDrive, allows you to post to facebook etc. It allows choice. Suprisingly, Windows has always allowed choice. Web mail won for consumers for a bit. Now it's heading towards apps and sync which are omnipresent on all platforms now. WinFS was replaced with metadata indexing which didn't incur the system-level performance hit. Metadata indexing is there on iOS, Windows at least. Edit: I don't actually take photos with my phone unless it's a quick shot of my car parked to make sure I don't get a dodgy ticket.
- aaronbrethorst 13y ago
  There are many of us who don't do social photography and use desktop photographic software and Windows' built in image management stuff. I fall into a different category. I do some 'social photography' as a means of personal promotion, but almost never with my iPhone's camera. Virtually 100% of the photos I upload to, say, Facebook and Instagram came out of one of my dedicated cameras, like my DSLR or digital rangefinder. All of my photos are shot in camera RAW, and I manage them in Lightroom. It wouldn't be possible for me to use built-in image management tools on OS X or Windows, or manage the photos through Finder or Explorer, respectively, since I need non-destructive photo editing. Lightroom provides non-destructive photo editing, and basically everything else I need, and provides it through their .lrcat catalog files, which, interestingly, are actually just sqlite3 databases.
  - csmuk 13y ago
    I use a DSLR, shoot raw and use Lightroom as well. In fact I just filled up 12Gb of Harry Potter studio tour today... However we're a pretty niche "prosumer" market. The really big sector is "cheapy digital compact" and use windows' tools because it's what they have and it's the least effort. The tooling is shit but it's good enough for most things. I mentioned the studio tour becuase there is a good distribution of "normal" camera owners there. The breakdown was pretty much 10% DSLR, 70% compact digital, 20% smartphone.
ams6110 13y ago
Facebook only exists because they got photo sharing "good enough" I think that's an oversimplification. There was photo-sharing before Facebook, and there were social networks with photo-sharing before Facebook. Facebook got the whole package to be "good enough" along with some luck at being in the right place at the right time.
- twoodfin 13y ago
  Necessary, but not sufficient. I agree.

kalleboo 13y ago

It's interesting to contrast this to what Apple ended up doing in OS X - HFS on the bottom with some indexed, extensible metadata bolted on, then the Core Data API for structured storage backed by SQLite, but still represented in "files". I wonder how far Core Data could be stretched - could they replace all of HFS/a traditional file hierarchy if they were willing to kill (or hack in backcompat) support for legacy formats?

csmuk 13y ago
Actually they did the same on Windows Vista and above.

amaks 13y ago

Few data points from someone who actually worked in the latest iteration of unified storage, WinFS (2004-2006). The first problem was the lack of clear vision, contradictory requirements (AKA solving all world problems). For example, the data model had to be changed very late in the game. The previous data model was incredibly complex. It had entities of different types, links between them (of different types IIRC), it allowed multiple entities to be a parent of another entity, and from security stand point this was not solvable in any meaningful way. Shame that hundreds of people written tons of code for the flawed model without even realizing that it won't work, despite all high level architects involved into the project. At the end they realized that they couldn't possibly ship that, and decided to do a micro reset of WinFS' data model (around mid of 2005). Second problem was SQL Server (actually a fork of what became SQL Server 2005) which was optimized to work as a dedicated service on a server machine and required lots of effort to massage it to work relatively well with other resource consumers on an average consumer grade machine. Third, WinFS APIs were all managed and Longhorn Explorer was written in managed code as well. The whole thing was slow as hell and extremely unstable. By the end the teams working on WinFS and related projects were fairly motivated though, we could see the light and the decision to kill WinFS (and do Longhorn reset) was simply a matter of time. Ironically, the news of cancelling the project came the next day after the team declared WinFS Beta 2 ready. Now I think it was the right decision.

frik 13y ago
WinFS teaser video was so cool: http://www.youtube.com/watch?v=UhWCH8ebShY http://www.youtube.com/watch?v=UhWCH8ebShY Watching various Channel9 videos with the cool demo apps, and then trying WinFS beta 2 on WinXP, I was shocked how alpha quality it still was. It was clear for me that it wouldn't ship with Whistler/Vista. Sadly, all cool WinFS demo apps like the "Life Journal" timeline app (where you could zoom in and see all photos and videos on a vertical timeline) were missing in WinFS beta 2. All related videos got removed from Channel9 :/ Life Journal Timeline app screenshot: http://siliconvalleysleuth.co.uk/photos/uncategorized/img_6110.jpg http://siliconvalleysleuth.co.uk/photos/uncategorized/img_61... The WinFS "store" was actually a sub-folder of the "System Volume Information" directory (a hidden dir on C: partition). You could access the WinFS using an UNC path and some .Net based utilities.

yuhong 13y ago

Personally my favorite is the Jet 4.0 story and how the Access team ended up having to fork it for Access 2007. I think for a year or so, any development on Jet 4.0 was stopped until it was eventually resumed due to security.

i386 13y ago
Do you have a link to that story?
- yuhong 13y ago
  I don't have a single source, but it is not hard to figure out. Some of it is in MichKa's posts on Google Groups, and notice for example the >1 year long gap between Jet 4.0 SP5 and SP6.

kazagistar 13y ago

I was hoping for more analysis for why WinFS failed from a technical perspective, but unfortunately it ended with a bunch of boring internal political maneuvering in microsoft.

alkonaut 13y ago

Sad to see how close we were to a windows release with the CLR being a first class citizen and a revolutionary storage system as well. Fast forward a decade, a couple of major windows releases, and we are still doing C++, NTFS and COM.

yajoe 13y ago

I love this read, and I would love to read entire books of people from the trenches making software. However, most of the specific details Hal cites are... well... dated from someone who learned the craft in the 1980. This line struck out to me as especially myopic: > Most of the world of commerce we are used to was made possible by the creation and growth of the concept of Structured Storage. The modern world of Credit Cards and ATMs is 100% predicated on this work. Amazon.com was in the realm of science fiction in the 1940s. By the 1970s the conceptual basis for everything you needed to create it was in place. It took until the 1990s for those concepts to mature sufficiently to let Amazon happen. I can't put my finger on why this seems wrong, and it could be such a strong contrarian view I need a moment to accept it. I feel like the 1990s .com boom AND the modern social boom may depend on structured storage, but structured storage certainly isn't the sufficient condition. Reading through Hal's prose, it struck me that HTTP is the very thing Hal set out to build, and it seems that since he was so focused on file systems he missed another obvious technology choice from which to draw. Would it have been dog slow for every application to make local HTTP requests to read files? Heck Yeah. Is it insane? Yeah. Would it have been slower than what they built? I don't know... And HTTP was so well understood and supported it would have allowed applications to start to mix content from local and remote sources, what we effectively have with pure JavaScript apps today. But HTTP was a standard, and Microsoft from those days was allergic to standards. Alas, someone will eventually build a JavaScript-based OS that treats all files as HTTP endpoints. And then we'll get photos to sync with metadata. Just saying.

MichaelGG 13y ago
Can you explain a bit of how HTTP would actually help here? HTTP doesn't magically fix the issues of mixing content local and remote (latency, availability, etc.) This sounds like "throw more XML at it" except with HTTP instead.
- geocar 13y ago
  No it doesn't. And those are important problems, but another very important problem that needs solving is getting everyone to put all of their content into WinFS and make all of their services available from WinFS. Using an existing system bypasses a huge part of that.

andor 13y ago

“I know I saw a spreadsheet a couple of weeks ago; when I want to find it again do I look in my file system or do I look in my email?”. Bill was trying to make multiple points with this simple example, but the primary one was not that there should be a way to search across disparate stores. (from part 1 of the series) My first thought was: put your mail in a Maildir instead of that pst file and you have "unified" storage ;-) It seems they wanted to build something like the opposite of Unix: "everything is a (structured) database" instead of "everything is a file".

viraptor 13y ago
Why would Maildir help here? Unless the file itself was saved separately and linked from the email in an app-specific way you still wouldn't be able to open it directly in your spreadsheet app. (excluding fusefs-style magic) Unless you have a well-defined structures stored somewhere, you just get extensions on top of extensions... see things as simple as 'ar' files for example. Also Maildirs have lots of different formats and ways of indexing - each app treating the files as "just files" and doing it in a very specific way :( Sometimes I really appreciate the "here's an API, you will never know where/how it's stored and you shouldn't care" approach.
- andor 13y ago
  Unless the file itself was saved separately and linked from the email in an app-specific way you still wouldn't be able to open it directly in your spreadsheet app. That's true. You might be able to find the right mail file using text-based tools, if the mail isn't MIME encoded, but then you still don't have the file. I just found it fun to think of it as the opposite of Unix. The Nepomuk framework in KDE seems to have similar goals by the way. They kept their database separate from the filesystem, but integrated it in quite a few KDE applications. Nepomuk uses a graph database, so the schema is easily extendable. But apparently they don't even use all features of the current schema, only rating, tagging and comments appear to be consistently implemented in applications. http://userbase.kde.org/Nepomuk http://userbase.kde.org/Nepomuk

philsnow 13y ago

Reading these articles, I'm struck by the refusal to admit influence from other systems (or even point out how Windows' features have inspired the same in other systems). > For example, Windows had implemented the TransmitFile function for optimizing transmission of files from a web server by doing all the work in kernel mode. Isn't that just sendfile(2) from BSD ? > An application could attach another stream to the file to hold whatever it wanted. The file system guys really didn’t care, and they didn’t interpret the stream. So this was a very natural extension. Isn't that just forks from HFS/+ ? I actually don't know the history behind all of these things, so the influence could have been the other way around (or the features could have been developed completely independently and contemporaneously). Edit: just saw the nod to DEC, so it's not that they're completely absent, just (IMHO conspicuously) missing in some places.

pak6 13y ago

is this helpful for wedding photography like this http://www.shakkyastudio.com/ http://www.shakkyastudio.com/