I think that the thriftdb folks (integrated to HN and hnsearch) do a great job with some of what you are looking for. If not, I'm happy to make data accessible

I think that the thriftdb folks (integrated to HN and hnsearch) do a great job with some of what you are looking for. If not, I'm happy to make data accessible. Actually... It's just using the unix fs as a db right now, the structure is open and pretty easy to decipher. The dir tree for the objects is just "db/substr(md5hashofurl, 0, 2)/substr(md5hashofurl, 2, 3)/md5hash)" and the bytime/yyyy/m/d/hr/min is just symlinks to the md5hash (0 len file).