3 ms·

one of the things I want to try (but have only done some preliminary trials with) is to keep the concept of the write only ledger, but to replicate the

by fiatjaf 10y ago

one of the things I want to try (but have only done some preliminary trials with) is to keep the concept of the write only ledger, but to replicate the db into several views of the data (some with severely restricted content). How is that different from the current concept of CouchDB views? You meant to replicate the DB to various different places and use the same CouchDB views from there? Or you meant something like a replication-view, in which some calculations are done with the documents in the source database and the target database receives the result of those calculations as their primary documents?

mikekchar 10y ago
Yes, the latter. One of the main problems I've seen is that indexing things that you will never query is both expensive in time and space. Also it's amazing how many views tend to have exactly the same data, only sorted differently. And the reason to sort it differently is because you only want to work on a subset of the data, but you can only restrict the query in a contiguous section of keys. An example of this might be that you have a large number of daily reports. They all need different aspects of the data, but you end up writing views that sort by date and then collate the result in the server. So you end up maintaining an index for data that you will never query again and you are doing lots of extra processing merging the data after the query. Much better to replicate one day's worth of data to a new db every evening (possibly setting up a continuous replication to keep it up to date) and then add views on that db to do what you want. Like I said a full replication of a million documents takes about 10 minutes, so it's a reasonable thing to do.
- fiatjaf 10y ago
  I like the concept of views. I think it is the best part of CouchDB. Saving data and later defining views (that should be very fast). Once I wrote an app that stored massively enormous documents with lots of data, the views were used to turn that data into queriable information later. I like what you suggested very much, because what CouchDB can currently do with views is very limited to what a powerful views implementation would, and the one you've suggested is a good suggestion of how to do it better. Thank you. I'll give this a lot of thinking when I restart writing my https://github.com/fiatjaf/summadb/ https://github.com/fiatjaf/summadb/

by fiatjaf 10y ago

mikekchar 10y ago

Yes, the latter. One of the main problems I've seen is that indexing things that you will never query is both expensive in time and space. Also it's amazing how many views tend to have exactly the same data, only sorted differently. And the reason to sort it differently is because you only want to work on a subset of the data, but you can only restrict the query in a contiguous section of keys. An example of this might be that you have a large number of daily reports. They all need different aspects of the data, but you end up writing views that sort by date and then collate the result in the server. So you end up maintaining an index for data that you will never query again and you are doing lots of extra processing merging the data after the query. Much better to replicate one day's worth of data to a new db every evening (possibly setting up a continuous replication to keep it up to date) and then add views on that db to do what you want. Like I said a full replication of a million documents takes about 10 minutes, so it's a reasonable thing to do.

fiatjaf 10y ago
I like the concept of views. I think it is the best part of CouchDB. Saving data and later defining views (that should be very fast). Once I wrote an app that stored massively enormous documents with lots of data, the views were used to turn that data into queriable information later. I like what you suggested very much, because what CouchDB can currently do with views is very limited to what a powerful views implementation would, and the one you've suggested is a good suggestion of how to do it better. Thank you. I'll give this a lot of thinking when I restart writing my https://github.com/fiatjaf/summadb/ https://github.com/fiatjaf/summadb/