3 ms·

I am interested in a concept close to a CRDT - but I think meaningfully different. Is there a 'data structure' or 'scheme/setup' which fits the below descripti

by chaxor 3y ago

I am interested in a concept close to a CRDT - but I think meaningfully different. Is there a 'data structure' or 'scheme/setup' which fits the below description? Have a repo (with a specific git commit hash), which is specifically constructed to reproducibly construct some data (database, dataset, set of examples files, etc) (which can then be checked by a checksum), such that when a different person pulls and executes that code, a decision could be made to either run the code again, or to pull it from the decentralized group of people that have that repo database. This is effectively "code as data" which is checked by a key value pairing: [git commit hash <==> data checksum hash]. So if you have a process that runs for 24 days on a good machine and produces a 30 MB file for people to use, they can check for reproducibilityif they want with the code, or just grab the file from everyone who pulled the repo to save time. Obviously having defaults for download or rerun could be determined by certain logic like execution time etc, but the base idea is something I've looked into for a while. Is there a name for this type of setup?

killthebuddha 3y ago
I'm very unsure that I understand your requirements, but if I squint really really hard it looks like you're asking for a blockchain.
- chaxor 3y ago
  Hmmm... Perhaps that could be one component, but I don't know if proof of work in the typical way would be required? As in 'guessing' a hash (I don't know much about that area so I could be wrong here)., I don't think would be needed - but perhaps the proof of work could simply be 'calculate the output that you want to make anyway' - such as the weights of a neural network, a read only database to do complex queries on, an operating system compilation with specific configuration, a dataset of estimates of some physical science data using a particular algorithm, etc. So instead of using CPU cycles to do finance/currency things (or whatever is done in that arena), this would calculate whatever data you actually want directly. In some cases it could be a ton of calculation (estimating electron density functional, or training NN weights) to receive a small file (DFT functionals are quite small). Or it could be some calculation for a large amount of data (ETL for big we scraped data). One important aspect which is similar to the currency/Blockchain realm is that the commits would specify the data state, so the data would likely be mostly read-only for people, but if they change it, it would simply branch off to a different hash pair.
scrumlord 3y ago
[dead]

by chaxor 3y ago

killthebuddha 3y ago

I'm very unsure that I understand your requirements, but if I squint really really hard it looks like you're asking for a blockchain.

chaxor 3y ago
Hmmm... Perhaps that could be one component, but I don't know if proof of work in the typical way would be required? As in 'guessing' a hash (I don't know much about that area so I could be wrong here)., I don't think would be needed - but perhaps the proof of work could simply be 'calculate the output that you want to make anyway' - such as the weights of a neural network, a read only database to do complex queries on, an operating system compilation with specific configuration, a dataset of estimates of some physical science data using a particular algorithm, etc. So instead of using CPU cycles to do finance/currency things (or whatever is done in that arena), this would calculate whatever data you actually want directly. In some cases it could be a ton of calculation (estimating electron density functional, or training NN weights) to receive a small file (DFT functionals are quite small). Or it could be some calculation for a large amount of data (ETL for big we scraped data). One important aspect which is similar to the currency/Blockchain realm is that the commits would specify the data state, so the data would likely be mostly read-only for people, but if they change it, it would simply branch off to a different hash pair.

scrumlord 3y ago

[dead]