4 ms·

One problem with offering a dataset as a torrent is that it's impossible to edit it after it's released. However, it seems like that doesn't matter at all in t

One problem with offering a dataset as a torrent is that it's impossible to edit it after it's released. However, it seems like that doesn't matter at all in this case, because any scenario I can think of which could be solved by editing the dataset (like redacting private info that was accidentally included) wouldn't avoid the original problem: that they accidentally released private info in the first place. Perhaps it'd be useful to edit the original dataset in order to add to it / enhance it with more info, but in that case they could just release a second dataset as an addendum. So the core idea seems solid. Thank you for this!

jzelinskie 13y ago
There are attempts to feel out a process for "updating torrents". However, this is long from becoming a standard practice in the BitTorrent ecosystem. Check this[0] out for more info. [0] http://www.bittorrent.org/beps/bep_0039.html http://www.bittorrent.org/beps/bep_0039.html
pixelcort 13y ago
BitTorrent Sync might be useful for that: http://www.bittorrent.com/sync http://www.bittorrent.com/sync
- whadar 13y ago
  Hopefully sharefest.me would be another alternative pretty soon
- rodolphoarruda 13y ago
  I'd used BT Sync for a couple of weeks to sync data between my own machines. It works neatly. One question here. When you modify some part of a big file, does the program send out only the difference to the other authorized machines, or entire file? Let's say a researcher exports her data to a 1GB CSV file of my interest. I download it. In the following week the same researcher updates her CVS with more data, now it has 1.01GB in size. How big my next download will be?
  - lojack 13y ago
    Seems as though it supports patching so only the parts that are changed would be synced. Of course, the download size is completely dependent on what parts of the file were changed. http://forum.bittorrent.com/topic/19228-latest-sync-build-1170/page-6#entry56304 http://forum.bittorrent.com/topic/19228-latest-sync-build-11...
dspillett 13y ago
> but in that case they could just release a second dataset as an addendum Or for some data it would make sense to partition the data into smaller chunks instead of one huge archive. That way adding a chunk (the new year's data for a multi-year dataset perhaps) just menas releasing a new torrent with the extra srchive in and a name meaningful enough to indicate the difference. Anyone with the last set could then just download the new partition (and any modified ones).
axman6 13y ago
If it's stored on ZFS, Copy on Write will let you edit a copy that only stores the changed files, and deduplication could give back even more space (if necessary and RAM permits).
dnautics 13y ago
One problem with offering a dataset as a torrent is that it's impossible to edit it after it's released. for scientific endeavours, this should be considered a feature, not a bug.

jzelinskie 13y ago

There are attempts to feel out a process for "updating torrents". However, this is long from becoming a standard practice in the BitTorrent ecosystem. Check this[0] out for more info. [0] http://www.bittorrent.org/beps/bep_0039.html http://www.bittorrent.org/beps/bep_0039.html

pixelcort 13y ago

BitTorrent Sync might be useful for that: http://www.bittorrent.com/sync http://www.bittorrent.com/sync

whadar 13y ago
Hopefully sharefest.me would be another alternative pretty soon
rodolphoarruda 13y ago
I'd used BT Sync for a couple of weeks to sync data between my own machines. It works neatly. One question here. When you modify some part of a big file, does the program send out only the difference to the other authorized machines, or entire file? Let's say a researcher exports her data to a 1GB CSV file of my interest. I download it. In the following week the same researcher updates her CVS with more data, now it has 1.01GB in size. How big my next download will be?
- lojack 13y ago
  Seems as though it supports patching so only the parts that are changed would be synced. Of course, the download size is completely dependent on what parts of the file were changed. http://forum.bittorrent.com/topic/19228-latest-sync-build-1170/page-6#entry56304 http://forum.bittorrent.com/topic/19228-latest-sync-build-11...

dspillett 13y ago

> but in that case they could just release a second dataset as an addendum Or for some data it would make sense to partition the data into smaller chunks instead of one huge archive. That way adding a chunk (the new year's data for a multi-year dataset perhaps) just menas releasing a new torrent with the extra srchive in and a name meaningful enough to indicate the difference. Anyone with the last set could then just download the new partition (and any modified ones).

axman6 13y ago

If it's stored on ZFS, Copy on Write will let you edit a copy that only stores the changed files, and deduplication could give back even more space (if necessary and RAM permits).

dnautics 13y ago

One problem with offering a dataset as a torrent is that it's impossible to edit it after it's released. for scientific endeavours, this should be considered a feature, not a bug.