3 ms·

It seems that you may have been able to extract specific information about things like "how much of an angle the fish made their tails flick" as opposed to stor

by spynxic 10y ago

It seems that you may have been able to extract specific information about things like "how much of an angle the fish made their tails flick" as opposed to storing the raw files. Such a technique would've definitely eliminated extra irrelevant information like "the dish [you] have the fishes in" as well as cut back on the size of the data.

Balgair 10y ago
True! That info is what the paper you are trying to write is all about. You take these TBs of data, and effectively compress it into a single graph of maybe 5kB, the thing you really care about. However, you still have to keep all the data around for possible re-analysis depending on the terms of the paper you submitted to, for up to 10 years or more in some cases, along with all the equipment and maybe some frozen fish samples in LN2 down in the basement. Now, not all labs are so well funded, and sometimes accidents happen and grad students are not always so informed about these rules, but you should be keeping it all ready for re-testing for some number of years. You simply cannot get rid of it all for the sake of science. If you really want to help out bio-peepz, then helping them program is a great way to do so. Spaghetti does not even come close to describing their 'code'. If anyone really wanted to reproduce the experiment, then they would have to wade into the fetid swamp of 'code' that produced it. Most bio people cannot even begin to tell you how to code or what it even means. They can PCR and Western Blot better than Jesus himself, but code? No way. It is a real hindrance to the sciences, actually. The helping hand that code was meant to be has become a ball-n-chain that drags peoples minds away from them and 'lets the computer do it' for them.
- spynxic 10y ago
  Wow, I failed to considered the need for validation.. but even still, that adds only another layer of testing to ensure that your techniques for converting the raw data to structured data are sound enough to be reproduced. I'd guess the model still wouldn't be considered "pure" enough by the scientific community unless it can be proven to be free from bias samples of raw information.
  - jkimmel 10y ago
    > but even still, that adds only another layer of testing to ensure that your techniques for converting the raw data to structured data are sound enough to be reproduced. That's actually not correct. What if your validation method is broken? What if that unit test you wrote has a bug in the test itself? In science you have to keep the raw data available. This is not only for reproduction of your particular structured metric (i.e. tail flick angle), but for other groups to design novel metrics that may add to or trivialize the published result.
    - fnord123 10y ago
      >In science you have to keep the raw data available. There is nothing I'm aware of saying that you can't losslessly convert tiff to png or even tiff to tiff.gz. Nobody needs to collect pcap files for their sensors (the raw data). They just collect the data and stick it in an appropriate file.
      - Balgair 10y ago
        Most bio-peeps and medical personnel have an understanding that there is a difference between .png, .jpg, and .tif. But the diff of .tif and .tiff? No idea, let alone the +/- of a .png to a .jpg. It's all just a picture to them. So, trying to tell them that you can losslessly convert them, though maybe true, is not a good idea. .tiff can handle a stack of images and you only have to call it once in a MatLab script, .jpg cannot and will result in a shitfit as they try to load in an entire folder of images. Then you get into bit-depth and God help you if the conversation ever has you trying to pronounce 'int', 'float', 'str', or 'double'. The class of a variable? Dude, this person could care less and will get out a protractor and measure tail flicks off a print-out before any of that will ever sink in.
        fnord123 10y ago
        I think you underestimate people. If you're an admin doing user support for such a system and someone says they need more storage because of blah blah blah then you offer ways to alter the workflow so they don't need 500G of tiff files. It might even speed up their workflow since it's not streaming 500G off disk. No need to get into bit depth and int float str and double (though all of the biomedial researchers I've worked with knew these very well). Just: 'you can convert to png and back again without losing data. Just like zipping a file and unzipping it again. Let me show you how...'

by spynxic 10y ago

Balgair 10y ago

True! That info is what the paper you are trying to write is all about. You take these TBs of data, and effectively compress it into a single graph of maybe 5kB, the thing you really care about. However, you still have to keep all the data around for possible re-analysis depending on the terms of the paper you submitted to, for up to 10 years or more in some cases, along with all the equipment and maybe some frozen fish samples in LN2 down in the basement. Now, not all labs are so well funded, and sometimes accidents happen and grad students are not always so informed about these rules, but you should be keeping it all ready for re-testing for some number of years. You simply cannot get rid of it all for the sake of science. If you really want to help out bio-peepz, then helping them program is a great way to do so. Spaghetti does not even come close to describing their 'code'. If anyone really wanted to reproduce the experiment, then they would have to wade into the fetid swamp of 'code' that produced it. Most bio people cannot even begin to tell you how to code or what it even means. They can PCR and Western Blot better than Jesus himself, but code? No way. It is a real hindrance to the sciences, actually. The helping hand that code was meant to be has become a ball-n-chain that drags peoples minds away from them and 'lets the computer do it' for them.

spynxic 10y ago
Wow, I failed to considered the need for validation.. but even still, that adds only another layer of testing to ensure that your techniques for converting the raw data to structured data are sound enough to be reproduced. I'd guess the model still wouldn't be considered "pure" enough by the scientific community unless it can be proven to be free from bias samples of raw information.
- jkimmel 10y ago
  > but even still, that adds only another layer of testing to ensure that your techniques for converting the raw data to structured data are sound enough to be reproduced. That's actually not correct. What if your validation method is broken? What if that unit test you wrote has a bug in the test itself? In science you have to keep the raw data available. This is not only for reproduction of your particular structured metric (i.e. tail flick angle), but for other groups to design novel metrics that may add to or trivialize the published result.
  - fnord123 10y ago
    >In science you have to keep the raw data available. There is nothing I'm aware of saying that you can't losslessly convert tiff to png or even tiff to tiff.gz. Nobody needs to collect pcap files for their sensors (the raw data). They just collect the data and stick it in an appropriate file.
    - Balgair 10y ago
      Most bio-peeps and medical personnel have an understanding that there is a difference between .png, .jpg, and .tif. But the diff of .tif and .tiff? No idea, let alone the +/- of a .png to a .jpg. It's all just a picture to them. So, trying to tell them that you can losslessly convert them, though maybe true, is not a good idea. .tiff can handle a stack of images and you only have to call it once in a MatLab script, .jpg cannot and will result in a shitfit as they try to load in an entire folder of images. Then you get into bit-depth and God help you if the conversation ever has you trying to pronounce 'int', 'float', 'str', or 'double'. The class of a variable? Dude, this person could care less and will get out a protractor and measure tail flicks off a print-out before any of that will ever sink in.
      - fnord123 10y ago
        I think you underestimate people. If you're an admin doing user support for such a system and someone says they need more storage because of blah blah blah then you offer ways to alter the workflow so they don't need 500G of tiff files. It might even speed up their workflow since it's not streaming 500G off disk. No need to get into bit depth and int float str and double (though all of the biomedial researchers I've worked with knew these very well). Just: 'you can convert to png and back again without losing data. Just like zipping a file and unzipping it again. Let me show you how...'