3 ms·

what sorts of tools are you using to deal with this data?

mattkrause 8y ago
It's a hodgepodge. Mostly Matlab, Python, and R, with a few things that have tight time/memory requirements in C++. Matlab was really popular in neuroscience for a very long time, so we still have a lot of code in m-files, but most labs are moving towards Python (and a few towards R). The code quality varies a lot. Some of our "core" stuff is great, but there's also a lot of stuff that was written quickly and meant to be run once ("let's just try it"), which is cold comfort when you find it years later. People also come in with varying levels of programming skill. I'm going to try to do actual code reviews with the undergrads this summer to see if we can't make our stuff a little less embarrassing.
- codekilla 8y ago
  I'm interested in using kdb+/q for applications dealing with large scale scientific data. Do you think there would be much interest in the neuroscience community? I'll add, I work in a lab and have seen the stuff of nightmares myself.
  - mattkrause 8y ago
    Possibly! I've not played with the time series databases much. Most of our queries would be something like "give me signals from channels A-C that are +/- 1 second from all events of type X." If it would help with something like that, it'd be great.
- codekilla 8y ago
  I can't reply to your last comment...but yes..anything dealing with time series it's easily the best in the world. In a past life long ago I worked in finance and heard about it then. Recently, I was thinking about simulations and how to maybe deal with and analyze the timeseries data and I thought about it. Apparently, NASA is experimenting with it at their Ames Frontier Development Lab, astrophysics data is huge. Anyway, I am just kicking the tires at this point and trying to find ways to implement and play around with it a bit. If you or anyone you know develops an interest, feel free to drop me a line. grantroy _@_ caltech dot edu

It's a hodgepodge. Mostly Matlab, Python, and R, with a few things that have tight time/memory requirements in C++. Matlab was really popular in neuroscience for a very long time, so we still have a lot of code in m-files, but most labs are moving towards Python (and a few towards R). The code quality varies a lot. Some of our "core" stuff is great, but there's also a lot of stuff that was written quickly and meant to be run once ("let's just try it"), which is cold comfort when you find it years later. People also come in with varying levels of programming skill. I'm going to try to do actual code reviews with the undergrads this summer to see if we can't make our stuff a little less embarrassing.

codekilla 8y ago
I'm interested in using kdb+/q for applications dealing with large scale scientific data. Do you think there would be much interest in the neuroscience community? I'll add, I work in a lab and have seen the stuff of nightmares myself.
- mattkrause 8y ago
  Possibly! I've not played with the time series databases much. Most of our queries would be something like "give me signals from channels A-C that are +/- 1 second from all events of type X." If it would help with something like that, it'd be great.
codekilla 8y ago
I can't reply to your last comment...but yes..anything dealing with time series it's easily the best in the world. In a past life long ago I worked in finance and heard about it then. Recently, I was thinking about simulations and how to maybe deal with and analyze the timeseries data and I thought about it. Apparently, NASA is experimenting with it at their Ames Frontier Development Lab, astrophysics data is huge. Anyway, I am just kicking the tires at this point and trying to find ways to implement and play around with it a bit. If you or anyone you know develops an interest, feel free to drop me a line. grantroy _@_ caltech dot edu