4 ms·

I would agree most strongly with number 6. Having spent time in the biotech arena, there are so many fractured data format structures. A good example of this wo

I would agree most strongly with number 6. Having spent time in the biotech arena, there are so many fractured data format structures. A good example of this would be small variations in .fastq files. There are different formats generated from different equipment, and different software expects format A when you have format B, and I've seen people trying to shove the wrong formats into pipelines that blow up or create a tone of work as a result. Honestly though, the biggest problem with people using python for bid data though is just the opposite of number 2, sometimes you need to use that framework, but when you rely too heavily on various frameworks, that's how you end up in dependency hell, largely due to the aforementioned format issues. eg. Framework 1 update fixes bug X but creats bug y in framework 2's parsing.

bdevine 12y ago
Your "bid data" typo seems like a nice little Freudian slip! As far as your second point though: although it's not ideal, don't virtualenvs do the trick? If you really needed to, you could even set up a workflow of different sandboxes to pass data through. If the alternative to relying on frameworks is rolling your own, frankly I would probably choose the former, but that's just me.
- arca_vorago 12y ago
  Good point on virtualenvs, but it's a python specific fix (which does work). The problem is often you are tying in other kinds of tooling. The other thing to remember about roll your own is you can get efficiency that frameworks can't, and that time difference adds up fast. For example, we designed a worker distribution system, rolled our own worker management code on top of a framework, and reduced compute times from ~1-2days to ~4 hours. That's a huge increase in productivity that no tool or framework could give us. There is power in roll your own, I would just say whip your programmers into submission regarding good commenting/documentation though.

bdevine 12y ago

Your "bid data" typo seems like a nice little Freudian slip! As far as your second point though: although it's not ideal, don't virtualenvs do the trick? If you really needed to, you could even set up a workflow of different sandboxes to pass data through. If the alternative to relying on frameworks is rolling your own, frankly I would probably choose the former, but that's just me.

arca_vorago 12y ago
Good point on virtualenvs, but it's a python specific fix (which does work). The problem is often you are tying in other kinds of tooling. The other thing to remember about roll your own is you can get efficiency that frameworks can't, and that time difference adds up fast. For example, we designed a worker distribution system, rolled our own worker management code on top of a framework, and reduced compute times from ~1-2days to ~4 hours. That's a huge increase in productivity that no tool or framework could give us. There is power in roll your own, I would just say whip your programmers into submission regarding good commenting/documentation though.