3 ms·

Scale of 1-10, how much does the spaghetti code from my PhD make you want to self-harm? https://gtfiber.github.io https://gtfiber.github.io Follow-up: say I h

Scale of 1-10, how much does the spaghetti code from my PhD make you want to self-harm? https://gtfiber.github.io https://gtfiber.github.io Follow-up: say I have a lot of free time to patch it up, write better documentation, unit tests, etc. What's the first, most value-added thing to do?

roel_v 9y ago
Looks OK, but the work of the type of person to ask for feedback on code and has put it on Github without external motivation is generally not the sort of project that's most problematic. Without unit tests and this many options who (from the looks of it) can interact with each other, I'd worry about how confident you are yourself that the right results come out though. You've probably eyeballed the results of all sample images in the repo once or twice, but do you do so consistently every time you make a change somewhere? This is only a concern when this is a long-running product, of course. It's the most common problem I find too - someone starts out with part A, works on smallish (but not bare bones) dataset (so easier to verify manually than a full out dataset, but not trivial either). Then works on parts B, C and D, verifying those each time independently or on the most common use case or whatever boundary conditions it happens to be that your dataset hits. Then when fixing something unrelated, small errors are introduce somewhere in A or B or C but you have no way to tell because it's not feasible to manually verify everything every time. That and running only on real-world, way too big datasets. Turnaround time for running them is too long, only major problems show up, the effects of options x and y are only small so errors in them are largely unnoticeable unless you look for them - which you can't/don't on large datasets. As to nr2, depends on what your goal is. Do you want to make this into a more polished product someone who doesn't know how to code can use, do you want to open it up so that others can try variations of your algorithms, do you want to make it into a tool that becomes standard in your field and will act as a citation-generator or at least -attractor? Something else still? Defining your goal with the software to set a framework to make future decisions in is actually the most value-added thing of all, I'd say. For 1 and 3 you need a self-contained download, i.e. no 'download this and this compiler' etc. - just a zip with an exe. I haven't tried the tool, I can't tell if there's any functionality missing to do this (like a 'first use' tutorial, or even just some training slides/pdf and -exercises). Furthermore for 3 you need to have a command line version. It depends a bit though on how ubiquitous Matlab is in your field. Probably ubiquitous enough that it won't be a deal breaker for anyone sufficiently motivated, but some people have a natural dislike for it from their undergrad days, or others just are philosophically opposed to using for-pay tools (even if they don't have to pay for it themselves). I'm not, just saying that you might get more citations if it adheres more to a certain 'ideological purity benchmark'; you have to decide for yourself what that is in your field. This is not generally something most people are concerned with, but the type of person who would put their own stuff on Github is likely to be more sensitive to this. Again, it's a social thing - just bringing it up. For 2, I'd start with writing some documentation (not too much, just a few paragraphs) that explains the structure of your program, and where someone who'd want to add a new method should start; or better yet, explain the plugin framework you've build if you have one. Also for 2, you need some unit tests if any new functionality can't be implemented completely separate like through a plugin dll. Not so much for your potential contributors, but more for yourself so that you know everything keeps working when you get someone else's contributions that touch a lot of code. In general, you're more likely to be taken seriously as a thought leader in your field if you have other users and/or some social validation that others are using it, like when you're taught workshops at conferences or elsewhere. So some introductory material, exercises that teach you all aspects of using it, maybe a link to a video or mention of you giving a workshop somewhere. In the life cycle of your software (from what I can tell), that sort of effort will yield much higher ROI than most coding will (especially the 'refactor to detangle spaghetti' coding which is only needed when you actually have a reason for it, not just for the sake of it). All of this of course assuming you're still in academia and/or want to continue there.
- imperssonator 9y ago
  Thank you, seriously, from a very deep place, for taking the time to actually look at the code and write this. You are exactly right about how I went about testing it, at least in the early stages. It was always "test on the hardest edge cases, make sure they don't throw errors, move on to fixing another small piece". I actually found that making the GUI brought out most of the deeper, more bizarre errors and made it loads more stable. There's a much bigger validation dataset that I used later on that I just haven't uploaded yet. In the latest iteration I have manually traced examples that are used for training and validation, which makes me a lot more confident in the results. Adding an instructional video / workshop was one of the top things on my to-do list. I agree that that would help any potential users in a huge way. I did video chat with some people who used it, and they ultimately published a paper using it, so that was pretty cool. In the end, though, this was just a semi-polished byproduct of the actual experimental work I was doing, so like others have pointed out, I didn't have to make the code nice to pass peer review or graduate.
- imperssonator 9y ago
  I'm curious if you're aware of the Materials Genome Initiative. It's a group of materials scientists in the US who are very sympathetic to these problems and want to build out infrastructure for big data analysis of materials. We're based mostly at NIST, where most of the huge datasets are generated or stored.
  - roel_v 9y ago
    No never heard of this - I work in very different fields though. There are initiatives enough left and right, but they're mostly bottom-up, with the majority fizzling out after the one or two persons driving up get other positions or different types of projects. Which is a chicken and egg problem; many people don't want to risk their projects on something they have no control of and might collapse as they've seen happen often to others. Maybe I'm just cynical having been through a few cycles of this, it would be great if someone did succeed. I just don't see the incentives. Quality in software engineering is a hobby interest, it's not vital to the success of research, or of a researcher. I know this causes a major cognitive dissonance shock to those who are heavily software-focused like most on this site; and of course it did the same to me, for many years even. I guess I've evolve along the lines of most humanitarian aid workers. You start out thinking you'll fix this once and for all in the next year or two, and 20 years later you find yourself being quite happy that you managed to make some difference to a village of a few families.

roel_v 9y ago

Looks OK, but the work of the type of person to ask for feedback on code and has put it on Github without external motivation is generally not the sort of project that's most problematic. Without unit tests and this many options who (from the looks of it) can interact with each other, I'd worry about how confident you are yourself that the right results come out though. You've probably eyeballed the results of all sample images in the repo once or twice, but do you do so consistently every time you make a change somewhere? This is only a concern when this is a long-running product, of course. It's the most common problem I find too - someone starts out with part A, works on smallish (but not bare bones) dataset (so easier to verify manually than a full out dataset, but not trivial either). Then works on parts B, C and D, verifying those each time independently or on the most common use case or whatever boundary conditions it happens to be that your dataset hits. Then when fixing something unrelated, small errors are introduce somewhere in A or B or C but you have no way to tell because it's not feasible to manually verify everything every time. That and running only on real-world, way too big datasets. Turnaround time for running them is too long, only major problems show up, the effects of options x and y are only small so errors in them are largely unnoticeable unless you look for them - which you can't/don't on large datasets. As to nr2, depends on what your goal is. Do you want to make this into a more polished product someone who doesn't know how to code can use, do you want to open it up so that others can try variations of your algorithms, do you want to make it into a tool that becomes standard in your field and will act as a citation-generator or at least -attractor? Something else still? Defining your goal with the software to set a framework to make future decisions in is actually the most value-added thing of all, I'd say. For 1 and 3 you need a self-contained download, i.e. no 'download this and this compiler' etc. - just a zip with an exe. I haven't tried the tool, I can't tell if there's any functionality missing to do this (like a 'first use' tutorial, or even just some training slides/pdf and -exercises). Furthermore for 3 you need to have a command line version. It depends a bit though on how ubiquitous Matlab is in your field. Probably ubiquitous enough that it won't be a deal breaker for anyone sufficiently motivated, but some people have a natural dislike for it from their undergrad days, or others just are philosophically opposed to using for-pay tools (even if they don't have to pay for it themselves). I'm not, just saying that you might get more citations if it adheres more to a certain 'ideological purity benchmark'; you have to decide for yourself what that is in your field. This is not generally something most people are concerned with, but the type of person who would put their own stuff on Github is likely to be more sensitive to this. Again, it's a social thing - just bringing it up. For 2, I'd start with writing some documentation (not too much, just a few paragraphs) that explains the structure of your program, and where someone who'd want to add a new method should start; or better yet, explain the plugin framework you've build if you have one. Also for 2, you need some unit tests if any new functionality can't be implemented completely separate like through a plugin dll. Not so much for your potential contributors, but more for yourself so that you know everything keeps working when you get someone else's contributions that touch a lot of code. In general, you're more likely to be taken seriously as a thought leader in your field if you have other users and/or some social validation that others are using it, like when you're taught workshops at conferences or elsewhere. So some introductory material, exercises that teach you all aspects of using it, maybe a link to a video or mention of you giving a workshop somewhere. In the life cycle of your software (from what I can tell), that sort of effort will yield much higher ROI than most coding will (especially the 'refactor to detangle spaghetti' coding which is only needed when you actually have a reason for it, not just for the sake of it). All of this of course assuming you're still in academia and/or want to continue there.

imperssonator 9y ago
Thank you, seriously, from a very deep place, for taking the time to actually look at the code and write this. You are exactly right about how I went about testing it, at least in the early stages. It was always "test on the hardest edge cases, make sure they don't throw errors, move on to fixing another small piece". I actually found that making the GUI brought out most of the deeper, more bizarre errors and made it loads more stable. There's a much bigger validation dataset that I used later on that I just haven't uploaded yet. In the latest iteration I have manually traced examples that are used for training and validation, which makes me a lot more confident in the results. Adding an instructional video / workshop was one of the top things on my to-do list. I agree that that would help any potential users in a huge way. I did video chat with some people who used it, and they ultimately published a paper using it, so that was pretty cool. In the end, though, this was just a semi-polished byproduct of the actual experimental work I was doing, so like others have pointed out, I didn't have to make the code nice to pass peer review or graduate.
imperssonator 9y ago
I'm curious if you're aware of the Materials Genome Initiative. It's a group of materials scientists in the US who are very sympathetic to these problems and want to build out infrastructure for big data analysis of materials. We're based mostly at NIST, where most of the huge datasets are generated or stored.
- roel_v 9y ago
  No never heard of this - I work in very different fields though. There are initiatives enough left and right, but they're mostly bottom-up, with the majority fizzling out after the one or two persons driving up get other positions or different types of projects. Which is a chicken and egg problem; many people don't want to risk their projects on something they have no control of and might collapse as they've seen happen often to others. Maybe I'm just cynical having been through a few cycles of this, it would be great if someone did succeed. I just don't see the incentives. Quality in software engineering is a hobby interest, it's not vital to the success of research, or of a researcher. I know this causes a major cognitive dissonance shock to those who are heavily software-focused like most on this site; and of course it did the same to me, for many years even. I guess I've evolve along the lines of most humanitarian aid workers. You start out thinking you'll fix this once and for all in the next year or two, and 20 years later you find yourself being quite happy that you managed to make some difference to a village of a few families.