3 ms·

Interesting. When I was in high school, I wrote some software to denoise datasets with a genetic algorithm (it's what we had before deep learning was a thing ;

Interesting. When I was in high school, I wrote some software to denoise datasets with a genetic algorithm (it's what we had before deep learning was a thing ;). I showed it to my math teachers and they were horrified; they told me that you cannot change data once it's been collected. I realize now that their take is kind of wrong -- pretty much every instrument in the world performs filtering on what it samples before it presents you a number. Saying you can't filter data kind of misses the point, because almost every real-world system is bandwidth limited and acts as a low-pass filter. But sometimes there are different noise sources, and you can probably write an algorithm to attack them without being "inaccurate". (For example, ever measure very low voltages at relatively high frequency? You'll see a nice 60Hz component in there, because you are surrounded by a 60Hz electric field. That's noise, not data. Apparently averaging through multiple power line cycles is fine, but writing a program to do the same thing is wrong. Seems weird to me.) Anyway, I'm still bitter about it. (That, and how in 4th grade I got a C in science because we had to make a flipbook about earthquakes that had to be titled "quakin' shakin' earth" and I spelled "quakin'" and "shakin'" wrong. Still don't know how to spell either of those contractions, or what flipbooks have to do with science. They don't come up in real life much. But I sure am bitter, 26 years later.)

sildur 6y ago
Any education before college is training, not teaching. They train you, like a dog, to pass a set of exams. Learning something in the process is irrelevant for the "teachers".
- jrockway 6y ago
  Not really. I should have gone to the physics den instead of the math office to show it off, though. In math, there is no uncertainty. The real world has annoying quirks, however. (When I was in college, they built a new engineering laboratory complex, and everyone moved to it. Some people doing laser experiments had to go back to the old building, though, because the new one vibrated too much.)
  - TeMPOraL 6y ago
    Reminds me of a story a friend told me, about how their university management wanted to save money and shut the AC off during holiday break. This almost destroyed a $1M research laser; after the break, the researchers came back to their lab, only to discover the air inside is just a little short of hitting the dew point and covering all the expensive optics in water. (AFAIK that same team also had a problem when moving to a new building, because it wasn't constructed level enough.)
jdonaldson 6y ago
All of what you said was true. Scientists try and find "original sources" for data, but they're usually analyzing measurements that have been altered in some way before they receive them... usually in the measurement hardware itself. One can't really avoid it completely. The flip side is that noise can actually be helpful, especially in situations where the signal itself is too weak to reach some detectable threshold on its own. I generally try hard to remove bias/noise from data before analyzing it, but I also add noise back in later on... e.g. jittering data for plots, or putting a noise feature into a model to establish a threshold for useless features. I'm also a bit miffed that I had to discover this all on my own... Noise is useful and interesting! In business-oriented analysis, noise can be even more important, since changes in "data noise" can indicate important changes in business patterns, or problems with how they're collecting data (companies seldom share this information with analysts beforehand). I'd go so far as to say that understanding the noise in a dataset is the primary concern for most of my analyses these days. Here's a good book with more pop-sci noise examples: https://www.amazon.com/Noise-Bart-Kosko/dp/0670034959 https://www.amazon.com/Noise-Bart-Kosko/dp/0670034959
pmiller2 6y ago
Regarding power supply noise, specifically, the easiest way to get rid of it is pretty simple: use a battery instead of AC.
- nine_k 6y ago
  The wires in walls provide some of the noise. Your body, when you hold the oscilloscope probes, work as an antenna for them. The real remedy is using a differential input that cancels out identical noises induced in both wires of a pair.

sildur 6y ago

Any education before college is training, not teaching. They train you, like a dog, to pass a set of exams. Learning something in the process is irrelevant for the "teachers".

jrockway 6y ago
Not really. I should have gone to the physics den instead of the math office to show it off, though. In math, there is no uncertainty. The real world has annoying quirks, however. (When I was in college, they built a new engineering laboratory complex, and everyone moved to it. Some people doing laser experiments had to go back to the old building, though, because the new one vibrated too much.)
- TeMPOraL 6y ago
  Reminds me of a story a friend told me, about how their university management wanted to save money and shut the AC off during holiday break. This almost destroyed a $1M research laser; after the break, the researchers came back to their lab, only to discover the air inside is just a little short of hitting the dew point and covering all the expensive optics in water. (AFAIK that same team also had a problem when moving to a new building, because it wasn't constructed level enough.)

jdonaldson 6y ago

All of what you said was true. Scientists try and find "original sources" for data, but they're usually analyzing measurements that have been altered in some way before they receive them... usually in the measurement hardware itself. One can't really avoid it completely. The flip side is that noise can actually be helpful, especially in situations where the signal itself is too weak to reach some detectable threshold on its own. I generally try hard to remove bias/noise from data before analyzing it, but I also add noise back in later on... e.g. jittering data for plots, or putting a noise feature into a model to establish a threshold for useless features. I'm also a bit miffed that I had to discover this all on my own... Noise is useful and interesting! In business-oriented analysis, noise can be even more important, since changes in "data noise" can indicate important changes in business patterns, or problems with how they're collecting data (companies seldom share this information with analysts beforehand). I'd go so far as to say that understanding the noise in a dataset is the primary concern for most of my analyses these days. Here's a good book with more pop-sci noise examples: https://www.amazon.com/Noise-Bart-Kosko/dp/0670034959 https://www.amazon.com/Noise-Bart-Kosko/dp/0670034959

pmiller2 6y ago

Regarding power supply noise, specifically, the easiest way to get rid of it is pretty simple: use a battery instead of AC.

nine_k 6y ago
The wires in walls provide some of the noise. Your body, when you hold the oscilloscope probes, work as an antenna for them. The real remedy is using a differential input that cancels out identical noises induced in both wires of a pair.