5 ms·

I wish information theory was part of math/cs/engineering curriculum in more places. The basics are fundamental to many areas of science (especially if they to

I wish information theory was part of math/cs/engineering curriculum in more places. The basics are fundamental to many areas of science (especially if they touch probability in any way), intuitive, and mostly accessible with just a couple of handwaves.

duality 8y ago
I agree. Information Theory is _the_ ur-science. It's the science of matching mathematical models to data, and so it's where inference meets the deduction. Such an important and fascinating subject should be widely taught and appreciated.
- JoeAltmaier 8y ago
  Well, next to math. And logic. And ethics.
  - cossatot 8y ago
    I think a lot of us scientists wouldn't view any discipline not fundamentally based in collecting and interpreting observations as a science at all, much less an ur-science.
    - danharaj 8y ago
      Good luck observing anything without naming it and your observations. And once you have names, logic and mathematics are sure to follow.
      - harshreality 8y ago
        Logic and the rest of math are tools used in science, but I think the complaint was that science itself is about observation, developing theories, and testing theories against new observations. To elaborate: In pure subjects like math/logic, you have no observations, at least not really. Everything is abstract. Theorems follow the nomenclature of "hypothesis" and "conclusion" as if it's mimicking science, but every provable/disprovable statement in math, though it can be observed, formalized, and proved or disproved with enough effort or luck, is nevertheless merely a logical consequence of the axioms you started with. Theorems may provide insights, but they don't discover anything new. The conclusions of theorems are logically equivalent to the axioms. That's the essence of what it means to prove, mathematically. Whereas, in a science, there is no underlying theoretical truth. The only truths are quasi-truths that you develop based on observation and experiment, but they can all change tomorrow with a new experiment that is not beholden to any historical science precedent. Math/logic: everything is entirely based on axioms. The notion of "discovery" and "testing" of theorems is a cognitive fiction. Observation is merely the human process of deciding what things to try to prove. Everything is backstopped by abstract axioms and logic. Science: everything is entirely based on observation. The claim of scientific "truth" is a human fiction. We can never be absolutely sure of anything in science. There is no abstract truth backstopping science. Caveat: this does not imply anyone should be disputing scientific truth for practical purposes in everyday life. Disputing scientific consensus because it's not philosophically true is the refuge of spiritual quacks, religious apologists, and even rational people who are too in love with philosophy to take a step back and consider whether their philosophy is practical. "Oh, gravity isn't really true; quantum field theory isn't really true." Okay then, have fun not being able to move because you're floating in the air, and have fun without all modern electronics because they depend on the physics you're disputing, and have fun trying to find water with your dowsing rod.
        duality 8y ago
        I agree with this and would add that science (by which I mean areas of knowledge observing phenomena and seeking to describe them quantitatively) interacts with deductive logic (including math) via information theory. Science without modeling would be devoid of predictive power. And, personally, I've never heard ethics or logic described as sciences. Please correct me if this is common terminology. But of course it doesn't diminish their importance.
        cat199 8y ago
        You are confusing empirical science with science as a whole and missing formal science; The term 'science' generically should also be viewed within the philosophical context of 'art' e.g. an art vs a science.
saiya-jin 8y ago
We had it in our university, actually quite deep. It was done by head of IT department on our faculty, long-retired guy who was supposedly brilliant as theoretical scientist and had high reputation all over Europe in his field. It was done in most horrible and unmotivating way - A4 page or two densely covered with all greek letters and some more, and 98% of the content were just proofs of relatively simple statements. On all tests/exams, only the proofs were tested (so you either gave 1-2 pages of a single proof per question or blank page and could effectively go home as failed). Subjectively it was the worst set of classes during whole 5 years (and we had some serious IT-unrelated crap because were part of electro-engineering faculty back then), completely mandatory, no credit system back then to make it up via something else. Out of 100 people in 3rd and 4th year, at that point completely focused on Software engineering studies only, maybe 2-3 had proper clue and could do the stuff out of their head. Needless to say, most people thrown out of university failed exactly these courses, and quite a few were brilliant coders, very successful afterwards. They just couldn't be bothered with bad approach this guy took. It is very important topic, but should be taught in a sane way. This guy couldn't do it, it alienated the topic to every single student for years to come (even to those few who got it all), and nobody at school dared to challenge him and his methods.
- beagle3 8y ago
  If you are math inclined, Cover & Thomas’ “elements of information theory” is a very clear and readable introduction (more for coding, less for channels, but great overall). If you consider giving it another chance.
  - mturmon 8y ago
    I heartily agree with this. Cover and Thomas was a huge leap in clarity and digestibility over the canonical texts of the time - it still holds up today. I took two grad-level information theory classes, one from Blahut's book, and one from Cover and Thomas, and the difference was night and day. I definitely recommend the book -- even the first two chapters (that define entropy and related concepts, and that cover the AEP) would be a good first cut.
SilasX 8y ago
I saw a signature on slashdot: "Information theory is life. The rest is just the KL divergence."

duality 8y ago

I agree. Information Theory is _the_ ur-science. It's the science of matching mathematical models to data, and so it's where inference meets the deduction. Such an important and fascinating subject should be widely taught and appreciated.

JoeAltmaier 8y ago
Well, next to math. And logic. And ethics.
- cossatot 8y ago
  I think a lot of us scientists wouldn't view any discipline not fundamentally based in collecting and interpreting observations as a science at all, much less an ur-science.
  - danharaj 8y ago
    Good luck observing anything without naming it and your observations. And once you have names, logic and mathematics are sure to follow.
    - harshreality 8y ago
      Logic and the rest of math are tools used in science, but I think the complaint was that science itself is about observation, developing theories, and testing theories against new observations. To elaborate: In pure subjects like math/logic, you have no observations, at least not really. Everything is abstract. Theorems follow the nomenclature of "hypothesis" and "conclusion" as if it's mimicking science, but every provable/disprovable statement in math, though it can be observed, formalized, and proved or disproved with enough effort or luck, is nevertheless merely a logical consequence of the axioms you started with. Theorems may provide insights, but they don't discover anything new. The conclusions of theorems are logically equivalent to the axioms. That's the essence of what it means to prove, mathematically. Whereas, in a science, there is no underlying theoretical truth. The only truths are quasi-truths that you develop based on observation and experiment, but they can all change tomorrow with a new experiment that is not beholden to any historical science precedent. Math/logic: everything is entirely based on axioms. The notion of "discovery" and "testing" of theorems is a cognitive fiction. Observation is merely the human process of deciding what things to try to prove. Everything is backstopped by abstract axioms and logic. Science: everything is entirely based on observation. The claim of scientific "truth" is a human fiction. We can never be absolutely sure of anything in science. There is no abstract truth backstopping science. Caveat: this does not imply anyone should be disputing scientific truth for practical purposes in everyday life. Disputing scientific consensus because it's not philosophically true is the refuge of spiritual quacks, religious apologists, and even rational people who are too in love with philosophy to take a step back and consider whether their philosophy is practical. "Oh, gravity isn't really true; quantum field theory isn't really true." Okay then, have fun not being able to move because you're floating in the air, and have fun without all modern electronics because they depend on the physics you're disputing, and have fun trying to find water with your dowsing rod.
      - duality 8y ago
        I agree with this and would add that science (by which I mean areas of knowledge observing phenomena and seeking to describe them quantitatively) interacts with deductive logic (including math) via information theory. Science without modeling would be devoid of predictive power. And, personally, I've never heard ethics or logic described as sciences. Please correct me if this is common terminology. But of course it doesn't diminish their importance.
      - cat199 8y ago
        You are confusing empirical science with science as a whole and missing formal science; The term 'science' generically should also be viewed within the philosophical context of 'art' e.g. an art vs a science.

saiya-jin 8y ago

We had it in our university, actually quite deep. It was done by head of IT department on our faculty, long-retired guy who was supposedly brilliant as theoretical scientist and had high reputation all over Europe in his field. It was done in most horrible and unmotivating way - A4 page or two densely covered with all greek letters and some more, and 98% of the content were just proofs of relatively simple statements. On all tests/exams, only the proofs were tested (so you either gave 1-2 pages of a single proof per question or blank page and could effectively go home as failed). Subjectively it was the worst set of classes during whole 5 years (and we had some serious IT-unrelated crap because were part of electro-engineering faculty back then), completely mandatory, no credit system back then to make it up via something else. Out of 100 people in 3rd and 4th year, at that point completely focused on Software engineering studies only, maybe 2-3 had proper clue and could do the stuff out of their head. Needless to say, most people thrown out of university failed exactly these courses, and quite a few were brilliant coders, very successful afterwards. They just couldn't be bothered with bad approach this guy took. It is very important topic, but should be taught in a sane way. This guy couldn't do it, it alienated the topic to every single student for years to come (even to those few who got it all), and nobody at school dared to challenge him and his methods.

beagle3 8y ago
If you are math inclined, Cover & Thomas’ “elements of information theory” is a very clear and readable introduction (more for coding, less for channels, but great overall). If you consider giving it another chance.
- mturmon 8y ago
  I heartily agree with this. Cover and Thomas was a huge leap in clarity and digestibility over the canonical texts of the time - it still holds up today. I took two grad-level information theory classes, one from Blahut's book, and one from Cover and Thomas, and the difference was night and day. I definitely recommend the book -- even the first two chapters (that define entropy and related concepts, and that cover the AEP) would be a good first cut.

SilasX 8y ago

I saw a signature on slashdot: "Information theory is life. The rest is just the KL divergence."