4 ms·

Yes, I've found dashboards useful. Generalizations or problematic, since so much of this depends on industry, company, and role. I'm in an industry (education

by blagie 4y ago

Yes, I've found dashboards useful. Generalizations or problematic, since so much of this depends on industry, company, and role. I'm in an industry (education) where at-a-glance dashboards have a proven track record of dramatically improve learning outcomes. It turns out that: (1) Teachers, while believing they know their students, actually don't. Teachers also regularly fail to grasp what students do and don't understand. (2) Allowing teachers to monitor their students, both individually and classroom wide, leads to much better outcomes. In the context of a classroom, teachers are busy, non-technical (and sometimes incompetent), so an at-a-glance view is critical. For an example of real-time dashboarding done well, see Learning Catalytics. It grew out of a highly-respected physics education research project (Peer Instruction, by Eric Mazur), and allows teachers to monitor student understanding during lecture. Showing where misconceptions exist allows for deeper dives and deeper conversation. There are many systems like it, and ample evidence of improved student growth when they're used. That's very specific, I know, but there are similar examples in other industries I've worked in.

urthor 4y ago
Real time monitoring during a live task is certainly a use case I've never thought of. A very good point. I remember a piece on programming education research awhile ago. Said education outcomes improved considerably if you constantly bombard students with multiple choice questions. Administering that through a dashboard would be essential.
- falcor84 4y ago
  > education outcomes improved considerably if you constantly bombard students with multiple choice questions I would love to read more about that, if you happen to have a link to share.
  - urthor 4y ago
    https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006023 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...
    - falcor84 4y ago
      Much obliged. I see the relevant part seems to be the following, in the "Tip 4" section: """ The key to making demonstrations more effective is to make learners predict the outcome of the demonstration before performing it. Crucially, their prediction should be in some way recorded or public, e.g., by a show of hands, by holding up cue cards marked with A, B, C, or D, or by talking to their neighbour. We speculate that the sting of being publicly wrong leads learners to pay more attention and to reflect on what they are learning; regardless of whether this hypothesis is true, instructors should be careful not to punish or criticise students who predicted wrongly but rather to use those incorrect predictions as a spur to further exploration and explanation. """
      - blagie 4y ago
        That part has been replicated (see e.g. vicarious learning, from Derek Muller as well as Miki Chi, studies on Khan Academy in-video questions, etc.). However, it's not the most relevant part. The most relevant part is simple use of active learning. The classic paper here is Hake (Interactive-Engagement vs. Traditional Methods: A Six-Thousand-Student Survey of Mechanics Test Data for Introductory Physics Courses). However, it's been replicated discipline-by-discipline (lots of papers in PNAS). Best summary is ICAP by Miki Chi. Miki didn't invent it, and ICAP isn't the most rigorous, but she's pretty good at these summaries. Having students do anything beyond listening leads to huge improvements in student learning, basically no matter what it is. Some things lead to bigger improvements than others, of course. Even clicking a "next" button occasionally shows a (very modest) gain. For good interactive engagement, doubling of learning gains is a good lower-bound over passive listening.
        urthor 4y ago
        Note that it's Active Recall according to other education research. Not only active participation, but actively being forced to remember the knowledge, in a different format to the original, is the key. Teaching another student is definitely an approach to this. But a gentle touch multiple choice question also works.
mistrial9 4y ago
honestly this is a nightmare'ish point of view for anything but hard science at senior level. The emotional environment of being monitored constantly is toxic. Control-oriented people gravitate towards constant monitoring in similar ways as the Executive and their Dashboard are described above. How can thinking and feeling humans delegate interaction in a learning environment to machinery ?
- blagie 4y ago
  That can be true, but it's not true in this case. Eric Mazur's classroom is sort of a visitor sport -- people come from around the world to observe. I'd encourage you to do the same. It's very human. As originally implemented, the technique was pretty simple: - You give a traditional lecture - Every five minutes or so, students are asked a question to see if they understand concepts - The system peers students who answered differently to discuss. Students can change their answers at the end. - The instructor and class see aggregated statistics (e.g. 90% of people answered 27, 5% answered 54, and 5% gave other answers) - Based on the feedback, the instructor can do a deeper dive into places where there are common misconceptions Today's class is very different. Mazur juggles all sorts of peer learning models, 360 reviews, etc. in ways which are superhuman. But that's the original model. Most of the uses of formative assessment lead to classrooms feeling more personal, more personalized, and more human. There are control-oriented systems which scare the heck out of me, like GoGuardian and Securely, but this isn't it. There is a complex ethics question of how to build systems which don't dehumanize students.
  - urthor 4y ago
    The problem with voluntary question and answer is that the loud kids ask questions. The shy kids sit there and be quiet. "Prompt" questions, like multiple choice, are a good way of encouraging reluctant students to come out of their shell."
    - blagie 4y ago
      It's not voluntary. 100% of students are asked a question. In the eighties, this was entered into a classroom response system (a clicker where kids press a button on multiple choice) and an eighties-era computer would process their responses for an eighties-era dashboard, presumably using one of those $20,000 CRT projectors. Today, it's cell phones and laptops, and they do a lot more than multiple choice. Students might be asked to highlight text in a passage (and the system shows a heatmap), to draw vectors, etc.

by blagie 4y ago

urthor 4y ago

Real time monitoring during a live task is certainly a use case I've never thought of. A very good point. I remember a piece on programming education research awhile ago. Said education outcomes improved considerably if you constantly bombard students with multiple choice questions. Administering that through a dashboard would be essential.

falcor84 4y ago
> education outcomes improved considerably if you constantly bombard students with multiple choice questions I would love to read more about that, if you happen to have a link to share.
- urthor 4y ago
  https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006023 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...
  - falcor84 4y ago
    Much obliged. I see the relevant part seems to be the following, in the "Tip 4" section: """ The key to making demonstrations more effective is to make learners predict the outcome of the demonstration before performing it. Crucially, their prediction should be in some way recorded or public, e.g., by a show of hands, by holding up cue cards marked with A, B, C, or D, or by talking to their neighbour. We speculate that the sting of being publicly wrong leads learners to pay more attention and to reflect on what they are learning; regardless of whether this hypothesis is true, instructors should be careful not to punish or criticise students who predicted wrongly but rather to use those incorrect predictions as a spur to further exploration and explanation. """
    - blagie 4y ago
      That part has been replicated (see e.g. vicarious learning, from Derek Muller as well as Miki Chi, studies on Khan Academy in-video questions, etc.). However, it's not the most relevant part. The most relevant part is simple use of active learning. The classic paper here is Hake (Interactive-Engagement vs. Traditional Methods: A Six-Thousand-Student Survey of Mechanics Test Data for Introductory Physics Courses). However, it's been replicated discipline-by-discipline (lots of papers in PNAS). Best summary is ICAP by Miki Chi. Miki didn't invent it, and ICAP isn't the most rigorous, but she's pretty good at these summaries. Having students do anything beyond listening leads to huge improvements in student learning, basically no matter what it is. Some things lead to bigger improvements than others, of course. Even clicking a "next" button occasionally shows a (very modest) gain. For good interactive engagement, doubling of learning gains is a good lower-bound over passive listening.
      - urthor 4y ago
        Note that it's Active Recall according to other education research. Not only active participation, but actively being forced to remember the knowledge, in a different format to the original, is the key. Teaching another student is definitely an approach to this. But a gentle touch multiple choice question also works.

mistrial9 4y ago

honestly this is a nightmare'ish point of view for anything but hard science at senior level. The emotional environment of being monitored constantly is toxic. Control-oriented people gravitate towards constant monitoring in similar ways as the Executive and their Dashboard are described above. How can thinking and feeling humans delegate interaction in a learning environment to machinery ?

blagie 4y ago
That can be true, but it's not true in this case. Eric Mazur's classroom is sort of a visitor sport -- people come from around the world to observe. I'd encourage you to do the same. It's very human. As originally implemented, the technique was pretty simple: - You give a traditional lecture - Every five minutes or so, students are asked a question to see if they understand concepts - The system peers students who answered differently to discuss. Students can change their answers at the end. - The instructor and class see aggregated statistics (e.g. 90% of people answered 27, 5% answered 54, and 5% gave other answers) - Based on the feedback, the instructor can do a deeper dive into places where there are common misconceptions Today's class is very different. Mazur juggles all sorts of peer learning models, 360 reviews, etc. in ways which are superhuman. But that's the original model. Most of the uses of formative assessment lead to classrooms feeling more personal, more personalized, and more human. There are control-oriented systems which scare the heck out of me, like GoGuardian and Securely, but this isn't it. There is a complex ethics question of how to build systems which don't dehumanize students.
- urthor 4y ago
  The problem with voluntary question and answer is that the loud kids ask questions. The shy kids sit there and be quiet. "Prompt" questions, like multiple choice, are a good way of encouraging reluctant students to come out of their shell."
  - blagie 4y ago
    It's not voluntary. 100% of students are asked a question. In the eighties, this was entered into a classroom response system (a clicker where kids press a button on multiple choice) and an eighties-era computer would process their responses for an eighties-era dashboard, presumably using one of those $20,000 CRT projectors. Today, it's cell phones and laptops, and they do a lot more than multiple choice. Students might be asked to highlight text in a passage (and the system shows a heatmap), to draw vectors, etc.