12 ms·
On the efficacy of online proctoring using Proctorio (2021) [pdf]
- endisneigh 4y agoCheating software is silly. Seems simpler to force someone to a secure location with a trusted proctor. This can and should be shared amongst all users of this and costs amortized appropriately.
- VoidWhisperer 4y agoFor things like the SATs/ACTs in the US, this used to be the case. I think it still is, but less 'important' tests (read: probably in-school tests and such) were moved remotely because of covid and might be stuck in a kind of limbo now
- daurnimator 4y agoWhile I don't disagree, a secure location with a trusted proctor is not simple or easy for all cases: students may be studying remotely/at home, which could be in a rural area with no accessible proctoring, or internationally with no trusted proctoring. And that is not always due to their own wishes: covid lockdowns, unavailability of international flights or other governmental restrictions may be keeping them there.
- thewebcount 4y agoNot to mention people with disabilities.
- ceejayoz 4y agohttps://www.economist.com/asia/2022/05/26/indias-exams-are-plagued-by-cheating https://www.economist.com/asia/2022/05/26/indias-exams-are-p... > Yet even as officials come up with novel ideas, so do the cheats. In February, a medical student at Mahatma Gandhi Memorial College in Indore, a small city, was caught with a skin-coloured Bluetooth device surgically implanted in his ear. A phone linked to the device was sewn into a secret trouser pocket. Last year, ten students taking a trainee-teacher exam were arrested for attempting to use Bluetooth gadgets concealed in the soles of their flip-flops. At least 25 students had bought such footwear from a gang for 600,000 rupees ($7,700) a pair. It is often mandatory for students to remove shoes and socks before exams. Cheaters get pretty clever.
- bhaney 4y agoI don't think cheat-detecting software like Proctorio could catch any of these schemes either
- mannykannot 4y agoFor the most part, it is not the cheaters in these cases demonstrating cleverness; it's those who are selling the means and methods.
- tsol 4y agoThe only way to win is to not play, but to instead invent cheating decides for other players so that you can profit without risk.
- CamperBob2 4y agoOr just instruct the student to go to a location chosen at random from a list of nearby cooperating venues -- schools, churches, office buildings, whatever -- where they won't have had the chance to prepare any elaborate cheating schemes. That would represent a good compromise between an on-site proctor and allowing the student full control over the testing environment.
- TobyTheDog123 4y agoSimilar to the TSA, I've found a lot of these platforms and nanny-software (Life360 et al) to be little more than security theater. If the person is determined enough, they will always find ways around it -- and good on them for doing so, this software is not a net-benefit on society no matter how "noble" they try to appear. Not the most related example, but I was just watching a video about undetectable online cheating in video games with a camera that monitors the screen and a robot that will physically move the mouse to accomodate. https://www.pcworld.com/article/696507/this-real-aimbot-uses-a-physical-mouse-to-cheat-at-shooting-games.html https://www.pcworld.com/article/696507/this-real-aimbot-uses...
- RcouF1uZ4gsC 4y agoUp until now, the University has served a dual purpose of education and certification. I wonder if in the future, that will be split. Maybe the place where you study, and the place that tests are certifies your competency in the subject are two different entities. Medicine, engineering, accounting, law all have some sort of this split where it is actually a different entity (medical board, bar, etc) that certified your competence.
- bigmattystyles 4y agoOther than civil engineering, I don't think that's true in the US for engineering. Happy to be corrected though.
- dhosek 4y agoWhen I wanted to switch from doing PHP/Perl programming to Java, I found getting the Java certification was a helpful thing for getting that first job. I wouldn’t necessarily recommend the certifications in general as a means of anything, but it does serve to open doors which might be hard to open otherwise. I have to imagine that this sort of path isn’t unique to myself.
- RcouF1uZ4gsC 4y agoThe Professional Engineer certification exam covers a lot more than just civil engineering. https://ncees.org/engineering/pe/ https://ncees.org/engineering/pe/
- sarchertech 4y agoThat's true but most engineers outside of civil engineering don't bother with it.
- InitialLastName 4y agoIt's less common in the other fields (where an engineer's decisions are less likely to involve life-risking decisions, or enormous amounts of public money), but the Professional Engineer certification is certainly a thing in the other fields. In fact, it's ambiguous at least in my state whether it is legal to run a business offering "engineering" services in those fields without a PE on staff.
- dataflow 4y agoI feel like 6 is not a large sample size?
- kelseyfrog 4y agoHow would you determine an appropriate sample size?
- viraptor 4y agoThat's a whole thing in statistics: https://en.wikipedia.org/wiki/Power_of_a_test https://en.wikipedia.org/wiki/Power_of_a_test
- kelseyfrog 4y agoRight, I'm trying to do this sorcratically and we got a smarty-pants over here. HN constantly has sample size critiques without power analysis responses which is a shame.
- mmmpop 4y agoI'm quick to this sort of judgment myself, so I feel confident advising you that I don't necessarily think the person was trying to be snarky.
- kelseyfrog 4y agoOh no, not at all. It was hilarious to have someone jump in earnestly. If anything i was the one being duplicitous with a leading question.
- rrobukef 4y agoFirst, I did not find any statistical test in the paper. The sample size is not relevant. Further the power of a test already required you to know two parameters: variance and the size of the effect your testing for (difference between means). Proctorio is very careful in not claiming any effectiveness for detecting cheaters, how will you estimate these parameters? Also iirc, the rule of thumb was for a normal distribution 30 samples is enough for decent strength, 40 samples is enough for general distributions unless you're looking at very small effects, weird distributions or in very noisy experiments.
- liminalsunset 4y agoThe purpose of these kinds of tools isn't to prevent cheating. Just like most DRM schemes, the purpose of these tools is the same as any other external consultant - so that the administration or people in charge of the school can offload the responsibilities related to any cheating that may happen to an external party, and if questioned, can respond that "reasonably secure technical measures have been taken to guarantee the integrity of the exam". Whether the integrity of the exams is actually guaranteed is immaterial, because the company that made the nanny software went on the record and took responsibility, and/or by splitting the responsibility, the school admin and the nanny software vendor can point fingers at each other and the responsibility essentially disappears like it usually does. You can see this with most other security theater type things. As long the "general public" is unaware that they have been lied to, they will accept this kind of positioning as gospel, and not look into it further. For the most part, as expected, the world turns around just fine without real security because systems tend to be robust to a certain level of bad actors doing selfish and small scale bad things like exam cheating, and as long as nobody's feelings are hurt, it's all fine.
- JumpCrisscross 4y agoGuessing there is an assumption about deterrence, too.
- asldfkjaskljdf 4y ago
- liminalsunset 4y agoDespite this probably being effective for casual or opportunistic people, the problem is that the opposite of the responsibility hot-potato thing happens on the "client" side too. Companies dedicated to "getting around these tools" and their ads start popping up, and now you have these companies effectively (in students' minds anyway) taking over the responsibility of dealing with the tool. So it becomes an IT pros vs IT pros cat-and-mouse game, just like jailbreaking and every other game cheating service.
- 4y ago
- hughes 4y agoI don't think the point is to protect against cheating - it's to protect the perception of legitimacy of institutions that are threatened by irrelevance.
- Aperocky 4y agoThe traditional methods of measuring competence via a closed-book, fixed time test is being rapidly dated. Practical knowledge and rote memorization have held mostly equal status, since out in the field both used to be inaccessible. Our tests are still structured around this, however the reality has moved on and rote memorization is now easily a search away. I like to see more tests moving to open-book model where test takers are allowed to use all resources, as in work. Of course, this will be harder to administer because schools can't just reuse tests over and over again since they're bound to show up in the internet. Overall, more and more of evaluations will move from testing altogether into some project based evaluation, which I believe is superior in terms of accessing skill. Let's see how long the old format hangs on.
- skybrian 4y agoThis depends very much on the job. Memorization is of course needed when learning a foreign language, and for any job with a real-time performance aspect, you can't be looking stuff up all the time.
- mbreese 4y agoIn classes where I had open book tests, the value of the notes was always mitigated by the time allowed for the test. It was always made clear to us that while you could use any resources you wanted, you definitely didn’t want to do that. You just didn’t have time. But if you needed to look up a particular chemical structure (example), it was possible. But you had to really know the material in the first place.
- SoftTalker 4y agoBeing able to search for an answer is not the same thing as knowing the answer, and it does not contribute to being able to synthesize ideas from observation combined with knowledge. Otherwise I don't disagree with you that this is where things are heading. In a few more years we probably won't even be giving traditional grades in school anymore. What would they mean, other than that the student has enough Google prowess to look up the answers?
- blacksqr 4y agoWaiting for the announcement of an even more intrusive technology called Proctolio.
- bitwize 4y agoGiven the malarkey that academic cheaters get up to, that may be necessary.
- gjsman-1000 4y agoI was at a school which used "Lockdown Browser" to try to lock-down the Windows environment to only the exam. Unfortunately for the school, I had experience with Windows programming, and blew the doors off Lockdown Browser in about 10 minutes, with multiple apps and windows open on top of the befuddled browser. And later, I found a way to do it with no programming skill at all. I did it to make the point that if I could figure it out, who is to say that some rich student's parents wouldn't pay a nice sum of cash for an exploit like that? Or that they haven't? The school, well... just decided to ignore it and keep it anyway because it was a contract. https://gabrielsieben.tech/2020/10/28/the-lockdown-browser-is-not-very-good-at-locking-down/ https://gabrielsieben.tech/2020/10/28/the-lockdown-browser-i...
- trevcanhuman 4y agoI've also used it when I was taking exams for high school. It's pretty bad software if you ask me. rant/long comment First of all, they have a 'different' program for each institution, customized for you to 'sign-in' onto your school's auth page. From an institutional perspective it seems clever, since everything you do will most likely be logged along with your session. Another annoying thing was that there's different versions of the program for 'each' software. E.g. I used the Canvas platform for some exams and the ALEKS platform for math exams. And they have the same name under the windows programs so I can only really know which one I'm opening from looking at the desktop. I believe the Lockdown program is using some lame code to keep the program window 'active'. I never really tried to do any programming behind it. But I do remember that I had some AutoHotkey scripts running for completely unrelated stuff. On another note, I read their privacy policy and it seems very vague. I wonder if they can get away with letting a stranger view my face for two hours.
- 6stringmerc 4y agoOf course they got this result they had a sample size of 30 Dutch from various geographies - not the program set for 30 in a homogeneous group. If it’s a weakness out of 50 different groups, okay, but how about testing 30 other skin tones or bone structures or hair types… Study means nothing without breakdown of race / ethnicity (self reported) and age range because in its functional instead of academic gaming for publication environments, the algorithm is designed for a much greater sample size. Of diversity, because the absence of information on this condition the software must account for is woefully naive.
- ec109685 4y agoIt was also totally contrived with people pretending to be nervous and fake cheating.
- Aachen 4y agoAs a small note on the page labeled 281, they remark that they called for digital cheating methods since analog ones like cheat sheets were previously already found to be "virtually undetectable". Also interesting: you can create profiles to weigh things more heavily, like weigh a abnormal noises worse than normal if you are suspecting students of using audio calls. They made a profile specific to each of the cheater's method and retroactively applied those profiles to the collected data, and still half of them could not be made to show up near the top of the suspects list. And this is only possible if you already know the exact cheating method your class will use. The page labeled 288 does make a compelling case for how the spreading of fear, uncertainty, and doubt is effective, leading to less people trying to cheat. It just doesn't stop the cheaters that risk the perceived odds and I doubt this is sustainable once such systems are everywhere and they notice it's safe. For now, you might as well turn on a dummy system that claims to be real by doesn't actually invade privacy because it's a dummy. Per the results, it would be equally effective and thus achieve the best of both worlds.
- chrismorgan 4y agoI think this is rather important. I get the distinct impression they also called for single-device cheating methods; almost all of the cheating methods employed are unrealistically complicated and prone to detection: but you could just use another device (phone, laptop, whatever), and be vastly less likely to be detected or flagged as suspicious. (The only thing that might remain troublesome is ones where you want to speak to someone or play a video, so the microphone shenanigans might still be employed.) In practice I feel like this isn’t even “it failed to detect all six cheaters” but rather “it failed to detect all six cheaters, even though we got them to cheat in ways that are uncommonly detectable and there are much easier ways to cheat that it has absolutely no hope of detecting”.
- pophenat 4y agoWhile at university, we had one professor who would assess student assignment papers after conducting an interview. It would take him 5-10 minutes per person and cheaters really did get caught out. It is a common practice in some universities and it's very hard to cheat when being asked questions in person. It can be done online, as long as the identity of the person can be verified. Takes a lot longer and is a lot more engaging that written tests, but it works.
- MichaelCollins 4y agoI wonder how he handles foreign students who feign language difficulty when convenient. From what I've seen, such cheaters tend to get away with it.
- pophenat 4y agoThere was this case of a foreign student who submitted a perfectly written paper, obviously copy/pasted from some websites. At the interview, he could not answer any questions properly, and that made it even more clear that he could not have written the marketing-type paragraphs. I think he did just pass anyway, but that’s a different story - the university tended to let international students pass easier sometimes.
- solomonb 4y agoI took a class in college where the professor made us plagiarize our entire final paper. We couldn't use a single completely original sentence. The trick was that you had to highlight and source every plagarization. It turned out this was one of the most challenging papers I wrote in undergrad and it forced me to understand my source material on a much more intimate level. This was 15 years ago and access to information via the internet has only intensified since then. Perhaps our educators need to reconsider from first principles how we assess students in this era.
- gzer0 4y agoRemote proctor software is nothing but security theatre. It's so laughable the little effort they put at all to prevent any sort of cheating whatsoever. I created my own ring 3 rootkit (user land) and infected my own system to modify registry values to hide remote software to allow complete and total pwnage of the exam itself. TerminateBlacklist: mov [esp-4], esi lea esp, [esp-4] push offset loc_C81CF9 push ds:TerminateProcess jmp locret_6CCBD5 EnumerateBlacklist: mov edx, g_Globals mov ecx, [edx+0C904h] mov eax, [edx+0C8FCh] mov esi, [edx+0C91Ch] mov eax, [ecx+eax] mov [esi+edi+28h], eax add edi, 2Ch The pointer ‘g_Globals’ is a 4-byte pointer- seen multiple times in the main module. Opening ‘g_Globals’ in memory and browsing through the pointers accessed above such as 0x0C91C brings us to a custom array type containing hundreds of strings. Taking a look through the struct type and looking back at the array iteration code, wrote a basic structure and dumped all blacklisted processes: class BlacklistedProcess { public: PCHAR window_name; PCHAR process_name; char pad[0x24]; }; Here's a list of all current blacklisted processes: https://pastebin.com/raw/bidPLAU7 https://pastebin.com/raw/bidPLAU7
- simonebrunozzi 4y agoProctoring is testing overseen by an authorized, neutral, proctor, who ensures the identity of the test taker and the integrity of the test. I wasn't familiar with the term. (non-native English speaker, lived in the US for ~10 years)