4 ms·
One of the useful applications of trigram-based analysis I have done is the following: for a large web-based application form where about 200000 online applicat
by sn41 7y ago
One of the useful applications of trigram-based analysis I have done is the following: for a large web-based application form where about 200000 online applications were made, we had to filter out the dummy applications - often, people would try out the interface using "aaa" as a name, for example.
Since the names were mostly Indian, we did not even have a standard database of names to test against.
What we did was the following: go through the entire database of all applications, and build a trigram frequency table. Then, using that trigram table, do a second pass over the database of names to find names with anomalous trigrams - if the percentage of trigram frequency anomaly in a name was too high (if the name was long enough), or the absolute number of trigrams in name was too high (if the name was short), we flagged the application and examined it manually. Using this alone, we were able to filter out a large number of dummy application forms.
Of course, it is not a comprehensive tool since what forms a valid name is very vague, but I think this kind of a tool is useful and culture-neutral.
- amelius 7y agoThe problem with these methods is that you exclude everybody that deviates from the norm. Yes, it might make your life (as a developer) a little bit easier, but it makes the lives of some of the applicants a lot harder.
- pjc50 7y agoI think the "manual review" phase makes this OK, in a way that simply autobanning Mr Null from your system isn't.
- amelius 7y agoI'm not sure if that would work either. Why not send them an email to verify? Or use captcha tech developed by big companies that actually has some science behind it.
- sn41 7y agoThese are high school students in India. Many of them are from rural backgrounds and do not have personal email accounts. From our experience, the forms are many times filled up by employees at cyber cafes who fill their own email addresses and mobile numbers instead of the students. The only reliable means of communications back to students is by a government approved website, or newspapers, or official media. (The process also has to stand up in court in case some student says that (s)he did not get the communication, and newspaper ads are a documentable evidence of communications on a specified date.) (BTW, the forms do have captchas, the spurious forms are manually filled in by mischievous/malicious/curious applicants.)
- banana_giraffe 7y agoOften not in practice. These sort of tools to triage input data for human review generally means the humans will just auto-approve what the computer tells them to. Yes, they'll catch some extreme cases, but anything that looks like the normal bad case will go by unnoticed generally more so than if the human is doing all of the review work.