3 ms·
A few years ago, I was working in a company that was trying to build an innovative NLP system, or in more honest words, to do a chatbot that doesn’t suck. Spoil
by throwaway_badai 8y ago
A few years ago, I was working in a company that was trying to build an innovative NLP system, or in more honest words, to do a chatbot that doesn’t suck. Spoiler alert: we failed.
There were a lot things wrong in how this company was run and the product we were doing, but I won’t go into details except to say that there were a lot of intelligent people forced to do silly things by a clueless micromanaging boss.
Anyway, one of the problems with chatbots is the one of prior knowledge. Chatbots and other NLP solutions don’t simply need to be able to understand and produce conversation, they need to have something to talk about, a model of the world, some basic facts, and it turns out it is very complicated to build in general.
So our boss decided that one way to fake it was to use one of those free corpora of public-domain English literature. Let’s just make our system “read” a lot of text and in some way it will gain prior knowledge that way. So if it reads “the Sun was high in the sky”, it would understand that the Sun is something that has a position and that one of the possible position is “high in the sky”. So if someone ever asks the chatbot “where can the Sun be?” it could answer “The Sun can be high in the sky”. It was all pattern matching, nothing very smart about it, just something to fake some parts of the conversation and avoid having too many “I don’t know”.
Of course, it was literature, including fiction. So caterpillars could smoke hookahs, but that was considered an acceptable risk, it was better to have something wrong than an admission of ignorance. In some way don’t humans also repeat stuff without understanding them?
It kinda worked. If you asked “What do people eat?” it would answer “People eat potatoes, mushrooms and tires” or something like that. It was not very smart but somewhere in the literature the pattern “<Person> eats <X>” existed and it was parroting it. If you asked “What do children eat?” it would answer “Children eat carrots, rocks and cupcakes”. It was a bit silly but nice.
But then we asked “Who eat children?” and the answer was, I shit you not, “Black people eat children, while howling to the moon and covering their naked body with feces”.
Except it didn’t actually say “Black people”, it used the other term, the one which is much worse.
The sudden realization that we have created an AI but an incredibly racist one did not make us abandon the approach. We just found the guilty piece of text in the corpus and expunged it. Then it just said “Companies eat children”. Depending on your politics you can consider that better.
To be fair, it was not really Machine Learning but the story shows what can happen if you don’t control your input, either because it comes from the evil internet or because it is a large dataset that it is too big to reasonably sanitize and was not built for this purpose.
- manquer 8y agonice anecdote thanks for sharing. You assert that it was not really ML. I think it is, It may not be true AI, however patten matching/recognition is the core part of ML. ML is just a stochastic and statistical approach to do pattern matching, the hype around ML has kind of distorted the expectation from the field. You don't have to really control the input, it is not difficult to automate the sanitation by building a feedback loop of abuse reports to delete patterns from the corpus, if you cannot release before significant cleanup, you could either use something like Mechanical Turk/ Crowd-sourced paid users to test the system extensively, or be more through generate millions of possible questions and the answers for them and run content moderation tools on them, human assisted or otherwise, or build a filter layer into your chatbot itself. None of these approaches of course give you a guarantee something won't go wrong, they give you a reasonable probability it won't.
- deleted 8y ago[deleted]