3 ms·
Input is almost always filtered. Without mincing the semantics of that, let's say you receive free-form NL query as input. You don't filter that to reduce it
by birdcircle 14y ago
Input is almost always filtered. Without mincing the semantics of that, let's say you receive free-form NL query as input. You don't filter that to reduce it to core terms you send to the database? You don't remove prepositions? You don't tokenize at all? Input is ambiguous and reduced/filtered into pseudo-meaningful terms to return relevant output.
- unconed 14y agoThis is true for all structured data, but the problem always comes from freeform text fields, which you cannot filter for one context without mangling or destroying the data for all other purposes—except perhaps trivial things like trimming leading/trailing whitespace.
- prodigal_erik 14y agoIf I want to tweet I <3 O'Reilly books you could pre-encode that as safe to paste into raw SQL, or as well-formed (X)HTML, but you can't do both simultaneously. Either encoding would end up distorting the content in the other context. You have to encode during output (and writing to a database counts) using the rules of the system consuming that output. Lots of crappy web forums visibly mangle punctuation in a futile effort to avoid this.
- MartinCron 14y agoWhat a great example. I am going to remember that one.