5 ms·
Building a full-text search engine in 150 lines of Python code (2021)
- cocodill 2y ago> return [token for token in tokens if token] I love that kind of bullshit poetry.
- pastage 2y agoYeah, but what is worst? I tried to understand the Haskell way in an article last week[0] pure (n, guard (factor /= n) $> factor) Which I think is more or less the same as this python line. return [factor for factor in factors if factor and not factor == n] The article does fancy stuff with memory caches which I believe is easy to do in python but I need to understand the Haskell code better. [0] Haskell: A Great Procedural Language https://entropicthoughts.com/haskell-procedural-programming#leaning-into-the-first-classiness-of-effects https://entropicthoughts.com/haskell-procedural-programming#... https://news.ycombinator.com/item?id=42754098 https://news.ycombinator.com/item?id=42754098
- mrkeen 2y ago> pure (n, guard (factor /= n) $> factor) Returns a tuple: Left side is n, right side is (Just factor) if factor is not n, or Nothing if it is.
- itishappy 2y agoYou're being way to clever! The machinery from the article is only needed to deal with IO. A more direct translation: # py [token for token in tokens if token] -- hs filter (not . null) tokens The full functions: # py def analyze(text): tokens = tokenize(text) tokens = lowercase_filter(tokens) tokens = stopword_filter(tokens) tokens = stem_filter(tokens) return [token for token in tokens if token] -- hs analyze :: String -> [String] analyze = filter (not . null) . stem_filter . stopword_filter . lowercase_filter . tokenize
- pastage 2y agoThanks I will look at that too. Clever and ignorant are synonyms in my book, and I am ignorant here. In this case IO and memory was what was interesting in the article.
- itishappy 2y agoI'm being way to clever... I read your article when it was originally posted, but hadn't realized that you're quoting from it! My bad... If you'll humor me, I'd love to take another shot, but from the more interesting direction! Adding a type might clear stuff up a bit: -- hs pure (n, guard (factor /= n) $> factor) :: IO (Int, Maybe Int) # py print((n, if factor == n then factor else None)) So without the for loop, there's no list to comprehend. Adding the list back gives something more equivalent to a standard for loop in Python: -- hs for numbers $ \n -> pure (n, guard (factor /= n) $> factor) :: IO [(Int, Maybe Int)] # py for n in numbers: print((n, if factor == n then factor else None)) Which could also be written as a list comprehension in either language: -- hs [pure (n, guard (factor /= n) $> factor) | n <- numbers] :: IO [(Int, Maybe Int)] # py [print((n, if factor == n then factor else None)) for n in numbers] Note that they behave a bit differently though, and it's the reason I haven't included `return` in the python line (what does `print` return?). Python will loop through the list and actually run the `print` function on each element, while Haskell will loop through and collect all the `IO` into one function to run later. Although it's starting to get pretty un-Pythonic, you can hack the behavior into Python with something like: # py return lambda: [f for f in (print((n, if factor == n then factor else None)) for n in numbers)] Which would need to be run using something unholy like: # py factorize(numbers)()
- dkga 2y agoIt grew on me, too. Originally I frowned upon this kind of python shenanigans but now I must confess it makes me a bit happier inside whenever I have to type a similar thing.
- graemep 2y agoI think its lovely if its reasonably readable (which this is) but they can be convoluted. I have written Python list comprehensions that I could not read myself the next day and so I am more careful now.
- ks2048 2y agoIf you're used to this, it's nice and readable. Or you can do, return filter(None,tokens) Not obvious, but giving None is like giving "lambda x : x" to filter().
- chaos_emergent 2y agoHoly shit, I’ve been writing python for 15 years and it’s the first time I’ve seen None used as an identity function. That’s nuts! How does it work under the hood? Does filter have a special case for evaluating None?
- echoangle 2y agoYes, that’s just a special case. You can’t call None as a function. https://docs.python.org/3/library/functions.html#filter https://docs.python.org/3/library/functions.html#filter > If function is None, the identity function is assumed, that is, all elements of iterable that are false are removed.
- sroussey 2y agoYou can do the same thing in JavaScript with Boolean Eg, arr.filter(Boolean)
- evnp 2y agoIs this any different from `filter(bool, tokens)`? (similar to the JS equivalent mentioned in a sibling) If not, I'm now curious why the `None` special case was added to `filter`..
- hacked_greenhat 2y agohttps://stackoverflow.com/questions/8748036/is-there-a-builtin-identity-function-in-python https://stackoverflow.com/questions/8748036/is-there-a-built... While this doesn't explain why filter accepts None as the lambda, I think the fact that there are no builtin identity function kind of explains why. It's easier to write None rather than put an identity lambda. I reckon it also makes it easier for a compiler to optimize this specific case.
- jankovicsandras 2y agoThis is a good intro to text search. Shameless plug: If you throw in a bit more, ca. 250 SLOC, you can have BM25 search: https://github.com/jankovicsandras/bm25opt https://github.com/jankovicsandras/bm25opt
- benob 2y agoMine: bm25 I use for teaching (sorry for the French example) https://gist.github.com/benob/69d48421f88f5dcc2b26a204d3251d8a https://gist.github.com/benob/69d48421f88f5dcc2b26a204d3251d...
- heresie-dabord 2y agoMerci! Mais faut-il s'en excuser? L'exemple est en français, voilà tout!
- usrnm 2y agoThis is an English-speaking forum, so yes, it is expected to use English and link to other stuff in English (or provide a translation). It doesn't mean that French (or any other language) is a bad thing, it just isn't the language used by this community.
- heresie-dabord 2y agoSure... but for the record, the code that the parent posted is in Python -- one of the global tech economy's pre-eminent programming languages for more than a decade. The parent's example contains some variables with Unicode values that are French words. It strikes me as odd that the parent would say sorry at all. Maybe the parent has a better understanding than I of linguistic sensitivities.
- marginalia_nu 2y agoYou can probably have phrase matching in a hundred lines more, maybe less. Most of the difficulty in search is dealing with the sheer volume of data. The algorithms themselves are pretty trivial for the most part.
- deleted 2y ago[deleted]