3 ms·
I have a suggestion for improving clarity: word_bag: dict[str, int] = dict() # Multiset for line in raw_corpus: words = line.split()
by eesmith 3y ago
I have a suggestion for improving clarity:
word_bag: dict[str, int] = dict() # Multiset
for line in raw_corpus:
words = line.split()
for word in words:
if word in word_bag:
word_bag[word] += 1
else:
word_bag[word] = 1
Try:
word_bag = collections.Counter()
for line in raw_corpus:
word_bag.update(line.split())
then, to filter:
word_bag = {word: count for (word, count) in word_bag.items()
if count >= min_frequency}
or, if you can avoid printing the filtered word bag:
return set(word for (word, count) in word_bag.items()
if count >= min_frequency)
Also, that:
raw_corpus : list[list[str]],
should be list[str]. It's your corpus which is a list[list[str]]. (And you might want to use "line_words" instead of "line"
Could you explain why lines are processed independently?
for i, line in enumerate(corpus):
for j, target_word in enumerate(line):
Does it make a difference that some lines have fewer words than others?
I would think that
for positive_words in sliding_window(flatten(corpus), window_size):
using the sliding_window and flatting recipes at https://docs.python.org/3/library/itertools.html#itertools-recipes https://docs.python.org/3/library/itertools.html#itertools-r... would be more appropriate. Though using it directly would not handle the leading/trailing partial windows.
I liked seeing the 'η' - I haven't yet used non-ASCII in my code!
- extasia 3y agoHi eesmith, thanks for your insight! I've applied a lot of the improvements you mentioned, the collections.Counter one didn't even cross my mind- very neat:) The lines are processed independently as they are separate sentences. One line may be from one source, and the next from a completely different source. The problem with simply concatenating them is that the target words on the boundaries between sentences can end up with context words from unrelated sentences. >I liked seeing the 'η' - I haven't yet used non-ASCII in my code! Then you may enjoy this april fool's python issue I made https://github.com/python/cpython/issues/103172 https://github.com/python/cpython/issues/103172 Yeah, I like using utf-8 chars for mathematical symbols, it makes my brain hurt a little less when mapping between the two! I also like using ŷ for predictions in ML contexts as that's the canonical symbol used in the literature.
- eesmith 3y ago> The lines are processed independently as they are separate sentences. Ahh! I didn't understand that about the corpus. Thanks!