extasia

15 ms·

151.
Ask HN: Tools for LLM training time estimation?
6 points by extasia 3y ago|0 comments
152.
by extasia 3y ago
Trial implies some sort of study or analysis that the other two words you suggest don't.
153.
by extasia 3y ago
I have been to a Dennys. (euro enjoyer here)
154.
by extasia 3y ago
It's not a serious implementation but one for didactic purposes:) I've used gensim's one and it seems great for practical use but I wanted to build an understanding of the underlying algorithm, so built my own from scratch. I
155.
1-liner for viewing the shape of all local Tensors in Python(gist.github.com)
2 points by extasia 3y ago|1 comments
156.
by extasia 3y ago
Using torch and numpy a lot I find myself perplexed about the shape of some torch.Tensors, particularly if they have a non-obvious set of dimensions (e.g a batch dimension or something). I reach for this one-liner pretty regularly so perhap
157.
by extasia 3y ago
What do you mean by self regularisation?
158.
by extasia 3y ago
The context for one training step is the target word, plus a window of the `n` words that precede and follow that word. For example in the sentence: > Usually titles are more relevant to a document then a random sentence or word in the m
159.
by extasia 3y ago
Do you mean html tags? I'm not sure I follow!
160.
by extasia 3y ago
Hi eesmith, thanks for your insight! I've applied a lot of the improvements you mentioned, the collections.Counter one didn't even cross my mind- very neat:) The lines are processed independently as they are separate sentences. On
161.
by extasia 3y ago
In the skip-gram algorithm (aka word-2-vec), words vectors are global; i.e one word maps to one vector. In practice, as you intuit, contextual embeddings are in fact useful for obtaining more precise word vectors. The example given in[0] is
162.
Show HN: Word2vec Algorithm in ~100sloc with NumPy(github.com)
72 points by extasia 3y ago|30 comments
163.
by extasia 3y ago
What is thought is not what is said. What is said is not what is heard.
164.
by extasia 3y ago
Fwiw, I can't think of a single popular neural net architecture that takes advantage of sparsity.
165.
by extasia 3y ago
Interesting! I'm working through implementing each of these w/ only numpy, so the "it's just a dot product" should come in handy:) Case in point I just finished implementing the skip gram algorithm... In possibly th
166.
by extasia 3y ago
Love your username, OP.
167.
by extasia 3y ago
I wasn't aware of this, do you have a link you could share?
168.
by extasia 3y ago
You're correct! It's dot product reduces to cosine sim, not the other way around. Could you clarify your second statement about Mad and 3x3s? Afraid I don't follow.
169.
by extasia 3y ago
I use something similar: "answer concisely" or "be terse" and it works pretty well
170.
by extasia 3y ago
"Data driven decision making"
171.
by extasia 3y ago
I thought that they use the dot product during training[0] and then the cosine similarity during inference. Sometimes the vectors are pre-normalized for inference (thus reducing cosine sim. to just the dot product), is this what you're
172.
by extasia 3y ago
Could you give an example of an embedding space that is only meaningful locally? The word vector idea of modifying a word by some operation e.g King->Queen ~= man->woman ~= uncle->aunt Which intuitively seems non-local to me.
173.
by extasia 3y ago
Are you looking to make this a product? On an unrelated note I realized recently that the 'bag' in bag-of-words is another name for the multiset data structure... Which makes sense when you think about the text as being a _set_ of
174.
by extasia 3y ago
Are you tracking your satisfaction somewhere? Interested in your embedding based system - is that embedding layer + neural net? Sounds very cool overall:)
175.
by extasia 3y ago
Deepmind are still based in London. In fact they're pretty near to the Turing Institute HQ.
176.
by extasia 3y ago
The transformer architecture is arguably the most important breakthrough in NLP, and language is the predominant mode of communication between humans, so I fail to see how its "just a load of hype" Could you name a bigger breakthr
177.
by extasia 3y ago
I would add that some people working at the Turing institute are also tenured professors, so this is only one of their salaries. Not sure this applies to that exact position, but a person I know is a Turing fellow and professor at a univers
178.
by extasia 3y ago
Probably the software book that has influenced my practice the most. Fantastic writing, and not too long either.
179.
by extasia 3y ago
Political expediency is my understanding.
180.
by extasia 3y ago
There's a theory that these LLMs are learning these specialized internal structures anyways. If you could distill _those_, now that would be useful! (cheaper inference, ability to deploy on mobile etc.)

More ›