huac

11 ms·

61.
by huac 3y ago
the number 1 in Mandarin Chinese is usually pronounced "yi" which can be mistaken for the number 7 ("qi"). so in certain contexts, e.g. reading out a phone number, it would be pronounced "yao" instead.
62.
by huac 3y ago
agree, this is a nice example of generating synthetic data, and I believe that the synthetic data is helpful for generating useful embeddings for RAG, but not including an ablation with fine-tuned E5 or another commonly used embedding model
63.
by huac 3y ago
the BTL model is just a way to infer 'true' skill levels given some list of head to head comparisons. the head to head comparisons / rankings are the most important!!!! and in this case, the rankings come from GPT-4 itself. s
64.
by huac 3y ago
some catches I'm looking for: it remains to be seen if the generations are high quality (not just low perplexity) and if the architecture scales beyond 3B, but definitely super promising
65.
by huac 3y ago
I would, at the very least, wrap the errors being returned inside the process function https://github.com/neurocult/agency/blob/14b14e50a7570189388... Or, I suppose the user must handle exception behavior in
66.
by huac 3y ago
yeah, I just think it's a question of picking the right set of things to abstract away (and we're all figuring out what set that should be). I chose to not add much abstraction over the go-openai library to support maximum customi
67.
by huac 3y ago
definitely serve different purposes, I basically think abstracting away the flows (agency/langchain) is the wrong layer of abstraction, and that it's more useful to abstract away certain problems to solve (eg vector retrieval) or
68.
by huac 3y ago
I like Go a lot for working with OpenAI etc, it's 'just' API calls and Go is great at that. I've opensourced some bits here: https://github.com/stillmatic/gollum -- in particular, function dispatch
69.
by huac 3y ago
I concur that, for the use cases that OpenAI's voices cover, it is significantly better than Eleven.
70.
by huac 3y ago
It is theoretically possible to train a model that, given some speech, attempts to continue the speech, e.g. Spectron: https://michelleramanovich.github.io/spectron/spectron/ . Similarly, it is possible to train a
71.
by huac 3y ago
to zoom out a bit, this is known as the 'cocktail party problem' and is very much considered unsolved (particularly when there is an unknown number of speakers or overlapping speech).
72.
by huac 3y ago
nice work! generating good example data is the most important part of finetuning. imo summarization is also a fairly simple task -- I wouldn't be surprised if a fine-tuned open source model (eg llama 13 / mistral 7) would get to s
73.
by huac 3y ago
"it depends" https://kipp.ly/transformer-inference-arithmetic/
74.
by huac 3y ago
10k H100's cost around $400M at sticker price. Sometimes I wonder if a massive setup like this is less expensive (volume discount) or more.
75.
by huac 3y ago
the nice thing too is that because you are freezing almost all the parameters, and generally in lower precision (eg QLoRA loads the full model in 4-bit), it's super low gpu memory usage. a free Colab will suffice for finetuning a 7b de
76.
by huac 3y ago
I don't have any access to their financials, so this is speculative, but while they do 'give away' GPT-3.5-turbo in the free ChatGPT, the rest of the business is likely extremely profitable. If you want to estimate cost of se
77.
by huac 3y ago
I think this is one of the most important possible works for open source LLM's, really glad y'all pushed this forward! That's not hyperbole. Why is OpenAI able to charge so little for their API's? I have heard rival mega
78.
by huac 3y ago
you're thinking of 1455 market st. this is 1455 third st. (the coincidence amused me)
79.
by huac 3y ago
the 'low resource languages jailbreak gpt-4' paper ( https://arxiv.org/abs/2310.02446v1 ) shows that the models, when prompted in a low resource language, return highly coherent and sophisticated responses. it&
80.
by huac 3y ago
15 million GPU hours is interesting, llama 2 had about 1.7 million hours for the 70B (3.3M for all). Their GPU hours are on LUMI which is AMD MI250 GPU, which Mosaic reports is about 75% of the A100 ( https://www.mosaicml.com&#x2F
81.
by huac 3y ago
rebuttal: compute optimality matters https://arxiv.org/pdf/2203.15556.pdf
82.
by huac 3y ago
We used Triton Inference Server (with a Golang sidecar to translate requests) for model serving and a separate Go app that handled receiving the request, fetching features, sending to Triton, doing other stuff with the response, serving. Th
83.
by huac 3y ago
I would love some examples of how to do non-trivial data interop between Rust and Python. My experience is that PyO3/Maturin is excellent when converting between simple datatypes but conversions get difficult when there are non-standar
84.
by huac 3y ago
yeah, I hear you that the decoder-only infilling approach is 'weird' -- I just don't know if I agree that it's manifestly worse at language understanding / performance than the BERT appraoch
85.
by huac 3y ago
where is that reported? in Table 14 I see PSM performing much better than SPM. I also see a note about the SPM performance which attributes the degradation to the tokenizer edge cases > As an example, our model would complete the string
86.
by huac 3y ago
this comment is downvoted but it is correct. while fill-in-the-middle is less obvious in the decoder-only paradigm, it is still possible. one example is code-llama https://ai.meta.com/blog/code-llama-large-language-mode
87.
by huac 3y ago
I am, with many optimizations eg dynamic batching, tensorrt, but most important is batch size > 1 -- cheating a little bit in the comparison ;). batch size 1 is more like 200 tok/s, still a very large improvement. the reason why thi
88.
by huac 3y ago
The OP notes that they can infer a 7B model at 2s/tok using their CPU setup. Equivalent inference on a 4090 ($1600) is up to 400 tok/s, or 0.0025 s/tok. This is an 800x difference. (to be fair, you can do better than the OP&#
89.
by huac 3y ago
because on a per-inference level, it's _still_ cheaper to use GPU than it is to use CPU, even if you ignore latency and only consider offline batch applications.
90.
by huac 3y ago
I wanted a very simple feature flagging solution, so I wrote my own with Google Sheets as the backend: https://github.com/stillmatic/flagsheet It's slightly more complex than a code push but still very easy to use

More ›