ImprobableTruth

7 ms·

61.
by ImprobableTruth 3y ago
>GPT-3.5 has 175B parameters versus 70B parameters in Llama 2 We know that for the original version of GPT-3.5, but my assumption was that Turbo was a distilled smaller model (which is why it uses OAI's new vocab & is so much fa
62.
by ImprobableTruth 3y ago
It's "unnatural" because it was finetuned on generated data using another model, almost certainly gpt-4 (whose TOS forbid this).
63.
by ImprobableTruth 3y ago
Llama 2 is open source-ish. Weights are freely available and can be commercially used, but only if you have less than 700m users and agree to some "don't do naughty things" terms.
64.
by ImprobableTruth 3y ago
I use two 3090s to run the 70b model at a good speed. Takes 32 gigs of vram, more depending on context. I tried CPU+GPU (5900X + 3090) but with extended context it's slow enough that I wouldn't recommend it (~1 token/s). CPU
65.
by ImprobableTruth 3y ago
GPT-4 trounces everything else available. I'd say that 70b is about 3.5 level, unless you're doing something where finetuning greatly benefits you. 13b is alright for toy applications, but the difference to gpt 3.5 (let alone gpt4
66.
by ImprobableTruth 3y ago
"some" quality - naive quantization (that isn't just the usual full->half) absolutely mutilates the model, to the point that picking a smaller model is much better. gpt-q is also more than just group wise quantization and
67.
by ImprobableTruth 3y ago
They're talking about the meltdown he had on stream [1] (in front of the mentioned pirate flag), that ended with him saying he'd stop using AMD hardware [2]. He recanted this two weeks after talking with AMD [3]. Maybe he'll
68.
by ImprobableTruth 3y ago
Flagged "human" but actually "LLM" is not a false positive, but a false negative.
69.
by ImprobableTruth 3y ago
Best to just hear it straight from the horse's mouth. https://arxiv.org/abs/2307.01850
70.
by ImprobableTruth 3y ago
Trains for local transportation make no sense. Trams for high-throughput and buses for everything else is the only sensible thing for local public transport.
71.
by ImprobableTruth 3y ago
Arxiv link seems to be broken, here is the correct one: https://arxiv.org/pdf/2306.03038.pdf
72.
by ImprobableTruth 3y ago
If type level functions are a "second language", then so is Zig's comptime.
73.
by ImprobableTruth 3y ago
You still have to memorize the "design patterns" that replace 'missing' features. Especially annoying for something like interfaces where there's a bunch of variants and people often use implementations with awful e
74.
by ImprobableTruth 3y ago
This isn't a new result really. We already know through the gpt-4 paper that rlhf style fine-tuning just makes the model more compliant, not more capable.
75.
by ImprobableTruth 3y ago
>Which would interfere with Go's philosophy that zero should be a valid (usable) state for types. What would be the default value of a reference type without nil? ... but nil is not actually a usable reference. You can't do any
76.
by ImprobableTruth 3y ago
Deepmind is owned by Google, but it's British and they've been behind a lot of significant conceptual results in the last couple years. Most significant progress is just "engineering", so it's all done by US corpora
77.
by ImprobableTruth 3y ago
Turbo uses a different vocabulary (same one as gpt-4). Indicates that it's not the same model as the original 3.5, so I would be very surprised if it wasn't distilled.
78.
by ImprobableTruth 3y ago
The midjourney price would be equivalent to ~100 hours cloud time. How is that crazy expensive?
79.
by ImprobableTruth 3y ago
They ballooned massively in size after covid (when their IPO occured), in 2019 they were still smaller.
80.
by ImprobableTruth 3y ago
This is easy to say, but I think the issue is that getting an LLM right isn't easy, so it's not clear who should steward such a project. Something like BLOOM shows that even if you have the necessary compute, you can still get a m
81.
by ImprobableTruth 3y ago
Did you mess around with the settings? I'm getting a correct implementation and since it's deterministic (with default settings) it should be the same for you.
82.
by ImprobableTruth 3y ago
Not sure if it was intentional,but the title of this article is pretty hilarious considering she explicitly badmouths explainability (trying to peer inside a black box) and advocates for interpretability (building models that are less black
83.
by ImprobableTruth 3y ago
Midjourney and DALL-E were both announced and in closed beta a month before Imagen was announced, with filters already in place.
84.
by ImprobableTruth 3y ago
Except they did rush their oh so dangerous language model out when push came to shove - turns out it just sucks in comparison. Frankly, I expect something similar for image generation. For all the good research Google's labs do, they&#
85.
by ImprobableTruth 3y ago
They could just filter it like Midjourney or DALL-E.
86.
by ImprobableTruth 3y ago
I mean, most researchers I know are just very excitable about their research and love to share it. I think that's a lot more likely than this being some PR masterplay - if MS wanted to really push this, they wouldn't release like
87.
by ImprobableTruth 3y ago
Subscribe to ChatGPT Plus and you get GPT-4 access. I tested your scene with it, its response: In the above scene, Joseph determined that Allie's parents were dead by testing the bad guy's knowledge of Allie's upbringing. He
88.
by ImprobableTruth 3y ago
Is this with full or empty context?
89.
by ImprobableTruth 3y ago
Computers are (with a couple caveats) deterministic, so I think the better analogy would be to say that it's the same as saying that "computers are just state machines". But this is a bit nitpicky. The core idea is of course
90.
by ImprobableTruth 4y ago
I'm curious, do you not think this might have adverse effects? Namely, if NN weights aren't copyrightable, limited releases like Meta has done might not be possible anymore so they might just cease completely with releases, ultima

More ›