moconnor

11 ms·

91.
by moconnor 3y ago
Transformers do not maintain hidden state between tokens. In this sense they cannot have an inner monologue. If you force one to output only a single token, it is constrained to doing a single pass of each layer on the problem. If you enc
92.
Why 99% is NOT Enough – Marble Machine X Accuracy Test (2022)(youtube.com)
1 points by moconnor 3y ago|0 comments
93.
by moconnor 3y ago
Siri does use on-device voice recognition. Active airplane mode and you can still use it. Edit: for some commands only it seems. I agree with OP, this is odd.
94.
by moconnor 4y ago
Yeah, this is deliberately misleading. Note the use of "Open AI" vs OpenAI. I wonder if the aim is to provoke legal action over the right to use the phrase "Open AI" and have a "OpenAI isn't open so why do they
95.
by moconnor 4y ago
Why? I'm pretty sure it could do this kind of task - attention is computed between all pairs of tokens. Yes, it's a lot of compute.
96.
by moconnor 4y ago
Clojure - cool, I haven’t seen this in the wild for ages! https://github.com/chr15m/rogule.com/tree/master/src/rogule
97.
by moconnor 4y ago
We did releases that were only bug fixes and maintenance. There can be a long tail of bugs in a complex system, most of which are relatively harmless and only occur in very rare situations; it’s hard to justify fixing these when paying cust
98.
by moconnor 4y ago
You can try these things out in a GPT-3 playground. Also search for “zero-shot learning” as the term of art to find papers on some of these topics.
99.
by moconnor 4y ago
This is beautiful. Having worked with everything from nanoGPT to Megatron, sitting down and reading through picoGPT.py was clear and refreshing with just the essential details. Nothing left to add, nothing left to take away: perfection.
100.
by moconnor 4y ago
Very high-dimensional space is unintuitive. It can help to imagine a cube as N increases - 2^N corners and vanishingly small volume to surface area. This gives useful intuitions e.g. cosine distance ignores length and only looks at angles,
101.
by moconnor 4y ago
INFO: did you make them sit on that hard wooden chair for eight hours and watch you recline in your aeron? But seriously: are your graphs showing 21 productive hours a week? That’s like four a day? What _are_ you doing the rest of the time?
102.
by moconnor 4y ago
All that to end with “no meaningful improvement over the salesforce codegen model” is a bit disappointing. Negative results are interesting in their own right. I’d rather read about why this isn’t better at the 6B parameter level than e s
103.
by moconnor 4y ago
If pumping SO2 into the atmosphere becomes profitable via cooling offsets, we might struggle to get people to stop doing it. Industrial scale CO2 and SO2 production might be how you get to Venus.
104.
by moconnor 4y ago
If it helps, a blacksmith who made their daily living producing nails could not conceive of ways to earn money post-industrialisation other than owning a mine or a factory either. But it turns out there are many unimaginable but better ways
105.
by moconnor 4y ago
A brutal example of commoditise your complements: you can see an economy as capital, land and labour. Labour employs capital to extract value from land. The fear here is that capital replaces labour, leading to the value of labour dropping
106.
by moconnor 4y ago
I’ve worked a 4-day work week at three companies now, ranging from tens to thousands of employees. You just block the time on your calendar and remind people when they try to schedule something there anyway. I would recommend you try it. It
107.
by moconnor 4y ago
Seconded - greenfield development benefits more than anything else from experience.
108.
by moconnor 4y ago
The difficulty of coding plateaus. Algorithms that seemed hard to grasp when you first encountered them become second nature and after that there's not an ever-increasing tower of logical complexity to climb. After that, there are thre
109.
by moconnor 4y ago
100% this. Source: previously worked at Arm. No inside knowledge of this case but this explanation fits background perfectly, whereas the article’s suggestions are incoherent. Also agree the article is a Qualcomm hit piece.
110.
by moconnor 4y ago
Here in Germany there are deposits on plastic bottles for things like coke. The bottles are more sturdy and get re-used multiple times.
111.
by moconnor 4y ago
If you’re working remotely in the EU, you are treated like a local worker with respect to taxes etc. this can cause issues if your employer doesn’t have a legal entity in the country, which a whole range of third-parties will solve for a cu
112.
by moconnor 4y ago
36% larger radius than Earth means ~2.5x more volume. If it has a similar density to Earth 2.5g would probably stretch the definition of habitable for humans. This doesn't rule out habitation by other life, of course. Cool that we&#x
113.
by moconnor 4y ago
At last! I love Colab but the vague promises around availability and quota made it impossible to recommend for my team to use professionally. I even tried and failed to get it up and running with a Google cloud GPU recently, before just swi
114.
by moconnor 4y ago
Note: this video is sped up to look more dramatic.
115.
by moconnor 4y ago
Transferable tax credits is the most interesting idea for a universal basic income implementation I’ve come across! It also has the benefit of appealing more to libertarian thinking (tax reduction and market-determined value!) than traditi
116.
by moconnor 4y ago
But… the FCC literally is how antitrust regulation is enforced - what do you want here? Competing antitrust regulators? Anti-antitrust regulation?
117.
by moconnor 4y ago
Actually the architecture of many of these models is profoundly unsurprising. The real difficulty actually is in training them. Just preprocessing the data for a language model can take several hundred days of CPU time. Training takes month
118.
by moconnor 4y ago
LaMDA in this sense _cannot_ learn. It takes a few thousand word parts as input and produces an output. It doesn’t update its weights beyond that, each time it produces a new input the same process is run.
119.
by moconnor 4y ago
These interest yields are unsustainable even for GUSD. Take your money out now ahead of the rush.
120.
by moconnor 4y ago
I hope you’re right, but the author is already one counter example and it would be far from the most self-contradictory position in e.g. creationism…

More ›