nestorD

7 ms·

61.
by nestorD 2y ago
I am doing quite a bit of work with JAX (the Python library used here) in a high-performance numerical computing context. On GPU/TPU, it is not going to reach perfect 100% hardware usage, but it is going to get close enough (far above
62.
by nestorD 2y ago
My intuitive understanding of transformers is that each token (input+output) gives the model some "thinking space" that can be used to store reasoning information (independently of the token): thus, a transformer should be more cl
63.
by nestorD 2y ago
I have done it, generating a circle with symbols around it, it took a surprising amount of work to get what I wanted.
64.
by nestorD 2y ago
I have seen Rust people use it, it has a library: https://crates.io/crates/coz
65.
by nestorD 3y ago
This is a very nice website! I would love some Baye Area graphs.
66.
by nestorD 3y ago
> Early on in my engineering career, there are a handful of times I was assigned to a project where I was set up to fail. I didn't understand the situation until far too late. There are a few tell-tale symptoms: I didn't quite
67.
by nestorD 3y ago
Take your document, cut it (cleanly!) into pieces small enough to fit into your sentence embedder's context window, and generate several embeddings that all point to the same document. I would recommend against merging (averaging, et
68.
by nestorD 3y ago
In my experience, architecture stays. A nice modular codebase (where you can switch-out the various models and have control on the minutia of the pipeline from data to model) brings a lot of value and can fairly easily be kept up-to-date wi
69.
by nestorD 3y ago
Very much so! Someone clearly did not (or did?) read the book...
70.
by nestorD 3y ago
I believe that this is a non-problem pushed forward by small-scale experiments that are not representative of what people actually do with AI generation. A lot of new content, while AI generated, has been hand picked and polished by a human
71.
by nestorD 3y ago
Exciting! I have been wanting a Rust port of JAX for a long time! (I mostly care about the ability to write numerical GPU code via XLA, not differentiation) Is there a crate in the works? I would start using it (and contributing if needed)
72.
by nestorD 3y ago
MoE let's you use scale model size up with compute. That leads to hopefully more intelligent models. It, however, is independent with context size: the ability to process a lot of tokens / text.
73.
by nestorD 3y ago
It means they have significantly less means (to get a lot of GPUs letting them scale up in context length) and are likely less well-versed in optimization (which also helps with scaling up)[0]. I believe those two things together are likely
74.
by nestorD 3y ago
Regarding the 10M tokens context, RingAttention has been shown [0] recently (by researchers, not ML engineers in a FAANG) to be able to scale to comparable (1M) context sizes (it does take work and a lot of GPUs). [0]: https://
75.
by nestorD 3y ago
In my experience (I gave a demo as part of a talk today! It went great!), you want your code to be reliable (obviously... a 1 in 5 failure rate will bite you back on demo day), require as little thinking as possible (that can be worked arou
76.
by nestorD 3y ago
I can definitely relate to that. I found that using ChatGPT to help me code led me to get working, well documented, code done much faster but it tend to also be somewhat more "naive" code: going for the obvious solutions even if t
77.
by nestorD 3y ago
I remember writing a Mastermind solver in Ocaml, about a year after getting started with programming. The solution space is fairly small so I would start with a random guess then, of all solutions that are still legal, pick the one that wou
78.
by nestorD 3y ago
Part of the reason is that at the scale of a country like the US, if you are going to have several supercomputers build in the coming years, you want to diversify to avoid putting all your eggs in the same basket (and to giving a preferenti
79.
by nestorD 3y ago
I have not! Setting the reference aside to read it later...
80.
by nestorD 3y ago
I find the cover band metaphor to be the best description of what most magicians do. It is absolutely a performance, and the experience cannot be boiled down to the list of notes, but is it art? My take is that it becomes art when the perfo
81.
by nestorD 3y ago
The vast majority of them do tricks designed by other people (passed down from a mentor if you come from a living line of performers, out of books/video that genuinely innovative magicians share with the community, or custom made for a
82.
by nestorD 3y ago
Note that the vast majority of magicians are to magic what cover bands are to music. Only artists in that they are performing in front of an audience, not creators in any meaningful way. (Source: I am a consulting mentalist, I am the kind o
83.
by nestorD 3y ago
It is this exact realization, while looking for some of my old code, that first got me started using git and github.
84.
by nestorD 3y ago
Deemed non-proper in a lot of field and by many advisor (whose approval you need). But, also, technical vocabulary is specific which matter when you operate at the edge of knowledge.
85.
by nestorD 3y ago
Nowadays I would introduce chatgpt as an intermediate layer to use the language. I did some experiments recently with chatgpt4.0 and R and, while not perfect, it did great being able to go from specification to code to interpretation of th
86.
by nestorD 3y ago
Having used JAX quite a bit for numerical computing (and having lectured on this use-case) I would say that a surprisingly large number of algorithms can be expressed as array[0] operations (even if it sometimes takes a bit of thinking). An
87.
by nestorD 3y ago
Spending time on alchemy is, to me, symptomatic of a curious mind that was open to experimentation where his peers might reject things altogether on presuppositions. If he had no been that easily distracted by weird things I expect he would
88.
by nestorD 3y ago
I use it and I am happy with it! The interface is good, the plots provided are useful, I like their transparency on upcoming features, and the fact that the premium version is not needed and mostly a way to support them financially. On the
89.
by nestorD 3y ago
A lot of smart TVs do that. Being able to spot it (played a lot with edge detection when discovering computer vision) is a curse.
90.
by nestorD 3y ago
I believe the author is missing some data sources (bus lines?). There are areas I know are reachable within 30 minutes from downtown Berkeley that do not appear on the map.

More ›