shihab

12 ms·

61.
by shihab 2y ago
The speedup figures they report are compared to their own cutlass-based baseline. Has anyone done a performance comparison against cuBLAS? All cutlass results I have seen so far for Gemm are within ~10% of cuBLAS. If the 2x-2.5x speedup the
62.
by shihab 2y ago
For future readers, note that those 3x and 10x figures are compared to vLLM's own previous release, and NOT compared to Deepseek's implementation. I am very curious to see how well-optimized Deepseek's code is compared to lea
63.
by shihab 2y ago
The biggest victims of these non-scalable process is people without a good network. As an intl PhD student, I am that person. So now I have this weird dynamic: I get interview calls only from FAANG companies, the ones with the manpower to d
64.
by shihab 2y ago
I recently saw Sam Altman bragging about OpenAI's performance on Codeforces (leetcode-like website), which I consider just about the worst benchmark possible. 1. All problems are small- the prompt and solution (<100 LOC, often <6
65.
by shihab 2y ago
From the creator of Triton (OpenAI)- "PS: The name Triton was coined in mid-2019 when I released my PhD paper on the subject. I chose not to rename the project when the "TensorRT Inference Server" was rebranded as "Trito
66.
by shihab 2y ago
I understand this sentiment for experienced developers. It is an imperfect signal. But what is in your opinion a better signal for junior or new grads? Every alternative I can think of is either worse, or sounds nice but impractical to impl
67.
by shihab 2y ago
To get an idea of just how advanced cheating tools has become, take a look here: https://leetcodewizard.io/ I think every interviewer, hiring manager ought to know or be trained on these tools, your intuition about candidat
68.
by shihab 2y ago
yeah, that's very likely the explanation. All these functions are pretty high latency instructions, vs rejection sampling which only involves a multiplication. On Nvidia GPUs, mul has latency of 1-4 cycles while others are 16-32.
69.
by shihab 2y ago
On GPU, the rejection sampling approach wouldn't come close to analytical one.
70.
by shihab 2y ago
I had the impression that gpu isn’t a good fit for ultra low latency usecases. Can you please elaborate on what sort of work hft firms do with gpu?
71.
by shihab 2y ago
That doesn't necessarily mean final weights are 8-bit though. Tensor core ops are usually mixed precision- matmul happens in low precision but accumulation (i.e. final result) is done in much higher precision to reduce error. from deep
72.
by shihab 2y ago
Please note that it’s using pretty aggressive quantization (around 4 bits per weight)
73.
Google "We have no moat, and neither does OpenAI" (2023)(semianalysis.com)
95 points by shihab 2y ago|53 comments
74.
by shihab 2y ago
Here [1] is the leaderboard from chabot arena, where users vote on the output of two anonymous models. Deepseek R1 needs more data points- but it already climbed to No 1 with Style control ranking, which is pretty impressive. Link [2] to th
75.
by shihab 2y ago
I have been using it to implement some papers from a scientific domain I'm not expert in- I'd say there were around same in output quality, with R1 having a slight advantage for exposing it's thought process, which has been r
76.
by shihab 2y ago
Yes, polls are an imperfect tool. But I think they remain the only tool we have to gauge what decisions coming out of Washington are product of broad popular support vs ones product of intense lobbying from shadowy powers.
77.
by shihab 2y ago
I helped develop a hydro solver coupled with radiation at Los Alamos this summer. We observed from 7x upto 15x performance improvement on a single GPU compared to 64-core intel CPU. [1] Now granted, the flops to byte ratio for this program
78.
by shihab 2y ago
Is anyone aware of any opinion poll among US population about banning tiktok? This to me feels like one of the issues with potentially largest disconnect between voters and politicians Edit: found one from Pew. "The share of Americans
79.
by shihab 2y ago
That's the smallest of 4 experiments. It goes upto 140 million cells, where MI300X retains similar performance advantage of around 10% over Nvidia's H100.
80.
by shihab 2y ago
I was looking for a simple table recently- outlining say how the shared memory or total register size/SM varies between generations (Something like that Wiki table). It was surprisingly hard to find those info.
81.
by shihab 2y ago
I think that example says more about the company that chose to put that code as a demo in their homepage .
82.
by shihab 2y ago
First part is true here in Michigan too. Wish they stayed open late.
83.
by shihab 2y ago
CS Phd student here: took an online assesment recently (for internship role). 3 questions in total. None of them were puzzle-type question, or required advanced data structure or algorithm knowledge, but they were more of implementation foc
84.
by shihab 2y ago
The last OpenAI valuation I read about was 157 billion. I am struggling to understand what justifies this. To me, it feels like OpenAI is at best few months ahead of competitors in some areas. But even if I am underestimating the advantag
85.
by shihab 2y ago
I have come across quite few startups who are trying a similar idea: break the nvidia monopoly by utilizing AMD GPUs (for inference at least): Felafax, Lamini, tensorwave (partially), SlashML. Even saw optimistic claims like CUDA moat is on
86.
by shihab 2y ago
Excellent, amazing article. To the author, if you're lurking here, I have a tangential question- how long did it take you to write this article? From first line of code to the last line of this post? As someone who works in GPGPU space
87.
by shihab 2y ago
An otherwise valid point made using a terrible example.
88.
by shihab 2y ago
The biggest condition behind US aid to Jordan and Egypt is them continuing friendly relations with Israel. In 1970s when this aid was started- this condition was made very explicit by USA. So in other words, these two at least are nothing b
89.
by shihab 2y ago
EU foreign policy chief said the court's decision should be implemented. Ireland also indicated they would comply with the warrant.
90.
by shihab 2y ago
> Hamas is a terrorist group that was elected by Gaza’s residents. "Prime Minister Benjamin Netanyahu gambled that a strong Hamas (but not too strong) would keep the peace and reduce pressure for a Palestinian state." - From &q

More ›