2 ms·

Some small tips from superficially reading the code: https://github.com/arunsupe/semantic-grep/blob/b7dcc82a7cbab4fc4fdc1d4ef86092a071e48455/sgrep.go#L96 https

Some small tips from superficially reading the code: https://github.com/arunsupe/semantic-grep/blob/b7dcc82a7cbab4fc4fdc1d4ef86092a071e48455/sgrep.go#L96 https://github.com/arunsupe/semantic-grep/blob/b7dcc82a7cbab... You can read the vector all at once. See e.g.: https://github.com/danieldk/go2vec/blob/ee0e8720a8f518315f35322c119bec396c912bd0/go2vec.go#L113 https://github.com/danieldk/go2vec/blob/ee0e8720a8f518315f35... --- https://github.com/arunsupe/semantic-grep/blob/b7dcc82a7cbab4fc4fdc1d4ef86092a071e48455/sgrep.go#L115 https://github.com/arunsupe/semantic-grep/blob/b7dcc82a7cbab... You can compute the similarity much faster by using BLAS. Good BLAS libraries have SIMD-optimized implementations. Or if you do multiple tokens as once, you can do a matrix-vector multiplication (sgemv), which will be even faster in many implementations. Alternatively, there is probably also a SIMD implementation in Go using assembly (it has been 7 years since I looked at anything in the Go ecosystem). You could also normalize the vectors while loading. Then during runtime the cosine similarity is just the dot product of the vectors (whether it pays off depends on the size of your embedding matrix and the size of the haystack that you are going to search).

neonsunset 2y ago
SIMD situation in Go is still rather abysmal, it’s likely easier to just FFI (though FFI is very slow so I guess you're stuck with ugly go asm if you are using short vectors). As usual, there's a particular high-level language that does it very well, and has standard vector similarity function nowadays...
- bitexploder 2y ago
  Go ahead and share the language, it is good etiquette for HN :)
  - neonsunset 2y ago
    Actually, there's three: C#: https://github.com/dotnet/runtime/blob/main/docs/coding-guidelines/vectorization-guidelines.md https://github.com/dotnet/runtime/blob/main/docs/coding-guid... (and full family of other types: Vector2/3/4, Matrix3x2/4x4 and upcoming Tensor<T>), vector similarity I was talking about is this: https://learn.microsoft.com/en-us/dotnet/api/system.numerics.tensors.tensorprimitives.cosinesimilarity https://learn.microsoft.com/en-us/dotnet/api/system.numerics..., it uses a highly optimized SIMD kernel, for DotProduct just use an adjacent method Swift: https://developer.apple.com/documentation/swift/simd-vector-types https://developer.apple.com/documentation/swift/simd-vector-... it is also a competent language at portable SIMD by virtue of using LLVM and offering almost the same operators-based API (e.g. masked = vec1 & ~vec2) like C# Mojo: https://docs.modular.com/mojo/stdlib/builtin/simd https://docs.modular.com/mojo/stdlib/builtin/simd which follows the above two, it too targets LLVM so expect good SIMD codegen as long the lowering strategy does it in an LLVM-friendly way, which I have not looked at yet.
    - e12e 2y ago
      Not Julia?
      - neonsunset 2y ago
        Julia is less "general-purpose" and I know little about the quality of its codegen (it does target LLVM but I haven't seen numbers that place it exactly next to C or C++ which is the case with C#). Mojo team's blog posts do indicate they care about optimal compiler output, and it seems to have ambitions for a wider domain of application which is why it is mentioned.

neonsunset 2y ago

SIMD situation in Go is still rather abysmal, it’s likely easier to just FFI (though FFI is very slow so I guess you're stuck with ugly go asm if you are using short vectors). As usual, there's a particular high-level language that does it very well, and has standard vector similarity function nowadays...

bitexploder 2y ago
Go ahead and share the language, it is good etiquette for HN :)
- neonsunset 2y ago
  Actually, there's three: C#: https://github.com/dotnet/runtime/blob/main/docs/coding-guidelines/vectorization-guidelines.md https://github.com/dotnet/runtime/blob/main/docs/coding-guid... (and full family of other types: Vector2/3/4, Matrix3x2/4x4 and upcoming Tensor<T>), vector similarity I was talking about is this: https://learn.microsoft.com/en-us/dotnet/api/system.numerics.tensors.tensorprimitives.cosinesimilarity https://learn.microsoft.com/en-us/dotnet/api/system.numerics..., it uses a highly optimized SIMD kernel, for DotProduct just use an adjacent method Swift: https://developer.apple.com/documentation/swift/simd-vector-types https://developer.apple.com/documentation/swift/simd-vector-... it is also a competent language at portable SIMD by virtue of using LLVM and offering almost the same operators-based API (e.g. masked = vec1 & ~vec2) like C# Mojo: https://docs.modular.com/mojo/stdlib/builtin/simd https://docs.modular.com/mojo/stdlib/builtin/simd which follows the above two, it too targets LLVM so expect good SIMD codegen as long the lowering strategy does it in an LLVM-friendly way, which I have not looked at yet.
  - e12e 2y ago
    Not Julia?
    - neonsunset 2y ago
      Julia is less "general-purpose" and I know little about the quality of its codegen (it does target LLVM but I haven't seen numbers that place it exactly next to C or C++ which is the case with C#). Mojo team's blog posts do indicate they care about optimal compiler output, and it seems to have ambitions for a wider domain of application which is why it is mentioned.