DiederikVink

4 ms·

1. 1.▲
   by DiederikVink 11mo ago

   Super stoked to finally release this on HN. Looking forward to everyone's feedback!
2. 2.▲
   by DiederikVink 2y ago

   You're totally right and we are doing a mixed batch. What we changed was the priority of performing prefills over decodes. When looking at a variety of workloads, we realized that prioritizing finishing a query (priotizing decodes) lea
3. 3.▲
   by DiederikVink 2y ago

   If you deploy our solution on-prem, you would be able to handle 2x the workload on the same amount of hardware. This ensures you scale up your hardware 2x slower, giving you a ~50% reduction in your GPU Infrastructure bills.
4. 4.▲
   by DiederikVink 2y ago

   1. Yes that's correct to some degree. Depending on the model details we might need to do some manual tweaking to get everything up and running, but generally we can get a model up within a day. There's always optimizations and tes
5. 5.▲
   by DiederikVink 2y ago

   Thanks for the feedback and the specific model suggestion! Compared to the replicate/modal solutions our big focus is to ensure you don't experience rate limits. We want to ensure you get a good quality of service no matter what.
6. 6.▲
   by DiederikVink 2y ago

   Thanks for the suggestions! This is definitely something we'll be looking at. We're currently working on providing a more extensive interface to show users a variety of performance metrics of the models they're running. Havin
7. 7.▲
   by DiederikVink 2y ago

   Thanks for letting us know, we've updated it now! Although we're currently only supporting text models, we do definitely have image and video generation models in our roadmap as these are very compute intensive models meaning they
8. 8.▲
   by DiederikVink 2y ago

   That's a great question, but its hard to get enough insight into how Groq is serving models to properly know what's missing. If I had to hazard a guess, it would be that their system architecture (# of chips and chip architecture
9. 9.▲
   by DiederikVink 2y ago

   Hey, that links corresponds to the private models list that only works once you've created an account. If you'd like to see the public models page, please check it out here: https://console.ncompass.tech/public-mod
10. 10.▲
    by DiederikVink 2y ago

    Hey, I'm one of the co-founders, thanks for letting us know! I've just run it a few times and seems fine on my end. It does take a second to load the models, but feel free to let me know if this persists.