3 ms·

I am getting quite deep into coding with AI and cost of tokens is a bit of an issue indeed. Trivial issue because it saves me A LOT of time, but it could be an

by siscia 2y ago

I am getting quite deep into coding with AI and cost of tokens is a bit of an issue indeed. Trivial issue because it saves me A LOT of time, but it could be an issue for new people testing it. I would love to test this approach. Are you guys fine tuning for each codebase?

manishsharan 2y ago
>>cost of tokens is a bit of an issue indeed Their cost is $0.7 per 1M token. DeepSeek is $0.14 / 1M tokens ( cache miss)
- siscia 2y ago
  DeepSeek is an amazing product but has few issues: 1. Data is used for training 2. Context window is rather small and doesn't fit as well large codebase I keep saying this over and over in all the content I create, the valu of coding with AI will come from working on big, complex, legacy codebases. Not from flashy demo where you create a to-do app. For that you need solid models with big context and private inference.
  - MacsHeadroom 2y ago
    DeepSeek is open source and has a context length of 128k tokens.
    - siscia 2y ago
      Commercial service have a context of 64k tokens, which I find quite limiting. https://api-docs.deepseek.com/quick_start/pricing https://api-docs.deepseek.com/quick_start/pricing Running it locally is quite a bit beyond the scope of being productive while coding with AI. Beside that 128k is still significantly less than Claude
      - elashri 2y ago
        Shouldn't we be comparing with other open source model? In particular since this is about llama3.3 then they have the exact context limit which is 128k [1]. Also [1] https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct
        siscia 2y ago
        Why? Whenever using a model to be more effective as a developer I don't particularly care if the model is open source or closed source. I would love to use open source models as well, but the convenience to just plug an API against some endpoints in unbeatable.
samatdav 2y ago
Yes, we fine-tune for each codebase. Now we are focusing on larger enterprise codebases that would: 1. benefit from the fine-tuning the most. 2. have the budget to pay us for the service. For smaller projects that are price-sensitive we are probably not a good fit at this point.

by siscia 2y ago

manishsharan 2y ago

>>cost of tokens is a bit of an issue indeed Their cost is $0.7 per 1M token. DeepSeek is $0.14 / 1M tokens ( cache miss)

siscia 2y ago
DeepSeek is an amazing product but has few issues: 1. Data is used for training 2. Context window is rather small and doesn't fit as well large codebase I keep saying this over and over in all the content I create, the valu of coding with AI will come from working on big, complex, legacy codebases. Not from flashy demo where you create a to-do app. For that you need solid models with big context and private inference.
- MacsHeadroom 2y ago
  DeepSeek is open source and has a context length of 128k tokens.
  - siscia 2y ago
    Commercial service have a context of 64k tokens, which I find quite limiting. https://api-docs.deepseek.com/quick_start/pricing https://api-docs.deepseek.com/quick_start/pricing Running it locally is quite a bit beyond the scope of being productive while coding with AI. Beside that 128k is still significantly less than Claude
    - elashri 2y ago
      Shouldn't we be comparing with other open source model? In particular since this is about llama3.3 then they have the exact context limit which is 128k [1]. Also [1] https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct
      - siscia 2y ago
        Why? Whenever using a model to be more effective as a developer I don't particularly care if the model is open source or closed source. I would love to use open source models as well, but the convenience to just plug an API against some endpoints in unbeatable.

samatdav 2y ago

Yes, we fine-tune for each codebase. Now we are focusing on larger enterprise codebases that would: 1. benefit from the fine-tuning the most. 2. have the budget to pay us for the service. For smaller projects that are price-sensitive we are probably not a good fit at this point.