5 ms·
tinfoil doesn't seem to distinguish between cached and input tokens. I think that would make it untenably expensive
by itemize123 4mo ago
tinfoil doesn't seem to distinguish between cached and input tokens. I think that would make it untenably expensive
- traceroute66 4mo ago> tinfoil doesn't seem to distinguish between cached and input tokens. Correct. Privatemode correctly distinguish[1][2]. But with Tinfoil it appears to be a fundamental architectural limitation that they cannot.[3] [1] https://www.privatemode.ai/pricing https://www.privatemode.ai/pricing [2] https://www.privatemode.ai/blog/secure-prompt-caching https://www.privatemode.ai/blog/secure-prompt-caching [3] https://docs.tinfoil.sh/resources/caching https://docs.tinfoil.sh/resources/caching
- 3s 4mo agoHi, cofounder of Tinfoil here. We're working on implementing prompt caching. It just requires more work than with traditional, non-private inference and requires some careful considerations to actually implement it.