3 ms·
To sing the praises of Bedrock again, it does have continuous pre-training as well as RAG “knowledge bases”. The former is based on JSON fragments and the RAG
by benjaminwootton 3y ago
To sing the praises of Bedrock again, it does have continuous pre-training as well as RAG “knowledge bases”. The former is based on JSON fragments and the RAG stuff is PDFs and other document formats.
With regards to its efficacy, I haven’t gone to production with it yet but I was reasonably impressed.
I uploaded 100 legal case documents to Bedrock via Claude and could push it pretty hard asking about the various cases and for situations across the knowledge base.
It did feel like it broke down and got confused at a certain point of complexity of questioning, but I still think it’s already useful as a “copilot” or search engine and surely it will only improve over time.
- ilaksh 3y agoI forgot about the continuous pre-training thing. How long and how much did they cost on Bedrock? I had tried to suggest continuous pre-training to my client but it seemed expensive and when I mentioned that he lost interest and just kept wanting me to do fine tuning. Also to clarify, did you do the continuous pre-training or RAG? And did you compare the efficacy of one or the other or both?
- benjaminwootton 3y agoI used the RAG knowledge bases for most of my testing described above. I got a toy demo up and running with continuous pre-training but haven’t evaluated it unfortunately.
- hbamoria 3y agoOh Great! How did you evaluate the LLM responses? I'm cofounder of an evaluation and monitoring platform - Athina AI (www.athina.ai) You can use our monitoring dashboard and evals to check your LLM performance and iterate quickly.