5 ms·

Is there a limit? Could I create a knowledge base with 10,000 documents? 100k? 1M?

8organicbits 3y ago
https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html https://docs.aws.amazon.com/bedrock/latest/userguide/quotas....
- hrdwdmrbl 3y ago
  I’m sorry, I don’t understand those limits. It uses a lot of unfamiliar terms like “batch inference” and “modality”. I just want a nice UI that I can give my hard-drive to and then ask it questions.
  - deleted 3y ago
    [deleted]
  - nighthawk454 3y ago
    That’s probably unrealistic at this time
    - xuancanh 3y ago
      It's doable with Amazon Q, but it's in Preview phase now. https://aws.amazon.com/q/ https://aws.amazon.com/q/
      - hrdwdmrbl 3y ago
        Is Q the answer to this whole thread? People are talking about AWS Bedrock but that seems like something a startup would build upon and offer something like Q, eh?
        AndrewKemendo 3y ago
        I’m going to try it this week so maybe we have a follow up thread??
  - lopkeny12ko 3y ago
    [flagged]
    - ZephyrBlu 3y ago
      This attitude puzzles me. "If you wish to make an apple pie from scratch, you must first invent the universe" energy. We are not always makers. Oftentimes we're consumers as well. I don't want to read documentation and experiment with my phone, I just want it to work out of the box and do what I expect. This is standard consumer behaviour and you're lying to yourself if you don't think you act like this with some things.
benjaminwootton 3y ago
The documents are encoded as vectors and stored in a database, so I suspect it would be effectively unlimited. You would just pay for storage and compute. AWS OpenSearch has fairly good integration so you could look up costs for that. It’s not the cheapest AWS service to run and not exactly serverless as you pay by the hour.
nashadelic 3y ago
Even if you could, the problem is that these documents are first chunked into smaller pages, and then embeddings are created. When you ask a question, the algo searches for relevant chunks and passes them to the LLM's overall prompt. If there are too many chunks, or too many chunks with similar content, the search also coupled with the LLM's limited context window would mean only 1-3 chunks get passed. This isn't the same as the training data the LLM is trained on. As a result, it doesn't take advantage of the entire document set. So, if you have a billion documents, only 1-3 chunks will be picked for the final answer. When you know a question spans many many documents, the answer is never going to cover that. You could make a recursive also where you parse all the chunks, generate summaries of those and then pass them to the next chunk sequentially and so on. But you can imagine how expensive and slow that will be. It might still work for you but this is a very lossy approach.
- fennecbutt 3y ago
  Easy: if > n chunks returned query "too much information, ask user to clarify". If you have that much data in your vdb, and the user is querying a slice, it just won't do to have them ask something too generic, they need to be asked to be specific. Ie "what was that thing from last year?" "Sorry, could you be more specific?" "Oh uh the thing to do with the financial reports" "Financial reports, yes, what about them?" "It was something to do with the way we generate them" "Ah yes, the generation of financial reports; last year in December your team leader said on Slack...etc"

https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html https://docs.aws.amazon.com/bedrock/latest/userguide/quotas....

hrdwdmrbl 3y ago
I’m sorry, I don’t understand those limits. It uses a lot of unfamiliar terms like “batch inference” and “modality”. I just want a nice UI that I can give my hard-drive to and then ask it questions.
- deleted 3y ago
  [deleted]
- nighthawk454 3y ago
  That’s probably unrealistic at this time
  - xuancanh 3y ago
    It's doable with Amazon Q, but it's in Preview phase now. https://aws.amazon.com/q/ https://aws.amazon.com/q/
    - hrdwdmrbl 3y ago
      Is Q the answer to this whole thread? People are talking about AWS Bedrock but that seems like something a startup would build upon and offer something like Q, eh?
      - AndrewKemendo 3y ago
        I’m going to try it this week so maybe we have a follow up thread??
- lopkeny12ko 3y ago
  [flagged]
  - ZephyrBlu 3y ago
    This attitude puzzles me. "If you wish to make an apple pie from scratch, you must first invent the universe" energy. We are not always makers. Oftentimes we're consumers as well. I don't want to read documentation and experiment with my phone, I just want it to work out of the box and do what I expect. This is standard consumer behaviour and you're lying to yourself if you don't think you act like this with some things.

benjaminwootton 3y ago

The documents are encoded as vectors and stored in a database, so I suspect it would be effectively unlimited. You would just pay for storage and compute. AWS OpenSearch has fairly good integration so you could look up costs for that. It’s not the cheapest AWS service to run and not exactly serverless as you pay by the hour.

nashadelic 3y ago

Even if you could, the problem is that these documents are first chunked into smaller pages, and then embeddings are created. When you ask a question, the algo searches for relevant chunks and passes them to the LLM's overall prompt. If there are too many chunks, or too many chunks with similar content, the search also coupled with the LLM's limited context window would mean only 1-3 chunks get passed. This isn't the same as the training data the LLM is trained on. As a result, it doesn't take advantage of the entire document set. So, if you have a billion documents, only 1-3 chunks will be picked for the final answer. When you know a question spans many many documents, the answer is never going to cover that. You could make a recursive also where you parse all the chunks, generate summaries of those and then pass them to the next chunk sequentially and so on. But you can imagine how expensive and slow that will be. It might still work for you but this is a very lossy approach.

fennecbutt 3y ago
Easy: if > n chunks returned query "too much information, ask user to clarify". If you have that much data in your vdb, and the user is querying a slice, it just won't do to have them ask something too generic, they need to be asked to be specific. Ie "what was that thing from last year?" "Sorry, could you be more specific?" "Oh uh the thing to do with the financial reports" "Financial reports, yes, what about them?" "It was something to do with the way we generate them" "Ah yes, the generation of financial reports; last year in December your team leader said on Slack...etc"