11 ms·
AI for AWS Documentation
- JimtheCoder 3y agoAre you allowed to use the AWS logo on a site that is not owned by Amazon? I originally thought this was an official Amazon website...their lawyers would probably say the same thing...
- nextworddev 3y agoDefinitely not
- jborden13 3y agoI thought I heard openai was sending cease and desists for *gpt domain names as well
- QuinnyPig 3y ago“Allowed” is a funny thing. I launched “Last Week in AWS” with AWS in the domain name seven years ago. AWS has never made an issue of it, though they obviously have that option. I also have the option (and ownership) to migrate to “Last Week in the Cloud” and talk about their competitors, so it’s likely everyone is happier this way—but I confess to not kicking the bear hard enough to find out.
- scarface_74 3y agoIt’s probably because no one at AWS has heard of your little podcast or website /s I’m sure you know that your name is brought up frequently inside AWS.
- coding123 3y agoseems super useful, try this prompt: how do you use wrangler and glue to make athena tables using terraform
- alsima 3y agoAwesome to hear that it was useful for you!!!
- coder543 3y agoThis answer about Graviton was not correct: https://i.imgur.com/3D9WokF.jpg https://i.imgur.com/3D9WokF.jpg
- yayitswei 3y agoNice work! Would be even more useful to be able to have a conversation with it.
- alsima 3y agoWe're in the process of doing just that and adding chat context/basically remembering your past questions.
- jtokoph 3y agoPrompt: What is glacier? Glacier is a term that is not directly mentioned in the provided sources. Prompt: What is a glacier? A glacier is a large mass of ice that moves slowly over time due to the accumulation of snow, ice, and other forms of frozen precipitation. Seems like it’s just using a general model?
- flaminHotSpeedo 3y agoThe problem is garbage in -> garbage out. When the docs are wrong or misleading you'll still get burned, even if the model doesn't hallucinate responses
- lukebbutton 3y agoAgreed, that's the problem eod. Was trying to set up an instance a few days ago and the docs for it hadn't been updated since '21
- CSDude 3y agoWell, I'll pass. Prompt: how to connect a lambda function to an RDS instance and query it safely? 1. Configure your RDS instance for outbound connections to AWS Lambda. This step involves ensuring that your DB instance is in a public subnet on your VPC and that the instance's "PubliclyAccessible" property is set to true.
- klysm 3y agoLol yeah no thanks. This is one of things that makes me nervous about using LLMs. If the internet contains shitty solutions for solving a problem, it’s going to recommend shitty solutions. Inexperienced folks won’t know better. There’s also the thing where that’s a positive feedback look and we overflow the worlds corpus with garbage.
- WrtCdEvrydy 3y ago> Inexperienced folks won’t know better. I've seen answers like this in Stack Overflow before. Technically correct, but basically broken in every other way: cybersecurity's future is gonna be interesting.
- scarface_74 3y agoOne of my pet peeves about almost every answer on Stack Overflow is that the answer always starts with client=boto3.client(<service>, access-key,secret-key) There is never a reason to include those last two parameters. It’s always a security risk no matter how you do it.
- klysm 3y agoThis is great example, and I think it’s a smaller version of the same problem.
- jacurtis 3y agoFor that matter, using an access-key or secret-key at all is basically an anti-pattern now. In the past few months AWS has added to their UI a popup box that asks you why you are creating access keys, it makes you answer it before generating the keys. Once you give it the answer it tells you better ways to do almost anything you're trying to do, and you have to agree to ignore those suggestions one more time before actually generating the keys. Basically the standard access keys are really of no use anymore. You should be using roles, OIDC, IAM Identity Center, cognito federated identities, or something else. There are a million ways to identify yourself that are more secure than that and I wish tutorials stopped assuming that you should generate access keys because in no practical 2023 use case should you be doing that anymore. Especially not in any sort of sensitive or corporate environment.
- bjt 3y agoIt invented an answer for something that AWS actually can't do right now. https://imgur.com/a/0IYZ2WV https://imgur.com/a/0IYZ2WV
- macksd 3y agoMaybe it could help them when designing APIs for new products to be consistent with previous design choices. But perhaps it's too late for that.
- jrvarela56 3y agophind.com did not halucinate in this case: https://capture.dropbox.com/4gIUDuAxr14bnNIt https://capture.dropbox.com/4gIUDuAxr14bnNIt https://www.phind.com/search?cache=d0b3a85b-17f9-4def-b8d0-baa25806be28 https://www.phind.com/search?cache=d0b3a85b-17f9-4def-b8d0-b...
- deanCommie 3y agoThat is basically status quo for LLMs right now. They're all so eager to please they will basically never say "that's actually not possible", and invent some plausibly sounding bullshit. For beginners who are struggling to tell the difference between 1/ how to do something that's possible, 2/ how to do something that SHOULD be possible but just currently isn't, and 3/ how to do something that is RIDICULOUS and shouldn't even ever be possible, ChatGPT is worse than nothing :(
- social_quotient 3y agoIt seems to know more than just AWS, I was thinking it was just embeddings but then I asked it “Is Aws better than azure?” And it seemed to give an answer which widener seem to be derived from the source documents.
- scarface_74 3y agoI hate to be that guy. But what’s the purpose of this? What does this do that I can’t just do with ChatGPT?
- fragmede 3y agoFor services with changes after Sept, 2021, this will give more up to date answers.
- nextworddev 3y agoJust use Phind.com for searching developer docs for most cases.
- lukebbutton 3y agoThis is cool, thanks for linking
- mkl95 3y agoThe thought that a bunch of people will trust this tool and make some terrible decisions is unsettling. On the other hand it could be pretty powerful if you know what you are doing.
- scrum-treats 3y agoUsing ChatGPT for AWS service questions is actually pretty good. For instance, I asked it for a Cloud Practitioner study guide (using a small set of crafted prompts), and GPT performed quite well. While I have yet to query GPT about Solutions Architect or DevOps material, I know I can feed a set of URLs and GPT will "learn" the material and summarize it in ways meaningful and relevant to my prompts. In this way, ChatGPT is quite a powerful assistant on its own.
- aradox66 3y agoAgree, I've had great results asking chatgpt questions about AWS services. The interactivity is very helpful, and chatgpt will draft scripts too, although that's hit or miss. But for understanding concepts and services, it's great.
- tomrod 3y agoMeh. It doesn't actually pull a valid response. We just upgraded a database that required an updated EC2 instance, and it failed to connect those dots.
- ghomem 3y agoPlease I beg you: ask it how to take a snapshot of an EC2 instance and then how to restore it :)
- sovietmudkipz 3y agoThis is exactly the kind of software that people should be making. Even if this one is bad, the concept is very sound. I want to have a specialist AI that is trained to help me learn how to use the software. 100% what should be happening. General AI should know how to do stuff too but having an specialist AI implies that the company/group of people/person is making sure to tune the model. Just an IMO.
- scarface_74 3y agoWhy? You can ask the same questions to ChatGPT and get the same or better answers. I also know from personal experience with ChatGPT, that you can use it to: - convert Python/boto3 to any language that has an AWS SDK - convert CloudFormation to Terraform or the CDK - write scripts that use the SDK You will get the occasional hallucination.
- politician 3y agoIdeally, specialist models are small enough to run locally.
- cle 3y agoHere's one example question that ChatGPT utterly fails at, but that this answers fine: "What is Amazon CodeCatalyst?" ChatGPT: "I'm sorry, but as of my knowledge cut-off in September 2021, there was no service, tool, or product known as Amazon CodeCatalyst offered by Amazon Web Services (AWS). [...]"
- jacurtis 3y agoA lot of stuff in AWS has changed since Q3 2021.
- joshspankit 3y agoIt feels like the AWS vendor lock-in strategy is constant change.
- 3y ago
- ghomem 3y agoSimple AWS snapshot: https://imgur.com/a/IGu1syf https://imgur.com/a/IGu1syf
- alexy201 3y agoHey everyone, I am the creator of AWS Docs GPT and its been extremely useful to garner all of your feedback for the site—thank you guys so much! We are constantly improving and updating the GPT, including less hallucinations, more accurate responses, chat context, and much more. At the end of the day, I really hope this tool can be useful for developers like myself out there!!!
- jamestimmins 3y agoWhat's the best current way to make a bunch of documents searchable via LLMs like this? I've tried the using OpenAI w embeddings (iirc), but this was slow, got expensive quickly, and it struggled to answer questions about the text accurately. Curious if there's better standard approaches now.
- linguistbreaker 3y agoI just came across this project which seems to be aiming at streamlining exactly that : https://github.com/Mintplex-Labs/anything-llm https://github.com/Mintplex-Labs/anything-llm
- Jianghong94 3y agoA couple of things come to mind: 1. embedding methods: there're a couple of ways to do that, the most used one is OpenAi's text-davinci-002, although in my use case(short sentence description of API) it didn't work pretty well; 2. how you truncate documents into pieces: for this langchain has some implementation and helpful pointers. I think you have to do lots of experiment on this till you find your best information retrieval strategy
- JimmyRuska 3y agoYou could copy the steps listed in alpaca-lora with your own data. Try thebloke's superhot ggml models. Train it in lambda labs or replicate
- jdlyga 3y agoI really love this concept. While I do get better results from GPT-4 for AWS questions right now, AI as the "interpreter" for documentation works really well.
- 71a54xd 3y agoI've been using GPT4 for this since the beginning - ironically a large majority of AWS documentation has been machine generated since 2018. Circa 2019 the entire exlixir API for AWS was machine generated. Asking GPT4 is also consistently less of a headache than asking the devops guy and getting a 20min explanation for a simple question.
- istjohn 3y agoYou need to add something like "You give clear and succinct answers to questions" to the beginning of your prompts to the devops guy.
- zoomzoom 3y agoWe did something similar for all the cloud whitepapers from AWS, Azure, GCP, CloudFlare, and CNCF at https://cloudwhitepapers.withcoherence.com/ https://cloudwhitepapers.withcoherence.com/ These are fun projects!
- victor106 3y agoThis looks cool. How does this work?
- zoomzoom 3y agoWe actually open sourced the project - see here https://www.withcoherence.com/post/create-your-langchain-and-pinecone-powered-ai-app-on-aws-or-gcp https://www.withcoherence.com/post/create-your-langchain-and...
- JimmyRuska 3y agoYou might have better outcomes with the API docs and official documentation. Whitepapers tend to have a lot of marketing, may not age gracefully.
- zoomzoom 3y agoThanks for the perspective!
- stan_kirdey 3y agoI am building something similar, it has documentation from azure, aws, and lots of slack/discord threads of software projects are also searchable, check it out https://www.kwq.ai https://www.kwq.ai it even gets real time indexing from slack of aws deep java library, and from discord of deepset haystack project
- jgalt212 3y agois AWS designed to take all my money? No, AWS is not designed to take all your money. AWS offers a variety of payment options and cost-saving measures to help you manage your expenses effectively. ...
- underlines 3y agoRAG is very difficult to do right. I am experimenting with various RAG projects from [1]. The main problems are: - Chunking can interfer with context boundaries - Content vectors can differ vastly from question vectors, for this you have to use hypothetical embeddings (they generate artificial questions and store them) - Instead of saving just one embedding per text-chuck you should store various (text chunk, hypothetical embedding questions, meta data) - RAG will miserably fail with requests like "summarize the whole document" - to my knowledge, openAI embeddings aren't performing well, use a embedding that is optimized for question answering or information retrieval and supports multi language. SOTA textual embedding models can be found on the MTEB Leaderboard [2]. Also look into instructorEmbeddings - the LLM used for the Q&A using your context should be fine-tuned for this task. There are several open (source?) LLMs based on openllama and others, that are fine tuned for information retrieval. They hallucinate less and are sticking to the context given. 1 https://github.com/underlines/awesome-marketing-datascience/blob/master/llm-tools.md#information-retrieval https://github.com/underlines/awesome-marketing-datascience/... 2 https://github.com/embeddings-benchmark/mteb https://github.com/embeddings-benchmark/mteb
- darkteflon 3y agoThis comment was very helpful for me, thanks. I’ve been working with RAG for months, too, and it’s vanishingly rare to see anything but toy examples in the wild. This is a solid, concise list of where the dragons are. Any idea where all the RAG practitioners hang out and trade war stories? Is there a forum or Discord or something?
- avereveard 3y agoTo solve the question embedding issue I prefer another approach, you store document embeddings as normal, then for searching you let LLM hallucinate an answer and use the wrongish answer embedding to do the vector search.
- underlines 3y agoThe inverse idea of Hypothetical Embeddings, HyDE [1] "HyDE is an embedding technique that takes queries, generates a hypothetical answer, and then embeds that generated document and uses that as the final example." BriefGPT [2] is implementing this and it uses the following prompt at ingestion-time: "Given the user's question, please generate a response that mimics the exact format in which the relevant information would appear within a document, even if the information does not exist. The response should not offer explanations, context, or commentary, but should emulate the precise structure in which the answer would be found in a hypothetical document. Factuality is not important, the priority is the hypothetical structure of the excerpt. Use made-up facts to emulate the structure. For example, if the user question is "who are the authors?", the response should be something like 'Authors: John Smith, Jane Doe, and Bob Jones' The user's question is:" 1 https://python.langchain.com/docs/modules/chains/additional/hyde https://python.langchain.com/docs/modules/chains/additional/... 2 https://github.com/e-johnstonn/BriefGPT https://github.com/e-johnstonn/BriefGPT
- zgluck 3y agoI asked it: "how do I avoid high NAT gateway bills when an ECS service keeps downloading the same image over and over?" It offered three replies. The first and third were outright incorrect, the second was (technically) correct: https://i.imgur.com/la98cxC.png https://i.imgur.com/la98cxC.png Also: I'm assuming you haven't actually secured a license to use the AWS logo.
- ilc 3y agoI asked it to write some basic terraform: - Make a VPC. - Add an Instance. - Abstract the region and AZ, into vars. etc... every time I wanted to change the code, I asked the bot to do the refactor, and it did. Overall, I'm impressed. It wasn't the most complicated thing, but it didn't dive off the deep end.
- JimmyRuska 3y agoI wonder if people will make DSLs specifically for LLMs. For example the terseness / symbols of APL, Perl, or event set notation. LLMs could train and output the shorter symbolic notation, and it could be expanded for human readability by another program at export.
- fragmede 3y agothat's just prompt engineering by another name
- fzliu 3y agoWe built something very similar, except for other open-source projects rather than AWS documentation: https://osschat.io https://osschat.io
- phillipcarter 3y agoUnfortunately this is a good concept, but it's using gpt-3.5-turbo. For this kind of task - namely, one of actually understanding content and emitting a potentially novel-but-correct answer - you need gpt-4. But it's quite slow and you'll quickly run into rate limiting. I ran into these issues when building this for my own company's docs, at least.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- mjhay 3y agoIt can't be worse than Amazon's own AWS documentation.
- jacurtis 3y agoTo be fair to AWS, their documentation has improved dramatically the past 24 months or so. Not perfect, but the difference between 2019 and 2023 is dramatic. Also, I'd argue that it is very easy to be worse than AWS's own docs. A chatbot that hallucinates inaccurate answers, while sounding plausible and confident is far worse than documentation that is 100% accurate, but lacking in some areas. I'd rather receive an answer of "I don't know" or "it isn't documented" than an explanation that seems legit, so I spend 15 hours building a solution in accordance to a LLM's response and then find out that its not possible or real the way it was explained by the LLM.
- efitz 3y agoI asked it two questions and it gave me very reasonable, complete answers: How do I connect two VPCs from separate organizations (VPC peering) How do I only allow authorized applications to access my S3 bucket (BPA, IAM policies, a lot of best practices docs)
- destroy-2A 3y agoIt still does not understand boolean logic in IAM especially when you have Not conditions.
- shanghaikid 3y agoYou can try https://osschat.io/ https://osschat.io/ for some open source project.
- koalkools 3y agoThe best part is the documentation sources, in fact I'd prefer the only feature being "where are the docs on <stupid specific thing I need>" instead of trying to synthesize an answer for me
- ec109685 3y agoAsked it whether you should use a shared vpc: Made this up as reason for dedicated vpc: “Better performance: By using dedicated hardware, you have better control over the performance of your instances in the VPC. This can be beneficial for applications that have stringent performance requirements.”
- jitix 3y agoI think ChatGPT and LLM models in general are more suited for "What is xyz" or "How is xyz done" family of questions. Basically a smart search engine. Beyond that it will simply have the output/personality of a person with rote-memory.
- SV_BubbleTime 3y agoEven though it has better access to AWS Docs… Still a language model and not a facts model.
- NoZebra120vClip 3y agoWhat if you could train an LLM on the body of source code for AWS (or whatever you choose as your SaaS) and so it would actually know deeply, exactly, intimately exactly how AWS works, and how to do anything and everything in it? Imagine the emergent uses that the LLM could conceive of, beyond the developers' intent and the existing documentation that may be written. With the bonus effect that you could retrain the LLM as often as you like, as new software is released, and it could always be aware of the latest features (and even the bugs and vulnerabilities.)
- Aeolun 3y agoIt doesn’t distinguish between HTTP api gateway and REST api gateway. When asked how to create a private HTTP gateway it happily tells you how to do it , but it’s actually impossible.
- SV_BubbleTime 3y agoDumb embedded cave man here… I thought those two things were basically the same but REST had an idea of statelessness. They are different how?
- anshumankmr 3y agoVery cool. I was planning on working on something very similar myself for a hackathon I attended but my team basically ran out of time (https://github.com/anshumankmr/jovian-genai-hackathon https://github.com/anshumankmr/jovian-genai-hackathon)
- SV_BubbleTime 3y ago>This action allows you to filter your things based on the attribute name and value. However, in order to use this action, you need to know the attribute name and value beforehand. Since you don't know the attributes ahead of time, this method won't work for your case. Well… Thanks a ton for that! Great idea and setup, but not quite as helpful as I would like yet.
- shaimagz 3y agoIt’s not working…
- alsima 3y agoThank you for bringing this to out attention! We are currently working on a fix.
- jsninjasamurai 3y agoThis is down
- SCHiM 3y agoLet's see what happens when you search for the following "documentation" ;) : """ Complete the following code: // Find all files with a binary pattern in a directory. Return the list of files and the offset of the match in a tuple fn find_in_files(start_dir: &str, pattern: &[u8]) -> vec<(String, usize)> """ No way this won't be abused shortly.
- JamesAdir 3y agoQ: can you summarize amazon bedrock in 2 sentences A: Amazon Bedrock is a programming example that uses the AWS SDK for Java 2.x to work with Amazon EC2. It provides examples and code snippets for programming tasks related to Amazon EC2 using the AWS SDK for Java 2.x.