14 ms·
GPT3 Get answers to technical questions from your documentation site
- chandan_maruthi 4y agoUpdate: Still waiting on OpenAI to increase API limits after we hit hard limits after a popular HN post
- chandan_maruthi 4y agoSome Highlights - Twig takes static documentation sites and lets users ask questions about it - Custom/FineTuned GPT3 models are trained on your data - Can be embedded within your app or docs site. Seeking Feedback - What do you think about it. - Would you add this to your docs site or within your app when it's available - What can I add/change to make this better.
- sAbakumoff 4y agoI think it's a great idea and I would definitely try to replace support staff with a chat bot like this one. But i would need the functionality to be available via API and full access to statistics of usage
- chandan_maruthi 4y agoThank you. Appreciate the feedback. I agree APIs be built for - Learning about usage stats - What users are asking - Flow of questions/intent etc
- pvg 4y agoSeeking Feedback Show HN's area really for things that can be tried and you're not quite at that stage yet. Take a look at: https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html Generally, if you have 'add me to waitlist' it's not a Show HN (yet).
- chandan_maruthi 4y agoHi, If you see just below the "add me to waitlist", There are live Chat Bots created for several companies you can try out right now. These chat interfaces are live and work on custom/fine-tuned models that have been trained on documentation of these companies. Over the last 3hrs users exchanged over 1000 messages with the bots. So it's live, it works, and users can try and play with it right now and give feedback. It should check all the boxes. The waitlist is to allow us to onboard companies in an orderly way as each bot is trained on their custom documentation. I have changed the words to "Get Access". I hope that helps.
- Terretta 4y agoHow about "Get Private Access"
- spaniard89277 4y agoIt seems that technical writers will be even more important, as their job will be the imput for models. Unless GPT can learn directly from code and explain it properly. With chatGPT I've had some problems trying to deduce from code prompts, but this seems promising.
- chandan_maruthi 4y agoHah true! If any I would say this makes the content put out by technical writers more accessible to users. One founder I showed it to, suggested embedding this chat within their app so users get answers without leaving the product.
- raydiatian 4y agoNice, I have been thinking the same was possible to “teach” an AI about your codebase, and then ask it to make PRs for simple features and bug fixes.
- chandan_maruthi 4y agoI could see that comming soon with things like copilot
- httpteapot 4y agoSounds like https://githubnext.com/projects/ai-for-pull-requests/ https://githubnext.com/projects/ai-for-pull-requests/
- chandan_maruthi 4y agoThats a good project. My PR Comments are often less than good. I could use something like that.
- obert 4y agothough PR should explain Why more than What, ie I can figure out what changed looking at files, but without context and design details an autogenerated description doesnt make a review any easier
- immortalloom 4y agoBeing able to have conversations with a documentation site is the future. Awesome that you are tackling it!
- chandan_maruthi 4y agoThank you! I believe AI can be good at doing the boring stuff while helping humans be more productive.
- visarga 4y agoConversations with docs is one possible application. I foresee conversations with papers, articles, books, the transcript of a YT lecture, or even with Google (oh no!). The other major LLM improvement I expect is multi-round code execution. Let the model iterate on bug fixing, like us. We need the extra steps, the LLM does too.
- iamflimflam1 4y agoWas thinking of building something very similar - upload your technical docs - and create a chat bot help system. One of the tricky things to overcome is the length limits of the prompt you can feed into things like GPT-3. There are some suggestions on the OpenAI website on how to overcome this. The main one seems to be to filter the content using embeddings and then only feed "relevant" sections into the prompt. Would be interested to know what your approach is.
- chandan_maruthi 4y agoI was able to break down the content into smaller prompts/completions. And remove some recurring and low-information data. Your suggestion of using embeddings to identify and feed relevant completions is pretty good. I'll try that soon.
- iamflimflam1 4y agoHere’s the example notebook that shows how to do this. https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb https://github.com/openai/openai-cookbook/blob/main/examples...
- NicoJuicy 4y agoWhat's the main difference between using https://trainmy.ai/ https://trainmy.ai/ vs. with your app. Since I'm planning to use that for where i work, for the same use-case.
- iamflimflam1 4y agoFrom what I was thinking of bulding: Packaging it up as a nice widget that can be embedded on your site easily. So you just upload your docs and have a snippet of JavaScript for your website that gives you a help bot.
- chandan_maruthi 4y agoAgree, a snippet that you add to the application that trains every day and gets better. I am thinking we can ingest data every day/week as documentation updates.
- allisdust 4y agoHow did you train it. Does openai provide api for training chatgpt on specific sites or data?
- chandan_maruthi 4y agoOpen AI has a concept called fine-tuning. You can create a new model on top of OpenAIs own models using your data. These new models are only available to you. This allows you to build custom applications that are better than generic models in very specific scenarios like this.
- wcoenen 4y agoHere are the OpenAI API docs that explain how to do fine tuning: https://beta.openai.com/docs/guides/fine-tuning https://beta.openai.com/docs/guides/fine-tuning
- recuter 4y agoQuestDb; Columnar time-series database with high performance ingestion and SQL analytics you know and love from QuestDB open source, now on the cloud. Q: How high is the performance? GPT3: We have tested QuestDB against other databases using benchmarking frameworks and benchmarking tools. We have also developed our own performance tests to simulate real-world use cases involving large data sets, multiple ingestion streams, and complex SQL queries. The results of these efforts are summarized in the table below:QuestDB is the fastest open source time series database available today. It powers real-life applications such as Yahoo! which stores 1 billion records per day in a single instance of QuestDB with GPT3 says QuestDb is web scale and I for one trust it implicitly.
- deleted 4y ago[deleted]
- kindofabigdeal 4y agoThis is actually interesting, can it parse documentation from my code? Or comment? Or how is this exactly working?
- chandan_maruthi 4y agoHi, I am training a custom GPT model on documentation published by these companies on their documentation websites. Its not reading from code/comments. But their documentation/support sites.
- visarga 4y agoIt seems to do open domain question answering without restricting to the topic. > Is the word cat made of 4 or 5 letters? >> The word cat is made of 4 letters, 3 of which are in the stem.
- chandan_maruthi 4y agoI am still figuring that out. More updates soon.
- btbuildem 4y agoI don't think you can restrict it, in fact, I think it's in your interest not to. GPT's ability to work with natural language comes from the large corpus it's trained on. Adding specific product documentation would give it "domain expertise" of sorts (assuming this is what OP did), but training it ONLY on your documentation -- I don't know enough to call it, but my intuition is that it would weaken it's ability to interpret the users' questions.
- sammyd56 4y agoThat's probably because fine-tuning is not the right approach for this use-case. A better approach might look more like this: https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb https://github.com/openai/openai-cookbook/blob/main/examples...
- jinct 4y agoIs there any way a company could run this on prem trained against their internal Confluence instance?
- chandan_maruthi 4y agoGood idea, I have not considered on prem yet. But I can see the need in cases where data is sensitive.
- MilStdJunkie 4y agoThat's what I was thinking. For sensitive information or information that's under legal restriction, how are we going to train the models for on-prem AI? I guess we could train models on similar corpus scraped from the internet - mostly old General Aviation manuals, I guess, but a) that's not enough corpus, and b) those GA docs are so old I'm afraid my AI will start chain smoking and casually using the word "broad".
- cloudking 4y agoLooks interesting, but need more details about set up and pricing. This is another approach that doesn't require fine tuning https://github.com/jerryjliu/gpt_index https://github.com/jerryjliu/gpt_index
- btbuildem 4y agoWere you able to build something using this?
- isthisthingon99 4y agoThe problem with GPT3 is it creates BS answers. The function "Interval" does not seem to exist. how would I do a SQL query with questdb to grab every other second of data? This is a common use case for us. We have built SQL functions to easily do this task. The function below will return every other second of data:SELECT * FROM `time_series` WHERE (INTERVAL '2s')This query can be run directly from the web console or via Postgres wire protocol using Postgres client libraries such as pg4ce, postgres-java and others.The above query will return all records in the table with timestamp greater than 12
- samsonradu 4y agoIndeed I believe this is the blocking problem. I've been asking questions about EU legislation and it keeps answering BS and then apologizing: Me: When was the last consolidation of Directive 2013/34/EU? GPT: Directive 2013/34/EU, also known as the Annual Accounts Directive, was last consolidated on November 19, 2019. The consolidation process involves collecting all the amendments and updates to the directive and incorporating them into a single, updated version of the directive. ... Me: Are you sure about the date of November 19, 2019? I could not find the consolidation on the EU official journal. GPT: I apologize for the error. Upon further research, it appears that Directive 2013/34/EU has not been consolidated since it was originally adopted in 2013. Directive 2013/34/EU, also known as the Annual Accounts Directive,... Me: Can you give me a list of the dates it was amended? GPT: - April 15, 2014: Directive 2014/56/EU amended Directive 2013/34/EU to clarify and supplement certain provisions regarding the preparation and publication of annual accounts and consolidated accounts. - June 17, 2014: Directive 2014/65/EU amended Directive 2013/34/EU to introduce new rules on the provision of investment services and the performance of investment activities... Me: I think you're wrong. Delegated Directive 2014/71/EU amends Directive 2011/65/EU. GPT: I apologize for the error in my previous response. You are correct that Delegated Directive 2014/71/EU amends Directive 2011/65/EU, which is also known as the RoHS Directive. It never gets it right. See https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32013L0034 https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32... for the correct data.
- iamflimflam1 4y ago
- nutanc 4y agoTrying something similar. Using a mix of embeddings and generative AI(davinci) to answer questions from scrapped data of website. Scrapped data for our website (Ozonetel.com) and created this site. 1. Scraping website. Used default node scraper. 5 mins. 2. Generated huggingface embeddings. 10 mins. 3. Use code AI to generate basic website. 5 mins. 4. Created prompt to limit to answers known. 1 min. So in 30 mins we are able to create a site search powered by generative AI. Disclaimer. This is still a work in progress. http://speech-kws.ozonetel.com/ozosearch http://speech-kws.ozonetel.com/ozosearch
- iamflimflam1 4y agoSeems to be down at the moment.
- nutanc 4y agoIts working for me. Is it just slow or the web page itself is not loading?
- iamflimflam1 4y agoWorking now. Was getting a cloud flare error before.
- chandan_maruthi 4y agoThat must have been me :-), I realized the thumbnail and fav icons were not updated so made a push. Sorry about it. I saw people posting on Twitter and I had to update them for meta tags unfurling stuff.
- gingerlime 4y agoImpressive. It’s not that intelligent with typos however. I asked it “what is ozontel?” and it answered “ Sorry! I have not learnt enough from this website to answer this confidently. The below links may have an answer to your query”
- 4y ago
- monkeydust 4y agoSome great stuff here. Been thinking about doing this for enterprise grade software at work. What I want to do is feed it docs (pdf), text from support queries with answers and videos. What types of approaches should I be considered here? Have just started using Whisper to do ASR on videos.
- chandan_maruthi 4y agoThank you, Yeah thats a good idea, Like literally take everything from the shared drives and train on it.
- monkeydust 4y agoAgreed. If you want to explore as a collaboration drop me an email (on my profile)
- the_real_sparky 4y agoI think the actual interface with OpenAI’s platform is the easy part. Everybody and their dog will have a version of this. Just look at the comments so far - many of us have already been playing with it. If you want a real moat, figure out how to parse existing PDF documentation that is really badly formatted. Think diagrams and tables with text floating in various places, etc. Documentation of this style is very common in industries where physical things are being built in the real world. The standards documentation (IEEE, ANSI, NFPA, etc) doesn’t usually parse cleanly, much less the messier internal documentation within the businesses. Grobid is the best example of such a documentation parser, but it is so laser focused on academic papers that it fails to properly process industry-style standards and SOP documentation. What the world needs right now is a Grobid that works for other kinds of messy documentation.
- iamflimflam1 4y agoI think you are right, this will be the key differentiator for anyone building a service like this - I guess like with most machine learning/data science projects - the real work is on the data engineering side of things. One thing that all these models will lack is the ability to include diagrams (on both the input and output side). Working out a clever way to do that would be very cool. At the moment there are some difficulties with the GPT interface - the most tricky one being the limit on the length of the input prompt. I'm not sure at the moment how much fine tuning helps with this. But, my assumption is that OpenAI will improve this, so there's not a huge way to differentiate here.
- the_real_sparky 4y agoYep, my only idea so far has been to generically describe the figures in text format. Doing so through recognition in any level of detail will be extremely tough, as often the drawings differ by variations that would be difficult for a model to understand. It may not matter that much though, as usually the notes and headings around each figure provides a lot of context. So maybe you can get 75% of the way there by identifying the “block” and keeping the textual information in that area associated together so that it can be fed into the embeddings (and thus later the LLM) as a single unit of related information. It’s frustrating though as often there are hundreds to thousands of pages of this stuff with diagrams and drawings randomly situated together on the pages. Documentation like this was designed to be dense for printing and consumed by a human that is familiar with it from regular use. I’m a bit concerned that the only solution may be paying a technical expert to sit down and convert it all to blocks of text. It would be an expensive endeavor, and even after it’s complete any changes (which happen often) would have to be continually maintained. If that’s the only solution then I may still go for it, as I think the value to the business of having all knowledge instantly searchable and then automatically summarized will be considerable.
- awestroke 4y agoQ: can I self-host questdb in my own EKS? > QuestDB is a popular time series database used by many companies for their application monitoring and real-time analytics. We have built QuestDB as a system to track the progress of our own application, which has evolved from an experiment into production at scale. As we grow, so does our use of QuestDB; it's now the core data store for all things InfluxData.We are looking to hire talented engineers who share our mission of empowering data producers with tools to Not super impressed
- deleted 4y ago[deleted]
- titzer 4y agoLet's use a language model trained on hundreds of gigabytes of god-knows-what, has a tendency to hallucinate, has literally never used your product, is known as a consummate bullshitter, and unleash it unsupervised on your confused users. Sorry for the snark, but what is your plan for quality assurance that this this meeting users needs instead of just BS'ing them out of your hair?
- chandan_maruthi 4y agoHa ha ha ! You have a point, That would be the next problem to tackle. Probably try to determine if I should even try to answer a question before attempting it. And then not showing results that don't fit criteria. At a high level, it seems like some embedding works need investigation but I haven't done much evaluation there. There would be filtering at both question and answer levels I suppose.
- deleted 4y ago[deleted]
- ngc248 4y agoI had this idea a few years back just when ML was heating up. A service which would crawl all of your documentation/data to which you can then ask questions and get answers back. A domain specific search engine/Knowledge Base. I started reading some of the papers, got to know that even to build a model required a huge number of resources and just dropped it.
- chandan_maruthi 4y agoStuff that was very hard just a few years ago is now relatively easier to do. I guess that is the nature of technology.
- neophyt3 4y agowondering if this can be used on RFCs
- chandan_maruthi 4y agoThats an interesting use case. How would you think you would use it on RFCs
- neophyt3 4y agoduring implementation, we may need more context or rules/requirements defined in RFC so in this case we could ask questions on those parts or may be it could generate pseudo code or may be it could link other dependent RFCs and get some more context
- pruthvishetty 4y agoIsn't the answers api deprecated?
- chandan_maruthi 4y agoYes the only have a completions api now
- motohagiography 4y agoWhile I've been looking for this, the gap in most product documentation I have seen is that it's written by architects who want you to understand the whole framework and demonstrate its myriad parts instead of by users (e.g. product managers) who just need to solve a problem and they will learn from that. On this GPT chat, I asked, "provide an example oauth2 config for QuestDB" and "how do I configure oauth2 for signaldot?" and it responded like a search engine with paragraphs from documentation that matched the keywords, perhaps within an extra degree of reasoning. As someone who uses fantastically (farcically?) complex security technologies, I'd tell architects, PMs, documentation writers and developers: Your Cookbook Is Your Product. LLM's have a GIGO problem, and until you can train it on example configs, it's going to be a search engine over someone's architectural grandstanding.
- o_____________o 4y agoI was just looking for a natural language interface to construct API queries, possibly based on OpenAPI specs. Surprising dearth, but this keeps coming up: https://www.microsoft.com/en-us/research/wp-content/uploads/2017/08/cikm17_nl2api.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/...
- chandan_maruthi 4y agoHi All, We had over 6,000k interactions and have hit OpenAIs hard api usage limits :-). I have requested OpenAi to increase limits. Until then the live chat with bot may be restricted. Remember to get on the waitlist with "get access" if you want to give it a try when the live chat starts again.
- chandan_maruthi 4y agoJust saw someone tried to hack/denial-of-serive the bot by just putting too many words in the prompt. (This hit triggers as usage is token based) . I just set a prompt limit of 100 characters should be plenty. Bot is till down until OpenAI resets limits. https://twitter.com/chandanmaruthi/status/1612134496035569664?s=20&t=JvOWZPvxNGXwvLqKG_a3KQ https://twitter.com/chandanmaruthi/status/161213449603556966...
- dang 4y agoWaiting lists and signup pages can't be Show HNs. Please see https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html. I've taken Show HN out of the title now.
- chandan_maruthi 4y agoDang, Hi, If you see just below the "add me to waitlist", There are live Chat Bots created for several companies you can try out right now. These chat interfaces are live and work on custom/fine-tuned models that have been trained on the documentation of these companies. Over the last 9hrs users exchanged over 6000 messages with the bots. So it's live, it works, and users can try and play with it right now and give feedback. A few hrs ago the bots hit OpenAis usage limits, Else its was working actively with over several hundred unique users interacting with the bot asking 4-5 questions each
- swyx 4y agomaybe if you put the waitlist signup at the bottom/it only shows up after some usage?