yuhongsun

7 ms·

31.
by yuhongsun 3y ago
Ya, a lot of knowledge is locked away in metadata, how documents are structured, in non-textual representations, or sometimes even just in the head of experts. We definitely want to be more than a shallow access to documents and we're
32.
by yuhongsun 3y ago
Ah thanks for pointing it out, it look like an awesome project! We love open source so it's nice to see others going this direction as well
33.
by yuhongsun 3y ago
I think the explosion in interest and all the new software around GenAI owe their success to NLP advancements (coming from an NLP background myself, I may be biased though). I think the best projects are those pushing the frontier of NLP an
34.
by yuhongsun 3y ago
So private refers to two things here, sorry for any confusion. When we say "chat over private data" we mean that this data isn't publicly available and no LLMs have this knowledge in their training. Meaning that with our syst
35.
by yuhongsun 3y ago
Access controls is a non-glamorous but critical piece of what we're building. Currently implementing automatic access sync-ing for a few sources like Google Drive, Confluence, Jira, and Notion to start. By matching document-access in t
36.
by yuhongsun 3y ago
Being able to filter down the data deterministically is a big value add, especially as the number of documents scale into the range of multiple millions. We have filters by document-set, tags, time range, source type (ie. only include Slack
37.
by yuhongsun 3y ago
Exactly, there is a huge amount of value in being able to quickly get a holistic view of topics. Most topics don't exist in an isolated tool - most often there are the official discussions/designs which exist in place, there are c
38.
by yuhongsun 3y ago
Thanks so much for sharing! Excited to look into it!
39.
by yuhongsun 3y ago
Hello, thanks for the kind words! With regards to your questions: 1. Are you referring to the local NLP models or the LLM? The local models are already open source models or ones we've trained ourselves. If you're talking about th
40.
by yuhongsun 3y ago
So one of the main things we do is automatically sync-ing documents from different sources of knowledge from your team. So all of the data connectors as well as the user authentication and access systems would have to be built from scratch
41.
by yuhongsun 3y ago
Sounds like Danswer might just fit your needs. Also ya, we came up with this idea of chatting with documents that you can select on the fly, I think we're the only ones who do this still. People have really been liking that one! If you
42.
by yuhongsun 3y ago
We definitely recommend you trying it out! As far as how outdated information is handled - we pass the most relevant documents along with metadata to the LLM. So in the case you mentioned, the LLM will be provided the procedure doc and the
43.
by yuhongsun 3y ago
Got it, we considered it but most connectors that we pull from don't support this. Might bring it back for the ones that do, thanks for the suggestion! It's actually still in the code, just none of the connectors implement it atm:
44.
by yuhongsun 3y ago
Do you mean directly uploading files in the UI and chatting against those files? This one will be done within the next few weeks, it's a very high value item for sure. Alternatively, if you're talking about real time indexing to m
45.
by yuhongsun 3y ago
Ya, there's definitely a common thread there! A lot of consulting firms have given us very positive feedback from using Danswer. I think the nature of the time-framed projects and frequent scope changes mean that people have to always
46.
by yuhongsun 3y ago
A lot of teams we talk to switched from an in-house solution to either directly using Danswer or building on top of Danswer. Glad you liked the demo! We're using E5 base by default but there's an embedding model admin page to choo
47.
by yuhongsun 3y ago
I definitely see what you mean. They also have advantages in bundling co-pilot with their other offerings. This will in no means be an easy battle for us, but we have hope that we'll be able to build something people love and end up us
48.
by yuhongsun 3y ago
Thanks so much! It's always rewarding to hear people sharing our excitement!
49.
by yuhongsun 3y ago
We're leaning heavily into the open source aspect. We think that a solution like this will be useful to even smaller teams (10-50) that other companies won't want to target. There is some non-trivial amount of setup required, spec
50.
by yuhongsun 3y ago
Yes! And we'd love it if you contribute them back to the project! More than half of the connectors are community contributed at this point and it's by far the most common area of contribution. There's a simple Document interf
51.
by yuhongsun 3y ago
This is also another reason why we think OSS is the way to go here. Taking on the tech giants alone is definitely a daunting task (maybe even impossible for a small isolated team). The hope is that by working with the community, we'll
52.
by yuhongsun 3y ago
So the intent of the GitHub/GitLab connector in the current iteration is to help people easily find implementations that have been done before. For example, if a new bug comes up with some feature, I can easily search for the PRs relat
53.
by yuhongsun 3y ago
Glad you asked! We are actually the only project (open source or closed), as far as I know, that handles multilingual quite well. We have options to do multilingual query-expansion and also multilingual embedding models. Without dropping an
54.
by yuhongsun 3y ago
Code search is something we have in our sights in the next couple months. Currently the GitHub connector pulls in PRs and Issues but not the whole code base. We wanted to have the best RAG pipeline so we deep dived on that. Code search uses
55.
by yuhongsun 3y ago
A lot of people are very bullish on AI, it's very interesting to hear the opposite side. My opinion is that LLMs are very powerful at digesting and distilling knowledge which is why we built this project. I also think that LLMs are ter
56.
by yuhongsun 3y ago
I think these developer platforms like LlamaIndex and Langchain are super great for prototyping and understanding the crowdsourced best approaches in solving these new LLM related challenges. Depending on how custom the pipelines need to be
57.
by yuhongsun 3y ago
Thank you! Yes, there are several options for user authentication (Basic Auth, Google OAuth, OIDC, SAML). Currently the RBAC is managed via the Danswer and this controls who has access to which documents (it's done at the connector lev
58.
by yuhongsun 3y ago
Yes! There is Basic Auth (email + password with email verification) and Google OAuth available in the free version. We also do OIDC and SAML to integration with Identity Providers (IdPs) like Okta but that's part of the paid features.
59.
by yuhongsun 3y ago
Thanks! Yes, it does do PDFs. We don't do anything fancy with it though like Optical Character Recognition (OCR). So pictures of text, as well as images and graphs will be lost. This is something we will work on though. Is this somethi
60.
by yuhongsun 3y ago
We use LlamaIndex very sparingly. Specifically the context aware document chunking functionality is via LlamaIndex. We couldn't use the more involved pipelines because we needed significant custom logic to enforce permissions, filters

More ›