3 ms·
Congrats on the initial launch! Here a few thoughts, feedback, and questions: * you do a good job in this post of describing why you need more than just ChatG
by addisonj 3y ago
Congrats on the initial launch!
Here a few thoughts, feedback, and questions:
* you do a good job in this post of describing why you need more than just ChatGPT to get acceptable quality, but much less of that is in the readme. I wouldn't be afraid to sell the project a bit harder, even in early days
* likewise, I think a small visual of the architecture would be helpful, just to make clear what the relationship is to chatGPT, and the additional features of a context store, etc
* Between the notes here and your product pages, it seems a strategy for commercialization with this repo being a lower level tool, with your commercial service having UI, simplified integration, and a hosted version to make this easier for semi-techical teams? If that is the case, I wouldn't be afraid to make that more explicit. GitHub is more and more a place for discovery, even to semi-techical people, but to do that well, I think focusing on a readme that makes it clear who the open source is for is important
* How are you planning on solving the data access problem? Is this a full SaaS service that will need access to the customer data directly somehow? Do you deploy this in the customer's environment? In thinking beyond this open source release and to the commercial side, that would be what I would want to know
* This point has been hit by others in this thread, but I would be curious to know what your plans are to help protect against valid but incorrect queries? I am not as convinced that this problem is insurmountable, as it does seem like you could build features to remove ambiguity by asking questions, show alternatives, etc that most semi-techical people could reason through, but ultimately it seems like thinking about how to involve the teams that own the data might be an important part of the problem.
Anyways, congrats again! This is an area I am really excited to see how it evolves (and will be exploring more!). My email is in my profile if you are interested in chatting more :)
- aazo11 3y agoThanks for the feedback regarding the Readme. We have some additional detail in the docs here https://dataherald.readthedocs.io/en/latest/index.html https://dataherald.readthedocs.io/en/latest/index.html (including an architecture doc), but will definitely be incorporating some of the feedback as we iterate on the documentation which. You can connect the hosted version to your data warehouse using the SQLAlchemy connectors which are in the open source engine, and the connection info will be stored after being encrypted. An on-prem version of the enterprise product is also on the roadmap. For protecting against errors the approach is to allow for a human to be in the loop as you alluded to. One of the core pieces of the engine is the Evaluation module which combines white box and black box techniques. You can block the response from going back to the question asker if the confidence level is below a certain threshold until a human verifies the response manually (or always put the human from the data team in the loop).
- aazo11 3y agoWill shoot you an email to connect and chat further.
- saigal 3y agoI want to double-down on what AAzo11 replied. There is a human in the loop via an admin dashboard that allows quick approval / editing in the event of incorrect queries. This significantly decreases time for a member of the data team to field requests since they typically block out an hour to approve/edit these queries before going back to the question-asker.