12 ms·

Just to plug that we have built MirrorThink.ai: an AI assistant specifically designed to help professional researchers in their day-to-day work. (I am aware th

Just to plug that we have built MirrorThink.ai: an AI assistant specifically designed to help professional researchers in their day-to-day work. (I am aware that self-promotion is allowed but somewhat frowned-upon in HN. I believe this is relevant to the context and I am particularly qualified to comment on it. But I'll strive to be as transparent as possible.) We have intentionally steered away from assisting with writing work for the reasons outlined in the article. Our focus has been to save time in keeping up with their field: making literature reviews much faster, summarizing recent discoveries in each researchers' niche area of interest, and gathering how others have tackled similar highly-specialized research problems. Grounding LLMs in scientific fact and always providing verifiable sources. We have spent many years before this building a business around our "Map of Science" at Scitodate, so we have strong connections with the community and a lot of experience in this domain. Both in terms of how science is done and how to reliably extract valuable knowledge from millions of documents with NLP. We now have one of the largest datasets of papers, patents, funding grants... Updated on a daily basis. With plenty of enrichment, connections and aggregations: we have done a lot of in-house research on disambiguation and entity-linking. We particularly have focused on mapping out the expertise, interests and achievements of all professional scientists, as well as the institutions they have worked at.

icapybara 3y ago
How do you get the chatbot to tell the truth more? I’m a scientist and this is my biggest showstopper with AI tools. I just can’t know if I can trust what it’s telling me.
- unblough 3y ago
  You are unable to. “LLMs can’t self-correct in reasoning tasks, DeepMind study finds“ https://news.ycombinator.com/item?id=37823543 https://news.ycombinator.com/item?id=37823543 Anyone who says otherwise is either ignorant of the underlying function of llms or trying to sell you something.
  - mistermann 3y ago
    Or, they realize that absence cannot be scientifically proven, and that scientists use language loosely, confusing those who take their loose language literally. The study didn't "find" (discover) what they claim, rather, they didn't find validation that it "can" (the implementation of which varies per observer, sub-perceptually). If you had to code something like this at work for a different domain, I bet you'd have no problem realizing that a nullable boolean is required to accurately model the problem space.
    - unblough 3y ago
      I recognize your clarification of “discovery” and conclusion from that research, but I do think there is a strong argument that in terms of the stochastic usage of a nonlinear system the “undefined” state of your nullable boolean is itself a falsey state.
      - mistermann 3y ago
        You can argue whatever you like, but if the unknown IS actually known, why can't scientists tell us their secrets? How many people would have to be in on the scheme? And this isn't just a one off, this is a systemic, institutional shortcoming, I encounter several instances of it every day just in my regular social media feeds.
  - famouswaffles 3y ago
    >You are unable to. This is just wrong lol. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 https://arxiv.org/abs/2207.05221
    - unblough 3y ago
- jiggawatts 3y ago
  Don't try to use AIs as omniscient oracles. Use them to replace grad students / interns. In other words, AIs are useful when you can confidently evaluate their output for correctness. Note that this also applies in novel scenarios, such as new scientific discovery. It's often much more difficult to come up with an answer than it is to verify it. As a random example, modern AIs are very good at image recognition tasks, which would make them ideally suited for finding rare phenomena in all-sky surveys. The AI doesn't need to write the research paper! It can be valuable if it can just find candidates for detailed evaluation by human researchers. Similarly, I've found AIs are quite good at spotting errors and inconsistencies. Don't make it write the paper... just ask it to proofread it for you. Etc...
- oersted 3y ago
  Never rely on the AI's own "memory". GPT-4 can remember a lot of deep technical things correctly from its training, but it can also write non-factual statements that look perfectly reasonable. We only use LLMs to retrieve and extract information from reliable sources (mostly papers and patents). If you provide extensive background information in the prompt, it will stick to that and avoid making up facts, and it has the added benefit of including verifiable references for each statement in the answer. Again, we treat the AI mostly as an "automatic reader", it's very good at that, it's not hard to keep it factually grounded and that alone can save hours of research. But when asking it to do deep reasoning or original writing it can be a bit more tricky to keep it controlled.
- jstarfish 3y ago
  Treat it like you're dealing with some vendor's salesperson. You have to bring your own source of truth, like the interfaces that let you "talk to your PDFs". Prep it with data you know to be true, then start your discussion. If you show up unprepared and let it tell you what's real, it will bullshit you in its favor.
arijun 3y ago
You don't have any pricing on your site, what does it cost? And do you have a video of a typical workflow or something? As an aside, the scroll on your website stutters pretty badly for me (on Firefox).
- oersted 3y ago
  You can sign-up for free and see the options. There's a free version if you register with an institutional email (university or company email domain), we want to support scientists (both academia and private). For full access it's pay-what-you-want right now (with a minimum of $5 or $10 depending what roll of the dice you get in the current A/B test).
  - ZeroCool2u 3y ago
    Just a heads up, I'm getting an error when I sign up. I'm using the Scitodate activation code that was emailed to me and it's claiming it's invalid. Edit: Hmmm, worked when I signed up with a different email.
    - oersted 3y ago
      Did the emails have different domains? We use the domain as a simple filter for the free version, but there might be a bug around to that. It would be helpful if you could help reproduce it.
      - ZeroCool2u 3y ago
        They did! I used my personal first, which is just Firstname.Lastname@gmail.com which was the one that broke. The other one was Firstname@company.com which seemed to work fine.
ZeroCool2u 3y ago
This is exactly how I use ChatGPT right now. Like an extreme rubber ducky for research when I don't have anyone else I can talk to about something. Also, thank you for the self promotion, I'll check out MirrorThink!
godelski 3y ago
- What's the product actually do? There's just one page with some vague descriptions and then a signup. I'd love to actually be pitched the product before signing up. - What is the cost? - How does literature search happen? What are the sources? How often is it updated? How sub-niche can it actually get? You specifically mention the researcher's niche interests but your examples are "Why can't we sleep when excited", "Companies selling 3D printers", "3D printers in Eurpore", and "3D printers for medical purposes." These are not niche topics, this is pretty broad and doesn't look like scientific research. I'm sorry, but without more information this pitch and website just look like an information harvester. This may not be what you've actually built, but this is what it comes off as from someone who does scientific research.
- oersted 3y ago
  These are valid remarks, we'll see about including them in the landing page. Before going to the answers, just to note: if you are interested, we are looking to work closely with more researchers, to make sure it saves time for you in particular on a daily basis. What does it do? In very simple terms: It is ChatGPT (GPT-4) with awareness of our database of papers, patents, grants... It comes with many "tools" to use various information retrieval styles and sources, some more focused on question answering and some on wider reviews. There are also tools to gather "metadata" of science: researcher profiles, institutes, scientific companies... What is the cost? There's a free version if you register with an institutional email (university or company email). We generally want to make it open to professional researchers (academic or private). The full version is pay-what-you-want with a $5/$10 minimum. How does literature search happen? It depends on the "tool" you are using, but it generally involves retrieving a set of candidate relevant documents (papers, patents, websites...) and either read through full documents or select relevant snippets from them so it can do a wider review. What are the sources? Primarily papers (~100M right now), including ArXiv, PubMed, IEEE, the Elsevier suite and many more individual journals. As well as grants from most national and international funding organizations. And US patents, with ongoing expansion to other countries. Updated daily whenever possible. We also have the option to complement scientific literature with web research, it can be good for example to review public documentation from scientific instrumentation and consumable providers on experimental protocols. How sub-niche can it actually get? It's hard to convey, it would be easiest if you just try it, it really can get as niche as you want it to, it can adapt to the level of expertise you operate at. If you are a researcher with a university email you can use most of the relevant functionality for free. You are right, the examples are not very representative. The intent was to make them understandable to a wide array of researchers with different backgrounds, but still, yeah they could be better.

icapybara 3y ago

How do you get the chatbot to tell the truth more? I’m a scientist and this is my biggest showstopper with AI tools. I just can’t know if I can trust what it’s telling me.

unblough 3y ago
You are unable to. “LLMs can’t self-correct in reasoning tasks, DeepMind study finds“ https://news.ycombinator.com/item?id=37823543 https://news.ycombinator.com/item?id=37823543 Anyone who says otherwise is either ignorant of the underlying function of llms or trying to sell you something.
- mistermann 3y ago
  Or, they realize that absence cannot be scientifically proven, and that scientists use language loosely, confusing those who take their loose language literally. The study didn't "find" (discover) what they claim, rather, they didn't find validation that it "can" (the implementation of which varies per observer, sub-perceptually). If you had to code something like this at work for a different domain, I bet you'd have no problem realizing that a nullable boolean is required to accurately model the problem space.
  - unblough 3y ago
    I recognize your clarification of “discovery” and conclusion from that research, but I do think there is a strong argument that in terms of the stochastic usage of a nonlinear system the “undefined” state of your nullable boolean is itself a falsey state.
    - mistermann 3y ago
      You can argue whatever you like, but if the unknown IS actually known, why can't scientists tell us their secrets? How many people would have to be in on the scheme? And this isn't just a one off, this is a systemic, institutional shortcoming, I encounter several instances of it every day just in my regular social media feeds.
- famouswaffles 3y ago
  >You are unable to. This is just wrong lol. GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 https://arxiv.org/abs/2207.05221
  - unblough 3y ago
jiggawatts 3y ago
Don't try to use AIs as omniscient oracles. Use them to replace grad students / interns. In other words, AIs are useful when you can confidently evaluate their output for correctness. Note that this also applies in novel scenarios, such as new scientific discovery. It's often much more difficult to come up with an answer than it is to verify it. As a random example, modern AIs are very good at image recognition tasks, which would make them ideally suited for finding rare phenomena in all-sky surveys. The AI doesn't need to write the research paper! It can be valuable if it can just find candidates for detailed evaluation by human researchers. Similarly, I've found AIs are quite good at spotting errors and inconsistencies. Don't make it write the paper... just ask it to proofread it for you. Etc...
oersted 3y ago
Never rely on the AI's own "memory". GPT-4 can remember a lot of deep technical things correctly from its training, but it can also write non-factual statements that look perfectly reasonable. We only use LLMs to retrieve and extract information from reliable sources (mostly papers and patents). If you provide extensive background information in the prompt, it will stick to that and avoid making up facts, and it has the added benefit of including verifiable references for each statement in the answer. Again, we treat the AI mostly as an "automatic reader", it's very good at that, it's not hard to keep it factually grounded and that alone can save hours of research. But when asking it to do deep reasoning or original writing it can be a bit more tricky to keep it controlled.
jstarfish 3y ago
Treat it like you're dealing with some vendor's salesperson. You have to bring your own source of truth, like the interfaces that let you "talk to your PDFs". Prep it with data you know to be true, then start your discussion. If you show up unprepared and let it tell you what's real, it will bullshit you in its favor.

arijun 3y ago

You don't have any pricing on your site, what does it cost? And do you have a video of a typical workflow or something? As an aside, the scroll on your website stutters pretty badly for me (on Firefox).

oersted 3y ago
You can sign-up for free and see the options. There's a free version if you register with an institutional email (university or company email domain), we want to support scientists (both academia and private). For full access it's pay-what-you-want right now (with a minimum of $5 or $10 depending what roll of the dice you get in the current A/B test).
- ZeroCool2u 3y ago
  Just a heads up, I'm getting an error when I sign up. I'm using the Scitodate activation code that was emailed to me and it's claiming it's invalid. Edit: Hmmm, worked when I signed up with a different email.
  - oersted 3y ago
    Did the emails have different domains? We use the domain as a simple filter for the free version, but there might be a bug around to that. It would be helpful if you could help reproduce it.
    - ZeroCool2u 3y ago
      They did! I used my personal first, which is just Firstname.Lastname@gmail.com which was the one that broke. The other one was Firstname@company.com which seemed to work fine.

ZeroCool2u 3y ago

This is exactly how I use ChatGPT right now. Like an extreme rubber ducky for research when I don't have anyone else I can talk to about something. Also, thank you for the self promotion, I'll check out MirrorThink!

godelski 3y ago

- What's the product actually do? There's just one page with some vague descriptions and then a signup. I'd love to actually be pitched the product before signing up. - What is the cost? - How does literature search happen? What are the sources? How often is it updated? How sub-niche can it actually get? You specifically mention the researcher's niche interests but your examples are "Why can't we sleep when excited", "Companies selling 3D printers", "3D printers in Eurpore", and "3D printers for medical purposes." These are not niche topics, this is pretty broad and doesn't look like scientific research. I'm sorry, but without more information this pitch and website just look like an information harvester. This may not be what you've actually built, but this is what it comes off as from someone who does scientific research.

oersted 3y ago
These are valid remarks, we'll see about including them in the landing page. Before going to the answers, just to note: if you are interested, we are looking to work closely with more researchers, to make sure it saves time for you in particular on a daily basis. What does it do? In very simple terms: It is ChatGPT (GPT-4) with awareness of our database of papers, patents, grants... It comes with many "tools" to use various information retrieval styles and sources, some more focused on question answering and some on wider reviews. There are also tools to gather "metadata" of science: researcher profiles, institutes, scientific companies... What is the cost? There's a free version if you register with an institutional email (university or company email). We generally want to make it open to professional researchers (academic or private). The full version is pay-what-you-want with a $5/$10 minimum. How does literature search happen? It depends on the "tool" you are using, but it generally involves retrieving a set of candidate relevant documents (papers, patents, websites...) and either read through full documents or select relevant snippets from them so it can do a wider review. What are the sources? Primarily papers (~100M right now), including ArXiv, PubMed, IEEE, the Elsevier suite and many more individual journals. As well as grants from most national and international funding organizations. And US patents, with ongoing expansion to other countries. Updated daily whenever possible. We also have the option to complement scientific literature with web research, it can be good for example to review public documentation from scientific instrumentation and consumable providers on experimental protocols. How sub-niche can it actually get? It's hard to convey, it would be easiest if you just try it, it really can get as niche as you want it to, it can adapt to the level of expertise you operate at. If you are a researcher with a university email you can use most of the relevant functionality for free. You are right, the examples are not very representative. The intent was to make them understandable to a wide array of researchers with different backgrounds, but still, yeah they could be better.