6 ms·

Hi Mark, One question out of academic curiosity: I'm exploring ways to use these tools for research projects in econ and am struggling to see a good angle. Fo

Hi Mark, One question out of academic curiosity: I'm exploring ways to use these tools for research projects in econ and am struggling to see a good angle. For instance, suppose I have lots of PDF reports on how firms have evolved on each quarter (10,000 reports or any other number beyond what I can read). Can LLMs be used to spit out variables based on these reports? EG: indicator variables (optimistic-vs-pesimistic), categories, etc. that I can then use as data to test economic models? I tried simple cases with ChatGPT3.5 and got the feeling it's great for outputting narrative text, but felt mediocre for when the output was more narrowly defined into categories (which was surprising).

mark_l_watson 4y ago
Look at the LangChain and Llama-Index (used to be called GPT-Index) projects that make smaller projects that need to use a large amount of text data do-able. There is also support for reading PDF files (and many other data sources), and pre-computing embeddings. If you spend a short while looking at example code in the documentation, find something that is similar to your requirements (e.g., semantic search, conversational chat about a set of documents, etc.), and build on that.
- zzleeper 4y ago
  Amazing; thanks a lot!