5 ms·

Structured Output with LangChain and Llamafile

46 points by brakmic 1y ago

dcreater 1y ago
People still use langchain?
- zingababba 1y ago
  What should be used instead?
  - Hugsun 1y ago
    I gave up after it didn't let me see the prompt that went into the LLM, without using their proprietary service. I'd recommend just using the API directly. They're very simple. There might be some simpler wrapper library if you want all the providers and can't be bothered to implement the support for each. Vercel's ai-sdk seems decent for JS.
    - halyconWays 1y ago
      >I gave up after it didn't let me see the prompt that went into the LLM, without using their proprietary service. Haha, really?
  - ebonnafoux 1y ago
    httpx to make the call yourself, or if you really want a wrapper the openAI python https://github.com/openai/openai-python https://github.com/openai/openai-python.
  - Jimmc414 1y ago
    pydanticai, dspy or deal directly with the provider sdks
    - dcreater 1y ago
      DSPy seems like the right, developed approach but its far too convoluted and I find the grammar is ugly.
  - dcreater 1y ago
    Plain old http requests and your own functions. Its almost always the better choice
- owebmaster 1y ago
  No
- deleted 1y ago
  [deleted]
- anshumankmr 1y ago
  Its good for quickly developing something but for production, I do not think so.We used it for a RAG application I built last year with a client, ended up removing it piece by piece, and found our app responded faster. But orgs think its some sort of flagbearer of LLMs.As I am interviewing for other roles now, HRs from other companies still ask for how many years of exp I have with Langchain and Agentic AI.
- codestank 1y ago
  i do because i don't know any better since i'm new to the AI space.
  - nilamo 1y ago
    My experience, as someone who is also new and trying to figure things out, is that langchain works great as long as everything you want to do has an adapter. Try to step off the path, and things get really complex really fast. After hitting that several times, I've found it's easier to just do things directly instead of trying to figure out the langchain way of doing things. I've found dspy to work closer to how I think, which has made working with pipelines so much easier for me.
- screye 1y ago
  It is useful if you keep swapping things out. Langchain's wrappers stay stable and up-to-date because of their popularity. In production, it's ideal startups that undergo a lot of flux. I would suggest against using their orchestration tooling, DSLs or default prompts. Those components are either underbaked or require deep adoption in a way that is harder to strip out later. We change models, providers and search tooling quite often. Having consistent interfaces helps speed things up and reduce legacy buildup. Their stream callbacks, function calling integration, RAG primitives and logging solutions are nice. One way of another, it is useful to have a langchain-like solution for these needs. Pydanticai + logfire seems like a better version of what I like about langchain. Haven't tried it, but I bet it's good.
reedlaw 1y ago
The use case in the article is relatively simple. For more complex structures, BAML (https://www.boundaryml.com/ https://www.boundaryml.com/) is a better option.
pcwelder 1y ago
``` try: answer = chain.invoke(question) # print(answer) # raw JSON output display_answer(answer) except Exception as e: print(f"An error occurred: {e}") chain_no_parser = prompt | llm raw_output = chain_no_parser.invoke(question) print(f"Raw output:\n\n{raw_output}") ``` Wait, are you calling LLM again if parsing fails just to get what LLM has sent to you already? The whole thing is not difficult to do if you directly call API without Lang chain, it'd also help you avoid such inefficiency.
- moribunda 1y ago
  I don't get the langchain hate, but I agree that this "blog post" is bad. Langchain has a way to return raw output, aside "with structured output": https://python.langchain.com/docs/how_to/structured_output/#advanced-raw-outputs https://python.langchain.com/docs/how_to/structured_output/#... It's pretty common to use a cheaper model to fix these errors to match the schema if it fails with a tool call.
  - crystal_revenge 1y ago
    > It's pretty common to use a cheaper model to fix these errors to match the schema if it fails with a tool call. This has not be true for a while. For open models there's 0 need for these kind of hacks with libraries like Xgrammar and Outlines (and several others) both existing as a solution on their own and being used by a wide range of open source tools to ensure structured generation happens at the logit levels. There's no-need to add multiples to your inference cost, when in some cases (xgrammar) they can reduce inference cost. For proprietary models more and more providers are using proper structured generation (i.e. constrained decoding) under-the-hood. Most notably OpenAI's current version of structure outputs makes use of logit based methods to guarantee the structure of the output.
Hugsun 1y ago
The version of llama.cpp that Llamafile uses supports structured outputs. Don't waste your time with bloat like langchain. Think about why langchain has dozens of adapters that are all targeting services that describe themselves as OAI compatible, Llamafile included. I'd bet you could point some of them at Llamafile and get structured outputs. Note that they can be made 100% reliable when done properly. They're not done properly in this article.
- halyconWays 1y ago
  >Don't waste your time with bloat like langchain. Amen. See also: "Langchain is Pointless" https://news.ycombinator.com/item?id=36645575 https://news.ycombinator.com/item?id=36645575
kristjansson 1y ago
It's right there. In the screenshot in the blog post. Grammar > 'JSON Schema + Convert'. That's what structured output is. ... it's going to be september forever, isn't it?

46 points by brakmic 1y ago

dcreater 1y ago

People still use langchain?

zingababba 1y ago
What should be used instead?
- Hugsun 1y ago
  I gave up after it didn't let me see the prompt that went into the LLM, without using their proprietary service. I'd recommend just using the API directly. They're very simple. There might be some simpler wrapper library if you want all the providers and can't be bothered to implement the support for each. Vercel's ai-sdk seems decent for JS.
  - halyconWays 1y ago
    >I gave up after it didn't let me see the prompt that went into the LLM, without using their proprietary service. Haha, really?
- ebonnafoux 1y ago
  httpx to make the call yourself, or if you really want a wrapper the openAI python https://github.com/openai/openai-python https://github.com/openai/openai-python.
- Jimmc414 1y ago
  pydanticai, dspy or deal directly with the provider sdks
  - dcreater 1y ago
    DSPy seems like the right, developed approach but its far too convoluted and I find the grammar is ugly.
- dcreater 1y ago
  Plain old http requests and your own functions. Its almost always the better choice
owebmaster 1y ago
No
deleted 1y ago
[deleted]
anshumankmr 1y ago
Its good for quickly developing something but for production, I do not think so.We used it for a RAG application I built last year with a client, ended up removing it piece by piece, and found our app responded faster. But orgs think its some sort of flagbearer of LLMs.As I am interviewing for other roles now, HRs from other companies still ask for how many years of exp I have with Langchain and Agentic AI.
codestank 1y ago
i do because i don't know any better since i'm new to the AI space.
- nilamo 1y ago
  My experience, as someone who is also new and trying to figure things out, is that langchain works great as long as everything you want to do has an adapter. Try to step off the path, and things get really complex really fast. After hitting that several times, I've found it's easier to just do things directly instead of trying to figure out the langchain way of doing things. I've found dspy to work closer to how I think, which has made working with pipelines so much easier for me.
screye 1y ago
It is useful if you keep swapping things out. Langchain's wrappers stay stable and up-to-date because of their popularity. In production, it's ideal startups that undergo a lot of flux. I would suggest against using their orchestration tooling, DSLs or default prompts. Those components are either underbaked or require deep adoption in a way that is harder to strip out later. We change models, providers and search tooling quite often. Having consistent interfaces helps speed things up and reduce legacy buildup. Their stream callbacks, function calling integration, RAG primitives and logging solutions are nice. One way of another, it is useful to have a langchain-like solution for these needs. Pydanticai + logfire seems like a better version of what I like about langchain. Haven't tried it, but I bet it's good.

reedlaw 1y ago

The use case in the article is relatively simple. For more complex structures, BAML (https://www.boundaryml.com/ https://www.boundaryml.com/) is a better option.

pcwelder 1y ago

``` try: answer = chain.invoke(question) # print(answer) # raw JSON output display_answer(answer) except Exception as e: print(f"An error occurred: {e}") chain_no_parser = prompt | llm raw_output = chain_no_parser.invoke(question) print(f"Raw output:\n\n{raw_output}") ``` Wait, are you calling LLM again if parsing fails just to get what LLM has sent to you already? The whole thing is not difficult to do if you directly call API without Lang chain, it'd also help you avoid such inefficiency.

moribunda 1y ago
I don't get the langchain hate, but I agree that this "blog post" is bad. Langchain has a way to return raw output, aside "with structured output": https://python.langchain.com/docs/how_to/structured_output/#advanced-raw-outputs https://python.langchain.com/docs/how_to/structured_output/#... It's pretty common to use a cheaper model to fix these errors to match the schema if it fails with a tool call.
- crystal_revenge 1y ago
  > It's pretty common to use a cheaper model to fix these errors to match the schema if it fails with a tool call. This has not be true for a while. For open models there's 0 need for these kind of hacks with libraries like Xgrammar and Outlines (and several others) both existing as a solution on their own and being used by a wide range of open source tools to ensure structured generation happens at the logit levels. There's no-need to add multiples to your inference cost, when in some cases (xgrammar) they can reduce inference cost. For proprietary models more and more providers are using proper structured generation (i.e. constrained decoding) under-the-hood. Most notably OpenAI's current version of structure outputs makes use of logit based methods to guarantee the structure of the output.

Hugsun 1y ago

The version of llama.cpp that Llamafile uses supports structured outputs. Don't waste your time with bloat like langchain. Think about why langchain has dozens of adapters that are all targeting services that describe themselves as OAI compatible, Llamafile included. I'd bet you could point some of them at Llamafile and get structured outputs. Note that they can be made 100% reliable when done properly. They're not done properly in this article.

halyconWays 1y ago
>Don't waste your time with bloat like langchain. Amen. See also: "Langchain is Pointless" https://news.ycombinator.com/item?id=36645575 https://news.ycombinator.com/item?id=36645575

kristjansson 1y ago

It's right there. In the screenshot in the blog post. Grammar > 'JSON Schema + Convert'. That's what structured output is. ... it's going to be september forever, isn't it?