5 ms·

The Only Way to Know If GPT-4 Got Dumber

16 points by swyx 3y ago

maest 3y ago
Oh, it's an ad.
- swyx 3y ago
  this is very cynical and hurts everything i am trying to do with this podcast. humanloop did not pay for this, i invited them on because they are one of the top companies in their space, and thought it might be of interest to HN because of their YC connection + perspective from working extremely closely with OpenAI (many might remember https://news.ycombinator.com/item?id=36177895 https://news.ycombinator.com/item?id=36177895 which we chose not to discuss on the pod but I asked him literally everything else). it hurts when people dismiss my work so casually as it was done in complete good faith and genuine interest in how founders are building in the new AI world.
  - onethought 3y ago
    It has an editorialised title, it doesn’t explain what this is really about. It’s basically click bait. It’s either an ad for human loop or an ad for your podcast. It’s not meant to be hurtful, it’s just the truth. If the title was instead more descriptive of what the content was. You wouldn’t get the cynicism.
    - swyx 3y ago
      as mentioned by @htk above - it links to the exact section of the essay, with that exact title, i wanted to offer for HN interest because it directly addresses active conversation topics that have been here before, which i know from myself being a longtime community member.
      - onethought 3y ago
        I see, so anchor tagging a clickbait piece of text within an article makes it not clickbait? Sorry, while I don’t like hurt feelings and I feel bad if that is the case. And I believe you are being genuine. I think it’s still something a bit off here.
        true_religion 3y ago
        The argument wasn’t over it being clickbait. It was claimed that this is an advertisement (it being paid is implied). The NY Times uses blatant clickbait, but I still respect them as real journalism and not just rewritten PR ad pieces producers.
        onethought 3y ago
        Sorry I don’t draw a distinction here. Clickbaiting is an advertising technique. This was advertising the author/affiliate. The main point isn’t wasn’t about the content (for me). Agree I guess there is a borderline where the title is both compelling to click, but still informative of what the overall article is. That wasn’t the case here. NYT does this better, agree.
swyx 3y ago
Hi HN! proud to present my latest interview with Raza Habib of Humanloop (YCS20). he went viral on the YC youtube a few months ago (https://www.youtube.com/watch?v=hQC5O3WTmuo https://www.youtube.com/watch?v=hQC5O3WTmuo) and in general has been the first and best known LLMOps/PromptOps/Foundation model ops platform out there (having bet on NLP before GPT3 and then ChatGPT blew up). I caught up with him in london and covered the origin story of how they pivoted from an automated labeling product into Humanloop today, and what they are seeing in both the AI Engineering and UK/Europe tech scene vs SF. They also launched their new pricing (free tier) and evals product (which a lot of other startups are trying to build) so we got into that a bit and I chose to highlight the "GPT4 nerfing" debate as that has been a topic of interest here before (https://news.ycombinator.com/item?id=36134249 https://news.ycombinator.com/item?id=36134249). happy to take qtns!
- j_crick 3y ago
  > They also launched Couldn't not mention that, could ya
  - swyx 3y ago
    why is that bad? it literally is what they just did lol
sublinear 3y ago
> please use the original title, unless it is misleading or linkbait; don't editorialize. Original title is: "Building the Foundation Model Ops Platform — with Raza Habib of Humanloop"
- htk 3y ago
  The anchored section linked by the OP has the same title.
pico_creator 3y ago
LLMOps / PromptOps, seems to be a new monitoring tool segment which is heating up. And pretty much essential as AI models are constantly being updated "in production" these days. Seeing that you interviewed LangChain, which just launched LangSmith back to back? How do you think this space will turn out - cause I find it very hard to tell how the two of them differ. Additionally, what happens when incumbent monitoring tools start entering the scene (DataDog, Elastic, etc) ?
- swyx 3y ago
  thanks! i address that a bit in the essay - basically langchain will try to build a verticalized solution for langchain, humanloop will be more general but for llms, datadog will be offering the same thing but for existing datadog customers. i think all three groups can win for a long time but when they start competing it will come down to customer personal preference and ability to sell and ship compelling features
  - pico_creator 3y ago
    makes sense, considering there lacks an open std to port back and forth. people stick to datadog / elastic / etc passionately, to avoid "redoing their charts" also low hanging fruit to evals prompt and check (does it JSON or not), is significantly more useful then folks give credit for Q_Q
    - swyx 3y ago
      i would love to help define an open standard with the AIEF...
s9w 3y ago
[dead]

16 points by swyx 3y ago

maest 3y ago

Oh, it's an ad.

swyx 3y ago
this is very cynical and hurts everything i am trying to do with this podcast. humanloop did not pay for this, i invited them on because they are one of the top companies in their space, and thought it might be of interest to HN because of their YC connection + perspective from working extremely closely with OpenAI (many might remember https://news.ycombinator.com/item?id=36177895 https://news.ycombinator.com/item?id=36177895 which we chose not to discuss on the pod but I asked him literally everything else). it hurts when people dismiss my work so casually as it was done in complete good faith and genuine interest in how founders are building in the new AI world.
- onethought 3y ago
  It has an editorialised title, it doesn’t explain what this is really about. It’s basically click bait. It’s either an ad for human loop or an ad for your podcast. It’s not meant to be hurtful, it’s just the truth. If the title was instead more descriptive of what the content was. You wouldn’t get the cynicism.
  - swyx 3y ago
    as mentioned by @htk above - it links to the exact section of the essay, with that exact title, i wanted to offer for HN interest because it directly addresses active conversation topics that have been here before, which i know from myself being a longtime community member.
    - onethought 3y ago
      I see, so anchor tagging a clickbait piece of text within an article makes it not clickbait? Sorry, while I don’t like hurt feelings and I feel bad if that is the case. And I believe you are being genuine. I think it’s still something a bit off here.
      - true_religion 3y ago
        The argument wasn’t over it being clickbait. It was claimed that this is an advertisement (it being paid is implied). The NY Times uses blatant clickbait, but I still respect them as real journalism and not just rewritten PR ad pieces producers.
        onethought 3y ago
        Sorry I don’t draw a distinction here. Clickbaiting is an advertising technique. This was advertising the author/affiliate. The main point isn’t wasn’t about the content (for me). Agree I guess there is a borderline where the title is both compelling to click, but still informative of what the overall article is. That wasn’t the case here. NYT does this better, agree.

swyx 3y ago

Hi HN! proud to present my latest interview with Raza Habib of Humanloop (YCS20). he went viral on the YC youtube a few months ago (https://www.youtube.com/watch?v=hQC5O3WTmuo https://www.youtube.com/watch?v=hQC5O3WTmuo) and in general has been the first and best known LLMOps/PromptOps/Foundation model ops platform out there (having bet on NLP before GPT3 and then ChatGPT blew up). I caught up with him in london and covered the origin story of how they pivoted from an automated labeling product into Humanloop today, and what they are seeing in both the AI Engineering and UK/Europe tech scene vs SF. They also launched their new pricing (free tier) and evals product (which a lot of other startups are trying to build) so we got into that a bit and I chose to highlight the "GPT4 nerfing" debate as that has been a topic of interest here before (https://news.ycombinator.com/item?id=36134249 https://news.ycombinator.com/item?id=36134249). happy to take qtns!

j_crick 3y ago
> They also launched Couldn't not mention that, could ya
- swyx 3y ago
  why is that bad? it literally is what they just did lol

sublinear 3y ago

> please use the original title, unless it is misleading or linkbait; don't editorialize. Original title is: "Building the Foundation Model Ops Platform — with Raza Habib of Humanloop"

htk 3y ago
The anchored section linked by the OP has the same title.

pico_creator 3y ago

LLMOps / PromptOps, seems to be a new monitoring tool segment which is heating up. And pretty much essential as AI models are constantly being updated "in production" these days. Seeing that you interviewed LangChain, which just launched LangSmith back to back? How do you think this space will turn out - cause I find it very hard to tell how the two of them differ. Additionally, what happens when incumbent monitoring tools start entering the scene (DataDog, Elastic, etc) ?

swyx 3y ago
thanks! i address that a bit in the essay - basically langchain will try to build a verticalized solution for langchain, humanloop will be more general but for llms, datadog will be offering the same thing but for existing datadog customers. i think all three groups can win for a long time but when they start competing it will come down to customer personal preference and ability to sell and ship compelling features
- pico_creator 3y ago
  makes sense, considering there lacks an open std to port back and forth. people stick to datadog / elastic / etc passionately, to avoid "redoing their charts" also low hanging fruit to evals prompt and check (does it JSON or not), is significantly more useful then folks give credit for Q_Q
  - swyx 3y ago
    i would love to help define an open standard with the AIEF...

s9w 3y ago

[dead]