7 ms·
I recently used Playwright for Python [0] and pypandoc [1] to build a scraper that fetches a webpage and turns the content into sane markdown so that it can be
by anotherpaulg 3y ago
I recently used Playwright for Python [0] and pypandoc [1] to build a scraper that fetches a webpage and turns the content into sane markdown so that it can be passed into an AI coding chat [2].
They are both powerful yet pragmatic dependencies to add to a project. I really like that both packages contain wheels or scriptable methods to install their underlying platform-specific binary dependencies. This means you don't need to ask end users to figure out some complex, platform-specific package manager to install playwright and pandoc.
Playwright let's you scrape pages that rely on js. Pandoc is great at turning HTML into sensible markdown.
For example, below is an excerpt of the openai pricing docs [3] that have been scraped to markdown [4] in this manner.
[0] https://playwright.dev/python/docs/intro https://playwright.dev/python/docs/intro
[1] https://github.com/JessicaTegner/pypandoc https://github.com/JessicaTegner/pypandoc
[2] https://github.com/paul-gauthier/aider https://github.com/paul-gauthier/aider
[3] https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turb...
[4] https://gist.githubusercontent.com/paul-gauthier/95a1434a28d9e5c7b62c4e1539db9955/raw/589efa9498203776f9b40797817022c8ed82a409/tmp.pricing.md https://gist.githubusercontent.com/paul-gauthier/95a1434a28d...
## GPT-4 and GPT-4 Turbo
GPT-4 is a large multimodal model (accepting text or image inputs and
outputting text) that can solve difficult problems with greater accuracy
than any of our previous models, thanks to its broader general knowledge
and advanced reasoning capabilities. GPT-4 is available in the OpenAI
API to [paying
customers](https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4).
Like `gpt-3.5-turbo`, GPT-4 is optimized for chat but works well for
traditional completions tasks using the [Chat Completions
API](/docs/api-reference/chat). Learn how to use GPT-4 in our [text
generation guide](/docs/guides/text-generation).
+-----------------+-----------------+-----------------+-----------------+
| Model | Description | Context window | Training data |
+=================+=================+=================+=================+
| gpt | | 128,000 tokens | Up to Dec 2023 |
| -4-0125-preview | | | |
| | New | | |
| | | | |
| | | | |
| | | | |
| | **GPT-4 | | |
| | Turbo**\ | | |
| | The latest | | |
| | GPT-4 model | | |
| | intended to | | |
| | reduce cases of | | |
| | "laziness" | | |
| | where the model | | |
| | doesn't | | |
| | complete a | | |
| | task. Returns a | | |
| | maximum of | | |
| | 4,096 output | | |
| | tokens. [Learn | | |
| | more](ht | | |
| | tps://openai.co | | |
| | m/blog/new-embe | | |
| | dding-models-an | | |
| | d-api-updates). | | |
+-----------------+-----------------+-----------------+-----------------+
| gpt- | Currently | 128,000 tokens | Up to Dec 2023 |
| 4-turbo-preview | points to | | |
| | `gpt-4 | | |
| | -0125-preview`. | | |
+-----------------+-----------------+-----------------+-----------------+
...