4 ms·

When making HTTP requests with Go (or any language) you should at least consider the following: - Setting a User Agent (req.Header.Set(Key, Value)) - Setting

When making HTTP requests with Go (or any language) you should at least consider the following: - Setting a User Agent (req.Header.Set(Key, Value)) - Setting Timeouts (https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/ https://blog.cloudflare.com/the-complete-guide-to-golang-net...) - Avoiding ioutil.ReadAll() and checking body length (or using https://golang.org/pkg/io/#LimitReader https://golang.org/pkg/io/#LimitReader) - Caching DNS lookups (https://github.com/viki-org/dnscache https://github.com/viki-org/dnscache)

tedmiston 10y ago
+ respecting expires or cache-control headers
thatusernametho 10y ago
Quick question: Since Go's html parsing requires utf-8 encoding is there a way to convert the response to utf-8 without ioutil.ReadAll()? (Still pretty new to Go).
- zalmoxes 10y ago
  html.Parse takes a io.Reader https://github.com/golang/net/blob/master/html/parse.go#L2016 https://github.com/golang/net/blob/master/html/parse.go#L201... an http Response.Body is a io.ReadCloser so you can pass it directly with `html.Parse(response.Body)` But don't forget to defer io.Close the body.
  - thatusernametho 10y ago
    I guess I wasn't specific enough. I'd like to stream the response so as to use less memory. I also need to ensure the body is utf-8 encoded because that makes Go happy. https://mionskowski.pl/html-golang-stream-processing https://mionskowski.pl/html-golang-stream-processing
niftich 10y ago
If only there were libraries that encapsulate best-practice plumbing and glue logic and offer a high-level API with sane defaults! A perusal of the awesome-go list [1] (allegedly, a "curated list", which may merit a debate) names a few: https://github.com/parnurzeal/gorequest https://github.com/parnurzeal/gorequest https://github.com/smallnest/goreq https://github.com/smallnest/goreq https://github.com/franela/goreq https://github.com/franela/goreq https://github.com/sethgrid/pester https://github.com/sethgrid/pester https://github.com/mozillazg/request https://github.com/mozillazg/request https://github.com/levigross/grequests https://github.com/levigross/grequests https://github.com/go-resty/resty https://github.com/go-resty/resty https://github.com/dghubble/sling https://github.com/dghubble/sling [1] https://github.com/avelino/awesome-go https://github.com/avelino/awesome-go
anonfunction 10y ago
I would also add retries to the list (https://github.com/sethgrid/pester https://github.com/sethgrid/pester)
piotrkubisa 10y ago
Thank you very much for your comment - I've realized that caching DNS should be considered in any "connected" service. I would add my two cents: - Calculate checksum and compare it with the latest entry (storing single 32/40-bytes long checksum takes lower amount of memory). - Respect various HTTP headers like Expires, Cache-control, Accept-Encoding (accepting compressed content may relieve strain on bandwidth), Content-length, Cookie-length etc. - If you were downloading a web-page and sending it via e-mail (for later usage?) you may gzip it - it may reduce costs of attachments or S3. - Attach header "User-Agent" and "From" (with an e-mail address to contact developer) - if you don't mind. - As you wrote, avoiding ioutil.ReadAll should be a good practice - Datadog published a blogpost which is worth reading - https://www.datadoghq.com/blog/crossing-streams-love-letter-gos-io-reader/ https://www.datadoghq.com/blog/crossing-streams-love-letter-... - Use logrotate or implement it on your own with i.e. Mutex. Eventually you use fluentd, Kafka, Pub/Sub pipeline or some SaaS offer specialized in storing logs. Most (or all) of Go packages with "request" in the name that I know doesn't cover aforementioned scenarios automatically - it is up to developer.
lobster_johnson 10y ago
Wouldn't it be more unixy to run a caching DNS resolver such as nscd locally on the machine, instead of building it into every app? (I would accept an answer such as "there's no caching resolver out there that properly handles thundering herd situations".)
lobster_johnson 10y ago
To avoid accidentally fetching a huge resource into RAM and exploding in a rain of fire, automatically spooling large bodies to disk is a good idea (as opposed to relying on swap, which is rarely a good idea). One could use a fixed threshold to avoid spooling smaller files.
spraak 10y ago
Do you mean > Avoiding ioutil.ReadAll AND > checking body length or do you mean, > Avoid ioutil.ReadAll() > and [instead you should be] checking the body length ? Do you have an example of that, or an example of LimitReader?
- deleted 10y ago
  [deleted]
- piotrkubisa 10y ago
  You made me curious, so I've prepared a simple benchmark - https://gist.github.com/piotrkubisa/8ccc308086378da7d2f6c0d550aded33 https://gist.github.com/piotrkubisa/8ccc308086378da7d2f6c0d5...
  - spraak 10y ago
    That's super awesome, thank you

tedmiston 10y ago

+ respecting expires or cache-control headers

thatusernametho 10y ago

Quick question: Since Go's html parsing requires utf-8 encoding is there a way to convert the response to utf-8 without ioutil.ReadAll()? (Still pretty new to Go).

zalmoxes 10y ago
html.Parse takes a io.Reader https://github.com/golang/net/blob/master/html/parse.go#L2016 https://github.com/golang/net/blob/master/html/parse.go#L201... an http Response.Body is a io.ReadCloser so you can pass it directly with `html.Parse(response.Body)` But don't forget to defer io.Close the body.
- thatusernametho 10y ago
  I guess I wasn't specific enough. I'd like to stream the response so as to use less memory. I also need to ensure the body is utf-8 encoded because that makes Go happy. https://mionskowski.pl/html-golang-stream-processing https://mionskowski.pl/html-golang-stream-processing

niftich 10y ago

If only there were libraries that encapsulate best-practice plumbing and glue logic and offer a high-level API with sane defaults! A perusal of the awesome-go list [1] (allegedly, a "curated list", which may merit a debate) names a few: https://github.com/parnurzeal/gorequest https://github.com/parnurzeal/gorequest https://github.com/smallnest/goreq https://github.com/smallnest/goreq https://github.com/franela/goreq https://github.com/franela/goreq https://github.com/sethgrid/pester https://github.com/sethgrid/pester https://github.com/mozillazg/request https://github.com/mozillazg/request https://github.com/levigross/grequests https://github.com/levigross/grequests https://github.com/go-resty/resty https://github.com/go-resty/resty https://github.com/dghubble/sling https://github.com/dghubble/sling [1] https://github.com/avelino/awesome-go https://github.com/avelino/awesome-go

anonfunction 10y ago

I would also add retries to the list (https://github.com/sethgrid/pester https://github.com/sethgrid/pester)

piotrkubisa 10y ago

Thank you very much for your comment - I've realized that caching DNS should be considered in any "connected" service. I would add my two cents: - Calculate checksum and compare it with the latest entry (storing single 32/40-bytes long checksum takes lower amount of memory). - Respect various HTTP headers like Expires, Cache-control, Accept-Encoding (accepting compressed content may relieve strain on bandwidth), Content-length, Cookie-length etc. - If you were downloading a web-page and sending it via e-mail (for later usage?) you may gzip it - it may reduce costs of attachments or S3. - Attach header "User-Agent" and "From" (with an e-mail address to contact developer) - if you don't mind. - As you wrote, avoiding ioutil.ReadAll should be a good practice - Datadog published a blogpost which is worth reading - https://www.datadoghq.com/blog/crossing-streams-love-letter-gos-io-reader/ https://www.datadoghq.com/blog/crossing-streams-love-letter-... - Use logrotate or implement it on your own with i.e. Mutex. Eventually you use fluentd, Kafka, Pub/Sub pipeline or some SaaS offer specialized in storing logs. Most (or all) of Go packages with "request" in the name that I know doesn't cover aforementioned scenarios automatically - it is up to developer.

lobster_johnson 10y ago

Wouldn't it be more unixy to run a caching DNS resolver such as nscd locally on the machine, instead of building it into every app? (I would accept an answer such as "there's no caching resolver out there that properly handles thundering herd situations".)

lobster_johnson 10y ago

To avoid accidentally fetching a huge resource into RAM and exploding in a rain of fire, automatically spooling large bodies to disk is a good idea (as opposed to relying on swap, which is rarely a good idea). One could use a fixed threshold to avoid spooling smaller files.

spraak 10y ago

Do you mean > Avoiding ioutil.ReadAll AND > checking body length or do you mean, > Avoid ioutil.ReadAll() > and [instead you should be] checking the body length ? Do you have an example of that, or an example of LimitReader?

deleted 10y ago
[deleted]
piotrkubisa 10y ago
You made me curious, so I've prepared a simple benchmark - https://gist.github.com/piotrkubisa/8ccc308086378da7d2f6c0d550aded33 https://gist.github.com/piotrkubisa/8ccc308086378da7d2f6c0d5...
- spraak 10y ago
  That's super awesome, thank you