7 ms·

How to implement a disk cache plugin for Elixir's Req HTTP client?

38 3y ago
Req.get!("https://hex.pm/api/packages/req").body["meta"]["description"] wow. coming from Go I have to say I do not like this. no error checking at all (I assume it just panics or exception?) no mention of JSON at all, even though thats what its doing. and if "body" is JSON, how do you even get the raw body, or can you? just seems over engineered/over fitted whatever you want to call it.
- jolux 3y ago
  > no error checking at all First of all: this example is a one-off script, so error handling is not as much of a concern (you'll notice the first expression installs packages, this is not how production apps should handle dependencies). Second, functions that raise always end in `!` in Elixir, or at least they are supposed to. Most have alternatives that return error tuples instead which you can pattern match on (this is what I recommend). You'll hear "let it crash" a lot as it applies to Erlang/OTP programming (and by extension Elixir), and this is a good high-level organizing principle. You shouldn't go out of your way to program against the unknown in Elixir. However, HTTP requests failing is common enough that in my opinion at least you really should just check the error. You can read the docs for `get/2` (as opposed to `get!/2` which raises) here: https://hexdocs.pm/req/Req.html#get/2 https://hexdocs.pm/req/Req.html#get/2. A common pattern is for the `!` version to call the version that doesn't raise, check the result, and raise on error, which is the case here: https://github.com/wojtekmach/req/blob/9de30de0df481ee557ccce230ab7a047917cab8a/lib/req.ex#L377-L380 https://github.com/wojtekmach/req/blob/9de30de0df481ee557ccc... > and if "body" is JSON, how do you even get the raw body, or can you? You would set `decode: false` when calling `get!/2: https://hexdocs.pm/req/Req.html#new/1 https://hexdocs.pm/req/Req.html#new/1. You can also set this as configuration with https://hexdocs.pm/req/Req.html#default_options/1 https://hexdocs.pm/req/Req.html#default_options/1. As a closing note I'll mention that Req is intended to be a very high-level, scripting-friendly requests library, similar to Requests in Python (the author also mentions in the OP that his use case is scripting). If you don't want conveniences like Req provides, you can either turn them off or use something different, like Finch (which Req is based on, https://github.com/sneako/finch https://github.com/sneako/finch). Other than Req and Finch I'm personally only familiar with HTTPoison, which is significantly older than all of the libraries derived from Mint (like Finch and Req, https://github.com/elixir-mint/mint https://github.com/elixir-mint/mint) but is very popular and still works. There are many others though, like Gun and Tesla and such. I just think it's important for folks to know that this level of "magic" is far from the default in Elixir, but it's there if you want it.
- strken 3y ago
  From memory https://wiki.c2.com/?LetItCrash https://wiki.c2.com/?LetItCrash is a big part of Erlang's (and therefore Elixir's) philosophy. You put everything in a process, monitor it with a supervisor, and just crash and let the supervisor recover unless the error is immediately recoverable in the calling code, e.g. trying a second provider or skipping the cache or something.
  - metadat 3y ago
    How do you report on / bubble up what the precise nature of the problem was under this paradigm? E.g. Was it a network issue, content type issue, malformed JSON, unexpected JSON format, etc? I mostly enjoy Go, Python, and Java (in this order), but I must say the self-contained, self-dependency-installing script thing in TFA seems pretty appealing.
    - strken 3y ago
      It's been a while since I've touched Phoenix, but from memory it uses separate process pools (meaning lightweight BEAM VM processes) for the http connection, application code, database connections, and probably any other external dependency. A crash in an application process is handled by the http process that sent the work to it. That being said, you return errors in a similar way to other languages if they're expected during the system's operation rather than crashes. Again it's been a while since I had to work with it.
    - sb8244 3y ago
      People often take this paradigm too literally, especially if they have not programmed much in Elixir. It is most common to bubble up errors using error tuples, and exceptions are available. Fred Hebert's blog post called "it's about the guarantees" makes this more clear: https://ferd.ca/it-s-about-the-guarantees.html https://ferd.ca/it-s-about-the-guarantees.html
      - throwawaymaths 3y ago
        It's actually a pet peeve of mine that people use error tuples so often in elixir. Honestly most of the time the correct thing is to raise. There really isn't anything you can do anyways, and avoiding the temptation to do something about it and make spaghetti out of the decision tree is worth it.
- metadat 3y ago
  It's actually not just Go, most procedural languages have similar HTTP request flows (e.g. Java, Python, C++, TCL). Cool GitHub, btw, though I couldn't help but notice you have only Go repositories, which I found amusing (never seen a profile quite like yours before!). Cheers.
- ricketycricket 3y ago
  > no error checking at all (I assume it just panics or exception?) In Elixir, bang functions per convention will raise on error. `get/2` will return error tuples allowing you to handle errors. In fact, get!/2 just calls get/2 and raises for you[^1]. > no mention of JSON at all Req is the most "batteries included" Elixir HTTP lib out there. I can't speak for Wojtek, but I believe the goal was to make Req extremely easy to use in scripting or things like LiveBook without having to do much work. That being said, the automatic decoding is mentioned in the readme[^2] and the docs[^3]. > if "body" is JSON, how do you even get the raw body, or can you? Per the docs[^3], you can either skip with the `:decode_body` option, or just build your own request using only the steps you want. > just seems over engineered/over fitted whatever you want to call it. Fair, but again, this library is designed to be on that end of the spectrum. There are plenty of other libraries further down the stack that you can use. I am partial to Finch[^4], upon which Req is built. To address the sibling comment about "Let it Crash", the language allows you to easily recover from crashes, but that is for resiliency, not error handling. In practice you would use the non-bang get/2, pattern match on the response, handle any errors, perhaps use Kernel.get_in/2 to safely traverse the map, etc. The example provided by the author is not "production ready". [^1]: https://github.com/wojtekmach/req/blob/v0.3.11/lib/req.ex#L376 https://github.com/wojtekmach/req/blob/v0.3.11/lib/req.ex#L3... [^2]: https://github.com/wojtekmach/req https://github.com/wojtekmach/req [^3]: https://hexdocs.pm/req/Req.Steps.html#decode_body/1 https://hexdocs.pm/req/Req.Steps.html#decode_body/1 [^4]: https://github.com/sneako/finch https://github.com/sneako/finch
  - thibaut_barrere 3y ago
    Thanks for detailing while I was asleep :-) Indeed, that + pattern matching does error checking just right. I must admit that the automatic JSON decoding initially turned me off a bit!
- antran22 3y ago
  It's a piece of example code to demo a concept. You are asking people to error check a something akin to a curl call in a bash command.
- deleted 3y ago
  [deleted]
- nesarkvechnep 3y ago
  Mind that this is not an idiomatic code. One would usually use pattern matching. case Req.get!("https://hex.pm/api/packages/req") do %{body: %{"meta" => %{"description" => desc}}} -> # use the value of desc here %{body: body} -> # report for unexpected body end As others mentioned, the ! at the end of the function name tells that the user should expect an exception.
  - jolux 3y ago
    That depends. I think that's a bit deep for pattern matching personally but I've evolved on this over time.
    - nesarkvechnep 3y ago
      Who decides when pattern matching is a bit deep?
      - tentacleuno 3y ago
        Well, the person you're replying to did, to be fair.
  - thibaut_barrere 3y ago
    I would not say this is not idiomatic - it depends on the situation. On production code or scripts where I want to be sure (which is the case mostly), I would indeed use something like: %{status: 200, body: json} = Req.get!(".....") And then later restructure as needed. Sometimes it would be: meta = json |> Map.fetch!("meta") |> Map.fetch!("description") Or pattern matching as you did!

38 3y ago

Req.get!("https://hex.pm/api/packages/req").body["meta"]["description"] wow. coming from Go I have to say I do not like this. no error checking at all (I assume it just panics or exception?) no mention of JSON at all, even though thats what its doing. and if "body" is JSON, how do you even get the raw body, or can you? just seems over engineered/over fitted whatever you want to call it.

jolux 3y ago
> no error checking at all First of all: this example is a one-off script, so error handling is not as much of a concern (you'll notice the first expression installs packages, this is not how production apps should handle dependencies). Second, functions that raise always end in `!` in Elixir, or at least they are supposed to. Most have alternatives that return error tuples instead which you can pattern match on (this is what I recommend). You'll hear "let it crash" a lot as it applies to Erlang/OTP programming (and by extension Elixir), and this is a good high-level organizing principle. You shouldn't go out of your way to program against the unknown in Elixir. However, HTTP requests failing is common enough that in my opinion at least you really should just check the error. You can read the docs for `get/2` (as opposed to `get!/2` which raises) here: https://hexdocs.pm/req/Req.html#get/2 https://hexdocs.pm/req/Req.html#get/2. A common pattern is for the `!` version to call the version that doesn't raise, check the result, and raise on error, which is the case here: https://github.com/wojtekmach/req/blob/9de30de0df481ee557ccce230ab7a047917cab8a/lib/req.ex#L377-L380 https://github.com/wojtekmach/req/blob/9de30de0df481ee557ccc... > and if "body" is JSON, how do you even get the raw body, or can you? You would set `decode: false` when calling `get!/2: https://hexdocs.pm/req/Req.html#new/1 https://hexdocs.pm/req/Req.html#new/1. You can also set this as configuration with https://hexdocs.pm/req/Req.html#default_options/1 https://hexdocs.pm/req/Req.html#default_options/1. As a closing note I'll mention that Req is intended to be a very high-level, scripting-friendly requests library, similar to Requests in Python (the author also mentions in the OP that his use case is scripting). If you don't want conveniences like Req provides, you can either turn them off or use something different, like Finch (which Req is based on, https://github.com/sneako/finch https://github.com/sneako/finch). Other than Req and Finch I'm personally only familiar with HTTPoison, which is significantly older than all of the libraries derived from Mint (like Finch and Req, https://github.com/elixir-mint/mint https://github.com/elixir-mint/mint) but is very popular and still works. There are many others though, like Gun and Tesla and such. I just think it's important for folks to know that this level of "magic" is far from the default in Elixir, but it's there if you want it.
strken 3y ago
From memory https://wiki.c2.com/?LetItCrash https://wiki.c2.com/?LetItCrash is a big part of Erlang's (and therefore Elixir's) philosophy. You put everything in a process, monitor it with a supervisor, and just crash and let the supervisor recover unless the error is immediately recoverable in the calling code, e.g. trying a second provider or skipping the cache or something.
- metadat 3y ago
  How do you report on / bubble up what the precise nature of the problem was under this paradigm? E.g. Was it a network issue, content type issue, malformed JSON, unexpected JSON format, etc? I mostly enjoy Go, Python, and Java (in this order), but I must say the self-contained, self-dependency-installing script thing in TFA seems pretty appealing.
  - strken 3y ago
    It's been a while since I've touched Phoenix, but from memory it uses separate process pools (meaning lightweight BEAM VM processes) for the http connection, application code, database connections, and probably any other external dependency. A crash in an application process is handled by the http process that sent the work to it. That being said, you return errors in a similar way to other languages if they're expected during the system's operation rather than crashes. Again it's been a while since I had to work with it.
  - sb8244 3y ago
    People often take this paradigm too literally, especially if they have not programmed much in Elixir. It is most common to bubble up errors using error tuples, and exceptions are available. Fred Hebert's blog post called "it's about the guarantees" makes this more clear: https://ferd.ca/it-s-about-the-guarantees.html https://ferd.ca/it-s-about-the-guarantees.html
    - throwawaymaths 3y ago
      It's actually a pet peeve of mine that people use error tuples so often in elixir. Honestly most of the time the correct thing is to raise. There really isn't anything you can do anyways, and avoiding the temptation to do something about it and make spaghetti out of the decision tree is worth it.
metadat 3y ago
It's actually not just Go, most procedural languages have similar HTTP request flows (e.g. Java, Python, C++, TCL). Cool GitHub, btw, though I couldn't help but notice you have only Go repositories, which I found amusing (never seen a profile quite like yours before!). Cheers.
ricketycricket 3y ago
> no error checking at all (I assume it just panics or exception?) In Elixir, bang functions per convention will raise on error. `get/2` will return error tuples allowing you to handle errors. In fact, get!/2 just calls get/2 and raises for you[^1]. > no mention of JSON at all Req is the most "batteries included" Elixir HTTP lib out there. I can't speak for Wojtek, but I believe the goal was to make Req extremely easy to use in scripting or things like LiveBook without having to do much work. That being said, the automatic decoding is mentioned in the readme[^2] and the docs[^3]. > if "body" is JSON, how do you even get the raw body, or can you? Per the docs[^3], you can either skip with the `:decode_body` option, or just build your own request using only the steps you want. > just seems over engineered/over fitted whatever you want to call it. Fair, but again, this library is designed to be on that end of the spectrum. There are plenty of other libraries further down the stack that you can use. I am partial to Finch[^4], upon which Req is built. To address the sibling comment about "Let it Crash", the language allows you to easily recover from crashes, but that is for resiliency, not error handling. In practice you would use the non-bang get/2, pattern match on the response, handle any errors, perhaps use Kernel.get_in/2 to safely traverse the map, etc. The example provided by the author is not "production ready". [^1]: https://github.com/wojtekmach/req/blob/v0.3.11/lib/req.ex#L376 https://github.com/wojtekmach/req/blob/v0.3.11/lib/req.ex#L3... [^2]: https://github.com/wojtekmach/req https://github.com/wojtekmach/req [^3]: https://hexdocs.pm/req/Req.Steps.html#decode_body/1 https://hexdocs.pm/req/Req.Steps.html#decode_body/1 [^4]: https://github.com/sneako/finch https://github.com/sneako/finch
- thibaut_barrere 3y ago
  Thanks for detailing while I was asleep :-) Indeed, that + pattern matching does error checking just right. I must admit that the automatic JSON decoding initially turned me off a bit!
antran22 3y ago
It's a piece of example code to demo a concept. You are asking people to error check a something akin to a curl call in a bash command.
deleted 3y ago
[deleted]
nesarkvechnep 3y ago
Mind that this is not an idiomatic code. One would usually use pattern matching. case Req.get!("https://hex.pm/api/packages/req") do %{body: %{"meta" => %{"description" => desc}}} -> # use the value of desc here %{body: body} -> # report for unexpected body end As others mentioned, the ! at the end of the function name tells that the user should expect an exception.
- jolux 3y ago
  That depends. I think that's a bit deep for pattern matching personally but I've evolved on this over time.
  - nesarkvechnep 3y ago
    Who decides when pattern matching is a bit deep?
    - tentacleuno 3y ago
      Well, the person you're replying to did, to be fair.
- thibaut_barrere 3y ago
  I would not say this is not idiomatic - it depends on the situation. On production code or scripts where I want to be sure (which is the case mostly), I would indeed use something like: %{status: 200, body: json} = Req.get!(".....") And then later restructure as needed. Sometimes it would be: meta = json |> Map.fetch!("meta") |> Map.fetch!("description") Or pattern matching as you did!