7 ms·

Fundamentals of Object Storage in Elixir

122 points by lawik 3y ago

leftsaidtim 3y ago
Oh boy another underjord post. It's always such a joy getting to read his writing.
thibaut_barrere 3y ago
Good write-up! I will add: - make sure to “stream” your transfers in general, instead of keeping whole file blobs in RAM. This to avoid refactoring at the least comfortable moment when a file gets too big in production - it is quite easy to use Mox for isolated testing in general, and it works nicely with ExAws, so make sure to integrate those tests early on
- sph 3y ago
  Seconding the Mox recommendation, it's pretty good and enforces good separation of concerns in your application. It was created by José himself, so you know it's got a well-designed API. You just need to get used with using behaviours, and using the Application config to tell your code when to use the mock vs the real thing. I use it to separate and test external API calls (i.e. to test the payment layer without making any request), to maintain separation between my code and an external library I might want to swap out at a later date, or even between various components of the application. Then writing unit tests becomes a breeze, and you don't need many integration tests. https://hexdocs.pm/mox/Mox.html https://hexdocs.pm/mox/Mox.html
  - sarchertech 3y ago
    I hate Mox with a passion. Because I hate separating out the interface and implementation of something that is only ever going to have one real implementation. The extra layer of indirection just for testing bugs me so much. And I just can’t justify it with “one day I might want to swap out the implementation”. YAGNI and if you do, deal with it then. So much legacy elixir code is an unreadable mess because of all the extra layers of interfaces that only get used for Mox. There’s nothing I dislike more than diving into an unfamiliar section of code and having to keep 15 files open to follow the logic for something that should have been in one module.
    - sitkack 3y ago
      Kinda horribly half serious, but can you use Elixir's Metaprogramming features to demoxify mox? https://www.youtube.com/watch?v=2Bjzml_Hpvk https://www.youtube.com/watch?v=2Bjzml_Hpvk
      - deleted 3y ago
        [deleted]
    - sph 3y ago
      That just means you're putting a boundary when it's not necessary. Mox exists to make tests simpler to write and reason about, if you keep hitting extra layers of indirections then, "you're holding it wrong." Most things are best used sparingly. That said, the vast majority of developers, even senior, have a very limited understanding of testing practices, and tend to create millions of separated, isolated modules, for no reason whatsoever. That's not Mox's fault. A stupid example in practice: a naive blog app does not need Mox at all. If you add SSO sign up, you might want to put a boundary there to unit test without needing to set up a fake auth server, but 99% of your code won't ever touch it, just the authentication tests. Tomorrow you add payments, you might want to put a boundary just before your StripeRestAPI module to test the entire payment flow without contacting Stripe in your unit tests. Again, only the payment module tests ever need to touch Mox, the rest of your code doesn't. You place it before modules in the outer edge that have real-world side effects, not to isolate your functional and mostly pure core. I recommend the DestroyAllSoftware talks, especially https://www.destroyallsoftware.com/talks/boundaries https://www.destroyallsoftware.com/talks/boundaries > And I just can’t justify it with “one day I might want to swap out the implementation”. YAGNI and if you do, deal with it then. I agree with that. Using Mox to swap out implementations is a possible use case, but in practice it's better to hardcode things rather than trying to make it generic and swappable the first time around. Junior devs have a tendency to over-engineer code "because we might need to change it later." My motto is "copy and paste it three times before adding a layer of abstraction"
      - sarchertech 3y ago
        I hope you see the irony in using “you’re holding it wrong” here? If real world use of a library tends to push towards a certain outcome that I dislike, I’m going to tend to dislike that library. You’re also never going to convince me that creating an interface for a single implementation is the best way to facilitate unit testing. Mox is nothing more than a kludge to get around the limitations of the language and I think there are much less invasive kludges available. To be honest when I’m working with devs who are new to elixir I’m embarrassed when I have to explain how Mox works. >but in practice it's better to hardcode things rather than trying to make it generic and swappable the first time around. Junior devs have a tendency to over-engineer code "because we might need to change it later." My motto is "copy and paste it three times before adding a layer of abstraction" This I agree with 100%
SSLy 3y ago
what's up with the swedes and their insistence of using : in lieu of apostrophe?
- bobwaycott 3y ago
  Even in English, the use case you’re referring to would not have an apostrophe—apostrophes are used for contractions and possession, not plurality.
  - penguin_booze 3y ago
    Meet our hero: https://www.bbc.co.uk/news/av/uk-39459831 https://www.bbc.co.uk/news/av/uk-39459831.
- emj 3y ago
  It is often used in swedish technical writing to separate and foreign word or an acronym from the plural so "VPS:es" in english is VPS:erna in swedish. I do not know if this is a swedish tradition, it most probably exists in Germany or the US. I've seen this use in Sweden for several decades.
  - SSLy 3y ago
    I've seen it used by Swedes and Swedish speaking Finns, at least.
- atombender 3y ago
  It's a fairly old tradition specific to Swedish and Finnish. Wikipedia has an explanation. [1] [1] https://en.wikipedia.org/wiki/Colon_(punctuation) https://en.wikipedia.org/wiki/Colon_(punctuation)
- lawik 3y ago
  If I knew what was up with us swedes I'd be consulting on that instead.
losvedir 3y ago
Nice post. One part that raised a red flag for me was: defp bucket!, do: System.fetch_env!("BUCKET_NAME") This will look at that environment variable at build time. For many workflows this doesn't work because you may build the app in CI and run it with different buckets in production and staging. I always default to using Application.get_env/2 and setting the value in runtime.exs, now that that's a thing in recent Elixir. For things where performance matters in a hot loop, it might be worth compiling in the value, but I wouldn't imagine that's the case here where it's a part of a network call to AWS.
- ilikehurdles 3y ago
  This will look at it at runtime not build time. `System.fetch_env!("BUCKET_NAME")` is still in the body of the function and won't get executed until it's called after application start. It is still a good reason to use Application.get_env, and it should be preferred over peppering system environment variable fetches all over the application.
  - losvedir 3y ago
    Oh, of course. Brain fart moment. I pattern matched (ha!) to code I've seen too much where that's at compile time. Yeah, runtime here works but I agree is still not as good as Application.get_env. Aside from (IMO) code clarity reasons, last I checked it is (or can be) much quicker. Application is a quick ETS lookup, while System reaches out to the system and I forget the details but at one point that was much slower.
- thibaut_barrere 3y ago
  As pointed out, it's not compile time but runtime here unless I'm missing anything. And it would be a good reason to build a Credo check exactly for this (I've done a bit of research about this). Any calls to System in regular compile time config (e.g. config/prod/dev/test or included) should be forbidden, same for attribute-level / meta code. I hope we will get this one day!
- lawik 3y ago
  There are a bunch of things that are kind of quick and dirty about these snippets. I think I mentioned that in the post, that these are not best practices. But not quite so ill as compile-time in this case :)
29athrowaway 3y ago
All Elixir projects I've seen, when you run "mix xref" you will see spaghetti.
- josevalim 3y ago
  You are not wrong but you may be hinting at the wrong cause. It is hard to generate a useful graph or summary that gives actionable feedback once you reach a certain amount of nodes. Even mix app.tree, which has no cycles, is barely useful after 30-50 deps because it becomes dense. Using mix xref to evaluate your project will, at best, give you a very blurry picture. Think of it more of a database: it will give you good information when you need to answer certain queries.
Proven 3y ago
[dead]

122 points by lawik 3y ago

Oh boy another underjord post. It's always such a joy getting to read his writing.

Good write-up! I will add: - make sure to “stream” your transfers in general, instead of keeping whole file blobs in RAM. This to avoid refactoring at the least comfortable moment when a file gets too big in production - it is quite easy to use Mox for isolated testing in general, and it works nicely with ExAws, so make sure to integrate those tests early on

sph 3y ago
Seconding the Mox recommendation, it's pretty good and enforces good separation of concerns in your application. It was created by José himself, so you know it's got a well-designed API. You just need to get used with using behaviours, and using the Application config to tell your code when to use the mock vs the real thing. I use it to separate and test external API calls (i.e. to test the payment layer without making any request), to maintain separation between my code and an external library I might want to swap out at a later date, or even between various components of the application. Then writing unit tests becomes a breeze, and you don't need many integration tests. https://hexdocs.pm/mox/Mox.html https://hexdocs.pm/mox/Mox.html
- sarchertech 3y ago
  I hate Mox with a passion. Because I hate separating out the interface and implementation of something that is only ever going to have one real implementation. The extra layer of indirection just for testing bugs me so much. And I just can’t justify it with “one day I might want to swap out the implementation”. YAGNI and if you do, deal with it then. So much legacy elixir code is an unreadable mess because of all the extra layers of interfaces that only get used for Mox. There’s nothing I dislike more than diving into an unfamiliar section of code and having to keep 15 files open to follow the logic for something that should have been in one module.
  - sitkack 3y ago
    Kinda horribly half serious, but can you use Elixir's Metaprogramming features to demoxify mox? https://www.youtube.com/watch?v=2Bjzml_Hpvk https://www.youtube.com/watch?v=2Bjzml_Hpvk
    - deleted 3y ago
      [deleted]
  - sph 3y ago
    That just means you're putting a boundary when it's not necessary. Mox exists to make tests simpler to write and reason about, if you keep hitting extra layers of indirections then, "you're holding it wrong." Most things are best used sparingly. That said, the vast majority of developers, even senior, have a very limited understanding of testing practices, and tend to create millions of separated, isolated modules, for no reason whatsoever. That's not Mox's fault. A stupid example in practice: a naive blog app does not need Mox at all. If you add SSO sign up, you might want to put a boundary there to unit test without needing to set up a fake auth server, but 99% of your code won't ever touch it, just the authentication tests. Tomorrow you add payments, you might want to put a boundary just before your StripeRestAPI module to test the entire payment flow without contacting Stripe in your unit tests. Again, only the payment module tests ever need to touch Mox, the rest of your code doesn't. You place it before modules in the outer edge that have real-world side effects, not to isolate your functional and mostly pure core. I recommend the DestroyAllSoftware talks, especially https://www.destroyallsoftware.com/talks/boundaries https://www.destroyallsoftware.com/talks/boundaries > And I just can’t justify it with “one day I might want to swap out the implementation”. YAGNI and if you do, deal with it then. I agree with that. Using Mox to swap out implementations is a possible use case, but in practice it's better to hardcode things rather than trying to make it generic and swappable the first time around. Junior devs have a tendency to over-engineer code "because we might need to change it later." My motto is "copy and paste it three times before adding a layer of abstraction"
    - sarchertech 3y ago
      I hope you see the irony in using “you’re holding it wrong” here? If real world use of a library tends to push towards a certain outcome that I dislike, I’m going to tend to dislike that library. You’re also never going to convince me that creating an interface for a single implementation is the best way to facilitate unit testing. Mox is nothing more than a kludge to get around the limitations of the language and I think there are much less invasive kludges available. To be honest when I’m working with devs who are new to elixir I’m embarrassed when I have to explain how Mox works. >but in practice it's better to hardcode things rather than trying to make it generic and swappable the first time around. Junior devs have a tendency to over-engineer code "because we might need to change it later." My motto is "copy and paste it three times before adding a layer of abstraction" This I agree with 100%

SSLy 3y ago

what's up with the swedes and their insistence of using : in lieu of apostrophe?

bobwaycott 3y ago
Even in English, the use case you’re referring to would not have an apostrophe—apostrophes are used for contractions and possession, not plurality.
- penguin_booze 3y ago
  Meet our hero: https://www.bbc.co.uk/news/av/uk-39459831 https://www.bbc.co.uk/news/av/uk-39459831.
emj 3y ago
It is often used in swedish technical writing to separate and foreign word or an acronym from the plural so "VPS:es" in english is VPS:erna in swedish. I do not know if this is a swedish tradition, it most probably exists in Germany or the US. I've seen this use in Sweden for several decades.
- SSLy 3y ago
  I've seen it used by Swedes and Swedish speaking Finns, at least.
atombender 3y ago
It's a fairly old tradition specific to Swedish and Finnish. Wikipedia has an explanation. [1] [1] https://en.wikipedia.org/wiki/Colon_(punctuation) https://en.wikipedia.org/wiki/Colon_(punctuation)
lawik 3y ago
If I knew what was up with us swedes I'd be consulting on that instead.

losvedir 3y ago

Nice post. One part that raised a red flag for me was: defp bucket!, do: System.fetch_env!("BUCKET_NAME") This will look at that environment variable at build time. For many workflows this doesn't work because you may build the app in CI and run it with different buckets in production and staging. I always default to using Application.get_env/2 and setting the value in runtime.exs, now that that's a thing in recent Elixir. For things where performance matters in a hot loop, it might be worth compiling in the value, but I wouldn't imagine that's the case here where it's a part of a network call to AWS.

ilikehurdles 3y ago
This will look at it at runtime not build time. `System.fetch_env!("BUCKET_NAME")` is still in the body of the function and won't get executed until it's called after application start. It is still a good reason to use Application.get_env, and it should be preferred over peppering system environment variable fetches all over the application.
- losvedir 3y ago
  Oh, of course. Brain fart moment. I pattern matched (ha!) to code I've seen too much where that's at compile time. Yeah, runtime here works but I agree is still not as good as Application.get_env. Aside from (IMO) code clarity reasons, last I checked it is (or can be) much quicker. Application is a quick ETS lookup, while System reaches out to the system and I forget the details but at one point that was much slower.
thibaut_barrere 3y ago
As pointed out, it's not compile time but runtime here unless I'm missing anything. And it would be a good reason to build a Credo check exactly for this (I've done a bit of research about this). Any calls to System in regular compile time config (e.g. config/prod/dev/test or included) should be forbidden, same for attribute-level / meta code. I hope we will get this one day!
lawik 3y ago
There are a bunch of things that are kind of quick and dirty about these snippets. I think I mentioned that in the post, that these are not best practices. But not quite so ill as compile-time in this case :)

29athrowaway 3y ago

All Elixir projects I've seen, when you run "mix xref" you will see spaghetti.

josevalim 3y ago
You are not wrong but you may be hinting at the wrong cause. It is hard to generate a useful graph or summary that gives actionable feedback once you reach a certain amount of nodes. Even mix app.tree, which has no cycles, is barely useful after 30-50 deps because it becomes dense. Using mix xref to evaluate your project will, at best, give you a very blurry picture. Think of it more of a database: it will give you good information when you need to answer certain queries.

Proven 3y ago

[dead]