6 ms·

Django 3.2 – News on compressed fixtures and fixtures compression

86 points by pauloxnet 6y ago

theptip 6y ago
Is anyone making extensive use of static fixtures? If so what do you like about them vs. using setUpTestData on dynamically generated models (optionally using something like FactoryBoy to make your fixture code more flexible)? I found static fixtures to be brittle and hard-to-maintain, because you need to manually edit them, or have code to regenerate them, every time your schema changes. Recently I've been tempted to revisit them and build out a manage.py command to run the domain scenarios that generate these fixtures (to make it easier to keep them up to date), since when testing DB migrations you often actually do want to have stale/old-schema DB state, instead of the default of using the post-migration code to generate your test fixtures. But I'm still not sure I'd prefer them in many tests outside of the migration-testing usecase.
- danpalmer 6y ago
  We use them for tests where we a) need representative data that has some sort of interesting format/dependencies that could be tricky (or slow?) to capture in factories, or that we need enough of that writing out ~100s of factories would be a bit nasty to maintain. I don't think we'd miss them a lot if we didn't have them for this, but they are handy for this. We use setUpTestData and factories or models directly for most of our testing. That said, I've just written a new tool for exporting a subset of production data for local use (we have a 2TB database but want a <5GB realistic looking dev database, we export a subset, anonymise, maintain FK integrity, install a bunch of Factories for special test cases). I initially started with loaddada/dumpdata but found them inflexible so I'm using the same de/serialisers with custom code around them. That has been somewhat useful, and if that's the level that this compression code is written at I'll probably use it.
  - windyfriendy 6y ago
    > writing out ~100s of factories would be a bit nasty to maintain I've found that I really like Model Bakery (https://model-bakery.readthedocs.io/en/latest/ https://model-bakery.readthedocs.io/en/latest/) a lot more than Factory Boy for this reason - no factories to maintain. It's very intuitive and easy to use.
- woile 6y ago
  In a small project I had, I was using fixtures to populate the local dev environment. ``` make dev/init ## runs compose in dev mode make dev/populate ## initializes data ``` would give you an "initialized" data, ready to start working on. When the project has some data in it, I consider it a closer reflection of prod. If you can (data is not sensible), you can update the fixtures with data from prod.
  - jlg23 6y ago
    > If you can (data is not sensible), you can update the fixtures with data from prod. Just make sure all devs use the same days from prod, then. I once was consulting for a dysfunctional team and everyone having different test data drawn from prod made reproducing even easy bugs hard and impossible for more complicated bugs.
- brianwawok 6y ago
  Not extensive, it's kind of a last ditch then when I can't generate random data. For example, I have some "integration" tests that I spin through and generate a huge pile of objects, and validate them against XSDs, to make sure the XML I am making is valid. I don't really want to factory boy this, I just want a smoke screen to make sure things are sane.. and as XSD changes, I can re-run the test and catch weird stuff quick. But this is like... < 1% of my total tests.
- tingletech 6y ago
  I've used them in production before inside of AWS beanstalk (.ebextensions). Was pretty much a static django site on top of a solr index. There was a hidden django where the content team would work on the content. When it was ready, we would dumpdata on the backend server and upload to S3. New beanstalks would grab the file from S3 and loaddata.
- michaelpb 6y ago
  I use Django test fixtures in one way! I have a series of E2E tests with Splinter [0] that go through setting up a variety of things from scratch and testing core features. Instead of having really long, complicated test that goes through the steps, I instead wrote a little helper function that "freezes" the DB after each step as a JSON fixture (which is then loaded by the next test), to allow me to break it down into many smaller tests that "start" from various reasonable database states. Works pretty well! I don't use them however for unit tests or any non-E2E tests. I just use helper factory functions with hardcoded test data. [0] https://splinter.readthedocs.io/en/latest/ https://splinter.readthedocs.io/en/latest/
zdw 6y ago
Looks like .gz is basically a free lunch in terms of CPU/memory impact with a decent size savings.
- jonstewart 6y ago
  I was surprised to see xz and bzip2, but not LZ4 or Snappy.
  - minusf 6y ago
    the algorithms chosen are a mix of "most common" and what comes out of the box with python's standard library to avoid extra dependencies.
    - pauloxnet 6y ago
      Exactly. I replied also before reading you answer.
  - pauloxnet 6y ago
    The Django code use only compression algorithms already included in the Python standard library to avoid unnecessary dependencies.
- pauloxnet 6y ago
  You're totally right. But you can choose also to use xz or bz2 for smaller files if you have spare cpu/memory.
CraigJPerry 6y ago
If it wasn't for Haskell's IHP framework then Django, Ruby on Rails and CakePHP would be fighting to come dead last in the techempower composite benchmark https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=composite https://www.techempower.com/benchmarks/#section=data-r20&hw=... Django, RoR - they're almost 2 orders of magnitude behind the fastest frameworks on the composite test and they're an order of magnitude behind the majority of Java / Go - i was going to say C# here but i see asp.net is much faster than the surprisingly homogenous performing "usual suspects" of enterprise web frameworks. And yet, if i had to launch a site tomorrow, it'd be Django i'd reach for - and with zero hesitation. Instagram, Youtube, Github, Facebook, Dropbox etc. etc. all launched on these "slowest of the slow" stacks and then optimised. Did anyone start out on a lithium (c++) or actix (rust) or other incredibly high performance stack? This is all a big diversion anyway, I think these benchmarks measure the wrong thing entirely. Instead - show me some measure of how easy & cheap it will be for me to change or add a behaviour to my site 3 years down the road if i've stuck to the prescribed idiomatic approach.
- brianwawok 6y ago
  Some of it may be B2B vs B2C, and volume of usage. I am B2B. My server costs are like 1% of my revenue using Django. Okay, I could shrink this by 95% by using raw C.. but who cares? Now if I were an image hosting CDN and Django bill was 50% of my revenue.. it might be a different story. I think it's the classic right tool for the right job story..
  - kasey_junk 6y ago
    You are a veteran performance software engineer, besides your entrepreneurial bona fides. What is the number you start migrating off django? What is the number you wished you’d picked something else? Is there a value to Django you could have got with a different Lang without the downside?
- travisjungroth 6y ago
  I’m tempted to go FastAPI over Django on my next project, even though I’ve been a big Django user. A little more performance and a little less coupling, and maybe you can survive longer as a Python monolith. Then convert to Rust/Go/something in the unlikely event of wild success.
ctur 6y ago
Not including Zstandard is a lost opportunity -- it is far more appropriate than XZ or LZMA for real-time applications like serving web pages. Should be easy to add, though, and likely would be a very nice compromise between space, memory, and speed (including tuning various levels of Zstandard compression). gzip is basically never worth using anymore except for backwards compatibility; zstd is both faster and more space efficient. lz4 similarly is worth adding for utmost speed but I tend to think zstd is better in practice (particularly when the control plane is Python where perf is somewhat limited anyway).
- pauloxnet 6y ago
  > Not including Zstandard is a lost opportunity -- it is far more appropriate than XZ or LZMA for real-time applications like serving web pages. The article is about offline creation of compressed data dumps and not about real-time application. The Django code here use only compression algorithms already present in the Python standard library to avoid unnecessary dependencies.

86 points by pauloxnet 6y ago

Is anyone making extensive use of static fixtures? If so what do you like about them vs. using setUpTestData on dynamically generated models (optionally using something like FactoryBoy to make your fixture code more flexible)? I found static fixtures to be brittle and hard-to-maintain, because you need to manually edit them, or have code to regenerate them, every time your schema changes. Recently I've been tempted to revisit them and build out a manage.py command to run the domain scenarios that generate these fixtures (to make it easier to keep them up to date), since when testing DB migrations you often actually do want to have stale/old-schema DB state, instead of the default of using the post-migration code to generate your test fixtures. But I'm still not sure I'd prefer them in many tests outside of the migration-testing usecase.

danpalmer 6y ago
We use them for tests where we a) need representative data that has some sort of interesting format/dependencies that could be tricky (or slow?) to capture in factories, or that we need enough of that writing out ~100s of factories would be a bit nasty to maintain. I don't think we'd miss them a lot if we didn't have them for this, but they are handy for this. We use setUpTestData and factories or models directly for most of our testing. That said, I've just written a new tool for exporting a subset of production data for local use (we have a 2TB database but want a <5GB realistic looking dev database, we export a subset, anonymise, maintain FK integrity, install a bunch of Factories for special test cases). I initially started with loaddada/dumpdata but found them inflexible so I'm using the same de/serialisers with custom code around them. That has been somewhat useful, and if that's the level that this compression code is written at I'll probably use it.
- windyfriendy 6y ago
  > writing out ~100s of factories would be a bit nasty to maintain I've found that I really like Model Bakery (https://model-bakery.readthedocs.io/en/latest/ https://model-bakery.readthedocs.io/en/latest/) a lot more than Factory Boy for this reason - no factories to maintain. It's very intuitive and easy to use.
woile 6y ago
In a small project I had, I was using fixtures to populate the local dev environment. ``` make dev/init ## runs compose in dev mode make dev/populate ## initializes data ``` would give you an "initialized" data, ready to start working on. When the project has some data in it, I consider it a closer reflection of prod. If you can (data is not sensible), you can update the fixtures with data from prod.
- jlg23 6y ago
  > If you can (data is not sensible), you can update the fixtures with data from prod. Just make sure all devs use the same days from prod, then. I once was consulting for a dysfunctional team and everyone having different test data drawn from prod made reproducing even easy bugs hard and impossible for more complicated bugs.
brianwawok 6y ago
Not extensive, it's kind of a last ditch then when I can't generate random data. For example, I have some "integration" tests that I spin through and generate a huge pile of objects, and validate them against XSDs, to make sure the XML I am making is valid. I don't really want to factory boy this, I just want a smoke screen to make sure things are sane.. and as XSD changes, I can re-run the test and catch weird stuff quick. But this is like... < 1% of my total tests.
tingletech 6y ago
I've used them in production before inside of AWS beanstalk (.ebextensions). Was pretty much a static django site on top of a solr index. There was a hidden django where the content team would work on the content. When it was ready, we would dumpdata on the backend server and upload to S3. New beanstalks would grab the file from S3 and loaddata.
michaelpb 6y ago
I use Django test fixtures in one way! I have a series of E2E tests with Splinter [0] that go through setting up a variety of things from scratch and testing core features. Instead of having really long, complicated test that goes through the steps, I instead wrote a little helper function that "freezes" the DB after each step as a JSON fixture (which is then loaded by the next test), to allow me to break it down into many smaller tests that "start" from various reasonable database states. Works pretty well! I don't use them however for unit tests or any non-E2E tests. I just use helper factory functions with hardcoded test data. [0] https://splinter.readthedocs.io/en/latest/ https://splinter.readthedocs.io/en/latest/

zdw 6y ago

Looks like .gz is basically a free lunch in terms of CPU/memory impact with a decent size savings.

jonstewart 6y ago
I was surprised to see xz and bzip2, but not LZ4 or Snappy.
- minusf 6y ago
  the algorithms chosen are a mix of "most common" and what comes out of the box with python's standard library to avoid extra dependencies.
  - pauloxnet 6y ago
    Exactly. I replied also before reading you answer.
- pauloxnet 6y ago
  The Django code use only compression algorithms already included in the Python standard library to avoid unnecessary dependencies.
pauloxnet 6y ago
You're totally right. But you can choose also to use xz or bz2 for smaller files if you have spare cpu/memory.

CraigJPerry 6y ago

If it wasn't for Haskell's IHP framework then Django, Ruby on Rails and CakePHP would be fighting to come dead last in the techempower composite benchmark https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=composite https://www.techempower.com/benchmarks/#section=data-r20&hw=... Django, RoR - they're almost 2 orders of magnitude behind the fastest frameworks on the composite test and they're an order of magnitude behind the majority of Java / Go - i was going to say C# here but i see asp.net is much faster than the surprisingly homogenous performing "usual suspects" of enterprise web frameworks. And yet, if i had to launch a site tomorrow, it'd be Django i'd reach for - and with zero hesitation. Instagram, Youtube, Github, Facebook, Dropbox etc. etc. all launched on these "slowest of the slow" stacks and then optimised. Did anyone start out on a lithium (c++) or actix (rust) or other incredibly high performance stack? This is all a big diversion anyway, I think these benchmarks measure the wrong thing entirely. Instead - show me some measure of how easy & cheap it will be for me to change or add a behaviour to my site 3 years down the road if i've stuck to the prescribed idiomatic approach.

brianwawok 6y ago
Some of it may be B2B vs B2C, and volume of usage. I am B2B. My server costs are like 1% of my revenue using Django. Okay, I could shrink this by 95% by using raw C.. but who cares? Now if I were an image hosting CDN and Django bill was 50% of my revenue.. it might be a different story. I think it's the classic right tool for the right job story..
- kasey_junk 6y ago
  You are a veteran performance software engineer, besides your entrepreneurial bona fides. What is the number you start migrating off django? What is the number you wished you’d picked something else? Is there a value to Django you could have got with a different Lang without the downside?
travisjungroth 6y ago
I’m tempted to go FastAPI over Django on my next project, even though I’ve been a big Django user. A little more performance and a little less coupling, and maybe you can survive longer as a Python monolith. Then convert to Rust/Go/something in the unlikely event of wild success.

ctur 6y ago

Not including Zstandard is a lost opportunity -- it is far more appropriate than XZ or LZMA for real-time applications like serving web pages. Should be easy to add, though, and likely would be a very nice compromise between space, memory, and speed (including tuning various levels of Zstandard compression). gzip is basically never worth using anymore except for backwards compatibility; zstd is both faster and more space efficient. lz4 similarly is worth adding for utmost speed but I tend to think zstd is better in practice (particularly when the control plane is Python where perf is somewhat limited anyway).

pauloxnet 6y ago
> Not including Zstandard is a lost opportunity -- it is far more appropriate than XZ or LZMA for real-time applications like serving web pages. The article is about offline creation of compressed data dumps and not about real-time application. The Django code here use only compression algorithms already present in the Python standard library to avoid unnecessary dependencies.