10 ms·

OpenZFS – dRAID, Finally

165 points by throw0101a 6y ago

oarsinsync 6y ago
This is primarily aimed at those running very large vdevs comprised of lots of disks. FTA: > The dRAID offers a solution for large arrays, vdevs with fewer than 20 spindles will have limited benefits from the new option. The performance and resilver result will be similar to RAIDZ for small numbers of spindles. If you're doing a 6-8 disk array, this probably isn't relevant to your interests. If you're doing a 20+ disk array, well firstly, damn, you've been lucky if you've avoided any serious issues. Secondly, you really want to look into this ASAP.
- tutfbhuf 6y ago
  > If you're doing a 20+ disk array ... Then you might want to consider ceph.
  - secabeen 6y ago
    You might, but you might not. I have an 140 disk array that just works. Ceph would require multiple nodes, lots of tuning, maintenance, etc.
    - tutfbhuf 6y ago
      But multiple nodes offers you resilience against node failures not just drive failures. And with Ceph you have the flexibility to add or remove single disks as you like. And you have multiple storage interface not just FS, but also S3 or block device if you like.
      - mbreese 6y ago
        But that's a lot of extra overhead too. You need monitors, OSDs, metadata servers, etc... If you're comfortable with the node being a single point of failure, you can do a lot with a single storage node connected to a JBOD and serving data over NFS. And even if the node is the unit of failure, you can attach a SAS JBOD to multiple servers if you'd like. In my position, in addition to the added cost, I just don't have the physical rack space for the extra servers required for ceph.
        tutfbhuf 6y ago
        > don't have the physical rack space for the extra servers required for ceph Sure, it depends but the 140 drives (mentioned by secabeen) alone will take up a huge amount of space. Maybe I just wanted to point out that there might be a point when you have maxed out the storage capability of a single node and might want to consider a distributed storage setup.
        secabeen 6y ago
        They take about 14U total. (3x4U 60-drive JBOds, and a 2U server.) Big, but not huge.
      - secabeen 6y ago
        And I need none of those things. This is bulk storage over CIFS. I have two systems in two different buildings, and we bought enough capacity to exceed the projected lifetime of the system. It should run till drives start reaching EOL, at which point, we'll migrate to a replacement, or shut it down.
- mbreese 6y ago
  I’m incredibly interested in this. I run multiple 60+ disk arrays, but never with enough of a budget for the extra servers required for a better distributed system. Basically these are single large JBODs that store large genomics data. (Cheap and big are the only priorities, speed is a distant third). Right now, I use ZFS with multiple raidz2 or raidz3 vdevs with a number of hot spares waiting to take over. Resilvering can take a day for a 4TB disk. I’m terrified how long a 12TB disk will take (hence the move to raidz3). Disk loss is a normal fact of life, and it’s something we prepare for. But this would help me sleep better at night. I’m seriously excited about this and I’m definitely the target audience for this.
  - secabeen 6y ago
    I'm in the same boat, but I don't run hot-spares. If a drive fails, I want a human in the loop before a resilver begins. Ive seen too many communication/cabling errors over the years, and I'd rather try to confirm the root cause before assuming that the problem is a bad disk.
    - rsync 6y ago
      I really like this line of thought and I think your instinct is very good here. Remember - if you are in the middle of a potentially failing array, every single bit of parity data is pure gold and if you fail out a drive, that drive worth of parity is gone forever. So if you fail out a drive but it was actually weird bus errors getting spammed by some other drive that is actually failing, now you're in trouble ... While on the topic, in filesystem emergencies, don't forget the ability to mount, and even run, read-only. Your filesystem might be puking on mount and backing you into a corner, but perhaps if it were mounted read-only ...
  - rsync 6y ago
    Even with 16TB disks I would not worry about 12 or 15 disk vdevs running in raidz3. Which is to say, beat the shit out of the disks before deploying them[1] to weed out early failures, then aggressively fail them out at the first sign of trouble[2]. We don't use hot spares - we just very aggressively pull drives that give us any meaningful smart errors ... [1] perhaps with the badblocks tool ... [2] smartctl
fps_doug 6y ago
I felt a bit stupid; I re-read the important parts several times but failed to visualize what that looks like on the disks, and how blocks are reconstructed on disk failure. A little googling lead me to a paper by Intel[0] which has a simple visualization of how this works on page 11, in case I'm not the only one. [0] https://docplayer.net/117362530-Draid-declustered-raid-for-zfs-installation-and-configuration-guide-high-performance-data-division.html https://docplayer.net/117362530-Draid-declustered-raid-for-z...
- aaronax 6y ago
  Yeah that is a good link; I especially liked the text on page 7. I believe Windows Storage Spaces has long used a similar technique[0] too, though it has been a few years since I was looking into it. I wonder if ZFS DRAID will eventually allow full utilization of disks of mixed sizes too, that was always a very attractive feature of Storage Spaces to me. [0] https://techcommunity.microsoft.com/t5/storage-at-microsoft/deep-dive-the-storage-pool-in-storage-spaces-direct/ba-p/425959 https://techcommunity.microsoft.com/t5/storage-at-microsoft/...
- robert_foss 6y ago
  Ah, so a distributed redundancy drive instead of a dedicated one as with normal raid.
- knorker 6y ago
  Oh? When I first saw that illustration it hugely frustrated me about how poor that illustration is. The numbers are not explained, nor are the rows, or columns. And the numbers repeat. I saw the diagrams first outside of that pdf, though. Still, the way it goes "4, 8, 2, 16, 1, and so on..." is a boatload of "wat?".
  - abrookewood 6y ago
    I think this quote makes it clearer: "unlike the raidz1 configuration shown above, in the dRAID1 configuration the neighbor drives of the failed drive 0 (i.e. drives in a same data+parity group) are not fixed". So the RAIDZ is set up with fixed groups of drives, but the dRAID is not, even though the parity ratio is the same.
the8472 6y ago
> This will saturate the replacement disk with writes while scattering seeks over the rest of the vdev. For 14 wide RAID-Z2 vdevs using 12TB spindles, rebuilds can take weeks. Didn't sequential resilver already solve the read half of the problem? Because that would already cut it down from weeks.
- magicalhippo 6y ago
  When replacing a disk with a hotspare in a regular RAID-Z vdev, the process is limited to the sequential write speed of that single new disk. The point with DRAID is that the spare capacity is distributed across all the disks in the vdev, hence the name. Thus all disks are read from and written to when "replacing" a failed disk with the spare capacity. The combined throughput of the multiple disks is typically much higher than that of the single disk. In addition they have an additional constraint which allows DRAID to also support sequential resilver. At least that's my understanding. There's been several presentations of DRAID on the OpenZFS channel, but the latest one is here[1]. There's also a writeup here[2], though not sure if it's 100% up to date, though should be close enough for getting the concept. [1]: https://www.youtube.com/watch?v=jdXOtEF6Fh0 https://www.youtube.com/watch?v=jdXOtEF6Fh0 [2]: https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAI...
  - the8472 6y ago
    What I was questioning was that it shouldn't be taking weeks because the write speed of a single disk is enough to fill it in a day or two with sequential IO. I.e. the sequential resilver alone should be enough to fix the "can take weeks" problem. Additional speed on top of that is nice of course, if you're using such wide pools.
    - magicalhippo 6y ago
      With DRAID you'd cut down the time to an hour or so though. Not sure how much that matters in practice, I've just got a home NAS with 6 disks to play with.
    - 6y ago
gbrown_ 6y ago
It's going to be so good to finally have an open source option for large storage building blocks. It's of particular interest to those of us in HPC as if you are building a large parallel file system and you don't want re-build times to suck your options are basically one of the following. - IBM's GPFS - Panasas' PanFS - Xyratex's -> Seagate's -> Cray's -> HPE's GridRAID in Clusterstor/Sonexion prodcuts Though ZFS and dRAID will likely show up in that last one in the future.
- deleted 6y ago
  [deleted]
- p_l 6y ago
  Lustre+ZFS seems to have its fans, and iirc ZFSonLinux was heavily funded by HPC use case
  - gbrown_ 6y ago
    Yup in particular Brian Behlendorf from Lawrence Livermore National Laboratory has been a major contributor to ZoL/OpenZFS.
  - KaiserPro 6y ago
    It makes sense for lustre, as you can remove the hardware raid controller and just use "dumb" jbods. As lustre is mostly a network raid0 (its got better now) it makes sense as you really shouldn't be using it for unreconstructable data.(gross simplification alert)
- throw0101a 6y ago
  Dell-EMC Isilon OneFS
  - iforgotpassword 6y ago
    It's what we're mainly using currently. Apart from a serious bug after an update that kept crashing nfsd on nodes about two years ago, it has been a very pleasant experience overall. We also use some netapp, but it's a relatively small setup. We tried ceph before, and it seemed to support two modes of operation: Completely broken or rebalancing and thus very slow.
cbsmith 6y ago
Curiously, Facebook's "Australian filter" is blocking posting of this article.
7e 6y ago
Doesn't this violate Isilon's patent on the virtual hot spare? https://patents.google.com/patent/US7146524 https://patents.google.com/patent/US7146524 It expires in nine months, which perhaps explains the timing.
- pinewurst 6y ago
  I don't think so - 3PAR used a very similar paradigm starting in 2000 or so. Plus declustered RAID in general was invented by Garth Gibson (and used by Panasas) before that parent.
Naac 6y ago
Aren't the parity blocks distributed across the disks in raidz2? So resilvering/rebuilding already takes advantage of reading from all the disks? If so what's the different between that and dRAID?
- diegocg 6y ago
  It's not about parity The essence of draid is that, instead of keeping a spare drive unused in case one of the working drives fail, it incorporates the spare drive to the array and uses it, but one drive worth of free space is reserved randomly across the entire array. That way, if one disk fails, the reserved space is used to write the data necessary to keep the array consistent. Because the free space is distributed randomly across the array, the write performance of a single drive doesn't become a bottleneck.
  - Naac 6y ago
    So if my understanding is correct this is orthogonal to the raidz type one is using? For example I can have a hot spare with a dRAID setup as part of my raidz2 pool?
  - louwrentius 6y ago
    This is exactly how most ‘old-school’ enterprise storage solutions work.
KaiserPro 6y ago
Oooo this is exciting. One of the annoying things about "traditional" raid is that making a raid group gives you performance, but it also kneecaps your rebuild time. This is because you need to do a coordinated read from each disk in the raid stripe by stripe. a rebuild is effectively a single threaded synchronous operation We don't really need to have such rigid mapping of data to raid groups anymore. which is why most large storage systems partition your data up into small chunks (a few megs) apply some forward error correction to it, and smear it all over the place. This means that you can do multi-threaded rebuilds. This means that as your array gets bigger, not only does the rebuild time get smaller, but its much more resilient to data loss. A good example of this is GPFS's native raid: https://www.usenix.org/legacy/events/lisa11/tech/slides/deenadhayalan.pdf https://www.usenix.org/legacy/events/lisa11/tech/slides/deen... (https://www.youtube.com/watch?v=2g5rx4gP6yU https://www.youtube.com/watch?v=2g5rx4gP6yU) A word of warning though. Don't do this for small arrays (<50 disks) its not worth the hassle, yet.
syoc 6y ago
dRAID looks super cool, was not familiar with it. It sadly looks like, according to 3 min googleing, like you will not be able to expand ZFS dRAID vdevs.
hinkley 6y ago
After reading about Reed Solomon codes I was a bit surprised to learn that parity disk was a literal rather than a figurative thing in RAID5 and RAID6. After diving into consistent hashing I was surprised that all blocks are striped across all disks instead of redundancy+1,2,3 disks, in pretty much any drive array solution I could find architecture docs for. Seems like bigger blocks on fewer disks would have better utilization numbers. You could be doing 2 unrelated reads at the same time from different spindles and tracks. With a dead drive, you could do nothing and wait for a new drive, or start making backup copies across the rest of the array as if the array were now n-1 disks, or some of both, like this solution seems to do.
matheusmoreira 6y ago
Is this feature related to the proposed RAID-Z expansion? https://github.com/openzfs/zfs/pull/8853 https://github.com/openzfs/zfs/pull/8853 It's mentioned in the pull request comments. Maybe it will help?
- rincebrain 6y ago
  Not...really? I think they reused some overlapping work involving more flexible data structures, but that's all, as far as I know. (cf. slide 11 of https://docs.google.com/presentation/d/1uo0nBfY84HIhEqGWEx-Tbm8fPbJKtIP3ICo4toOPcJo/edit https://docs.google.com/presentation/d/1uo0nBfY84HIhEqGWEx-T... )

165 points by throw0101a 6y ago

This is primarily aimed at those running very large vdevs comprised of lots of disks. FTA: > The dRAID offers a solution for large arrays, vdevs with fewer than 20 spindles will have limited benefits from the new option. The performance and resilver result will be similar to RAIDZ for small numbers of spindles. If you're doing a 6-8 disk array, this probably isn't relevant to your interests. If you're doing a 20+ disk array, well firstly, damn, you've been lucky if you've avoided any serious issues. Secondly, you really want to look into this ASAP.

tutfbhuf 6y ago
> If you're doing a 20+ disk array ... Then you might want to consider ceph.
- secabeen 6y ago
  You might, but you might not. I have an 140 disk array that just works. Ceph would require multiple nodes, lots of tuning, maintenance, etc.
  - tutfbhuf 6y ago
    But multiple nodes offers you resilience against node failures not just drive failures. And with Ceph you have the flexibility to add or remove single disks as you like. And you have multiple storage interface not just FS, but also S3 or block device if you like.
    - mbreese 6y ago
      But that's a lot of extra overhead too. You need monitors, OSDs, metadata servers, etc... If you're comfortable with the node being a single point of failure, you can do a lot with a single storage node connected to a JBOD and serving data over NFS. And even if the node is the unit of failure, you can attach a SAS JBOD to multiple servers if you'd like. In my position, in addition to the added cost, I just don't have the physical rack space for the extra servers required for ceph.
      - tutfbhuf 6y ago
        > don't have the physical rack space for the extra servers required for ceph Sure, it depends but the 140 drives (mentioned by secabeen) alone will take up a huge amount of space. Maybe I just wanted to point out that there might be a point when you have maxed out the storage capability of a single node and might want to consider a distributed storage setup.
        secabeen 6y ago
        They take about 14U total. (3x4U 60-drive JBOds, and a 2U server.) Big, but not huge.
    - secabeen 6y ago
      And I need none of those things. This is bulk storage over CIFS. I have two systems in two different buildings, and we bought enough capacity to exceed the projected lifetime of the system. It should run till drives start reaching EOL, at which point, we'll migrate to a replacement, or shut it down.
mbreese 6y ago
I’m incredibly interested in this. I run multiple 60+ disk arrays, but never with enough of a budget for the extra servers required for a better distributed system. Basically these are single large JBODs that store large genomics data. (Cheap and big are the only priorities, speed is a distant third). Right now, I use ZFS with multiple raidz2 or raidz3 vdevs with a number of hot spares waiting to take over. Resilvering can take a day for a 4TB disk. I’m terrified how long a 12TB disk will take (hence the move to raidz3). Disk loss is a normal fact of life, and it’s something we prepare for. But this would help me sleep better at night. I’m seriously excited about this and I’m definitely the target audience for this.
- secabeen 6y ago
  I'm in the same boat, but I don't run hot-spares. If a drive fails, I want a human in the loop before a resilver begins. Ive seen too many communication/cabling errors over the years, and I'd rather try to confirm the root cause before assuming that the problem is a bad disk.
  - rsync 6y ago
    I really like this line of thought and I think your instinct is very good here. Remember - if you are in the middle of a potentially failing array, every single bit of parity data is pure gold and if you fail out a drive, that drive worth of parity is gone forever. So if you fail out a drive but it was actually weird bus errors getting spammed by some other drive that is actually failing, now you're in trouble ... While on the topic, in filesystem emergencies, don't forget the ability to mount, and even run, read-only. Your filesystem might be puking on mount and backing you into a corner, but perhaps if it were mounted read-only ...
- rsync 6y ago
  Even with 16TB disks I would not worry about 12 or 15 disk vdevs running in raidz3. Which is to say, beat the shit out of the disks before deploying them[1] to weed out early failures, then aggressively fail them out at the first sign of trouble[2]. We don't use hot spares - we just very aggressively pull drives that give us any meaningful smart errors ... [1] perhaps with the badblocks tool ... [2] smartctl

fps_doug 6y ago

I felt a bit stupid; I re-read the important parts several times but failed to visualize what that looks like on the disks, and how blocks are reconstructed on disk failure. A little googling lead me to a paper by Intel[0] which has a simple visualization of how this works on page 11, in case I'm not the only one. [0] https://docplayer.net/117362530-Draid-declustered-raid-for-zfs-installation-and-configuration-guide-high-performance-data-division.html https://docplayer.net/117362530-Draid-declustered-raid-for-z...

aaronax 6y ago
Yeah that is a good link; I especially liked the text on page 7. I believe Windows Storage Spaces has long used a similar technique[0] too, though it has been a few years since I was looking into it. I wonder if ZFS DRAID will eventually allow full utilization of disks of mixed sizes too, that was always a very attractive feature of Storage Spaces to me. [0] https://techcommunity.microsoft.com/t5/storage-at-microsoft/deep-dive-the-storage-pool-in-storage-spaces-direct/ba-p/425959 https://techcommunity.microsoft.com/t5/storage-at-microsoft/...
robert_foss 6y ago
Ah, so a distributed redundancy drive instead of a dedicated one as with normal raid.
knorker 6y ago
Oh? When I first saw that illustration it hugely frustrated me about how poor that illustration is. The numbers are not explained, nor are the rows, or columns. And the numbers repeat. I saw the diagrams first outside of that pdf, though. Still, the way it goes "4, 8, 2, 16, 1, and so on..." is a boatload of "wat?".
- abrookewood 6y ago
  I think this quote makes it clearer: "unlike the raidz1 configuration shown above, in the dRAID1 configuration the neighbor drives of the failed drive 0 (i.e. drives in a same data+parity group) are not fixed". So the RAIDZ is set up with fixed groups of drives, but the dRAID is not, even though the parity ratio is the same.

the8472 6y ago

> This will saturate the replacement disk with writes while scattering seeks over the rest of the vdev. For 14 wide RAID-Z2 vdevs using 12TB spindles, rebuilds can take weeks. Didn't sequential resilver already solve the read half of the problem? Because that would already cut it down from weeks.

magicalhippo 6y ago
When replacing a disk with a hotspare in a regular RAID-Z vdev, the process is limited to the sequential write speed of that single new disk. The point with DRAID is that the spare capacity is distributed across all the disks in the vdev, hence the name. Thus all disks are read from and written to when "replacing" a failed disk with the spare capacity. The combined throughput of the multiple disks is typically much higher than that of the single disk. In addition they have an additional constraint which allows DRAID to also support sequential resilver. At least that's my understanding. There's been several presentations of DRAID on the OpenZFS channel, but the latest one is here[1]. There's also a writeup here[2], though not sure if it's 100% up to date, though should be close enough for getting the concept. [1]: https://www.youtube.com/watch?v=jdXOtEF6Fh0 https://www.youtube.com/watch?v=jdXOtEF6Fh0 [2]: https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAI...
- the8472 6y ago
  What I was questioning was that it shouldn't be taking weeks because the write speed of a single disk is enough to fill it in a day or two with sequential IO. I.e. the sequential resilver alone should be enough to fix the "can take weeks" problem. Additional speed on top of that is nice of course, if you're using such wide pools.
  - magicalhippo 6y ago
    With DRAID you'd cut down the time to an hour or so though. Not sure how much that matters in practice, I've just got a home NAS with 6 disks to play with.
  - 6y ago

gbrown_ 6y ago

It's going to be so good to finally have an open source option for large storage building blocks. It's of particular interest to those of us in HPC as if you are building a large parallel file system and you don't want re-build times to suck your options are basically one of the following. - IBM's GPFS - Panasas' PanFS - Xyratex's -> Seagate's -> Cray's -> HPE's GridRAID in Clusterstor/Sonexion prodcuts Though ZFS and dRAID will likely show up in that last one in the future.

deleted 6y ago
[deleted]
p_l 6y ago
Lustre+ZFS seems to have its fans, and iirc ZFSonLinux was heavily funded by HPC use case
- gbrown_ 6y ago
  Yup in particular Brian Behlendorf from Lawrence Livermore National Laboratory has been a major contributor to ZoL/OpenZFS.
- KaiserPro 6y ago
  It makes sense for lustre, as you can remove the hardware raid controller and just use "dumb" jbods. As lustre is mostly a network raid0 (its got better now) it makes sense as you really shouldn't be using it for unreconstructable data.(gross simplification alert)
throw0101a 6y ago
Dell-EMC Isilon OneFS
- iforgotpassword 6y ago
  It's what we're mainly using currently. Apart from a serious bug after an update that kept crashing nfsd on nodes about two years ago, it has been a very pleasant experience overall. We also use some netapp, but it's a relatively small setup. We tried ceph before, and it seemed to support two modes of operation: Completely broken or rebalancing and thus very slow.

cbsmith 6y ago

Curiously, Facebook's "Australian filter" is blocking posting of this article.

7e 6y ago

Doesn't this violate Isilon's patent on the virtual hot spare? https://patents.google.com/patent/US7146524 https://patents.google.com/patent/US7146524 It expires in nine months, which perhaps explains the timing.

pinewurst 6y ago
I don't think so - 3PAR used a very similar paradigm starting in 2000 or so. Plus declustered RAID in general was invented by Garth Gibson (and used by Panasas) before that parent.

Naac 6y ago

Aren't the parity blocks distributed across the disks in raidz2? So resilvering/rebuilding already takes advantage of reading from all the disks? If so what's the different between that and dRAID?

diegocg 6y ago
It's not about parity The essence of draid is that, instead of keeping a spare drive unused in case one of the working drives fail, it incorporates the spare drive to the array and uses it, but one drive worth of free space is reserved randomly across the entire array. That way, if one disk fails, the reserved space is used to write the data necessary to keep the array consistent. Because the free space is distributed randomly across the array, the write performance of a single drive doesn't become a bottleneck.
- Naac 6y ago
  So if my understanding is correct this is orthogonal to the raidz type one is using? For example I can have a hot spare with a dRAID setup as part of my raidz2 pool?
- louwrentius 6y ago
  This is exactly how most ‘old-school’ enterprise storage solutions work.

KaiserPro 6y ago

Oooo this is exciting. One of the annoying things about "traditional" raid is that making a raid group gives you performance, but it also kneecaps your rebuild time. This is because you need to do a coordinated read from each disk in the raid stripe by stripe. a rebuild is effectively a single threaded synchronous operation We don't really need to have such rigid mapping of data to raid groups anymore. which is why most large storage systems partition your data up into small chunks (a few megs) apply some forward error correction to it, and smear it all over the place. This means that you can do multi-threaded rebuilds. This means that as your array gets bigger, not only does the rebuild time get smaller, but its much more resilient to data loss. A good example of this is GPFS's native raid: https://www.usenix.org/legacy/events/lisa11/tech/slides/deenadhayalan.pdf https://www.usenix.org/legacy/events/lisa11/tech/slides/deen... (https://www.youtube.com/watch?v=2g5rx4gP6yU https://www.youtube.com/watch?v=2g5rx4gP6yU) A word of warning though. Don't do this for small arrays (<50 disks) its not worth the hassle, yet.

syoc 6y ago

dRAID looks super cool, was not familiar with it. It sadly looks like, according to 3 min googleing, like you will not be able to expand ZFS dRAID vdevs.

hinkley 6y ago

After reading about Reed Solomon codes I was a bit surprised to learn that parity disk was a literal rather than a figurative thing in RAID5 and RAID6. After diving into consistent hashing I was surprised that all blocks are striped across all disks instead of redundancy+1,2,3 disks, in pretty much any drive array solution I could find architecture docs for. Seems like bigger blocks on fewer disks would have better utilization numbers. You could be doing 2 unrelated reads at the same time from different spindles and tracks. With a dead drive, you could do nothing and wait for a new drive, or start making backup copies across the rest of the array as if the array were now n-1 disks, or some of both, like this solution seems to do.

matheusmoreira 6y ago

Is this feature related to the proposed RAID-Z expansion? https://github.com/openzfs/zfs/pull/8853 https://github.com/openzfs/zfs/pull/8853 It's mentioned in the pull request comments. Maybe it will help?

rincebrain 6y ago
Not...really? I think they reused some overlapping work involving more flexible data structures, but that's all, as far as I know. (cf. slide 11 of https://docs.google.com/presentation/d/1uo0nBfY84HIhEqGWEx-Tbm8fPbJKtIP3ICo4toOPcJo/edit https://docs.google.com/presentation/d/1uo0nBfY84HIhEqGWEx-T... )