4 ms·

While reading the parent comment ("stay online for ever") I was reminded of VMware lockstep VMs, which nowadays support up to 4 cores. I wouldn't be surprised i

While reading the parent comment ("stay online for ever") I was reminded of VMware lockstep VMs, which nowadays support up to 4 cores. I wouldn't be surprised if a few places do OpenVMS on lockstep. I am very curious what you mean by "in duplication". Do you mean "3 copies of production and a router" (probably not) - or are you saying you did something like lockstep for QEMU? Incidentally I just found https://wiki.qemu.org/Features/COLO https://wiki.qemu.org/Features/COLO, which looks like it may be being successfully used privately in one or two places (looking at the email addresses).

shakna 6y ago
It was a little while ago, but it was 3 production servers running with the mirror backend, with a dirty-bitmap [0], and a complicated routing setup (which could hold packets instead of dropping them, depending on their importance). I didn't really have anything to do with the routing though, so I can't say much. I'd probably use COLO if I was to do it again today. [0] https://kashyapc.fedorapeople.org/QEMU-Docs/_build/html/docs/interop/live-block-operations.html#live-disk-synchronization-drive-mirror-and-blockdev-mirror https://kashyapc.fedorapeople.org/QEMU-Docs/_build/html/docs...
- exikyut 6y ago
  Oooh, disk synchronization. Live migration but just for the block layer! Cool. I wonder what sorts of scenarios would benefit from using this instead of any of the dozen distributed database architectures it seems are out there. Incidentally while poking through https://www.qemu.org/docs/master/system/invocation.html https://www.qemu.org/docs/master/system/invocation.html I noticed that there are some COLO-related options in there, which is a bit exciting.
my123 6y ago
Some Arm processors that you can buy today also support dual-core lock step in hardware, such as the Jetson AGX Xavier from Nvidia. (however, not enabled by default for perf reasons, because you halve your cores by design in that mode) It's there for safety reasons for automotive. (see https://blogs.nvidia.com/blog/2020/05/20/xavier-achieves-industry-first-safety-assessment/ https://blogs.nvidia.com/blog/2020/05/20/xavier-achieves-ind... )
- jeffreygoesto 6y ago
  Could you point us to the source of your information please? The Xavier does not support dual-core lockstep afaik. There are other methods in place to detect random faults, but not that one. The first widely available ARM cores providing it are fairly new (at least in the automotive domain). [0] https://developer.arm.com/ip-products/processors/cortex-a/cortex-a76ae https://developer.arm.com/ip-products/processors/cortex-a/co...
  - my123 6y ago
    Hello, As a person who worked with Xavier for quite a while, dual-core lockstep is supported. Nvidia uses their own CPU cores, not Arm's. See: https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3231/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/mb1_platform_config_xavier.html https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-323... for how to enable it. > enable_ccplex_lock_step: Boolean; enables or disables CCPLEX dual-core lock step.
    - jeffreygoesto 6y ago
      Very interesting, thank you very much! I know the lockstepped R5s and the dual execution feature of the Carmels (close but no cigar for ISO26262), but not that one. Gonna ask our FAE about it eventually. ;-D
      - my123 6y ago
        Not to get your hopes down... but dual-execution in the Carmels and DCLS for CCPLEX should be references to the same thing. :)
        jeffreygoesto 6y ago
        Aaaah! Ok, so I did not overlook a big feature but knew it under a different name. Thanks for the clarification!

shakna 6y ago

It was a little while ago, but it was 3 production servers running with the mirror backend, with a dirty-bitmap [0], and a complicated routing setup (which could hold packets instead of dropping them, depending on their importance). I didn't really have anything to do with the routing though, so I can't say much. I'd probably use COLO if I was to do it again today. [0] https://kashyapc.fedorapeople.org/QEMU-Docs/_build/html/docs/interop/live-block-operations.html#live-disk-synchronization-drive-mirror-and-blockdev-mirror https://kashyapc.fedorapeople.org/QEMU-Docs/_build/html/docs...

exikyut 6y ago
Oooh, disk synchronization. Live migration but just for the block layer! Cool. I wonder what sorts of scenarios would benefit from using this instead of any of the dozen distributed database architectures it seems are out there. Incidentally while poking through https://www.qemu.org/docs/master/system/invocation.html https://www.qemu.org/docs/master/system/invocation.html I noticed that there are some COLO-related options in there, which is a bit exciting.

my123 6y ago

Some Arm processors that you can buy today also support dual-core lock step in hardware, such as the Jetson AGX Xavier from Nvidia. (however, not enabled by default for perf reasons, because you halve your cores by design in that mode) It's there for safety reasons for automotive. (see https://blogs.nvidia.com/blog/2020/05/20/xavier-achieves-industry-first-safety-assessment/ https://blogs.nvidia.com/blog/2020/05/20/xavier-achieves-ind... )

jeffreygoesto 6y ago
Could you point us to the source of your information please? The Xavier does not support dual-core lockstep afaik. There are other methods in place to detect random faults, but not that one. The first widely available ARM cores providing it are fairly new (at least in the automotive domain). [0] https://developer.arm.com/ip-products/processors/cortex-a/cortex-a76ae https://developer.arm.com/ip-products/processors/cortex-a/co...
- my123 6y ago
  Hello, As a person who worked with Xavier for quite a while, dual-core lockstep is supported. Nvidia uses their own CPU cores, not Arm's. See: https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3231/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/mb1_platform_config_xavier.html https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-323... for how to enable it. > enable_ccplex_lock_step: Boolean; enables or disables CCPLEX dual-core lock step.
  - jeffreygoesto 6y ago
    Very interesting, thank you very much! I know the lockstepped R5s and the dual execution feature of the Carmels (close but no cigar for ISO26262), but not that one. Gonna ask our FAE about it eventually. ;-D
    - my123 6y ago
      Not to get your hopes down... but dual-execution in the Carmels and DCLS for CCPLEX should be references to the same thing. :)
      - jeffreygoesto 6y ago
        Aaaah! Ok, so I did not overlook a big feature but knew it under a different name. Thanks for the clarification!