8 ms·

RaptorCS's redemption: the POWER9 machine works

174 points by ddevault 7y ago

nicoburns 7y ago
> I am quite impressed with it so far. Installation was a breeze, it compiles the kernel on 32 cores from spinning rust in 4m15s Does anyone know how this compares to a recent x86 machine? This seems quite fast for the entire kernel, but I have no idea what typical build times are like.
- Sir_Cmpwn 7y ago
  It's competitive with x86_64. Here are some more comprehensive benchmarks: https://www.phoronix.com/scan.php?page=article&item=blackbird-power9-4c&num=1 https://www.phoronix.com/scan.php?page=article&item=blackbir...
  - greatpatton 7y ago
    This one is a little more relevant for a server: https://www.phoronix.com/scan.php?page=article&item=rome-power9-arm&num=1 https://www.phoronix.com/scan.php?page=article&item=rome-pow...
- mrslave 7y ago
  2m3.673s; default config with loadable modules disabled; make -m 32. This is on a second-last generation 24C/48T Xeon (albeit a slower one) with all of the known speculative execution vulnerabilities mitigated except spec_store_bypass, and SMT still enabled (because htop is prettier with more pretend CPUs).
  - Sir_Cmpwn 7y ago
    What's the I/O situation? I did my build from an HDD, WD Blue. Also, kernel config matters a lot - I used yesterday's master branch from Linus, with Alpine Linux 3.10's ppc64le config, plus olddefconfig to catch it up.
    - deleted 7y ago
      [deleted]
  - tpearson-raptor 7y ago
    24 core ppc64le system, 5.4-rc2, defconfig, loadable modules on and built: real 1m55.267s Run from tmpfs, all protections from Spectre etc. on. Not too bad, really!
- JackRabbitSlim 7y ago
  The trick is far fewer modules and compile time options. A fun benchmark would be a Power9 Vs a X86_64 cross compiling the same ARM64 kernel. I wouldn't bet heavily on the Power9...
  - smcl 7y ago
    I agree, but I’m really curious if this is just because the commonly used compilers & libraries aren’t particularly optimised for ppc64le (which is a guess) or whether the hardware itself cannot match the equivalent Xeon and Ryzen offerings Not that this matters to the end user, but it is still interesting IMO
  - tpearson-raptor 7y ago
    I might take that bet core for core, matched node size (i.e. 14nm), run out of tmpfs or NVMe storage, and with both systems protected against Spectre, Meltdown etc. (i.e. no cheating by ignoring the ISA specifications)... Slightly OT, but if you limit the playing field to open ISA / fully owner controlled systems, POWER9 is so far out front in terms of performance that it looks like an outlier. ;)
    - Sir_Cmpwn 7y ago
      A race of one is easily won :)
      - tpearson-raptor 7y ago
        Well, to be fair I was thinking of the SiFive RISC-V board as one of the competitors, since my understanding was they did finally open the rest of their firmware a while back. However, yes, by far most of the remaining owner controlled systems are even smaller -- some of the smaller (older) ARM Chromebooks (sans GPU, of course), the BeagleBone black, those kinds of systems.
        Sir_Cmpwn 7y ago
        Yeah, I also have a HiFive Unleashed which is totally open, but it doesn't really aim to compete with Raptor iiuc. Until recently it also won out on open ISA, too, and the Blackbird has to be bootstrapped by the ARM BMC - which is a nonfree ISA. Power consumption is another story as well. Raptor systems is definitely the only realistic competitor to x86_64, though, for desktop and server use-cases.
- rwmj 7y ago
  It'll depend on the kernel configuration. I don't know about the Raptor, however I have some experience with IBM's POWER9 servers and they do have tremendous single thread performance and many cores, and are quite a bit faster than Intel Xeons (although at a considerably higher price, but the customers usually need the performance and aren't too sensitive about price).
- loudmax 7y ago
  Incidentally, I'm pretty sure that Sir_Cmpwn meant 32 threads, not 32 cores. Raptor sells an 8-core CPU bundle, and those cores are SMT4 capable, meaning 4 threads per core. So it will appear as 32 CPUs if you run `htop` or `cat /proc/cpuinfo` in Linux. Raptor does sell a 22-core SMT4 capable CPU, which, Yikes! that's a lot of computing power. That Blackbird bundle seems like a beast. The Phoronix benchmarks show it outperformed by high end x86 chips though. I wonder if some of that comes down to gcc optimizations because x86 has been around longer.
- loeg 7y ago
  Spinning rust may well be the bottleneck here.
equalunique 7y ago
Source hut build slave for ppc64? Groundbreaking!
- kop316 7y ago
  To ask the dumb question, what is source hut? It looks essentially like Gitlab to me.
  - Sir_Cmpwn 7y ago
    Not a dumb question. https://sourcehut.org https://sourcehut.org should tell you what you need to know. If you have any specific questions, I can answer them.
    - kop316 7y ago
      Hello! I went through that website before posting. That's why I thought it was a Gitlab-like service. I like that it works without JavaScript, and it appears to have a wiki-like integration too. It looks like a neat service.
      - aidenn0 7y ago
        It's designed so that most (all?) of the tasks you might do are "web optional" so e.g. old fogies like me can submit PRs via e-mail, and have the young whippersnappers review those requests on the web.
      - bjoli 7y ago
        I migrated after bitbucket decided to kill hg support. I could probably get by using git, but I like that I don't have to :) All in all, I have been very happy.
  - Un1corn 7y ago
    It can do the same tasks like Gitlab (Git hosting, issue tracking, CI and more) but it does it in a different way that is more similar to the workflow of the Linux kernel and other big open source projects. For example instead of pull requests, you have mailing lists in which you send patches. You can see more in the website of Sourcehut: https://sourcehut.org https://sourcehut.org
yjftsjthsd-h 7y ago
> However, I refuse to give any company credit for waking up their support team only when a scathing article about them frontpages on Hacker News. I told them I wouldn’t publish a positive follow-up unless they also convinced me that the support experience had been fixed for the typical user as well. You are a gem and I wish more people were like you.
- tpmx 7y ago
  A very old, but still key lesson here for product makers: when you know that you're doing new, early-gen work with a kinda reasonably high probability of something faulting - you need to provide stellar, high touch support, whatever it costs. And you need to plan to be ready for that from the point when you ship your first thing. (I think this applies both to hardware and software products.) Of course, this is easier said than done, but ultimately this is the core function - are your customers happy or not? As a CTO you need to prepare the company and your investors for this failure mode. You need to be able to explain to them that sometimes making sure that the product actually works and the existing customers are happy is more important than that sexy new feature of the month. It's not always a very easy balance. In my experience, even semi-technical people with a more sales/marketing kind of focus in the company often get blindsided by this. It's on us to prep them for this. It seems like this company learned it the hard way. The key thing is that they did learn.
  - tpearson-raptor 7y ago
    For what it's worth, we do have a focus on making sure the hardware works properly etc.; I've always been pushing for that and haven't had any real problems there. This particular board was a test escape. A very unique, thankfully singular (so far as I know) test escape, but there's always a chance for that -- the larger failure in my mind was the breakdown in the support process. With our new controls in place (including rapid escalation / RMA of "odd" faults observed by customer) I am confident this won't happen again. And yes, we're updating tests to catch the newly discovered failure mode. It's an interesting one, the board is a sort of a "zombie" in that if you boot it up by looking at it just right, it'll run stably and pass our stress tests, but it should never have left our factory in that condition due to the other faults. Full stop. :)
trollied 7y ago
Original issue thread: https://news.ycombinator.com/item?id=21049093 https://news.ycombinator.com/item?id=21049093
Athas 7y ago
I really like these machines. Unfortunately, I'm in Europe, where they seem to be harder (or riskier) to get hold of. I suppose I should also try to think of something reasonable to do with them, given their cost...
- smcl 7y ago
  Fellow EU here: I was curious too, but the high cost + the insane import duty + risk of return. I'm 99% sure if I RMA'd it like OP did, the local customs would somehow not understand that it's a replacement/repair and sting me for the import duty once more :(
  - gawin 7y ago
    This is the main reason I didn’t buy one.
    - smcl 7y ago
      Worth noting that I actually feel quite bad about this, I really wanted to try one out, and vote with my wallet in favour of more open hardware, an alternative architecture and to support a smaller company like Raptor. I eagerly followed all the news around Raptor (both here on HN, lobste.rs and elsewhere) and read all the guy "ClassicHasClass"'s blogs @ talospace.com - but in the end I unfortunately can't justify to myself the expense (when I spec out a modest Blackbird system it consumes the better part of a month's salary, I've recently bought my first flat and I sense a recession around the corner...). That actually makes me feel a bit hypocritical and part of the "problem", but maybe I'm not actually the target market.
      - tpearson-raptor 7y ago
        Anyone wanting control of a powerful modern computer is part of the target market. We just haven't been able to drive the price down further at this time -- doing hardware design around fully open source firmware is hard and a lot of the typical shortcuts to lower costs (including the always-concerning "post-sale monetization" concepts) simply aren't something we find acceptable on any level. Basically, it doesn't do anyone any good for us to lower cost by giving up the full owner control experience that is centric to our product lines. :)
dragontamer 7y ago
> I have christened it “flandre”3, which I think is fitting. Youmu Konpaku is best character. Just gotta love the sword-characters in a bullet-game. Although Flandre was an "extra" endgame boss character, never actually playable IIRC. Really good music for her stage though. ---------- Glad to hear everything is working out for your machine. One thing I'm curious about: > Installation was a breeze, it compiles the kernel on 32 cores from spinning rust in 4m15s Is that 8-core / 32-thread CPU? I don't think there's actually a 32-real core CPU available from them. If so, I think that's an impressive speed for 8-real cores.
- Sir_Cmpwn 7y ago
  Yes, I meant threads. Thanks for catching that.
classichasclass 7y ago
I hope this means some future work on Wayland performance on ppc64le or at least on 2D only is coming. My GPU-less 4-core Blackbird was unusable in Wayland with the basic BMC graphics (whereas it is quite sprightly in X.org on the same version of Fedora and with the exact same hardware loadout). I'll be honest and say I'm a Wayland skeptic generally for reasons I won't derail this post with, but I'm resigned to it probably being the future, and I would like to see it do better.
- rjsw 7y ago
  What is stopping people from adding a GPU to this class of system ?
  - classichasclass 7y ago
    From a hardware perspective, nothing. I have one in my Talos II and it runs rather better in Wayland than my Blackbird, though I'm usually in X.org. My only objection here is why performance would be so terrible without one. It was also part of an experiment to see how cheap I could make the loadout and still have an adequate system. However, from a libre computing perspective, the WX7100 in my Talos II requires firmware which is non-open, even though it is included in Linux. I imagine that this market would include people interested in reducing the amount of firmware they don't control as much as possible, so for this niche, that is relevant.
- emersion 7y ago
  Which compositor are you using? If you don't have a GPU you'll want one with software rendering support. Using a software OpenGL implementation like llvmpipe will work but will be very slow. We don't support this yet in wlroots, but it's definitely something we want to add, and not a limitation of the Wayland protocol at all.
  - classichasclass 7y ago
    It was whatever was out of the box with Fedora 30 and GNOME (I guess gnome-shell itself?), and indeed was very slow with llvmpipe. I look forward to the future work on this.

174 points by ddevault 7y ago

> I am quite impressed with it so far. Installation was a breeze, it compiles the kernel on 32 cores from spinning rust in 4m15s Does anyone know how this compares to a recent x86 machine? This seems quite fast for the entire kernel, but I have no idea what typical build times are like.

Sir_Cmpwn 7y ago
It's competitive with x86_64. Here are some more comprehensive benchmarks: https://www.phoronix.com/scan.php?page=article&item=blackbird-power9-4c&num=1 https://www.phoronix.com/scan.php?page=article&item=blackbir...
- greatpatton 7y ago
  This one is a little more relevant for a server: https://www.phoronix.com/scan.php?page=article&item=rome-power9-arm&num=1 https://www.phoronix.com/scan.php?page=article&item=rome-pow...
mrslave 7y ago
2m3.673s; default config with loadable modules disabled; make -m 32. This is on a second-last generation 24C/48T Xeon (albeit a slower one) with all of the known speculative execution vulnerabilities mitigated except spec_store_bypass, and SMT still enabled (because htop is prettier with more pretend CPUs).
- Sir_Cmpwn 7y ago
  What's the I/O situation? I did my build from an HDD, WD Blue. Also, kernel config matters a lot - I used yesterday's master branch from Linus, with Alpine Linux 3.10's ppc64le config, plus olddefconfig to catch it up.
  - deleted 7y ago
    [deleted]
- tpearson-raptor 7y ago
  24 core ppc64le system, 5.4-rc2, defconfig, loadable modules on and built: real 1m55.267s Run from tmpfs, all protections from Spectre etc. on. Not too bad, really!
JackRabbitSlim 7y ago
The trick is far fewer modules and compile time options. A fun benchmark would be a Power9 Vs a X86_64 cross compiling the same ARM64 kernel. I wouldn't bet heavily on the Power9...
- smcl 7y ago
  I agree, but I’m really curious if this is just because the commonly used compilers & libraries aren’t particularly optimised for ppc64le (which is a guess) or whether the hardware itself cannot match the equivalent Xeon and Ryzen offerings Not that this matters to the end user, but it is still interesting IMO
- tpearson-raptor 7y ago
  I might take that bet core for core, matched node size (i.e. 14nm), run out of tmpfs or NVMe storage, and with both systems protected against Spectre, Meltdown etc. (i.e. no cheating by ignoring the ISA specifications)... Slightly OT, but if you limit the playing field to open ISA / fully owner controlled systems, POWER9 is so far out front in terms of performance that it looks like an outlier. ;)
  - Sir_Cmpwn 7y ago
    A race of one is easily won :)
    - tpearson-raptor 7y ago
      Well, to be fair I was thinking of the SiFive RISC-V board as one of the competitors, since my understanding was they did finally open the rest of their firmware a while back. However, yes, by far most of the remaining owner controlled systems are even smaller -- some of the smaller (older) ARM Chromebooks (sans GPU, of course), the BeagleBone black, those kinds of systems.
      - Sir_Cmpwn 7y ago
        Yeah, I also have a HiFive Unleashed which is totally open, but it doesn't really aim to compete with Raptor iiuc. Until recently it also won out on open ISA, too, and the Blackbird has to be bootstrapped by the ARM BMC - which is a nonfree ISA. Power consumption is another story as well. Raptor systems is definitely the only realistic competitor to x86_64, though, for desktop and server use-cases.
rwmj 7y ago
It'll depend on the kernel configuration. I don't know about the Raptor, however I have some experience with IBM's POWER9 servers and they do have tremendous single thread performance and many cores, and are quite a bit faster than Intel Xeons (although at a considerably higher price, but the customers usually need the performance and aren't too sensitive about price).
loudmax 7y ago
Incidentally, I'm pretty sure that Sir_Cmpwn meant 32 threads, not 32 cores. Raptor sells an 8-core CPU bundle, and those cores are SMT4 capable, meaning 4 threads per core. So it will appear as 32 CPUs if you run `htop` or `cat /proc/cpuinfo` in Linux. Raptor does sell a 22-core SMT4 capable CPU, which, Yikes! that's a lot of computing power. That Blackbird bundle seems like a beast. The Phoronix benchmarks show it outperformed by high end x86 chips though. I wonder if some of that comes down to gcc optimizations because x86 has been around longer.
loeg 7y ago
Spinning rust may well be the bottleneck here.

equalunique 7y ago

Source hut build slave for ppc64? Groundbreaking!

kop316 7y ago
To ask the dumb question, what is source hut? It looks essentially like Gitlab to me.
- Sir_Cmpwn 7y ago
  Not a dumb question. https://sourcehut.org https://sourcehut.org should tell you what you need to know. If you have any specific questions, I can answer them.
  - kop316 7y ago
    Hello! I went through that website before posting. That's why I thought it was a Gitlab-like service. I like that it works without JavaScript, and it appears to have a wiki-like integration too. It looks like a neat service.
    - aidenn0 7y ago
      It's designed so that most (all?) of the tasks you might do are "web optional" so e.g. old fogies like me can submit PRs via e-mail, and have the young whippersnappers review those requests on the web.
    - bjoli 7y ago
      I migrated after bitbucket decided to kill hg support. I could probably get by using git, but I like that I don't have to :) All in all, I have been very happy.
- Un1corn 7y ago
  It can do the same tasks like Gitlab (Git hosting, issue tracking, CI and more) but it does it in a different way that is more similar to the workflow of the Linux kernel and other big open source projects. For example instead of pull requests, you have mailing lists in which you send patches. You can see more in the website of Sourcehut: https://sourcehut.org https://sourcehut.org

yjftsjthsd-h 7y ago

> However, I refuse to give any company credit for waking up their support team only when a scathing article about them frontpages on Hacker News. I told them I wouldn’t publish a positive follow-up unless they also convinced me that the support experience had been fixed for the typical user as well. You are a gem and I wish more people were like you.

tpmx 7y ago
A very old, but still key lesson here for product makers: when you know that you're doing new, early-gen work with a kinda reasonably high probability of something faulting - you need to provide stellar, high touch support, whatever it costs. And you need to plan to be ready for that from the point when you ship your first thing. (I think this applies both to hardware and software products.) Of course, this is easier said than done, but ultimately this is the core function - are your customers happy or not? As a CTO you need to prepare the company and your investors for this failure mode. You need to be able to explain to them that sometimes making sure that the product actually works and the existing customers are happy is more important than that sexy new feature of the month. It's not always a very easy balance. In my experience, even semi-technical people with a more sales/marketing kind of focus in the company often get blindsided by this. It's on us to prep them for this. It seems like this company learned it the hard way. The key thing is that they did learn.
- tpearson-raptor 7y ago
  For what it's worth, we do have a focus on making sure the hardware works properly etc.; I've always been pushing for that and haven't had any real problems there. This particular board was a test escape. A very unique, thankfully singular (so far as I know) test escape, but there's always a chance for that -- the larger failure in my mind was the breakdown in the support process. With our new controls in place (including rapid escalation / RMA of "odd" faults observed by customer) I am confident this won't happen again. And yes, we're updating tests to catch the newly discovered failure mode. It's an interesting one, the board is a sort of a "zombie" in that if you boot it up by looking at it just right, it'll run stably and pass our stress tests, but it should never have left our factory in that condition due to the other faults. Full stop. :)

trollied 7y ago

Original issue thread: https://news.ycombinator.com/item?id=21049093 https://news.ycombinator.com/item?id=21049093

Athas 7y ago

I really like these machines. Unfortunately, I'm in Europe, where they seem to be harder (or riskier) to get hold of. I suppose I should also try to think of something reasonable to do with them, given their cost...

smcl 7y ago
Fellow EU here: I was curious too, but the high cost + the insane import duty + risk of return. I'm 99% sure if I RMA'd it like OP did, the local customs would somehow not understand that it's a replacement/repair and sting me for the import duty once more :(
- gawin 7y ago
  This is the main reason I didn’t buy one.
  - smcl 7y ago
    Worth noting that I actually feel quite bad about this, I really wanted to try one out, and vote with my wallet in favour of more open hardware, an alternative architecture and to support a smaller company like Raptor. I eagerly followed all the news around Raptor (both here on HN, lobste.rs and elsewhere) and read all the guy "ClassicHasClass"'s blogs @ talospace.com - but in the end I unfortunately can't justify to myself the expense (when I spec out a modest Blackbird system it consumes the better part of a month's salary, I've recently bought my first flat and I sense a recession around the corner...). That actually makes me feel a bit hypocritical and part of the "problem", but maybe I'm not actually the target market.
    - tpearson-raptor 7y ago
      Anyone wanting control of a powerful modern computer is part of the target market. We just haven't been able to drive the price down further at this time -- doing hardware design around fully open source firmware is hard and a lot of the typical shortcuts to lower costs (including the always-concerning "post-sale monetization" concepts) simply aren't something we find acceptable on any level. Basically, it doesn't do anyone any good for us to lower cost by giving up the full owner control experience that is centric to our product lines. :)

dragontamer 7y ago

> I have christened it “flandre”3, which I think is fitting. Youmu Konpaku is best character. Just gotta love the sword-characters in a bullet-game. Although Flandre was an "extra" endgame boss character, never actually playable IIRC. Really good music for her stage though. ---------- Glad to hear everything is working out for your machine. One thing I'm curious about: > Installation was a breeze, it compiles the kernel on 32 cores from spinning rust in 4m15s Is that 8-core / 32-thread CPU? I don't think there's actually a 32-real core CPU available from them. If so, I think that's an impressive speed for 8-real cores.

Sir_Cmpwn 7y ago
Yes, I meant threads. Thanks for catching that.

classichasclass 7y ago

I hope this means some future work on Wayland performance on ppc64le or at least on 2D only is coming. My GPU-less 4-core Blackbird was unusable in Wayland with the basic BMC graphics (whereas it is quite sprightly in X.org on the same version of Fedora and with the exact same hardware loadout). I'll be honest and say I'm a Wayland skeptic generally for reasons I won't derail this post with, but I'm resigned to it probably being the future, and I would like to see it do better.

rjsw 7y ago
What is stopping people from adding a GPU to this class of system ?
- classichasclass 7y ago
  From a hardware perspective, nothing. I have one in my Talos II and it runs rather better in Wayland than my Blackbird, though I'm usually in X.org. My only objection here is why performance would be so terrible without one. It was also part of an experiment to see how cheap I could make the loadout and still have an adequate system. However, from a libre computing perspective, the WX7100 in my Talos II requires firmware which is non-open, even though it is included in Linux. I imagine that this market would include people interested in reducing the amount of firmware they don't control as much as possible, so for this niche, that is relevant.
emersion 7y ago
Which compositor are you using? If you don't have a GPU you'll want one with software rendering support. Using a software OpenGL implementation like llvmpipe will work but will be very slow. We don't support this yet in wlroots, but it's definitely something we want to add, and not a limitation of the Wayland protocol at all.
- classichasclass 7y ago
  It was whatever was out of the box with Fedora 30 and GNOME (I guess gnome-shell itself?), and indeed was very slow with llvmpipe. I look forward to the future work on this.