14 ms·

PCI Express on the Raspberry Pi 4

619 points by trollied 7y ago

Jonnax 7y ago
That's really cool. I'm curious, would it be possible to use a modern GPU (running at 1x) on an ARM based board? Would the open source drivers that are part of the Kernel work out the box on ARM?
- edude03 7y ago
  Yes, if you look around, people have even recompiled opensource drivers for RISC-V to use AMD cards.
- qdot_me 7y ago
  Hack’s creator here - it’s on my list of things to try. GPUs are notoriously hard to get to work on non-intel, having tried to get a few up on Alpha and Itaniums in the past. VideoBIOS expects to run and expects a well behaving Intel CPU to do the power-up. That said X can sometimes emulate these quite well. On ARM we’d also run into alignment issues and likely other quirks - but in principle...
  - jchw 7y ago
    I always wondered what the deal was with Video BIOS on modern PCI-express graphics cards. Obviously it must exist, since int 10h still works. Does video BIOS even need to be executed, though? I always assumed it was only necessary for the primary card to be able to display output during boot on PCs. (Otherwise, wouldn't two different cards trample over eachother's implementation of int 10h?) Sorry for the probably-obvious questions, it's sometimes tricky to find good sources for weird information like this that are also up to date.
    - qdot_me 7y ago
      It’s all, cough, implementation dependent. You can run two VideoBIOS implementations, and chances are only one will install an int10h handler, but both will set up the registers and whatnot in the GPUs. My AMD TAHITIs for instance need VideoBIOS to start some form of thermal management loop - otherwise they just run full-throttle on the fan. Then whichever card prevails (BIOS has the ability to select the initialization order) becomes the boot display device. X11 has some (generally working, for well behaved GPUs) emulation of this environment, so that the GPU can initialize late, and even reset under X control. This is how sane cards can work under headless ARM etc. Now, some manufacturers assume you get something like SSE or MMX - VideoBIOS spec technically mandates 386 instruction set only. That crap gets badly emulated. On top of this, drivers can sometimes reinit anyways, from native kernel code. If that happens, the VideoBIOS concerns are moot.
      - jchw 7y ago
        Thanks for the information. I’ve been really curious about this with the incoming ARM dev boards that have full PCI-e edge connectors. Pretty fascinating that X11 of all things is dealing with Video BIOS initialization, though. The sheer amount of functionality that was overloaded into X...
        floatboth 7y ago
        You don't need X, and I wouldn't assume its remnants of userspace mode setting would work on non-x86. The amdgpu kernel driver POSTs the GPUs when loaded just fine! I run an RX 480 on a MACCHIATObin with FreeBSD and Wayland only :) Plus, there are two ways to get the GPU in UEFI already – X86EmulatorPkg and native aarch64 GOP driver builds provided by AMD.
        jchw 7y ago
        X86EmulatorPkg is pretty cool. I guess if you have a board with UEFI it only makes sense to use something like that.
      - sebazzz 7y ago
        I always assumed the VideoBIOS runs on the GPU, but if I understand you correctly this is code run on the CPU?
        floatboth 7y ago
        Yes — "VBIOS", and these days mostly "UEFI GOP driver", is code that runs on the CPU and allows the system firmware to initialize the card and preset video output before any operating system is booted. Without it, you cannot enter the setup screen graphically, only over a serial port :D
        ComputerGuru 7y ago
        Yes, the PC (or whatever) hands off control of the entire system to the VideoBIOS which must do its thing and then pass control back.
  - pm215 7y ago
    People have got at least some cards working on server-class arm setups. On the other hand I think they might be using the 'emulate x86 uefi rom in an arm uefi system' hack, so there might be more effort required on a non-uefi setup. Still, it should have shaken out most of the alignment and memory-cacheability-attibutes issues already, hopefully.
    - floatboth 7y ago
      Yep, the "hack" (X86EmulatorPkg) allows running the card in the UEFI out of the box. AMD also provides native aarch64 builds of their UEFI GOP driver though. And none of this is necessary for running the GPU in the OS – amdgpu POSTs the GPU just fine. > memory-cacheability-attibutes issues Recently I've added aarch64 support to FreeBSD's port of the DRM/KMS drivers :) Took a couple hours to realize that our implementations of Linux's mapping functions used normal uncacheable memory instead of device memory – fixing that stopped the hangs on driver load and allowed everything to work. Then there was some corruption on the screen – our drm-kms is from Linux 5.0 for now, and I've had to cherry-pick a fix that only landed in 5.1 I think: https://patchwork.kernel.org/patch/10778815/ https://patchwork.kernel.org/patch/10778815/
      - pm215 7y ago
        You wouldn't believe how much discussion it took to decide that that minor drm_cache.h change really was the right fix for that issue :-)
  - timthorn 7y ago
    Some AMD cards have an Arm build of their UEFI firmware available, which does allow video from boot: https://developer.arm.com/architectures/system-architectures/software-standards/uefi-drivers https://developer.arm.com/architectures/system-architectures...
  - floatboth 7y ago
    VBIOS is often not necessary for running a GPU in the OS. The amdgpu driver can POST a GPU by itself just fine. Still… X86EmulatorPkg allows running an amd64 VBIOS in UEFI on an aarch64 machine :) AFAIK the bigger problem on embedded boards is half assed Synopsys Designware host controllers. I have a Radeon running on my Marvell MACCHIATObin, on FreeBSD even. But from what I've heard the Rockchip RK3399 has a worse version of the controller, and people trying GPUs on the ROCKPro64 saw errors related to not large enough BAR space or something. UPD: yeah, someone in the thread mentioned BAR space issues wrt NXP i.MX SoCs, that's probably what's happening on Rockchip. Would be amazing if the Broadcom chip in the Pi turns out to be the one with enough BAR space! :D
    - markus92 7y ago
      Could you clarify to me/give a short definition of BAR space? For obvious reasons it's a bit hard to search for :)
      - cesarb 7y ago
        In PCI, BAR is Base Address Register, which is a register in the PCI device's configuration space which defines where in the machine's physical memory address space that particular window of memory and/or I/O will be mapped (a single device can have several BARs, for instance a simple graphics card could have one for its control registers and one for the framebuffer). So the "BAR space" would be a shorthand for "the region of the physical memory address space which can be used to map the PCI devices memory through their Base Address Registers". The size of this region is limited, and graphics cards in particular tend to have somewhat large BARs. (See for yourself in your machine: run "lspci -v", the lines starting with "Memory at ..." or "I/O ports at ..." are the BARs.)
      - bluecmd 7y ago
        Search "PCIe bar space" and you should be fine
  - Teknoman117 7y ago
    SiFive has a few videos where they have their RISC-V core running an AMD gpu (via the in kernel amdgpu driver).
  - bstar77 7y ago
    "GPUs are notoriously hard to get to work on non-intel" I don't understand this statement... accelerated graphics have no issues working on any x86 system as well as ARM based linux systems (not to mention every cell phone in the past 10 years). Itanium is also Intel so that statement contradicts your previous statement.
    - qdot_me 7y ago
      To a limited extent. The accelerated graphics on ARM (and cell phones) has always been a system integrator problem. Until very recently, they were not user customizable - which means the initialisation was part of the proprietary firmware and hard to port across even different implementations of the same SoC. The x86 world had the advantage of user upgradable GPUs, which necessitates standardization and common firmware. On top of which, the accelerated graphics of cell phones is a horrible kludge of various standards.
  - notaplumber 7y ago
    OpenBSD has radeondrm enabled on mips64, powerpc and arm64. It can initialize the board just fine without emulating x86, since KMS. Working 3D acceleration is another story. OpenBSD even has (or had, build breaks often) packages for chromium browser on arm64 that were tested on this.
    - floatboth 7y ago
      > Working 3D acceleration is another story Why? I recently enabled amdgpu on FreeBSD arm64 — as soon as KMS happened, 3D just worked. There was some buffer corruption (https://user-images.githubusercontent.com/208340/60443774-97243900-9c24-11e9-861a-6dd29101c235.png https://user-images.githubusercontent.com/208340/60443774-97...) for which I've had to cherry-pick this tiny Linux patch: https://patchwork.kernel.org/patch/10778815/ https://patchwork.kernel.org/patch/10778815/ > packages for chromium browser on arm64 And we have Firefox, which doesn't break, has a working JS JIT, works on Wayland, renders with WebRender :)
  - vardump 7y ago
    > On ARM we’d also run into alignment issues... Just don't use ARMv5 or older. ARMv6+ supports unaligned access.
    - planteen 7y ago
      At least on ARMv7, it's up to the MMU, not the CPU. I've worked with Cortex-A9s that trap the unaligned access. You can write a handler or crash.
      - vardump 7y ago
        Interesting. I've kept my accesses aligned, but noticed that unaligned ones worked. Didn't know about the exception mechanism behind, but yeah, should have err... expected it. By the way, you can also opt to crash on x86. Just need to enable bit AM (Alignment Mask) bit 18) in CR0 (kernel stuff) and AC (Alignment Check, also bit 18) in EFLAGS. After that point, unaligned accesses trap. I use this in my CPU JIT/dynamic compiler to trap target unaligned accesses. https://en.wikipedia.org/wiki/Control_register#CR0 https://en.wikipedia.org/wiki/Control_register#CR0
- mntmn 7y ago
  You might run into address space issues. I haven’t checked Broadcom PCIe documentation for RPi4 (is there any?), but I tried a very similar hack with i.MX6 and older AMD and nVidia cards. They get recognized fine, but BARs cannot be mapped because they don’t fit in i.MX6’s tiny 16MB PCIe space.
- jamesfmilne 7y ago
  NVIDIA drivers might even work too Linux 32-bit ARM drivers https://www.nvidia.co.uk/Download/driverResults.aspx/137468/en-uk https://www.nvidia.co.uk/Download/driverResults.aspx/137468/...
  - whitelabrat 7y ago
    This would be incredible if gotten to work with a pi. I've been eyeing the Jetson Nano, but if I could use a pi with a V100, that would be hilarious. And awesome.
    - cr0sh 7y ago
      Based on what I've seen - it's on my list of projects to "try out", but rapidly becoming a more obsolete idea the longer I wait (but then again, I've got an Altair on my todo list, so meh)... Taking a low-cost Mini ITX board and pairing it with a decent mini-sized GTX 750 TI SC video card (I prefer the EVGA one) should yield a system that is somewhat on par with some of NVidia's embedded ML offerings, with the tradeoff of size and efficiency - but at a significantly reduced cost. Add a very short PCIe riser ribbon (or, if you can find one tall enough, a right-angle riser), and you could lay the card over "flat" above the CPU (using a 1U cooler/heatsink); you'd have to make a custom mounting frame of course, but I think you could make the whole package relatively compact. From my limited experience using Tensorflow with the 750 - it is a very capable card in that capacity and relatively inexpensive today. If you were willing to spend a bit more, there are mini-sized NVidia GPUs available in more recent models; of course if space is not an issue, then full-length GPUs can be substituted (my "goal" was to build such a system as close to within the footprint of a Mini-ITX board as possible). As far as the Mini-ITX motherboard is concerned, again, size to your budget. That said, if you decided to build this system using cheap pre-owned components, maybe an I5 with 8 gig - you could probably get it going for under $250.00 USD, maybe less.
- true_tuna 7y ago
  If you want it for ML you could check out the Coral TPU. (Tensor Processing Unit over USB) I picked up a couple of those yesterday.
yetihehe 7y ago
Hmm, how about using this for fast interconnect for making rpi clusters?
- hvidgaard 7y ago
  Wouldn't the gigabit LAN be a better fit for this? If you want to make a cluster, you need to make some custom hardware to connect to, that can facilitate the communication. At this point you're likely spending more that if you just bought a real desktop for more performance. I can see the fun factor in hacking the system together, though.
  - qdot_me 7y ago
    There are two ways of doing clusters - one is a message passing paradigm, which you can do over Ethernet (to an extent - I’d still take USB3 for 4x the bandwidth) - and the other is direct memory access a’la Cray. What really motivated me to do this hack is the relative abundance of stuff I can now plug into an FPGA :)
    - hvidgaard 7y ago
      I was thinking Ethernet because, A:It's cheap to buy a switch and cluster 100 RPi, B: You can have a desktop with a faster NIC and keep the RPis busy. But as with everything high performance, it depends entirely on the use case.
      - qdot_me 7y ago
        True. And with RPi4 having a 1000baseT, it’s not as painful as it seems. Perhaps even the driver can be coaxed into some form of DMA and MPI that is a bit lower latency than IP stack. With secondary IP layer on 802.11, it might actually work reasonably well.
        consp 7y ago
        Or use the PCIe mod and use an infiniband card for low latency and high thoughput.
segfaultbuserr 7y ago
It looks like an unreliable modification. Running a GHz-level interface with jumpers is almost impossible to control the impedance, it's a cool Proof-of-Concept though. But is it possible to bring the project to the next level? Is it possible to make a daughterboard with QFN connector? If so, one can make a pin-compatible daughterboard with an extension connector. To use it, just desolder the USB chip and solder a new daughterboard on it, and you're ready to go. It would be one of the coolest Raspberry Pi projects!
- qdot_me 7y ago
  Yup. The daughterboard is on my mind. Likely flex-PCB and that’s gonna take a week or two to respin. Hence I’m collecting ideas for various daughtercards I could cram in a panel before sending it off - straight-through to riser via USB3, expresscard SMT, maybe through-hole 1x? That said, PCIe phy’s are extremely robust - they do most of the impedance matching and delay mismatch training. And if you don’t ruin the onboard caps, this could be jumpered straight across.
  - segfaultbuserr 7y ago
    Great! I was thinking about using a SMT ribbon cable connector because of the limited available space, but apparently it won't be an issue if you raise the board high enough? Anyway, if this project ever goes to batch-production, make sure to update your blog when the funding campaign starts!
- mkesper 7y ago
  Maybe a new raspberry compute module (https://www.raspberrypi.org/blog/raspberry-pi-compute-module-new-product/ https://www.raspberrypi.org/blog/raspberry-pi-compute-module...) featuring PCIe could be implemented?
  - HeWhoLurksLate 7y ago
    Oh heck yeah, that'd be amazing. I'm unfamiliar with this- do the compute modules have any sequential unused pins on them?
boyadjian 7y ago
Why not, but I will wait for a classic ATX ARM motherboard instead. It should happen this year.
- steevdave 7y ago
  https://store.avantek.co.uk/arm-desktops.html https://store.avantek.co.uk/arm-desktops.html bit pricey but it exists already
- imtringued 7y ago
  Those will never be as cheap as an SBC with a SoC that was meant for some random discontinued TV box.
- mpoloton 7y ago
  Now there are many ARM single board computers such as RockPi with PCI-E connectors.
- hawski 7y ago
  Is Mini-ITX for $550 acceptable? https://www.solid-run.com/nxp-lx2160a-family/honeycomb-workstation/ https://www.solid-run.com/nxp-lx2160a-family/honeycomb-works... 16 core A72 CPU, up to 64GB RAM, PICe x8, 10GbE SFP+ ports and 1GbE RJ45, SATA. It's early access hardware, so it may have some pain points. Normal units will be available from the end of the year for $750. If I would have time and money right now, that's what I would buy.
  - floatboth 7y ago
    It's only pre-order right now, and the firmware isn't done. They promise SBSA compliance (which includes ECAM PCIe working via a generic ACPI attachment) but they haven't passed the full test suite yet. Some experts are skeptical about whether full compliance is possible on that NXP chip… I hope the PCIe works fine. And I hope the firmware will be FOSS like on their MACCHIATObin. One thing they revealed is that the chip is overclockable (including memory), which is awesome. IIRC they got 2.5ish GHz core clock working. Would be amazing if it does like 3GHz with a voltage boost. (I don't expect software voltage control… but there's always hard mods :D)
zaro 7y ago
For anybody interested in PCI Express on Arm there are already boards with PCI express connector, like https://store.pine64.org/?product=rockpro64-4gb-single-board-computer https://store.pine64.org/?product=rockpro64-4gb-single-board...
peterburkimsher 7y ago
Nice work! Would this be compatible with an M.2 to PCIe adaptor? [1] Being able to attach an Intel 660P and get 2 TB of fast SSD storage on a Raspberry Pi would be sweet. [1] https://www.amazon.com/EZDIY-FAB-Express-Adapter-Support-22110/dp/B01GCXCR7W https://www.amazon.com/EZDIY-FAB-Express-Adapter-Support-221...
- elFarto 7y ago
  Technically yes, but it's only a PCIe 1x Gen2 slot, so only 500MB/s of bandwidth (4x Gen3 is ~4GB/s). You'd be better off with a USB 3.0 to M.2 adapter.
  - londons_explore 7y ago
    The bandwidth works out about the same, but the USB controller and AHCI controllers will add quite some latency (and CPU load). I'd like to see benchmarks, but my guess is single-thread random 4k read performance will more than double via PCI Express rather than the USB.
noobermin 7y ago
Now this is the content I come to HN for. A serious hack just days after the 4 was released. Kudos to the OP. I envy people like OP for their tenacity. I barely have time to follow what's happening in IT, much less get ahead of the pack in doing cools hacks like this.
- userbinator 7y ago
  On the other hand, the fact that the RPi ecosystem remains notoriously proprietary (even the USB controller is a bastard variant that has next to no documentation --- of all the ones available, they had to choose that one) continues to be disappointing. I definitely like this sort of hack, but such hacks with documentation already available (and doing more than documented, basically) are certainly preferable.
- anderspitman 7y ago
  I'm sure you are but just in case are you aware of hackaday.com?
  - noobermin 7y ago
    Of course! HN is just a good digest of things on the maker side to the fairly abstract CS stuff to math and physics and all in between. It sure beats the political stuff that becomes tiresome (even though I succumb and engage in it like many posters here).
mrb 7y ago
Ohh I see where this is going qdot_me — would love to be able to hook up a GPU to a RPi4 to crypto-mine. So useful for so many applications ! I will seriously fund you if you can make this happen, hit me up by email. My contact info is on zorinaq.com
- mrb 7y ago
  Whoa, downvoted to minus 4. No idea why. In my 9 years of HN, this is my worst downvoted comment. Maybe people thought I was being off-topic? Let me clarify a bit for the public here why the comment is relevant: in the crypto mining community, some groups are looking into what minimal single board computer can provide a PCIe signal to connect a single GPU. Idea is to be cheap and reliable. If you have a many-PCIe board failure (http://bitcoin.zorinaq.com/many_pcie/ http://bitcoin.zorinaq.com/many_pcie/) you have 10-20 GPUs going down at once. Not good. By isolating each GPU on its own motherboard, you can isolate failures, thus increase mining profits. When I saw the OP mention cryptocurrency in the blog post, I thought hey maybe that's what he is looking to do...
teamski 7y ago
Hijacking the thread: I develop remotely on VPSes because I like to have an always-on box reachable from any client. I am wondering if a RP4 offers a similar experience at lower cost. Does anyone use a RP for this?
- numlock86 7y ago
  I do this with an RPi3 and it's doing good, so it's doable. It strongly depends on your setup and develeopment environment, though. Do you want VI to work over SSH or want full VNC access to a machine with Gnome and Eclipse? Or something in between like X forwarding? Also, is aarch64 even an option as a host system? (compilations, software availability etc.)
  - teamski 7y ago
    I use tmux and vim over ssh. You mean Arch? Yeah why not. Then also a lot webpack, Docker: wondering if they would get the Pi stuttering when compiling/building? And if vim is still smooth then (which isn't the case with my 20$ vps).
    - Narishma 7y ago
      AArch64 is what the 64-bit version of the ARM instruction set is called.
      - teamski 7y ago
        ah right, think all software I use are available as AArch64 binaries.
        floatboth 7y ago
        You can get AArch64 in Amazon EC2 by the way… up to 16 A72 cores, which is nice.
- sbr464 7y ago
  Here's a link to a user with an ipad pro setup using the RP4 https://twitter.com/jxnblk/status/1147555688933154816 https://twitter.com/jxnblk/status/1147555688933154816
AkashSKY 7y ago
That's really cool. I'm curious,I envy people like OP for their tenacity. I barely have time to follow what's happening in IT, much less get ahead of the pack in doing cools hacks like this. http://bestbuyonlin.com/%E0%A6%B8%E0%A7%8D%E0%A6%AF%E0%A6%BE%E0%A6%AE%E0%A6%B8%E0%A6%BE%E0%A6%82-%E0%A6%97%E0%A7%8D%E0%A6%AF%E0%A6%BE%E0%A6%B2%E0%A6%BE%E0%A6%95%E0%A7%8D%E0%A6%B8%E0%A6%BF-%E0%A6%8F-70/ http://bestbuyonlin.com/%E0%A6%B8%E0%A7%8D%E0%A6%AF%E0%A6%BE...
mng2 7y ago
Wow, that was quick. Given that this is Broadcom, I don't suppose there is any visibility into the Root Complex? When troubleshooting PCIe it'd be nice to have the LTSSM state at least. Would be really cool to get eye diagrams...
baybal2 7y ago
Raspberry has 2x Gigabit RGMIIs on those SoCs, but they don't wire them out. It is a waste I think
buildbuildbuild 7y ago
Fun. If you want PCIe on a SBC without the soldering, I highly recommend perusing Hackerboards. I'm very happy with my RockPi 4 (4GB RAM, PCIe, USB 3, 6 cores), which I discovered through their excellent database. https://www.hackerboards.com/search_boarddb.php https://www.hackerboards.com/search_boarddb.php
- beatgammit 7y ago
  I think you mean the RockPro64. The Rock64 only has 4 cores and no PCIe. That being said, I missed the PCIe on the specs last time I was comparing SOCs and I had forgotten about hackerboards, thanks for the reminder!
  - buildbuildbuild 7y ago
    http://rockpi.org/ http://rockpi.org/ :)
    - justinclift 7y ago
      Not seeing a mention of PCIe there on the board?
      - gtyras2mrs 7y ago
        Scroll down a bit. M.2 on the backside. Listed under Storage in the specs.
        justinclift 7y ago
        Isn't M.2 storage specific? eg not a useful PCIe slot for anything other than plugging in an SSD
        hipboi 7y ago
        M.2 has full four lanes PCIe, you can use M.2 to PCIe x4 adapter and use all the standard PCIe cards.
mallets 7y ago
I think I will wait for the pi 4 compute model, tyvm.
- pickle-wizard 7y ago
  I really hope the Pi 4 Compute Module breaks out the PCIe on the edge connector.
wang_li 7y ago
Aren't PCIe lanes shared? Why would I need to remove the USB 3.0 chip rather than just hooking right to the pins on the device where it's soldered in place? E: Apparently it's the PCI bus that is shared, not PCI Express lanes. Ty.
- blep-arsh 7y ago
  Nope, point-to-point.
- strmpnk 7y ago
  AFAICT lanes are not shared but there are chipsets which can break lanes out into other sets of lanes which are then routed back onto the original set of lanes. So if your CPU has 16 lanes you can hang a chip off of it which then provides more lanes which are then signaled back to the CPU over some subset of those lanes. It’s not clear if the lanes themselves can be multiplexed with packets from many devices but they can change the number of assigned lanes after initialization so a clever chipset could probably dynamically allocate lanes as used.
  - dgaudet 7y ago
    motherboard features such as x16 or 2x8 are achieved with "pcie mux" chips. these are devices which select which of N pairs of differential wires is attached to the input/output differential pair. search for "pcie mux" will find many, such as [0]. if you look at the diagram you'll see that it connects wire pair A+/A- to either B+/B- or C+/C- based on the value of the SEL line. these generally basic passive devices operating at analog signals level, no higher layer activity required. however some may exist which operate as "retimers", which do participate in the lowest layer of the PCIe electrical protocols (generally to extend reach). these are unlikely to be used for a typical x16 <-> 2x8 sort of motherboard feature though. the example i picked here is 4 lanes, and you would need 4 such chips to do a x16 <-> 2x8. (spoiler: you mux lanes 8-15 from slot X to lanes 0-7 of slot Y, and there are both TX and RX pairs which need muxing.) there do exist devices called "pcie switches" which operate at all layers of the pcie protocols, and allow for all sorts of sharing of the point-to-point links. examples at microsemi [1] ... for example a 48 lane switch could be used to connect two 16 lane GPUs to a 16 lane slot. this would allow either of the GPUs to burst to the full 16 lanes, or on average if both GPUs are communicating with the host then they would see 8 lanes of bandwidth. there's a picture of such a dual GPU card in this article [2], you can see the PCIe switch ASIC centered in between the two GPUs, above and to the right of the edge connector. [0] http://www.ti.com/product/HD3SS3412 http://www.ti.com/product/HD3SS3412 [1] https://www.microsemi.com/product-directory/ics/3724-pcie-switches https://www.microsemi.com/product-directory/ics/3724-pcie-sw... [2] https://graphicscardhub.com/dual-gpu-graphics-cards/ https://graphicscardhub.com/dual-gpu-graphics-cards/
epynonymous 7y ago
this is too awesome! that's quite a lot of work to get the pcie exposed, soldering and such i try to stay away from so great for the author. the form factor of pcie devices doesn't really play well with rpi, but there's definitely a need for faster, more stable persistent storage. i have heard a lot of issues with microsd cards based on wear leveling and such. it would be really nice if rpi could develop like an m2 interconnect where i could install an nvme ssd within the form factor of an rpi, that would make for a truly incredible little machine.
moftz 7y ago
These closeup shots of the VL805 are probably good enough to figure out most of the pins. https://www.viagallery.com/via-labs-vl805/ https://www.viagallery.com/via-labs-vl805/
dwheeler 7y ago
It'd be nice if there was an easier way to do this (vs. removing a chip!). E.g., maybe a dedicated pinout and an easy way to disable the existing use (since the pins can't be shared).
nereid 7y ago
Interesting install a SATA card and make a NAS. I think better than a usb.
- kingosticks 7y ago
  What makes it better than USB for this purpose?
- beatgammit 7y ago
  I would love something with 4x SATA ports for a NAS like this one[1]. I've seen PCIe on devices like these, but I've heard there are issues getting drivers to work properly. I haven't actually tried it, but other limitations (RAM, CPU, Ethernet) have prevented me from actually giving it a shot (I want ZFS, which is a bit memory hungry). The Pi has just enough that I think it's doable. I would absolutely love it if the Raspberry Pi foundation made a version with PCIe instead of USB. - [1] https://www.newegg.com/syba-si-pex40064-sata-iii/p/N82E16816124064 https://www.newegg.com/syba-si-pex40064-sata-iii/p/N82E16816...
  - IronWolve 7y ago
    That rockpi order page had a 4x sata to m2 that looks interesting. https://shop.allnetchina.cn/products/m-2-pci-e-protocol-to-4x-sata3-0-expansion https://shop.allnetchina.cn/products/m-2-pci-e-protocol-to-4...
Millennium 7y ago
If a Pi is capable of this already, why not replace the Ethernet, charging, micro-HDMI, and USB ports with a boatload of type-C Thunderbolt ports (plus support for the HDMI 1.4 alt mode)? Would 8xUSB-C cost that much more than 1xUSB-C+1xEthernet+2xMicro-HDMI+2xUSB3+2xUSB2 (with no PCI Express), in exchange for a considerably more flexible device?
- theonlyklas 7y ago
  Considering the fact that 1 USB C Thunderbolt port can power a dock that could do multiple HDMI, Ethernet, USB, VGA, etc, I agree completely and I hope that they release a Raspberry Pi4C.
  - kingosticks 7y ago
    Remember the 1xUSB-C they have is power only, adding a proper USB-C port is something else entirely.
  - jankotek 7y ago
    Dongles would cost more than Pi
- zokier 7y ago
  Because people do not want to deal with gazillion dongles
  - gambiting 7y ago
    And not just because all those dongles would significantly increase the cost of a relatively cheap device.
- joey_bob 7y ago
  A lot of the benefit of the rpi is the built in IO. Would the rpi still be cost effective without it? Not for many of the IoT projects that make the rpi so popular.
Zenst 7y ago
This is something that will probably prove more accessible with the Zero flavour of the 4.

619 points by trollied 7y ago

That's really cool. I'm curious, would it be possible to use a modern GPU (running at 1x) on an ARM based board? Would the open source drivers that are part of the Kernel work out the box on ARM?

edude03 7y ago
Yes, if you look around, people have even recompiled opensource drivers for RISC-V to use AMD cards.
qdot_me 7y ago
Hack’s creator here - it’s on my list of things to try. GPUs are notoriously hard to get to work on non-intel, having tried to get a few up on Alpha and Itaniums in the past. VideoBIOS expects to run and expects a well behaving Intel CPU to do the power-up. That said X can sometimes emulate these quite well. On ARM we’d also run into alignment issues and likely other quirks - but in principle...
- jchw 7y ago
  I always wondered what the deal was with Video BIOS on modern PCI-express graphics cards. Obviously it must exist, since int 10h still works. Does video BIOS even need to be executed, though? I always assumed it was only necessary for the primary card to be able to display output during boot on PCs. (Otherwise, wouldn't two different cards trample over eachother's implementation of int 10h?) Sorry for the probably-obvious questions, it's sometimes tricky to find good sources for weird information like this that are also up to date.
  - qdot_me 7y ago
    It’s all, cough, implementation dependent. You can run two VideoBIOS implementations, and chances are only one will install an int10h handler, but both will set up the registers and whatnot in the GPUs. My AMD TAHITIs for instance need VideoBIOS to start some form of thermal management loop - otherwise they just run full-throttle on the fan. Then whichever card prevails (BIOS has the ability to select the initialization order) becomes the boot display device. X11 has some (generally working, for well behaved GPUs) emulation of this environment, so that the GPU can initialize late, and even reset under X control. This is how sane cards can work under headless ARM etc. Now, some manufacturers assume you get something like SSE or MMX - VideoBIOS spec technically mandates 386 instruction set only. That crap gets badly emulated. On top of this, drivers can sometimes reinit anyways, from native kernel code. If that happens, the VideoBIOS concerns are moot.
    - jchw 7y ago
      Thanks for the information. I’ve been really curious about this with the incoming ARM dev boards that have full PCI-e edge connectors. Pretty fascinating that X11 of all things is dealing with Video BIOS initialization, though. The sheer amount of functionality that was overloaded into X...
      - floatboth 7y ago
        You don't need X, and I wouldn't assume its remnants of userspace mode setting would work on non-x86. The amdgpu kernel driver POSTs the GPUs when loaded just fine! I run an RX 480 on a MACCHIATObin with FreeBSD and Wayland only :) Plus, there are two ways to get the GPU in UEFI already – X86EmulatorPkg and native aarch64 GOP driver builds provided by AMD.
        jchw 7y ago
        X86EmulatorPkg is pretty cool. I guess if you have a board with UEFI it only makes sense to use something like that.
    - sebazzz 7y ago
      I always assumed the VideoBIOS runs on the GPU, but if I understand you correctly this is code run on the CPU?
      - floatboth 7y ago
        Yes — "VBIOS", and these days mostly "UEFI GOP driver", is code that runs on the CPU and allows the system firmware to initialize the card and preset video output before any operating system is booted. Without it, you cannot enter the setup screen graphically, only over a serial port :D
      - ComputerGuru 7y ago
        Yes, the PC (or whatever) hands off control of the entire system to the VideoBIOS which must do its thing and then pass control back.
- pm215 7y ago
  People have got at least some cards working on server-class arm setups. On the other hand I think they might be using the 'emulate x86 uefi rom in an arm uefi system' hack, so there might be more effort required on a non-uefi setup. Still, it should have shaken out most of the alignment and memory-cacheability-attibutes issues already, hopefully.
  - floatboth 7y ago
    Yep, the "hack" (X86EmulatorPkg) allows running the card in the UEFI out of the box. AMD also provides native aarch64 builds of their UEFI GOP driver though. And none of this is necessary for running the GPU in the OS – amdgpu POSTs the GPU just fine. > memory-cacheability-attibutes issues Recently I've added aarch64 support to FreeBSD's port of the DRM/KMS drivers :) Took a couple hours to realize that our implementations of Linux's mapping functions used normal uncacheable memory instead of device memory – fixing that stopped the hangs on driver load and allowed everything to work. Then there was some corruption on the screen – our drm-kms is from Linux 5.0 for now, and I've had to cherry-pick a fix that only landed in 5.1 I think: https://patchwork.kernel.org/patch/10778815/ https://patchwork.kernel.org/patch/10778815/
    - pm215 7y ago
      You wouldn't believe how much discussion it took to decide that that minor drm_cache.h change really was the right fix for that issue :-)
- timthorn 7y ago
  Some AMD cards have an Arm build of their UEFI firmware available, which does allow video from boot: https://developer.arm.com/architectures/system-architectures/software-standards/uefi-drivers https://developer.arm.com/architectures/system-architectures...
- floatboth 7y ago
  VBIOS is often not necessary for running a GPU in the OS. The amdgpu driver can POST a GPU by itself just fine. Still… X86EmulatorPkg allows running an amd64 VBIOS in UEFI on an aarch64 machine :) AFAIK the bigger problem on embedded boards is half assed Synopsys Designware host controllers. I have a Radeon running on my Marvell MACCHIATObin, on FreeBSD even. But from what I've heard the Rockchip RK3399 has a worse version of the controller, and people trying GPUs on the ROCKPro64 saw errors related to not large enough BAR space or something. UPD: yeah, someone in the thread mentioned BAR space issues wrt NXP i.MX SoCs, that's probably what's happening on Rockchip. Would be amazing if the Broadcom chip in the Pi turns out to be the one with enough BAR space! :D
  - markus92 7y ago
    Could you clarify to me/give a short definition of BAR space? For obvious reasons it's a bit hard to search for :)
    - cesarb 7y ago
      In PCI, BAR is Base Address Register, which is a register in the PCI device's configuration space which defines where in the machine's physical memory address space that particular window of memory and/or I/O will be mapped (a single device can have several BARs, for instance a simple graphics card could have one for its control registers and one for the framebuffer). So the "BAR space" would be a shorthand for "the region of the physical memory address space which can be used to map the PCI devices memory through their Base Address Registers". The size of this region is limited, and graphics cards in particular tend to have somewhat large BARs. (See for yourself in your machine: run "lspci -v", the lines starting with "Memory at ..." or "I/O ports at ..." are the BARs.)
    - bluecmd 7y ago
      Search "PCIe bar space" and you should be fine
- Teknoman117 7y ago
  SiFive has a few videos where they have their RISC-V core running an AMD gpu (via the in kernel amdgpu driver).
- bstar77 7y ago
  "GPUs are notoriously hard to get to work on non-intel" I don't understand this statement... accelerated graphics have no issues working on any x86 system as well as ARM based linux systems (not to mention every cell phone in the past 10 years). Itanium is also Intel so that statement contradicts your previous statement.
  - qdot_me 7y ago
    To a limited extent. The accelerated graphics on ARM (and cell phones) has always been a system integrator problem. Until very recently, they were not user customizable - which means the initialisation was part of the proprietary firmware and hard to port across even different implementations of the same SoC. The x86 world had the advantage of user upgradable GPUs, which necessitates standardization and common firmware. On top of which, the accelerated graphics of cell phones is a horrible kludge of various standards.
- notaplumber 7y ago
  OpenBSD has radeondrm enabled on mips64, powerpc and arm64. It can initialize the board just fine without emulating x86, since KMS. Working 3D acceleration is another story. OpenBSD even has (or had, build breaks often) packages for chromium browser on arm64 that were tested on this.
  - floatboth 7y ago
    > Working 3D acceleration is another story Why? I recently enabled amdgpu on FreeBSD arm64 — as soon as KMS happened, 3D just worked. There was some buffer corruption (https://user-images.githubusercontent.com/208340/60443774-97243900-9c24-11e9-861a-6dd29101c235.png https://user-images.githubusercontent.com/208340/60443774-97...) for which I've had to cherry-pick this tiny Linux patch: https://patchwork.kernel.org/patch/10778815/ https://patchwork.kernel.org/patch/10778815/ > packages for chromium browser on arm64 And we have Firefox, which doesn't break, has a working JS JIT, works on Wayland, renders with WebRender :)
- vardump 7y ago
  > On ARM we’d also run into alignment issues... Just don't use ARMv5 or older. ARMv6+ supports unaligned access.
  - planteen 7y ago
    At least on ARMv7, it's up to the MMU, not the CPU. I've worked with Cortex-A9s that trap the unaligned access. You can write a handler or crash.
    - vardump 7y ago
      Interesting. I've kept my accesses aligned, but noticed that unaligned ones worked. Didn't know about the exception mechanism behind, but yeah, should have err... expected it. By the way, you can also opt to crash on x86. Just need to enable bit AM (Alignment Mask) bit 18) in CR0 (kernel stuff) and AC (Alignment Check, also bit 18) in EFLAGS. After that point, unaligned accesses trap. I use this in my CPU JIT/dynamic compiler to trap target unaligned accesses. https://en.wikipedia.org/wiki/Control_register#CR0 https://en.wikipedia.org/wiki/Control_register#CR0
mntmn 7y ago
You might run into address space issues. I haven’t checked Broadcom PCIe documentation for RPi4 (is there any?), but I tried a very similar hack with i.MX6 and older AMD and nVidia cards. They get recognized fine, but BARs cannot be mapped because they don’t fit in i.MX6’s tiny 16MB PCIe space.
jamesfmilne 7y ago
NVIDIA drivers might even work too Linux 32-bit ARM drivers https://www.nvidia.co.uk/Download/driverResults.aspx/137468/en-uk https://www.nvidia.co.uk/Download/driverResults.aspx/137468/...
- whitelabrat 7y ago
  This would be incredible if gotten to work with a pi. I've been eyeing the Jetson Nano, but if I could use a pi with a V100, that would be hilarious. And awesome.
  - cr0sh 7y ago
    Based on what I've seen - it's on my list of projects to "try out", but rapidly becoming a more obsolete idea the longer I wait (but then again, I've got an Altair on my todo list, so meh)... Taking a low-cost Mini ITX board and pairing it with a decent mini-sized GTX 750 TI SC video card (I prefer the EVGA one) should yield a system that is somewhat on par with some of NVidia's embedded ML offerings, with the tradeoff of size and efficiency - but at a significantly reduced cost. Add a very short PCIe riser ribbon (or, if you can find one tall enough, a right-angle riser), and you could lay the card over "flat" above the CPU (using a 1U cooler/heatsink); you'd have to make a custom mounting frame of course, but I think you could make the whole package relatively compact. From my limited experience using Tensorflow with the 750 - it is a very capable card in that capacity and relatively inexpensive today. If you were willing to spend a bit more, there are mini-sized NVidia GPUs available in more recent models; of course if space is not an issue, then full-length GPUs can be substituted (my "goal" was to build such a system as close to within the footprint of a Mini-ITX board as possible). As far as the Mini-ITX motherboard is concerned, again, size to your budget. That said, if you decided to build this system using cheap pre-owned components, maybe an I5 with 8 gig - you could probably get it going for under $250.00 USD, maybe less.
true_tuna 7y ago
If you want it for ML you could check out the Coral TPU. (Tensor Processing Unit over USB) I picked up a couple of those yesterday.

yetihehe 7y ago

Hmm, how about using this for fast interconnect for making rpi clusters?

hvidgaard 7y ago
Wouldn't the gigabit LAN be a better fit for this? If you want to make a cluster, you need to make some custom hardware to connect to, that can facilitate the communication. At this point you're likely spending more that if you just bought a real desktop for more performance. I can see the fun factor in hacking the system together, though.
- qdot_me 7y ago
  There are two ways of doing clusters - one is a message passing paradigm, which you can do over Ethernet (to an extent - I’d still take USB3 for 4x the bandwidth) - and the other is direct memory access a’la Cray. What really motivated me to do this hack is the relative abundance of stuff I can now plug into an FPGA :)
  - hvidgaard 7y ago
    I was thinking Ethernet because, A:It's cheap to buy a switch and cluster 100 RPi, B: You can have a desktop with a faster NIC and keep the RPis busy. But as with everything high performance, it depends entirely on the use case.
    - qdot_me 7y ago
      True. And with RPi4 having a 1000baseT, it’s not as painful as it seems. Perhaps even the driver can be coaxed into some form of DMA and MPI that is a bit lower latency than IP stack. With secondary IP layer on 802.11, it might actually work reasonably well.
      - consp 7y ago
        Or use the PCIe mod and use an infiniband card for low latency and high thoughput.

segfaultbuserr 7y ago

It looks like an unreliable modification. Running a GHz-level interface with jumpers is almost impossible to control the impedance, it's a cool Proof-of-Concept though. But is it possible to bring the project to the next level? Is it possible to make a daughterboard with QFN connector? If so, one can make a pin-compatible daughterboard with an extension connector. To use it, just desolder the USB chip and solder a new daughterboard on it, and you're ready to go. It would be one of the coolest Raspberry Pi projects!

qdot_me 7y ago
Yup. The daughterboard is on my mind. Likely flex-PCB and that’s gonna take a week or two to respin. Hence I’m collecting ideas for various daughtercards I could cram in a panel before sending it off - straight-through to riser via USB3, expresscard SMT, maybe through-hole 1x? That said, PCIe phy’s are extremely robust - they do most of the impedance matching and delay mismatch training. And if you don’t ruin the onboard caps, this could be jumpered straight across.
- segfaultbuserr 7y ago
  Great! I was thinking about using a SMT ribbon cable connector because of the limited available space, but apparently it won't be an issue if you raise the board high enough? Anyway, if this project ever goes to batch-production, make sure to update your blog when the funding campaign starts!
mkesper 7y ago
Maybe a new raspberry compute module (https://www.raspberrypi.org/blog/raspberry-pi-compute-module-new-product/ https://www.raspberrypi.org/blog/raspberry-pi-compute-module...) featuring PCIe could be implemented?
- HeWhoLurksLate 7y ago
  Oh heck yeah, that'd be amazing. I'm unfamiliar with this- do the compute modules have any sequential unused pins on them?

boyadjian 7y ago

Why not, but I will wait for a classic ATX ARM motherboard instead. It should happen this year.

steevdave 7y ago
https://store.avantek.co.uk/arm-desktops.html https://store.avantek.co.uk/arm-desktops.html bit pricey but it exists already
imtringued 7y ago
Those will never be as cheap as an SBC with a SoC that was meant for some random discontinued TV box.
mpoloton 7y ago
Now there are many ARM single board computers such as RockPi with PCI-E connectors.
hawski 7y ago
Is Mini-ITX for $550 acceptable? https://www.solid-run.com/nxp-lx2160a-family/honeycomb-workstation/ https://www.solid-run.com/nxp-lx2160a-family/honeycomb-works... 16 core A72 CPU, up to 64GB RAM, PICe x8, 10GbE SFP+ ports and 1GbE RJ45, SATA. It's early access hardware, so it may have some pain points. Normal units will be available from the end of the year for $750. If I would have time and money right now, that's what I would buy.
- floatboth 7y ago
  It's only pre-order right now, and the firmware isn't done. They promise SBSA compliance (which includes ECAM PCIe working via a generic ACPI attachment) but they haven't passed the full test suite yet. Some experts are skeptical about whether full compliance is possible on that NXP chip… I hope the PCIe works fine. And I hope the firmware will be FOSS like on their MACCHIATObin. One thing they revealed is that the chip is overclockable (including memory), which is awesome. IIRC they got 2.5ish GHz core clock working. Would be amazing if it does like 3GHz with a voltage boost. (I don't expect software voltage control… but there's always hard mods :D)

zaro 7y ago

For anybody interested in PCI Express on Arm there are already boards with PCI express connector, like https://store.pine64.org/?product=rockpro64-4gb-single-board-computer https://store.pine64.org/?product=rockpro64-4gb-single-board...

peterburkimsher 7y ago

Nice work! Would this be compatible with an M.2 to PCIe adaptor? [1] Being able to attach an Intel 660P and get 2 TB of fast SSD storage on a Raspberry Pi would be sweet. [1] https://www.amazon.com/EZDIY-FAB-Express-Adapter-Support-22110/dp/B01GCXCR7W https://www.amazon.com/EZDIY-FAB-Express-Adapter-Support-221...

elFarto 7y ago
Technically yes, but it's only a PCIe 1x Gen2 slot, so only 500MB/s of bandwidth (4x Gen3 is ~4GB/s). You'd be better off with a USB 3.0 to M.2 adapter.
- londons_explore 7y ago
  The bandwidth works out about the same, but the USB controller and AHCI controllers will add quite some latency (and CPU load). I'd like to see benchmarks, but my guess is single-thread random 4k read performance will more than double via PCI Express rather than the USB.

noobermin 7y ago

Now this is the content I come to HN for. A serious hack just days after the 4 was released. Kudos to the OP. I envy people like OP for their tenacity. I barely have time to follow what's happening in IT, much less get ahead of the pack in doing cools hacks like this.

userbinator 7y ago
On the other hand, the fact that the RPi ecosystem remains notoriously proprietary (even the USB controller is a bastard variant that has next to no documentation --- of all the ones available, they had to choose that one) continues to be disappointing. I definitely like this sort of hack, but such hacks with documentation already available (and doing more than documented, basically) are certainly preferable.
anderspitman 7y ago
I'm sure you are but just in case are you aware of hackaday.com?
- noobermin 7y ago
  Of course! HN is just a good digest of things on the maker side to the fairly abstract CS stuff to math and physics and all in between. It sure beats the political stuff that becomes tiresome (even though I succumb and engage in it like many posters here).

mrb 7y ago

Ohh I see where this is going qdot_me — would love to be able to hook up a GPU to a RPi4 to crypto-mine. So useful for so many applications ! I will seriously fund you if you can make this happen, hit me up by email. My contact info is on zorinaq.com

mrb 7y ago
Whoa, downvoted to minus 4. No idea why. In my 9 years of HN, this is my worst downvoted comment. Maybe people thought I was being off-topic? Let me clarify a bit for the public here why the comment is relevant: in the crypto mining community, some groups are looking into what minimal single board computer can provide a PCIe signal to connect a single GPU. Idea is to be cheap and reliable. If you have a many-PCIe board failure (http://bitcoin.zorinaq.com/many_pcie/ http://bitcoin.zorinaq.com/many_pcie/) you have 10-20 GPUs going down at once. Not good. By isolating each GPU on its own motherboard, you can isolate failures, thus increase mining profits. When I saw the OP mention cryptocurrency in the blog post, I thought hey maybe that's what he is looking to do...

teamski 7y ago

Hijacking the thread: I develop remotely on VPSes because I like to have an always-on box reachable from any client. I am wondering if a RP4 offers a similar experience at lower cost. Does anyone use a RP for this?

numlock86 7y ago
I do this with an RPi3 and it's doing good, so it's doable. It strongly depends on your setup and develeopment environment, though. Do you want VI to work over SSH or want full VNC access to a machine with Gnome and Eclipse? Or something in between like X forwarding? Also, is aarch64 even an option as a host system? (compilations, software availability etc.)
- teamski 7y ago
  I use tmux and vim over ssh. You mean Arch? Yeah why not. Then also a lot webpack, Docker: wondering if they would get the Pi stuttering when compiling/building? And if vim is still smooth then (which isn't the case with my 20$ vps).
  - Narishma 7y ago
    AArch64 is what the 64-bit version of the ARM instruction set is called.
    - teamski 7y ago
      ah right, think all software I use are available as AArch64 binaries.
      - floatboth 7y ago
        You can get AArch64 in Amazon EC2 by the way… up to 16 A72 cores, which is nice.
sbr464 7y ago
Here's a link to a user with an ipad pro setup using the RP4 https://twitter.com/jxnblk/status/1147555688933154816 https://twitter.com/jxnblk/status/1147555688933154816

AkashSKY 7y ago

That's really cool. I'm curious,I envy people like OP for their tenacity. I barely have time to follow what's happening in IT, much less get ahead of the pack in doing cools hacks like this. http://bestbuyonlin.com/%E0%A6%B8%E0%A7%8D%E0%A6%AF%E0%A6%BE%E0%A6%AE%E0%A6%B8%E0%A6%BE%E0%A6%82-%E0%A6%97%E0%A7%8D%E0%A6%AF%E0%A6%BE%E0%A6%B2%E0%A6%BE%E0%A6%95%E0%A7%8D%E0%A6%B8%E0%A6%BF-%E0%A6%8F-70/ http://bestbuyonlin.com/%E0%A6%B8%E0%A7%8D%E0%A6%AF%E0%A6%BE...

mng2 7y ago

Wow, that was quick. Given that this is Broadcom, I don't suppose there is any visibility into the Root Complex? When troubleshooting PCIe it'd be nice to have the LTSSM state at least. Would be really cool to get eye diagrams...

baybal2 7y ago

Raspberry has 2x Gigabit RGMIIs on those SoCs, but they don't wire them out. It is a waste I think

buildbuildbuild 7y ago

Fun. If you want PCIe on a SBC without the soldering, I highly recommend perusing Hackerboards. I'm very happy with my RockPi 4 (4GB RAM, PCIe, USB 3, 6 cores), which I discovered through their excellent database. https://www.hackerboards.com/search_boarddb.php https://www.hackerboards.com/search_boarddb.php

beatgammit 7y ago
I think you mean the RockPro64. The Rock64 only has 4 cores and no PCIe. That being said, I missed the PCIe on the specs last time I was comparing SOCs and I had forgotten about hackerboards, thanks for the reminder!
- buildbuildbuild 7y ago
  http://rockpi.org/ http://rockpi.org/ :)
  - justinclift 7y ago
    Not seeing a mention of PCIe there on the board?
    - gtyras2mrs 7y ago
      Scroll down a bit. M.2 on the backside. Listed under Storage in the specs.
      - justinclift 7y ago
        Isn't M.2 storage specific? eg not a useful PCIe slot for anything other than plugging in an SSD
        hipboi 7y ago
        M.2 has full four lanes PCIe, you can use M.2 to PCIe x4 adapter and use all the standard PCIe cards.

mallets 7y ago

I think I will wait for the pi 4 compute model, tyvm.

pickle-wizard 7y ago
I really hope the Pi 4 Compute Module breaks out the PCIe on the edge connector.

wang_li 7y ago

Aren't PCIe lanes shared? Why would I need to remove the USB 3.0 chip rather than just hooking right to the pins on the device where it's soldered in place? E: Apparently it's the PCI bus that is shared, not PCI Express lanes. Ty.

blep-arsh 7y ago
Nope, point-to-point.
strmpnk 7y ago
AFAICT lanes are not shared but there are chipsets which can break lanes out into other sets of lanes which are then routed back onto the original set of lanes. So if your CPU has 16 lanes you can hang a chip off of it which then provides more lanes which are then signaled back to the CPU over some subset of those lanes. It’s not clear if the lanes themselves can be multiplexed with packets from many devices but they can change the number of assigned lanes after initialization so a clever chipset could probably dynamically allocate lanes as used.
- dgaudet 7y ago
  motherboard features such as x16 or 2x8 are achieved with "pcie mux" chips. these are devices which select which of N pairs of differential wires is attached to the input/output differential pair. search for "pcie mux" will find many, such as [0]. if you look at the diagram you'll see that it connects wire pair A+/A- to either B+/B- or C+/C- based on the value of the SEL line. these generally basic passive devices operating at analog signals level, no higher layer activity required. however some may exist which operate as "retimers", which do participate in the lowest layer of the PCIe electrical protocols (generally to extend reach). these are unlikely to be used for a typical x16 <-> 2x8 sort of motherboard feature though. the example i picked here is 4 lanes, and you would need 4 such chips to do a x16 <-> 2x8. (spoiler: you mux lanes 8-15 from slot X to lanes 0-7 of slot Y, and there are both TX and RX pairs which need muxing.) there do exist devices called "pcie switches" which operate at all layers of the pcie protocols, and allow for all sorts of sharing of the point-to-point links. examples at microsemi [1] ... for example a 48 lane switch could be used to connect two 16 lane GPUs to a 16 lane slot. this would allow either of the GPUs to burst to the full 16 lanes, or on average if both GPUs are communicating with the host then they would see 8 lanes of bandwidth. there's a picture of such a dual GPU card in this article [2], you can see the PCIe switch ASIC centered in between the two GPUs, above and to the right of the edge connector. [0] http://www.ti.com/product/HD3SS3412 http://www.ti.com/product/HD3SS3412 [1] https://www.microsemi.com/product-directory/ics/3724-pcie-switches https://www.microsemi.com/product-directory/ics/3724-pcie-sw... [2] https://graphicscardhub.com/dual-gpu-graphics-cards/ https://graphicscardhub.com/dual-gpu-graphics-cards/

epynonymous 7y ago

this is too awesome! that's quite a lot of work to get the pcie exposed, soldering and such i try to stay away from so great for the author. the form factor of pcie devices doesn't really play well with rpi, but there's definitely a need for faster, more stable persistent storage. i have heard a lot of issues with microsd cards based on wear leveling and such. it would be really nice if rpi could develop like an m2 interconnect where i could install an nvme ssd within the form factor of an rpi, that would make for a truly incredible little machine.

moftz 7y ago

These closeup shots of the VL805 are probably good enough to figure out most of the pins. https://www.viagallery.com/via-labs-vl805/ https://www.viagallery.com/via-labs-vl805/

dwheeler 7y ago

It'd be nice if there was an easier way to do this (vs. removing a chip!). E.g., maybe a dedicated pinout and an easy way to disable the existing use (since the pins can't be shared).

nereid 7y ago

Interesting install a SATA card and make a NAS. I think better than a usb.

kingosticks 7y ago
What makes it better than USB for this purpose?
beatgammit 7y ago
I would love something with 4x SATA ports for a NAS like this one[1]. I've seen PCIe on devices like these, but I've heard there are issues getting drivers to work properly. I haven't actually tried it, but other limitations (RAM, CPU, Ethernet) have prevented me from actually giving it a shot (I want ZFS, which is a bit memory hungry). The Pi has just enough that I think it's doable. I would absolutely love it if the Raspberry Pi foundation made a version with PCIe instead of USB. - [1] https://www.newegg.com/syba-si-pex40064-sata-iii/p/N82E16816124064 https://www.newegg.com/syba-si-pex40064-sata-iii/p/N82E16816...
- IronWolve 7y ago
  That rockpi order page had a 4x sata to m2 that looks interesting. https://shop.allnetchina.cn/products/m-2-pci-e-protocol-to-4x-sata3-0-expansion https://shop.allnetchina.cn/products/m-2-pci-e-protocol-to-4...

Millennium 7y ago

If a Pi is capable of this already, why not replace the Ethernet, charging, micro-HDMI, and USB ports with a boatload of type-C Thunderbolt ports (plus support for the HDMI 1.4 alt mode)? Would 8xUSB-C cost that much more than 1xUSB-C+1xEthernet+2xMicro-HDMI+2xUSB3+2xUSB2 (with no PCI Express), in exchange for a considerably more flexible device?

theonlyklas 7y ago
Considering the fact that 1 USB C Thunderbolt port can power a dock that could do multiple HDMI, Ethernet, USB, VGA, etc, I agree completely and I hope that they release a Raspberry Pi4C.
- kingosticks 7y ago
  Remember the 1xUSB-C they have is power only, adding a proper USB-C port is something else entirely.
- jankotek 7y ago
  Dongles would cost more than Pi
zokier 7y ago
Because people do not want to deal with gazillion dongles
- gambiting 7y ago
  And not just because all those dongles would significantly increase the cost of a relatively cheap device.
joey_bob 7y ago
A lot of the benefit of the rpi is the built in IO. Would the rpi still be cost effective without it? Not for many of the IoT projects that make the rpi so popular.

Zenst 7y ago

This is something that will probably prove more accessible with the Zero flavour of the 4.