Hmmm.... Wondering if this could be eventually used to emulate a PCIe card using another device, like a RaspberryPi or something more powerful... Thinking the idea of a card you could stick in a machine, anything from a 1x to 16x slot, that emulates a network card (you could run VPN or other stuff on the card and offload it from the host) or storage (running something with enough power to run ZFS and a few disks, and show to the host as a single disk, allowing ZFS on devices that would not support it). but this is probably not something easy...
Indeed, and even then, there's some sw-hw-codesign stuff that kinda helps you do what PCIem does but it's usually really pricey; so I kinda thought it'd be a good thing to have for free.
PCIe prototyping is usually not something super straightforward if you don't want to pay hefty sums IME.
immibis2 hours ago
The "DMA cards" used for video game cheating are generic PCIe cards and (at least the one I got) comes with open documentation (schematics, example projects etc).
the_biotjust now
What's this? Hardware specifically for game cheating? Got any links?
cakehonolulu3 hours ago
Hi! Author here! You can technically offload the transactions the real driver on your host does to wherever you want really. PCI is very delay-tolerant and it usually negotiates with the device so I see not much of an issue doing that proven that you can efficiently and performantly manage the throughput throughout the architecture. The thing that kinda makes PCIem special is that you are pretty much free to do whatever you want with the accesses the driver does, you have total freedom. I have made a simple NVME controller (With a 1GB drive I basically malloc'd) which pops up on the local PCI bus (And the regular Linux's nvme block driver attaches to it just fine). You can format it, mount it, create files, folders... it's kinda neat. I also have a simple dumb rasteriser that I made inside QEMU that I wanted to write a driver for, but since it doesn't exist, I used PCIem to help me redirect the driver writes to the QEMU instance hosting the card (Thus was able to run software-rendered DOOM, OpenGL 1.X-based Quake and Half-Life ports).
yndoendo1 hour ago
Just to hijack this thread on how resilient PCIe is. PS4 Linux hackers ran PCIe over UART serial connection to reverse engineer the GPU. [0] [1]
Can one make a PCIe analyzer out of your code base by proxy all transactions thru a virtual PCIem driver to a real driver?
jacquesm3 hours ago
Fantastic tool, thank you for making this it is one of those things that you never knew you needed until someone took the time to put it together.
gigatexal3 hours ago
This is really interesting. Could it be used to carve up a host GPU for use in a guest VM?
cakehonolulu3 hours ago
As in, getting the PCIem shim to show up on a VM (Like, passthrough)? If that's what you're asking for, then; it's something being explored currently. Main challenges come from the subsystem that has to "unbind" the device from the host and do the reconfiguration (IOMMU, interrupt routing... and whatnot). But from my initial gatherings, it doesn't look like an impossible task.
fc417fc8021 hour ago
> carve up
Passthru or time sharing? The latter is difficult because you need something to manage the timeslices and enforce process isolation. I'm no expert but I understand it to be somewhere between nontrivial and not realistic without GPU vendor cooperation.
Note that the GPU vendors all deliberately include this feature as part of their market segmentation.
benreesmanjust now
It would need to implement a few dozen ioctls, correctly stub the kernel module in guests, do a probably memory-safe assignment of GPU memory to guest, and then ultimately map that info to BAR/MSI-X semantics of a real kernel module. You could get VFIO pretty fast for a full start by correctly masking LTR bits, but to truly make it free you'd need a user space io_uring broker that had survived hundreds of millions of adversarial fuzz runs because there's only so fast the firmware blob can run even if it's preloaded into initramfs.
Serious work, detail intense, but not so different in design to e.g. Carmack's Trinity engine. Doable.
xerxes9016 hours ago
Something like the stm32mp2 series of MCUs can run Linux and act as a PCIe endpoint you can control from a kernel module on the MCU. So you can program an arbitrary PCIe device that way (although it won’t be setting any speed records, and I think the PHY might be limited to PCIe 1x)
tiernano5 hours ago
interesting... x1 would too slow for large amounts of storage, but as a test, a couple small SSDs could potentially be workable... sounds like im doing some digging...
jacquesm3 hours ago
There are many workloads that would not be able to saturate even an x1 link, it all depends on how much of the processing can be done internally to whatever lives on the other side of that link. Raw storage and layer-to-layer communications in AI applications are probably the worst cases but there are many more that are substantially better than that.
cakehonolulu3 hours ago
If there's any particular feature you feel you are missing on PCIem or anything, feel free to open an Issue and I'll look into it ;)
that is a huge win if you are developing drivers or even real hardware. it allows to iterate on protokols just with the press of a button
asimovDev3 hours ago
Could you explain in layman terms how it would help with developing PCIE hardware / drivers? I can immediately imagine something like writing more robust unit tests and maybe developing barebones drivers before you get access to actual hardware, but that's where my imagination runs out of fuel.
cakehonolulu2 hours ago
Sure! Let's say you (Or the company you work for) are trying to develop an NVME controller card, or a RAID card, or a NIC...
Usually, without actual silicon, you are pretty limited on what you can do in terms of anticipating the software that'll run.
What if you want to write a driver for it w/o having to buy auxiliary boards that act as your card?
What happens if you already have a driver and want to do some security testing on it but don't have the card/don't want to use a physical one for any specific reason (Maybe some UB on the driver pokes at some register that kills the card? Just making disastrous scenarios to prove the point hah).
What if you want to add explicit failures to the card so that you can try and make the driver as tamper-proof and as fault-tolerant as possible (Think, getting the PCI card out of the bus w/o switching the computer off)?
Testing your driver functionally and/or behaviourally on CI/CD on any server (Not requiring the actual card!)?
There's quite a bunch of stuff you can do with it, thanks to being in userspace means that you can get as hacky-wacky as you want (Heck, I have a dumb-framebuffer-esque and OpenGL 1.X capable QEMU device I wanted to write a driver for fun and I used PCIem to forward the accesses to it).
cakehonolulu3 hours ago
Indeed, the project has gone through a few iterations already (It was first a monolithic kernel module that required a secondary module to call into the API and whatnot). I've went towards a more userspace-friendly usage mainly so that you can iterate your changes much, much faster. Creating the synthetic PCI device is as easy as opening the userspace shim you program, it'll then appear on your bus. When you want to test new changes, you close the shim normally (Effectively removing it from the bus) and you can do this process as many times as needed.
LarsKrimi3 hours ago
Latching on to this thread, but can you make as simple as possible of an example?
Something like just a single BAR with a register that printfs whatever is written
Hi, thanks. That's almost it. The remaining problem is just how to tie it together (where do I put the handle_mmio_read pointer or which event should it be handled in?)
PCIEM_EVENT_MMIO_READ is defined but not used anywhere in the codebase
cakehonolulu1 hour ago
Hi! Sorry, this is an issue on my side; I forgot to update the documentation's example with the latest changes.
You basically have the kernel eventfd notify you about any access triggered (Based on your configuration), so from userspace, you have the eventfd and then you mmap the shared lock-less ring buffer that actually contains the events PCIem notifies (So you don't end up busy polling).
You basically mmap a struct pciem_shared_ring where you'll have your usual head/tail pointers.
From then on, on your main, you'd have a select() or a poll() for the eventfd; when PCIem notifies the userspace you'd check head != tail (Which means there are events to process) and you can basically do:
And that's it, don't forget to update the head pointer!
I'll go and update the docs now. Hopefully this clears stuff up!
agent0132 hours ago
I've been burned before by driver bugs that only manifested under very specific timing conditions or malformed responses from the device, tnx
cakehonolulu2 hours ago
Anytime, hopefully it fits your needs and helps you not spend more time than needed tracing issues like this. Thanks for the comment!
throwaway1324485 hours ago
Tangential question: PCIe is a pretty future-proof technology to learn/invest in,
right? As in,
it is very unlikely to become obsolete in the next 5-10 years (like USB)?
pjc505 hours ago
Neither of those is going to be obsolete in 5 years. Might get rebadged and a bunch of extensions, but there's such a huge install base that rapid change is unlikely. Neither Firewire nor Thunderbolt unseated USB.
formerly_proven4 hours ago
USB4 is the ~third USB protocol stack though (USB1/2 being basically the same iirc, USB3 being a completely separate protocol that neither logically nor physically interacts with USB1/2 at all), heavily based on Thunderbolt to the point of backwards compatibility.
p_l3 hours ago
USB4 is essentially thunderbolt with some new features and some features being optional instead of mandatory.
formerly_proven1 hour ago
A very noticeable feature is that USB4 can tunnel USB3, which means it works like an USB hub, instead of an external PCIe USB controller (like in Thunderbolt). USB2 is still just physically separately transported over the D+/D- pins.
CupricTea1 hour ago
PCIe is probably the most future proof technology we have right now. Even if it is upheaveled at the hardware level, from the software perspective it just exposes a device's arbitrary registers to some memory mapped location. Software drivers for PCIe devices will continue to work the same.
neocron5 hours ago
Might as well be replaced by optical connectors next years, but who knows in advance. Currently there is no competition
pjc502 hours ago
Hmm. What's the current maths on distance vs edge rate vs transceiver latency vs power consumption on when that would be a benefit? Not to mention how much of a pain it is to have good optical connectors.
I wouldn't expect that to be mainstream until after optical networking becomes more common, and for consumer hardware that's very rare (apart from their modem).
tiernano5 hours ago
even though it would be optical, it still is using PCIe protocols in the background...
bobmcnamara1 hour ago
PCIe is still using PCI protocol just over serdes
embedding-shape4 hours ago
How could you possibly know exactly what protocol they'd be using for the potential future optical PCIe connection? Your guess is as good as anyone's, no?
p_l3 hours ago
Probably because optical PCI-E is an old thing by now.
In fact, "zero~th generation" of thunderbolt used optical link, too. Also both thunderbolt and DisplayPort reuse a lot of common elements from PCI-E
GrowingSidewaysjust now
PCIe expertise will certainly outlive anyone on this forum.
checker6595 hours ago
Curious what you mean by learning? Learning about TLPs? Learning about FPGA DMA Engines like XDMA? Learning about PCIe switches / retimers? Learning about `lspci`?
throwaway1324483 hours ago
Nothing specific! I learned how to implement USB(-C) because there was some specific hardware I wanted to create. I could see the same thing happening with PCIe in future. With USB its longevity was fairly obvious to me, with PCIe I’m not well informed. Thanks for giving me some acronyms to explore!
The other existing solution to this is FPGA cards: https://www.fpgadeveloper.com/list-of-fpga-dev-boards-for-pc... - note the wide spread in price. You then also have to deal with FPGA tooling. The benefit is much better timing.
PCIe prototyping is usually not something super straightforward if you don't want to pay hefty sums IME.
[0] https://www.psdevwiki.com/ps4/PCIe
[1] https://fail0verflow.com/blog/2016/console-hacking-2016-post...
Passthru or time sharing? The latter is difficult because you need something to manage the timeslices and enforce process isolation. I'm no expert but I understand it to be somewhere between nontrivial and not realistic without GPU vendor cooperation.
Note that the GPU vendors all deliberately include this feature as part of their market segmentation.
Serious work, detail intense, but not so different in design to e.g. Carmack's Trinity engine. Doable.
Seems unlikely you'd emulate a real PCIe card in software because PCIe is pretty high-speed.
https://mikrotik.com/product/ccr2004_1g_2xs_pcie
and G-RAID
Usually, without actual silicon, you are pretty limited on what you can do in terms of anticipating the software that'll run.
What if you want to write a driver for it w/o having to buy auxiliary boards that act as your card? What happens if you already have a driver and want to do some security testing on it but don't have the card/don't want to use a physical one for any specific reason (Maybe some UB on the driver pokes at some register that kills the card? Just making disastrous scenarios to prove the point hah).
What if you want to add explicit failures to the card so that you can try and make the driver as tamper-proof and as fault-tolerant as possible (Think, getting the PCI card out of the bus w/o switching the computer off)?
Testing your driver functionally and/or behaviourally on CI/CD on any server (Not requiring the actual card!)?
There's quite a bunch of stuff you can do with it, thanks to being in userspace means that you can get as hacky-wacky as you want (Heck, I have a dumb-framebuffer-esque and OpenGL 1.X capable QEMU device I wanted to write a driver for fun and I used PCIem to forward the accesses to it).
Something like just a single BAR with a register that printfs whatever is written
Hopefully this is what you're searching for!
PCIEM_EVENT_MMIO_READ is defined but not used anywhere in the codebase
You basically have the kernel eventfd notify you about any access triggered (Based on your configuration), so from userspace, you have the eventfd and then you mmap the shared lock-less ring buffer that actually contains the events PCIem notifies (So you don't end up busy polling).
You basically mmap a struct pciem_shared_ring where you'll have your usual head/tail pointers.
From then on, on your main, you'd have a select() or a poll() for the eventfd; when PCIem notifies the userspace you'd check head != tail (Which means there are events to process) and you can basically do:
struct pciem_event *event = &event_ring->events[head]; atomic_thread_fence(memory_order_acquire); if (event->type == PCIEM_EVENT_MMIO_WRITE) handle_mmio_read(...);
And that's it, don't forget to update the head pointer!
I'll go and update the docs now. Hopefully this clears stuff up!
I wouldn't expect that to be mainstream until after optical networking becomes more common, and for consumer hardware that's very rare (apart from their modem).
In fact, "zero~th generation" of thunderbolt used optical link, too. Also both thunderbolt and DisplayPort reuse a lot of common elements from PCI-E