pcie dma transfer

[Christoph Böhmwalder]

Hi,

I'm not sure how on-topic this is on this list, but I have a question
regarding a device driver design issue.

For our Bachelor's project my team and I are tasked to optimize an
existing hardware solution. The design utilizes an FPGA to accomplish
various tasks, including a Triple Speed Ethernet controller that is linked to
the CPU via PCI express. Currently the implementation is fairly naive,
and the driver just does byte-by-byte reads directly from a FIFO on the
FPGA device. This, of course, is quite resource intensive and basically
hogs up the CPU completely (leading throughput to peak at around
10 Mbit/s).

Our plan to solve this problem is as follows:

* Keep a buffer on the FPGA that retains a number of Ethernet packages.
* Once a certain threshold is reached (or a period of time, e.g. 5ms, elapses),
  the buffer is flushed and sent directly to RAM via DMA.
* When the buffer is flushed and the data is in RAM and accessible by
  the CPU, the device raises an interrupt, signalling the CPU to read
  the data.
* In the interrupt handler, we `memcpy` the individual packets to
  another buffer and hand them to the upper layer in the network stack.

Our rationale behind keeping a buffer of packets rather than just
transmitting a single packet is to maximize the amount of data send
with each PCIe transaction (and in turn minimize the overhead).

However, upon reading a relevant LDD chapter [1] (which, admittedly, we
should have done in the first place), we found that the authors of the
book take a different approach:

> The second case comes about when DMA is used asynchronously. This happens,
> for example, with data acquisition devices that go on pushing data even if
> nobody is reading them. In this case, the driver should maintain a buffer so
> that a subsequent read call will return all the accumulated data to user
> space. The steps involved in this kind of transfer are slightly different:
>   1. The hardware raises an interrupt to announce that new data has arrived.
>   2. The interrupt handler allocates a buffer and tells the hardware where to transfer
>      its data.
>   3. The peripheral device writes the data to the buffer and raises another interrupt
>      when it’s done.
>   4. The handler dispatches the new data, wakes any relevant process, and takes care
>      of housekeeping.
> A variant of the asynchronous approach is often seen with network cards. These
> cards often expect to see a circular buffer (often called a DMAring buffer) established
> in memory shared with the processor; each incoming packet is placed in the
> next available buffer in the ring, and an interrupt is signaled. The driver then passes
> the network packets to the rest of the kernel and places a new DMA buffer in the
> ring.

Now, there are some obvious advantages to this method (not the least of
which being that it's much easier to implement), but I can't help but
feel like it would be a little inefficient.

Now here's my question: Is our solution to this problem sane? Do you
think it would be viable or that it would create more issues than it
would solve? Should we go the LDD route instead and allocate a new
buffer everytime an interrupt is raised?

Thanks for your help!

--
Regards,
Christoph

[1] https://static.lwn.net/images/pdf/LDD3/ch15.pdf (page 30)