Remote I/O bus

[Luca Ceresoli]

Hi Greg,

On 04/10/19 16:54, Greg KH wrote:
> On Fri, Oct 04, 2019 at 04:08:06PM +0200, Luca Ceresoli wrote:
>> Hi Greg,
>>
>> On 04/10/19 15:22, Greg KH wrote:
>>> On Fri, Oct 04, 2019 at 01:04:56PM +0200, Luca Ceresoli wrote:
>>>> Hi,
>>>>
>>>> on an embedded system I currently have a standard platform device:
>>>>
>>>> .-----.  data  .--------.
>>>> | CPU |--------| DEVICE |
>>>> '-----'   bus  '--------'
>>>>
>>>> The driver is a standard platform driver that uses ioread32() and
>>>> iowrite32() to access registers.
>>>>
>>>> So far, so good.
>>>>
>>>> Now in a new design I have the same device in an FPGA, external to the
>>>> SoC. The external FPGA is not reachable via an I/O bus, but via SPI (or
>>>> I2C). A microprocessor in the FPGA acts as a bridge: as an SPI client it
>>>> receives register read/write requests from the CPU, forwards them to the
>>>> devices on the in-FPGA data bus as a master, then sends back the replies
>>>> over SPI.
>>>>
>>>>                        SoC <- | -> FPGA
>>>>
>>>> .-----.  data   .---------.       .--------.  data   .--------.
>>>> | CPU |---------| SPI CTL |-------| BRIDGE |---------| DEVICE |
>>>> '-----'  bus A  '---------'  SPI  '--------'  bus B  '--------'
>>>>
>>>>
>>>> What would be a proper way to model this in the Linux kernel?
>>>>
>>>> Of course I can hack the drivers to hijack them on SPI, but I'm trying
>>>> to solve the problem in a better way. IMO "a proper way" implies that
>>>> the platform driver does not need to be aware of the existence of the
>>>> bridge.
>>>>
>>>> Note: in the real case there is more than one device to handle.
>>>>
>>>> At first sight I think this should be modeled with a "bridge" device that:
>>>>
>>>>  * is a SPI device
>>>>  * implements a "platform bus" where regular platform devices can be
>>>>    instantiated, similar to a "simple-bus"
>>>
>>> Yes, make your own "bus", and have the SPI device be your "host
>>> controller" in that it bridges the SPI bus to your "FPGA bus".
>>>
>>> The driver model is set up for this, it should be not that complex to do
>>> so.  If you have specific questions, just let me know.  "Clean" examples
>>> of what to do is the greybus code as that's probably one of the newest
>>> busses to be added to the kernel.
>>>
>>>> In device tree terms:
>>>>
>>>> &amba { /* data bus A in picture */
>>>>
>>>>     spi0: spi at 42000000 {
>>>>         reg = <0x42000000 0x1000>;
>>>>         #address-cells = <1>;
>>>>
>>>>         io-over-spi-bridge at 1 { /* data bus B */
>>>>             reg = <1>; /* slave select pin 1 */
>>>>             compatible = "linux,io-over-spi-bridge";
>>>>             #address-cells = <1>;
>>>>             #size-cells = <1>;
>>>>
>>>>             mydevice at 4000 {
>>>>                 /* 1 kB I/O space at 0x4000 on bus B */
>>>>                 reg = <0x4000 0x1000>;
>>>>             };
>>>>         };
>>>>     };
>>>> };
>>>>
>>>> The io-over-spi driver is supposed to request allocation of a virtual
>>>> memory area that:
>>>>  1. is as large as the address space on bus B
>>>>  2. is __iomem (non cached, etc)
>>>>  3. is not mapped on the physical CPU address space (bus A)
>>>>  4. page faults at every read/write access, triggering a callback
>>>>     that starts an SPI transaction, waits for the result and returns
>>>
>>> I don't think you can map memory to be "on an SPI bus", unless you have
>>> support for that in your hardware controller itself.  Trying to map
>>> memory in this way is odd, just treat the devices out off the bus as
>>> "devices that need messages sent to them", and you should be fine.  It's
>>> not memory-mapped iomemory, so don't think of it that way.
>>
>> If I got you correctly, this means I cannot reuse the existing device
>> drivers unmodified as I was hoping to.
> 
> You are switching from a "ioread/write" to "all data goes across an SPI
> link".  No, you can't reuse the existing drivers, but you can modify
> them to abstract out the "read/write data" functions to be transport
> agnositic.
> 
>> They won't be 'struct platform_device' instances anymore, they will
>> become 'struct mybus_device' instances. And as such they won't be
>> allowed to call ioread32() / iowrite32(), but will have to call
>> mybus_ioread32() and mybus_iowrite32(). Correct?
> 
> Yes.
> 
> But, if you do it right, the majority of your driver is the logic to
> control the hardware, and interact with whatever other subsystem those
> devices talk to.  Read/Write data and the bus the device talks to should
> just be a tiny shim that you can split out into a separate module/file.

Sure, the driver logic wouldn't be touched. Only the read/write helpers
and the entry point (I think I'd need two different probe functions, but
again sharing most of their code).

> Do you have a pointer to your existing code anywhere?

One of the drivers I'm looking at is:

https://github.com/Xilinx/linux-xlnx/blob/xilinx-v2019.1/drivers/media/platform/xilinx/xilinx-csi2rxss.c

Yes, the read/write helpers are nicely isolated. However this sits in a
vendor kernel that tends to change a lot from one release to another, so
most changes on these drivers need to be re-done each time I upgrade the
kernel. That's one of the reasons I would have preferred to not touch
the driver at all.

Thanks a lot,
-- 
Luca