Demystifying MIPI CSI2 TX-RX – FPGA Design

Author
Nikil Thapa
Senior FPGA Design Engineer
LogicTronix Technologies

Overview

This article(reference tutorial) outlines the steps and methodology required for MIPI CSI2 TX and RX using the Xilinx Zynq Ultrascale+ MPSoC FPGA. MIPI CSI2 is a widely used protocol for capturing and processing camera/vision sensor data. MIPI CSI2 TX- RX design is crucial for calibration and testing processing devices when creating custom vision sensor (camera) processing platform. It is also essential for verifying the functionality of ECU on ADAS Systems or Flight Controllers in drones or Navigation Controller of Mobile Robot Platform. This article(reference tutorial) will guide you through the MIPI CSI2 TX-RX design flow and implementation steps using the VIVADO/Vitis tool , as well as its testing setup between two Kria KV260 boards. We are using AMD Xilinx Kria KV260 platform which is based on MPSoC FPGA architecture for this MIPI CSI2 TX-RX design.

Here are some feature of Kria KV260 and this design:

  • The Kria KV260 board has a RPI Connector, which supports 2-lane MIPI CSI-2 configuration.
  • The board is based on the UltraScale+ FPGA architecture, which makes it possible to implement MIPI CSI-2 TX interface with the existing RPI-connector with a maximum line-rate of 2500Mbps.
  • It can support a variety of resolutions. The maximum supported resolutions are given below.
  • Xilinx MIPI CSI-2 RX Subsystem core does not natively support YUV422 10 bit, as it is the secondary data format. To support this format, one has to create a Video Format Bride(VFB) of their own.
  • However, with the use of existing features of MIPI CSI-2 RX Subsystem core, one can still be able to  achieve YUV422 10 bit color format.
  • For this, we can use the RAW10 format. This is one of the primary data formats.
  • YUV422 10 bit is one of the variants of YUV color format, where the chrominance component is interleaved. 
  • This results in the transmission of 1 pixel of data in the form of YU and YV format two times. Each of them is 10 bit wide thereby each transmission consumes 20 bit.
  • To transmit and receive YUV422 10 bit data in such format, both the MIPI CSI-2 RX and TX cores have to be customized with RAW10 with 2 PPC configuration while other transmitting and receiving pipelines in 1 PPC with 10 bit configuration
  • This means, in each clock cycle, two times of RAW10, that is, 20 bits are transmitted and received while other transmitting and receiving pipelines process this as a single data. This configuration brings the possibility for YUV422 10 bit support in MIPI CSI-2 TX and RX design.

MIPI Bandwidth Calculation for 2-lane interface

The following cases show the maximum resolution supported by the Kria KV260 MIPI RPI CAM connector.

Case 1: 3840×2160@20Hz, RGB888

Total Bandwidth: 4400x2250x20x24 = 4752 Mbps

Bandwidth Per Lane: 4752/2 = 2376 Mbps

Case 2: 3840×2160@30Hz, YUV422 8-bit

Total Bandwidth: 4752 Mbps

Bandwidth Per Lane: 2376 Mbps

Case 3: 3840×2160@25Hz, YUV422 10-bit

Total Bandwidth: 4950 Mbps

Bandwidth Per Lane: 2475 Mbps

—

Kria KV260 MIPI CSI-2 TX Pipeline Design

  • The following picture shows the block design for MIPI CSI-2 TX pipeline to transmit a 3840×2160@20Hz YUV422 10-bit frame.
Figure 1. Kria MIPI CSI2 TX Block Design

  • This design consists of a MIPI CSI-2 Transmission pipeline, which has a chain of Xilinx TPG, Video Processing Subsystem and MIPI CSI-2 TX Subsystem IP cores.

Clock Configuration

  • The design uses two clocks, obtained from the PS block.
  • One is 150MHz, used for AXI4-Stream to make sure data rate is still within the maximum rate supported by MIPI TX core. That is, 4K@20Hz.
  • Second is 200MHz, required by MIPI D-PHY.

TPG

  • This forms the stream source for the pipeline to generate 4K frames with 10-bit per component with 1PPC.
  • This IP gives the 4K frame in RGB with 10-bit per component.
Figure 2. Xilinx Video TPG IP Setting

Video processing Subsystem (VPSS)

  • There are two VPSS IP cores.
  • One is used to do color space conversion while another is used to do chroma sampling.
  • Both the cores support 4K resolution with 1PPC and 10-bit per component.
  • The color space conversion is used to convert a 4K frame from RGB space to YUV444 color space.
  • The chroma resampling is used to sample YUV444 pixel data into YUV422 pixel data with 20-bit data in each transmission.
Figure 3. Xilinx VPSS top level setting for color space converion

Configuration of VPSS CS

Figure 4. Xilinx VPSS IP Core Customization Setting for Chroma Resampling

MIPI CSI-2 TX Subsystem

  • The Kria KV260 board has a RPI Connector, which supports 2-lane MIPI CSI-2 configuration. Because of this, MIPI TX core is customized to set CSI-lanes value to 2.
  • The board is based on the UltraScale+ FPGA architecture, which makes it possible to implement MIPI CSI-2 TX interface with the existing RPI-connector with a maximum line-rate of 2500Mbps.
  • To support 4K YUV422 10 bit frame transmission, the following customizations are done:
    • Maximum Bits per Component: 10
    • Line Buffer Depth: 8192
    • Line rate: 2500Mbps
  • For the constraint mapping, the following customization is done, as shown in figure…
    • HP Bank Selection: 66
    • Clock Lane: D7
    • Data Lane0: E5
    • Date Lane1: G6
Figure 5. MIPI CSI2 TX Subsystem IP Configuration

Figure 6. MIPI CSI2 TX PIN Assignment for Kria KV260

  • The stream information for the MIPI TX core is given through the TUSER[95:0] pin. The values are instantiated by constant blocks.
Figure 7. MIPI CSI2 Tx connection
  • The information for each TUSER field can be found in the product guide.
Figure 8. MIPI CSI2 TX – TUSER field information
  • We only update the value for Word Count and Data Type field.
  • For 4K RAW10, the value for Word Count is 0x2580 and for Data Type is 0x2B. 
  • The bitstream is generated and exported in the XSA file format, for example, “kv260_mipi_tx_yuv422_10bit.xsa”

Software Design

  • In this part, the bare-metal software application is written using Vitis IDE.
  • It has a main function, which has the following functions to configure the IPs.
  • configPSGpio()
    • It does the configuration of PS GPIO for the IP Soft reset
    • The PS GPIO interface is initialized and configured its pins as output.
    • GPIO pins are set to do IP soft reset, as the reset pins of a few IP cores are made software-controllable in the hardware design
  • configTpg()
    • The TPG core is configured to generate color bar patterns in RGB color format.
  • configVpss()
    • This consists of separate configuration for each VPSS core for color space conversion and chroma resampling
    • For color space conversion, the VPSS core is initialized and its input and output stream parameters are created to set input color format XVIDC_CSF_RGB and output color format XVIDC_CSF_YCRCB_444 to convert RGB color space into YUV color space.
    • For chroma resampling, the VPSS core is initialized and its input and output stream parameters are created to set input color format XVIDC_CSF_YCRCB_444 and output color format XVIDC_CSF_YCRCB_422 to convert YUV444 into YUV420 respectively.
  • configMIPICsi2TX()
    • The MIPI CSI-2 TX SS core is initialized and activated.
    • Its stream parameters are configured using a TUSER pin from hardware.
  • This completes the software application design for MIPI CSI-2 TX hardware design, which is then built and run on the board.

Kria KV260 MIPI CSI-2 RX Pipeline Design

Figure 9. MIPI CSI2 RX Design

  • The pipeline design consists of Xilinx MIPI CSI-2 RX Subsystem, Video Frame Buffer Write, Buffer Read, Video Processing Subsystem, Video Timing Control, Axis Subset converter, AXI4-Stream-to-Video.
  • To properly receive and process, 4K YUV422 10 bit frame, only the MIPI CSI-2 RX subsystem core is configured with 2PPC while the rest of the IPs are configured with 1PPC with 10-bit per component
  • The stream clock is 300MHz with a pixel clock 297MHz for displaying 4K frames at 30FPS.

Clock Configuration

  • There are two clocking wizard IPs.
  • One is used to generate two clocks. 200MHz for MIPI D-PHY and 300MHz for AXI4-Stream interface.
  • Second is used to generate a pixel clock, which is 297MHz to support 4K@30Hz display.
Figure 10. Clocking Wizard for MIPI Design

MIPI CSI-2 RX Subsystem Core

  • The following are the customizations needed to be done for the proper reception of 4K YUV422 10 bit using RAW10 pixel format.
    • Pixel format: RAW10
    • Data Lanes: 2
    • Line Rate: 2500Mbps
    • Pixel per clock: 2
  • With this customization, the RX core gives YUV422, 10-bit output stream.
  • Pin Assignment
    • HP IO Bank Selection: 66
    • Clock Lane: D7
    • Data Lane0: E5
    • Data Lane1: G6
Figure 11. MIPI CSI2 RX subsystem IP config

Figure 12. MIPI CSI2 RX subsystem IP constrainting

Video Frame Buffer Write IP

  • It is used to buffer the received frames from the MIPI RX core.
  • This receives 4K, YUV422 10 bit frames and stores them with Semi-planer YUV422 10-bit memory format.
Figure 13: Video Frame Buffer IP customization

Video Frame Buffer Read

  • To display received frames, the display pipeline has a video frame buffer read IP.
  • This reads previously stored semi-planer YUV422 10-bit frames and streams them in YUV422 10-bit format.
  • The customization is shown in the figure below.
Figure 14. Video Frame Buffer Read IP setting

Video Processing Subsystem

  • There are two VPSS IP cores.
  • One is used to do chroma sampling while another is used to do color space conversion.
  • Both the cores are customized to support 4K resolution with 1PPC and 10-bit per component.
  • The chroma resampling is used to sample YUV422 pixel data into YUV444 pixel data.
  • The color space conversion is used to convert a 4K frame from YUV444 color space to RGB color space.
Figure 15. Xilinx VPSS at RX side

Figure 16. Xilinx VPSS doing CSC operation

Axis Subset Converter

  • There are two subset convert IPs,
  • One is to convert 10-bit RGB pixel data into 8-bit RGB pixel data by truncating MSB 2-bits from 10-bit pixel data.
  • This is done by specifying TData Remap String as tdata[27:20],tdata[17:10],tdata[7:0].
  • Second subset converter is used to swap the R and G color channels, as there were wrong RGB color outputs from PS DP noted during the design test when un-swapped RGB streams were directly fed to the PS DP controller.
  • The swapping is achieved by using tdata remapping string: tdata[15:8],tdata[23:16],tdata[7:0]

Subset converter customization for 10-bit to 8-bit RGB pixel data conversion

Figure 17. AXI Subset Converter for MIPI CSI Design

Subset converter customization for R and G color channel swapping

Figure 18. AXI Subset Converter

Video Timing Controller (VTC)

  • The Video timing required by the 4K stream is provided by VTC.
  • To generate video timing for a 4K stream, the MAX Clocks Per Line must be 8192.
  • The AXI-4 Lite Interface helps to configure this core from the software side to generate video timing.
Figure 19. Xilinx VTC IP Config for MIPI CSI2 RX Design

Axi4-Stream to Video Out

  • This synchronizes the 4K stream data and corresponding video timing to convert axi4-stream into native video with 12-bit per component. This configuration is required by the live DP interface of PS.
  • This core outputs 4K RGB streams in native format, synchronized with pixel clock.  This is fed into the PS Live DP interface.
Figure 20. AXI Stream to Video Out for displaying the RX MIPI data/frame

  • The bitstream is generated and exported in the XSA file format, for example, “kv260_mipi_rx_yuv422_10bit.xsa”

Software Design

  • In this part, the bare-metal software application is written using Vitis IDE.
  • The application has a main function, which houses the following functions to configure the pipeline.
  • configPSGpio()
    • It does the configuration of PS GPIO for the IP Soft reset
    • The PS GPIO interface is initialized and configured its pins as output.
    • GPIO pins are set to do IP soft reset, as the reset pins of a few IP cores are made software-controllable in the hardware design
  • configMIPICsi2Rx()
    • It is used to configure and activate the MIPI CSI-2 RX SS core.
  • configFBWr()
    • It configures Video Frame Buffer Write IP to buffer 4K YUV422 10-bit frames in the Semi-planer YUV422 format, specified by XVIDC_CSF_MEM_Y_UV10 parameter.
  • configFBRd()
    • It configures Video Frame Buffer Read IP to read back 4K Semi-planer YUV422 10-bit data from the buffer in YUV422 10-bit format, specified by XVIDC_CSF_MEM_Y_UV10 parameter.
  • configVpss()
    • This consists of separate configurations for each VPSS core for color space conversion and chroma resampling.
    • For the chroma resampling, the VPSS core is initialized and its input and output stream parameters are created to set input color format XVIDC_CSF_YCRCB_422 and output color format XVIDC_CSF_YCRCB_444 to convert YUV422 stream into YUV444 stream respectively.
    • For the color space conversion, the VPSS core is initialized and its input and output stream parameters are created to set input color format XVIDC_CSF_YCRCB_444 and output color format XVIDC_CSF_RGB to convert YUV444 color space into RGB color space.
  • configVtc()
    • This initializes VTC core and configures it to generate video timing for 3840×2160 resolution.
  • run_dp()
    • It calls the PS DP Controller driver.
    • The controller receives the 4K RGB stream from the pipeline and displays them on the DP monitor.
  • This completes the software application for MIPI CSI-2 RX hardware design, which is then built and run on the board.

Board Setup

Figure 21. MIPI CSI2 Tx and RX Design – KV260 Board setup

Disclaimer: in above setup we have tweaked some of lines of MIPI Ribbon Cable. You cant directly connect MIPI ribbon cable between two board like above setup without tweaking. For the Tweak and working design contact us at info@logictronix.com.

  • The above picture shows the board and connection setup.
  • The board on the left hand side is configured as MIPI CSI-2 TX. 
  • The board on the right hand side is configured as MIPI CSI-2 RX.  The HDMI cable is connected on this board to display the received frames on the HDMI monitor.
  • The boards are programmed using USB cables in JTAG mode.

Output

  • The bare metal applications for each board are loaded separately from the Vitis IDE.
  • The MIPI CSI-2 RX board is programmed first and then the MIPI CSI-2 TX board is programmed.
  • The following picture shows the output.
  • The TX board sends the TPG color bar pattern after RGB to YUV422 conversion through the MIPI CSI-2 TX interface in RAW10 format.
  • The RX board receives the frames in RAW10 format, which is buffered and then converted back to RGB from YUV422 format and displayed on the monitor.
Figure 22. TPG stream received at MIPI CSI2 RX (Receiving KV260) which is sent from MIPI CSI2 TX (Transmitting KV260)

Appendix

Kria KV260 MIPI CSI-2 TX-RX Design for RGB888, 3840×2160@20Hz

MIPI CSI-2 TX

Hardware Design

  • Pipeline has a chain of TPG, VPSS and MIPI CSI-2 TX SS cores.
  • The entire pipeline is configured at 1PPC with 8-bit per component.
  • Clock configuration:
    • 200MHz for MIPI D-PHY
    • 150MHz for AXI4-Stream to make sure data rate is still within the maximum rate supported by MIPI TX core. That is, 4K@20Hz
  • The pipeline consists of TPG for the 3840×2160 RGB Stream.
  • MIPI CSI-2 TX SS core has following customization
    • Maximum Bits per Component: 8
    • Line Buffer Depth: 4096
    • Line rate: 2500Mbps
    • Input Pixels per beat: 1
  • The stream information for the MIPI TX core is given through the TUSER[95:0] pin. The values are instantiated by constant blocks.
  • For 4K RGB888, the value for Word Count is 0x2D00 and for Data Type is 0x24.

Software Design

  • The main function has the following functions to configure the IPs.
  • configPSGpio()
  • configTpg()
  • configMIPICsi2TX()
  • This completes the software application design for MIPI CSI-2 TX hardware design,

MIPI CSI-2 RX

Hardware Design

  • The pipeline has MIPI CSI-2 RX SS, frame buffer write, frame buffer read, Video timing controller, Subset converter and Axis2video IP core.
  • The entire pipeline is configured at 1PPC with 8-bit per component.
  • Desired Clocks are achieved by two clocking wizards.
    • One is 200MHz for MIPI D-PHY
    • One is used to generate two clocks. 200MHz for MIPI DPHY and 300MHz for AXI4-Stream interface.
    • Second is used to generate pixel clock, which is 297MHz to support 4K@30Hz display.
  • MIPI CSI-2 RX Subsystem Core has following customizations
    • Pixel format: RGB888
    • Data Lanes: 2
    • Line Rate: 2500Mbps
    • Pixel per clock: 1
  • Video frame buffer write and read IP cores are used for buffering the received frames in RGBX8 format.
  • Subset converter is used to swap the R and G color channels . The swapping is achieved by using tdata remapping string: tdata[15:8],tdata[23:16],tdata[7:0]
  • Video Timing Controller is used to generate video timing required by 4K RGB streams.
  • AXI4-Stream-to-Video-Out IP core generates native 4K streams and is fed into the PS Live DP interface.

Software Design

  • The main function houses the following functions to configure the pipeline.
  • configPSGpio()
  • configMIPICsi2Rx()
  • configFBWr()
  • configFBRd()
  • configVtc()
  • run_dp()
  • This completes the software application for MIPI CSI-2 RX hardware design

Kria KV260 MIPI CSI-2 TX-RX Design for YUV422 8-Bit, 3840×2160@30Hz

MIPI CSI-2 TX

Hardware Design

  • Pipeline has a chain of TPG, VPSS and MIPI CSI-2 TX SS cores.
  • The entire pipeline is configured at 1PPC with 8-bit per component.
  • Clock configuration:
    • 200MHz for MIPI D-PHY
    • 297MHz (need confirmation) for AXI4-Stream to make sure data rate is still within the maximum rate supported by MIPI TX core. That is, 4K@30Hz
  • The pipeline consists of TPG for the 3840×2160 RGB Stream.
  • There are two VPSS cores.
    • One is used to convert a 4K frame from RGB space to YUV444 color space.
    • The chroma resampling is used to sample YUV444 pixel data into YUV422 pixel data with 16-bit data in each transmission.
  • MIPI CSI-2 TX SS core has following major customization
    • Maximum Bits per Component: 8
    • CSI Lanes: 2
    • Line Buffer Depth: 4096
    • Line rate: 2500Mbps
    • Input Pixels per beat: 1
  • The stream information for the MIPI TX core is given through the TUSER[95:0] pin. The values are instantiated by constant blocks.
  • For 4K YUV422, 8bit, the value for Word Count is 0x1E00 and for Data Type is 0x1E.

Software Design

  • The main function has the following functions to configure the IPs.
  • configPSGpio()
  • configTpg()
  • configVpss()
  • configMIPICsi2TX()
  • This completes the software application design for MIPI CSI-2 TX hardware design,

MIPI CSI-2 RX

Hardware Design

  • The pipeline has MIPI CSI-2 RX SS, frame buffer write, frame buffer read, VPSS core, Video timing controller, Subset converter and Axis2Video IP core.
  • The entire pipeline is configured at 1PPC with 8-bit per component.
  • Desired Clocks are achieved by two clocking wizards.
    • One is 200MHz for MIPI D-PHY
    • One is used to generate two clocks. 200MHz for MIPI D-PHY and 300MHz for AXI4-Stream interface.
    • Second is used to generate a pixel clock, which is 297MHz to support 4K@30Hz display.
  • MIPI CSI-2 RX Subsystem Core has following customizations
    • Pixel format: RGB888
    • Data Lanes: 2
    • Line Rate: 2500Mbps
    • Pixel per clock: 1
  • Video frame buffer write and read IP cores are used for buffering the received frames in RGBX8 format.
  • There are two VPSS IP cores.
    • One is used to do chroma sampling to sample YUV422 pixel data into YUV444 pixel data.
    • Second is used to convert a 4K frame from YUV444 color space to RGB color space.
  • Subset converter is used to swap the R and G color channels . The swapping is achieved by using tdata remapping string: tdata[15:8],tdata[23:16],tdata[7:0].
  • Video Timing Controller is used to generate video timing required by 4K RGB streams.
  • AXI4-Stream-to-Video-Out IP core generates native 4K streams and is fed into the PS Live DP interface.

Software Design

  • The main function houses the following functions to configure the pipeline.
  • configPSGpio()
  • configMIPICsi2Rx()
  • configFBWr()
  • configFBRd()
  • configVtc()
  • run_dp()
  • This completes the software application for MIPI CSI-2 RX hardware design

Revision History

Document Version:1.1
Document Date:November 29, 2024
Document Author(s):Nikil Thapa
Document Classification:Public Release
Version History:VersionDateComment
 1.0Nov 27First Version

Note: In this article we are explaining about how MIPI CSI2 TX and RX design can be developed and tested with Kria KV260.
There are few design customization/tweaks which we have not fully discussed in this article, for those design changes you either have to have good understanding of MIPI protocol and its signaling or you can contact us (info@logictronix.com) for reference design and support.

Also this reference design explains about Baremetal method for testing of TX-RX. For Petalinux/Yocto/Linux based design support (which needs custom driver for MIPI CSI2 TX) you can write us at email!

LogicTronix is Xilinx Certified Partner for FPGA Design and ML Acceleration!