Altera Arria V GZ User Manual Download Page 89 | Manualshive

Page: 89 / 300

background image

Chapter 5: IP Core Architecture

5–19

Avalon-MM Bridge TLPs

November 2012

Altera Corporation

Arria V GZ Hard IP for PCI Express

This design is consuming 1.25GB of PCIe address space when only 276 MBytes are
actually required. The solution is to edit the address map to place the base address of
each BAR at 0x0000_0000.

illustrates the optimized address map.

h

For more information about changing Qsys addresses using the Qsys address map,
refer to

Address Map Tab (Qsys)

in Quartus II Help.

shows the number of address bits required when the smaller memories

accessed by BAR2 and BAR4 have a base address of 0x0000_0000.

For cases where the BAR Avalon-MM RX master port connects to more than one
Avalon-MM slave, assign the base addresses of the slaves sequentially and place the
slaves in the smallest power-of-two-sized address space possible. Doing so minimizes
the system address space used by the BAR.

Avalon-MM-to-PCI Express Address Translation Algorithm

The Avalon-MM address of a received request on the TX Avalon-MM slave port is
translated to the PCI Express address before the request packet is sent to the
Transaction Layer. You can specify up to 512 address pages and sizes ranging from
4 KByte to 4 GBytes when you customize your Avalon-MM Arria V GZ Hard IP for
PCI Express as described in

“Avalon to PCIe Address Translation Settings” on

. This address translation process proceeds by replacing the MSB of the

Avalon-MM address with the value from a specific translation table entry; the LSB
remains unchanged. The number of MSBs to be replaced is calculated based on the
total address space of the upstream PCI Express devices that the Avalon-MM Hard IP
for PCI Express can access. The number of MSB bits is defined by the difference
between the maximum number of bits required to represent the address space
supported by the upstream PCI Express device minus the number of bits required to
represent the

Size of address pages

which are the LSB pass-through bits (

N

). The

Size

of address pages

(

N

) is applied to all entries in the translation table.

Figure 5–10. Optimized Address Map

Figure 5–11. Reduced Address Bits for BAR2 and BAR4

«
...
87
88
89
90
91
...
»

Summary of Contents for Arria V GZ

Page 1: ... altera com UG 01127 1 0 User Guide Arria V GZ Hard IP for PCI Express Document last updated for Altera Complete Design Suite version Document publication date 12 1 November 2012 Feedback Subscribe Arria V GZ Hard IP for PCI Express User Guide ...

Page 2: ...tor products to current specifications in accordance with Altera s standard warranty but reserves the right to make changes to any products and services at any time without notice Altera assumes no responsibility or liability arising out of the application or use of any information product or service described herein except as expressly agreed to in writing by Altera Altera customers are advised t...

Page 3: ... Understanding Channel Placement Guidelines 2 15 Quartus II Compilation 2 15 Compiling the Design in the MegaWizard Plug In Manager Design Flow 2 15 Compiling the Design in the Qsys Design Flow 2 16 Modifying the Example Design 2 19 Chapter 3 Getting Started with the Avalon MM Arria V GZ Hard IP for PCI Express Running Qsys 3 2 Customizing the Arria V GZ Hard IP for PCI Express IP Core 3 3 Adding ...

Page 4: ... PCI Express to Avalon MM Read Completions 5 14 PCI Express to Avalon MM Downstream Write Requests 5 14 PCI Express to Avalon MM Downstream Read Requests 5 15 Avalon MM to PCI Express Read Completions 5 15 PCI Express to Avalon MM Address Translation for Endpoints 5 16 Minimizing BAR Sizes and the PCIe Address Space 5 17 Avalon MM to PCI Express Address Translation Algorithm 5 19 Completer Only Si...

Page 5: ...64 Bit Bursting TX Avalon MM Slave Signals 6 52 Physical Layer Interface Signals 6 53 Transceiver Reconfiguration 6 54 Serial Interface Signals 6 54 Channel Placement for Gen1 and Gen2 Using CMU PLL 6 55 Channel Placement for Gen1 and Gen2 Using ATX PLL 6 58 Channel Placement for Gen3 Using Both CMU and ATX PLLs 6 60 PIPE Interface Signals 6 60 Test Signals 6 63 Making Pin Assignments 6 64 Chapter...

Page 6: ...ing the Avalon ST Application Interface 11 1 MSI Interrupts 11 1 MSI X 11 4 Legacy Interrupts 11 4 Interrupts for Root Ports Using the Avalon ST Interface to the Application Layer 11 5 Interrupts for Endpoints Using the Avalon MM Interface to the Application Layer 11 5 Enabling MSI or Legacy Interrupts 11 7 Generation of Avalon MM Interrupts 11 7 Interrupts for End Points Using the Avalon MM Inter...

Page 7: ...to the Application Layer 16 27 BFM Procedures and Functions 16 28 BFM Read and Write Procedures 16 28 ebfm_barwr Procedure 16 28 ebfm_barwr_imm Procedure 16 29 ebfm_barrd_wait Procedure 16 30 ebfm_barrd_nowt Procedure 16 30 ebfm_cfgwr_imm_wait Procedure 16 31 ebfm_cfgwr_imm_nowt Procedure 16 32 ebfm_cfgrd_wait Procedure 16 33 ebfm_cfgrd_nowt Procedure 16 33 BFM Configuration Procedures 16 34 ebfm_...

Page 8: ...cedure 16 48 ebfm_display_verb Procedure 16 48 Chapter 17 Debugging Hardware Bring Up Issues 17 1 Link Training 17 1 Link Hangs in L0 Due To Deassertion of tx_st_ready 17 4 Recommended Reset Sequence to Avoid Link Training Issues 17 6 Setting Up Simulation 17 6 Use the PIPE Interface for Gen1 and Gen2 Variants 17 6 Reduce Counter Values for Serial Simulations 17 7 Disable the Scrambler for Gen3 Si...

Page 9: ...nd 8 configurations Table 1 1 shows the aggregate bandwidth of a PCI Express link for Gen1 Gen2 and Gen3 for 1 4 and 8 lanes The protocol specifies 2 5 giga transfers per second for Gen1 5 giga transfers per second for Gen2 and 8 0 giga transfers per second for Gen3 Table 1 1 provides bandwidths for a single transmit TX or receive RX channel so that the numbers in Table 1 1 double for duplex opera...

Page 10: ...e 12 1 release Root Port support for Avalon MM Hard IP for PCI Express Native 2 support Multiple MSI and MSI X messages for the Avalon MM Hard IP for PCI Express Revised example design including a the Transceiver Reconfiguration Controller Qsys component and a driver for this component The Arria V GZ Hard IP for PCI Express offers different features for variants that use the Avalon ST and Avalon M...

Page 11: ... Memory Read Request Memory Write Request I O Read Request I O Write Request Configuration Read Request Root Port Configuration Write Request Root Port Message Request Message Request with Data Payload Completion without Data Completion with Data Memory Read Request single dword Memory Write Request single dword Payload size 128 2048 bytes 128 256 bytes Number of tags supported for non posted requ...

Page 12: ...ble overlap between these two purposes this document should be used in conjunction with an understanding of the PCI Express Base Specification 3 0 Release Information Table 1 3 provides information about this release of the PCI Express Compiler Expansion ROM Supported Not supported Notes to Table 1 2 1 Not recommended for new designs 2 Refer to Appendix A Transaction Layer Packet TLP Header Format...

Page 13: ... You can customize the Hard IP to meet your design requirements using either the MegaWizard Plug In Manager or the Qsys design flow When configured as an Endpoint the Avalon MM Arria V GZ Hard IP for PCI Express supports memory read and write requests and completions with or without data In Root Port mode the core also supports configuration reads and writes message transactions legacy interrupts ...

Page 14: ...s A Root Port that connects directly to a second FPGA that includes an Endpoint Two Endpoints that connect to a PCIe switch A host CPU that implements CvP using the PCI Express link connects through the switch For more information about configuration over a PCI Express link refer to Configuration via Protocol CvP on page 10 1 Figure 1 1 PCI Express Application with a Single Root Port and Endpoint ...

Page 15: ...d and pseudo random stimuli are applied to test the Application Layer interface Configuration Space and all types and sizes of TLPs Error injection tests that inject errors in the link TLPs and Data Link Layer Packets DLLPs and check for the proper responses PCI SIG Compliance Checklist tests that specifically test the items in the checklist Figure 1 2 PCI Express Application Including Arria V GZ ...

Page 16: ... Arria V GZ Hard IP for PCI Express using the current version of the Quartus II software targeting a Arria V GZ 5AGZME5K2F40C3 device With the exception of M20K memory blocks the numbers of ALMs and logic registers in Table 1 5 are rounded up to the nearest 50 Resource utilization numbers reflect changes to the resource utilization reporting starting in the Quartus II software v12 1 release 28 nm ...

Page 17: ...Optimization Technique settings refer to Area and Timing Optimization in volume 2 of the Quartus II Handbook Table 1 6 Arria V GZ Recommended Speed Grades for All Avalon ST Widths and Frequencies Link Rate Link Width Interface Width Application Clock Frequency MHz Recommended Speed Grades Gen1 1 64 bits 62 5 1 125 1 2 3 4 2 64 bits 125 1 2 3 4 4 64 bits 125 1 2 3 4 8 64 bits 250 1 2 3 2 8 128 Bits...

Page 18: ...Rate Link Width Interface Width Application Clock Frequency MHz Recommended Speed Grades Gen1 1 64 bits 62 5 1 125 1 2 3 4 2 64 bits 125 1 2 3 4 4 64 bits 125 1 2 3 4 8 64 bits 250 1 2 3 2 8 128 Bits 125 1 2 3 4 Gen2 1 64 bits 62 5 125 1 2 3 4 2 64 bits 125 1 2 3 2 4 128 bits 125 1 2 3 4 8 128 bits 250 1 2 3 2 Gen3 1 64 bits 125 1 2 3 4 2 64 bits 250 1 2 3 2 2 128 bits 125 1 2 3 2 4 128 bits 250 1...

Page 19: ...the Gen1 4 Endpoint Figure 2 1 illustrates the top level modules of the testbench in which the DUT a Gen1 8 Endpoint connects to a chaining DMA engine labeled APPS in Figure 2 1 and a Root Port model The Transceiver Reconfiguration Controller dynamically reconfigures analog settings to optimize signal quality of the serial interface The pcie_reconfig_driver drives the Transceiver Reconfiguration C...

Page 20: ...fer to System Design with Qsys in the Quartus II Handbook h For more information about the Qsys GUI refer to About Qsys in Quartus II Help Figure 2 2 illustrates the steps necessary to customize the Arria V GZHard IP for PCI Express and run the example design Figure 2 2 MegaWizard Plug In Manager and Qsys Design Flows Select Design Flow Review Qsys Example Design for PCIe Qsys Flow MegaWizard Plug...

Page 21: ...stomizing the Endpoint in the MegaWizard Plug In Manager Design Flow Understanding the Files Generated Reviewing the Qsys Example Design for PCIe Generating the Testbench Understanding Channel Placement Guidelines Simulating the Example Design Compiling the Design in the MegaWizard Plug In Manager Design Flow Modifying the Example Design Creating a Quartus II Project Follow these steps to copy the...

Page 22: ...he MegaWizard Plug In Manager design flow It specifies the same options that are chosen in Chapter 16 Testbench and Design Example Follow these steps to customize your variant in the MegaWizard Plug In Manager 1 On the Tools menu click MegaWizard Plug In Manager The MegaWizard Plug In Manager appears 2 Select Create a new custom megafunction variation and click Next 3 In Which device family will y...

Page 23: ...requency 100 MHz Use 62 5 MHz Application Layer clock for 1 Leave this option off Use deprecated RX Avalon ST data byte enable port rx_st_be Leave this option off Enable configuration via the PCIe link Leave this option off Enable byte parity ports on Avalon ST interface Leave this option off Multiple packets per cycle Leave this option off Enable configuration via the PCIe link Leave this option ...

Page 24: ...ers 19 On the PHY Characteristics tab specify the settings in Table 2 7 20 Click Finish The Generation dialog box appears 21 Turn on Generate Example Design to generate the Endpoint testbench and supporting files 22 Click Exit Table 2 4 Device Settings Parameter Value Maximum payload size 256 bytes Number of tags supported 32 Completion timeout range ABCD Implement completion timeout disable On Ta...

Page 25: ...d Table 2 8 provides an overview of directories and files generated Follow these steps to generate the chaining DMA testbench from the Qsys system design example 1 On the Quartus II File menu click Open 2 Navigate to the Qsys system in the altera_pcie_ device _hip_ast subdirectory Table 2 8 Qsys Generation Output Files Directory Description working_dir variant_name Includes the files for synthesis...

Page 26: ...g In Manager Design Flow Arria V GZ Hard IP for PCI Express November 2012 Altera Corporation User Guide 3 Click pcie_de_gen1_x8_ast128 qsys to bring up the Qsys design Figure 2 3 illustrates this Qsys system Figure 2 3 Qsys System Connecting the Endpoint Variant and Chaining DMA Testbench ...

Page 27: ... these parameters 5 To close the APPS component click the X in the upper right hand corner of the parameter editor Go to Simulating the Example Design on page 2 12 for instructions on system simulation Qsys Design Flow This section guides you through the steps necessary to customize the Arria V GZ Hard IP for PCI Express and run the example testbench in Qsys It includes the following steps Reviewi...

Page 28: ... the DUT and drives read and write TLPs to test DUT functionality An Endpoint BFM is available if your PCI Express design implements a Root Port pcie_reconfig_driver_0 This Avalon MM master drives the Transceiver Reconfiguration Controller The pcie_reconfig_driver_0 is implemented in clear text that you can modify if your design requires different reconfiguration functions After you generate your ...

Page 29: ...Create simulation model None This option generates a simulation model you can include in your own custom testbench Create testbench Qsys system Standard BFMs for standard Avalon interfaces Create testbench simulation model Verilog Synthesis Create HDL design files for synthesis Turn this option on Create block symbol file bsf Turn this option on Output Directory Path pcie_qsys pcie_de_gen1_x8_ast1...

Page 30: ...ence and Mentor simulation tools with the required libraries and simulation scripts testbench_dir variant_name testbench cad_vendor Includes the HDL source files and scripts for the simulation testbench Table 2 10 Qsys Generation Output Files Part 2 of 2 Directory Description Example 2 1 Excerpts from Transcript of Successful Simulation Run Time 56000 Instance top_chaining_testbench ep epmap pll_2...

Page 31: ...0040 ns 64 Bit Address Capable Supported INFO 20040 ns Messages Requested 4 INFO 20040 ns INFO 31208 ns EP PCI Express Link Status Register 1081 INFO 31208 ns Negotiated Link Width x8 INFO 31208 ns Slot Clock Config System Reference Clock Used INFO 33481 ns RP LTSSM State RECOVERY RCVRLOCK INFO 34321 ns EP LTSSM State RECOVERY RCVRLOCK INFO 34961 ns EP LTSSM State RECOVERY RCVRCFG INFO 35161 ns RP...

Page 32: ... INFO 84901 ns DMA Write INFO 84901 ns INFO 84901 ns TASK dma_wr_test INFO 84901 ns DMA Write INFO 84901 ns INFO 84901 ns TASK dma_set_wr_desc_data INFO 84901 ns INFO 84901 ns TASK dma_set_msi WRITE INFO 84901 ns Message Signaled Interrupt Configuration INFO 84901 ns msi_address RC memory 0x07F0 INFO 87109 ns msi_control_register 0x00A5 INFO 96005 ns msi_expected 0xB0FD INFO 96005 ns msi_capabilit...

Page 33: ... lists all files necessary to compile the project Follow these steps to add the Quartus II IP File qip to the project 1 On the Project menu select Add Remove Files in Project 2 Click the browse button next the File name box and browse to the gen1_x8_example_design altera_pcie_sv_hip_ast pcie_de_gen1_x8_ast128 synthesis directory Example 2 1 Excerpts from Transcript of Successful Simulation Run con...

Page 34: ...to the File Name box and browse to the synthesis directory that includes your Qsys project working_dir gen1_x8_example_design altera_pcie_sv_hip_ast pcie_de_gen1_x 8_ast128 synthesis 3 On the Quartus II File menu click New then New Quartus II Project then OK 4 Click Next in the New Project Wizard Introduction The introduction does not appear if you previously turned it off Example 2 2 Synopsys Des...

Page 35: ...d Files page 7 Complete the following steps to add the Quartus II IP File qip to the project a Click the browse button The Select File dialog box appears b In the Files of type list select IP Variation Files qip c Click pcie_de_gen1_x8_ast128 qip and then click Open d On the Add Files page click Add then click OK 8 Click Next to display the Device page 9 On the Family Device Settings page choose t...

Page 36: ...sign Example 2 3 Synopsys Design Constraint create_clock period 100 MHz name refclk_pci_express refclk_ derive_pll_clocks derive_clock_uncertainty PHY IP reconfig controller constraints Set reconfig_xcvr clock Modify to match the actual clock pin name used for this clock and also changed to have the correct period set create_clock period 125 MHz name reconfig_xcvr_clk reconfig_xcvr_clk HIP Soft re...

Page 37: ...nerate the transactions needed to test your Application Layer Figure 2 5 Testbench for PCI Express PCB Avalon MM slave Hard IP for PCI Express Altera FPGA PCB Transaction Layer Data Link Layer PHY MAC Layer x8 PCIe Link Physical Layer Lane 7 Unused Unused Lane 6 Lane 5 TX PLL PHY IP Core for PCI Express Lane 2 Lane 3 Lane 4 Lane 1 Lane 0 TX PLL Transceiver Bank Transceiver Bank S M Reconfig to and...

Page 38: ...2 20 Chapter 2 Getting Started with the Arria V GZ Hard IP for PCI Express Quartus II Compilation Arria V GZ Hard IP for PCI Express November 2012 Altera Corporation User Guide ...

Page 39: ...V GZ Hard IP for PCI Express 4 IP core On Chip memory DMA controller Transceiver Reconfiguration Controller In the Qsys design flow you select the Avalon MM Arria V GZ Hard IP for PCI Express as a component This component supports PCI Express 1 2 4 or 8 Endpoint applications with bridging logic to convert PCI Express packets to Avalon MM transactions and vice versa The design example included in t...

Page 40: ...GZHard IP for PCI Express IP Core 3 Adding the Remaining Components to the Qsys System 4 Completing the Connections in Qsys 5 Specifying Clocks and Interrupts 6 Specifying Exported Interfaces 7 Specifying Address Assignments 8 Simulating the Example Design 9 Understanding Channel Placement Guidelines 10 Adding Synopsis Design Constraints 11 Creating a Quartus II Project 12 Compiling the Design 13 ...

Page 41: ...ia V GZ Hard IP for PCI Express component then click Add Customizing the Arria V GZ Hard IP for PCI Express IP Core The parameter editor uses bold headings to divide the parameters into separate sections You can use the scroll bar on the right to view parameters that are not initially visible Follow these steps to parameterize the Hard IP for PCI Express IP core 1 Under the System Settings heading...

Page 42: ...s values for your final product 2 3 Under the PCI Express and PCI Capabilities heading specify the settings in Table 3 5 Table 3 4 Device Identification Registers Parameter Value Altera Value Vendor ID 0x00000000 0x00001172 Device ID 0x00000001 0x0000E001 Revision ID 0x00000001 0x00000001 Class Code 0x00000000 0x00FF0000 Subsystem Vendor ID 0x00000000 0x00001172 Subsystem Device ID 0x00000000 0x00...

Page 43: ...ted header credit Completion header credit and Completion data credit in the message area These values are computed based upon the values set for Maximum payload size and Desired performance for received requests Adding the Remaining Components to the Qsys System This section describes adding the DMA controller and on chip memory to your system 1 On the Component Library tab type the following tex...

Page 44: ...DMA Controller Parameters Parameter Value Width of the DMA length register 13 Enable burst transfers Turn on this option Maximum burst size Select 128 Data transfer FIFO depth Select 32 Construct FIFO from registers Turn off this option Construct FIFO from embedded memory blocks Turn on this option Advanced Allowed Transactions Turn on all options Table 3 9 On Chip Memory Parameters Part 1 of 2 Pa...

Page 45: ...Quartus II software compiles your design it merges logical channels After compilation the design has two reconfiguration interfaces one for the TX PLL and one for the channels however the number of logical channels is still five Enable In System Memory Content Editor feature D Turn off this option Instance ID Not required Table 3 10 Transceiver Reconfiguration Controller Parameters Parameter Value...

Page 46: ...gn create the following connections 1 Connect the pcie_sv_hip_avmm_0 Rxm_BAR0 Avalon Memory Mapped Master port to the onchip_memory2_0 s1 Avalon Memory Mapped slave port using the following procedure a Click the Rxm_BAR0 port then hover in the Connections column to display possible connections b Click the open dot at the intersection of the onchip_mem2_0 s1 port and the pci_express_compiler Rxm_BA...

Page 47: ... signals in this Qsys system connect to modules outside the design Follow these steps to export an interface 1 Click in the Export column 2 First accept the default name that appears in the Export column Then right click on the name select Rename and type the name shown in Table 3 12 DUT Txs Avalon Memory Mapped Slave dma_0 write_master Avalon Memory Mapped Master onchip_memory s1 Avalon Memory Ma...

Page 48: ...es In the design example you assign the base addresses manually The Avalon MM Arria V GZ Hard IP for PCI Express assigns base addresses to each BAR The maximum supported BAR size is 4 GByte or 32 bits Follow these steps to assign a base address to an Avalon MM slave interface manually 1 In the row for the Avalon MM slave interface base address you want to specify click the Base column 2 Type your ...

Page 49: ... base address For more information on optimizing BAR sizes refer to Minimizing BAR Sizes and the PCIe Address Space on page 5 17 Simulating the Example Design Follow these steps to generate the files for the testbench and synthesis 1 On the Generation tab in the Simulation section set the following options a For Create simulation model select None This option allows you to create a simulation mode...

Page 50: ...e your PCI Express system The simulation of the design example uses the following components and software The system you created using Qsys A testbench created by Qsys in the project_dir ep_g1_x4 testbench directory You can view this testbench in Qsys by opening project_dir ep_g1_x4 testbench s5_avmm_tb qsys which shown in Figure 3 2 The ModelSim software 1 You can also use any other supported thi...

Page 51: ...minal window a do msim_setup tcl r b ld debug r The debug argument stops optimizations improving visibility in the ModelSim waveforms c run 140000 ns r The driver performs the following transactions with status of the transactions displayed in the ModelSim simulation message window Various configuration accesses to the Avalon MM Arria V GZ Hard IP for PCI Express in your system after the link is i...

Page 52: ...tate CONFIG COMPLETE INFO 16001 ns EP LTSSM State CONFIG IDLE INFO 16093 ns RP LTSSM State CONFIG IDLE INFO 16285 ns RP LTSSM State L0 INFO 16545 ns EP LTSSM State L0 INFO 19112 ns Configuring Bus 001 Device 001 Function 00 INFO 19112 ns EP Read Only Configuration Registers INFO 19112 ns Vendor ID 0000 INFO 19112 ns Device ID 0001 INFO 19112 ns Revision ID 01 INFO 19112 ns Class Code 000000 INFO 1...

Page 53: ...ed INFO 37960 ns DLL Link Active Report Not Supported INFO 37960 ns INFO 37960 ns EP PCI Express Device Capabilities 2 Register 0000001F INFO 37960 ns Completion Timeout Rnge ABCD 50us to 64s INFO 39512 ns INFO 39512 ns EP PCI Express Device Control Register 0110 INFO 39512 ns Error Reporting Enables 0 INFO 39512 ns Relaxed Ordering Enabled INFO 39512 ns Error Reporting Enables 0 INFO 39512 ns Rel...

Page 54: ... to altera_pci_express sdc 1 Because altera_pci_express sdc is overwritten each time you regenerate your design you should save a copy of this file in an additional directory that the Quartus II software does not overwrite Creating a Quartus II Project You can create a new Quartus II project with the New Project Wizard which helps you specify the working directory for the project assign the projec...

Page 55: ... V GZ b In the Devices list select All c In the Available devices list select 5AGZME5K2F40C3 8 Click Next to close this page and display the EDA Tool Settings page 9 From the Simulation list select ModelSim From the Format list select the HDL language you intend to use for simulation 10 Click Next to display the Summary page 11 Check the Summary page to ensure that you have entered all the informa...

Page 56: ...3 18 Chapter 3 Getting Started with the Avalon MM Arria V GZ Hard IP for PCI Express Programming a Device Arria V GZ Hard IP for PCI Express November 2012 Altera Corporation User Guide ...

Page 57: ...able for the Arria V GZ Hard IP for PCI Express Parameter Group Application Layer Interface Avalon ST Avalon MM System Settings v v Base Address Register BAR and Expansion ROM Settings v v Base and Limit Registers for Root Ports v 1 Device Identification Registers v v PCI Express and PCI Capabilities Parameters v v Error Reporting v v Link Capabilities v v MSI and MSI X Capabilities v v Slot Capab...

Page 58: ...it allocation for the selected setting displays in the message pane Refer to Chapter 12 Flow Control for more information about optimizing performance The Flow Control chapter explains how the RX credit allocation and the Maximum payload size that you choose affect the allocation of flow control credits You can set the Maximum payload size parameter on the Device tab The Message window of the GUI ...

Page 59: ...lk Specifications for 8 0 GT s in the specification For Gen3 Altera recommends using a common reference clock 0 ppm because when using separate reference clocks non 0 ppm the PCS occasionally must insert SKP symbols potentially causes the PCIe link to go to recovery Arria V GZ PCIe Hard IP in Gen1 or Gen2 modes are not affected by this issue Systems using the common reference clock 0 ppm are not a...

Page 60: ... Hard IP Reconfiguration On Off When enabled you can use the Hard IP reconfiguration bus to dynamically reconfigure Hard IP read only registers For more information refer to Hard IP Reconfiguration Interface on page 15 1 Note to Table 4 2 1 The Gen1 and Gen2 simulation models support pipe and serial simulation The Gen3 simulation model supports serial simulation only with Phase 2 and Phase 3 Equal...

Page 61: ...I O addressing Specifies the address widths for the Prefetchable Memory Base register and Prefetchable Memory Limit register Table 4 5 Device ID Registers Register Name Offset Address Range Default Value Description Vendor ID 16 bits 0x0000 Sets the read only value of the Vendor ID register This parameter can not be set to 0xFFFF per the PCI Express Specification Address 0x000 Device ID 16 bits 0x...

Page 62: ...d 32 64 32 Avalon ST Indicates the number of tags supported for non posted requests transmitted by the Application Layer This parameter sets the values in the Device Control register 0x088 of the PCI Express capability structure described in Table 7 8 on page 7 5 The Transaction Layer tracks all outstanding completions for non posted requests made by the Application Layer This parameter configures...

Page 63: ...10 ms to 250 ms Range C 250 ms to 4 s Range D 4 s to 64 s Bits are set to show timeout value ranges supported The function must implement a timeout value in the range 50 s to 50 ms The following values are used to specify the range None Completion timeout programming is not supported 0001 Range A 0010 Range B 0011 Ranges A and B 0110 Ranges B and C 0111 Ranges A B and C 1110 Ranges B C and D 1111 ...

Page 64: ...from the Application Layer must contain the ECRC dword and have the TD bit set Track Receive Completion Buffer Overflow On Off Off When On the core includes the rxfx_cplbuf_ovf output status signal to track the RX posted completion buffer overflow status This signal is not available for the Avalon MM Hard IP for PCI Express IP Core Note to Table 4 7 1 Throughout the Arria V GZ Hard IP for PCI Expr...

Page 65: ...ystem software reads this field to determine the MSI X Table size n which is encoded as n 1 For example a returned value of 2047 indicates a table size of 2048 This field is read only Legal range is 0 2047 211 Address 0x068 26 16 MSI X Table Offset 31 0 Points to the base of the MSI X Table The lower 3 bits of the table BAR indicator BIR are set to zero by software to form a 32 bit qword aligned o...

Page 66: ...ed for the Slot power limit The following coefficients are defined 0 1 0x 1 0 1x 2 0 01x 3 0 001x The default value prior to hardware and firmware initialization is b 00 Writes to this register also cause the port to send the Set_Slot_Power_Limit Message Refer to Section 6 9 of the PCI Express Base Specification Revision 2 1 for more information Slot power limit 0 255 In combination with the Slot ...

Page 67: ...of 1 us Maximum of 2 us Maximum of 4 us Maximum of 8 us Maximum of 16 us Maximum of 32 us No limit This value indicates the acceptable latency that an Endpoint can withstand in the transition from the L1 to L0 state It is an indirect measure of the Endpoint s internal buffering It sets the read only value of the Endpoint L1 acceptable latency field of the Device Capabilities Register The Arria V G...

Page 68: ... a single request can be outstanding Single dword completer uses fewer resources than Completer Only This variant is targeted for systems that require simple read and write register accesses from a host CPU If you select this option the width of the data for RXM BAR masters is always 32 bits regardless of the Avalon MM width Control Register Access CRA Avalon slave port On Off Allows read and writ...

Page 69: ...on Number of address pages 1 2 4 8 16 32 64 128 256 512 Specifies the number of pages required to translate Avalon MM addresses to PCI Express addresses before a request packet is sent to the Transaction Layer Each of the 512 possible entries corresponds to a base address of the PCI Express memory segment of a specific size Size of address pages 4 KByte 4 GBytes Specifies the size of each memory s...

Page 70: ...4 14 Chapter 4 Parameter Settings Avalon to PCIe Address Translation Settings Arria V GZ Hard IP for PCI Express November 2012 Altera Corporation User Guide ...

Page 71: ...Generates all transmission cyclical redundancy code CRC values and checks all CRCs during reception Manages the retry buffer and retry mechanism according to received ACK NAK Data Link Layer packets Initializes the flow control mechanism for DLLPs and routes flow control credits to and from the Transaction Layer Physical Layer The Physical Layer initializes the speed lane numbering and lane width ...

Page 72: ...nd a Type 1 Configuration TLP is used to access the Configuration Space Registers of downstream components typically Endpoints on the other side of the link The Hard IP includes dedicated clock domain crossing logic CDC between the PHYMAC and Data Link Layers This chapter provides an overview of the architecture of the Arria V GZ Hard IP for PCI Express It includes the following sections Key Inter...

Page 73: ...s is partially supported Refer to the description of the rx_st_mask signal for further information about masking For more information about the RX datapath refer to Avalon ST RX Interface on page 6 4 TX Datapath The TX datapath transports data from the Application Layer s Avalon ST interface to the Transaction Layer The Hard IP provides credit information to the Application Layer for posted header...

Page 74: ...pulates that the frequency of this clock be 100 MHz the Hard IP also accepts a 125 MHz reference clock as a convenience You can specify the frequency of your input reference clock using the parameter editor under the System Settings heading The PCI Express Base Specification also requires a system configuration time of 100 ms To meet this specification the Arria V GZ Hard IP for PCI Express includ...

Page 75: ...allel interface to speed simulation however you cannot use the PIPE interface in actual hardware The Gen1 and Gen2 simulation models support pipe and serial simulation The Gen3 simulation model supports serial simulation only with equalization bypassed Transaction Layer The Transaction Layer is located between the Application Layer and the Data Link Layer It generates and receives Transaction Laye...

Page 76: ...and must be prepared to provide the entire data payload in consecutive cycles 3 The Transaction Layer verifies that sufficient flow control credits exist and acknowledges or postpones the request 4 The Transaction Layer forwards the TLP to the Data Link Layer Figure 5 3 Architecture of the Transaction Layer Dedicated Receive Buffer Transaction Layer TX Datapath Transaction Layer RX Datapath Avalon...

Page 77: ...ions are dependent upon the content of the PCI Express Configuration Space as described in the PCI Express Base Specification Revision 2 1 f Refer To Configuration Space Register Content on page 7 1 or Chapter 7 in the PCI Express Base Specification 2 1 for the complete content of these registers Data Link Layer The Data Link Layer is located between the Transaction Layer and the Physical Layer It...

Page 78: ...s block generates transmit packets generating a sequence number and a 32 bit CRC LCRC The packets are also sent to the retry buffer for internal storage In retry mode the TLP generator receives the packets from the retry buffer and generates the CRC for the transmit packet Retry Buffer The retry buffer stores TLPs and retransmits all unacknowledged packets in the case of NAK DLLP reception For ACK...

Page 79: ...rd IP for PCI Express It is the layer closest to the link It encodes and transmits packets across a link and accepts and decodes received packets The Physical Layer connects to the link through a high speed SERDES interface running at 2 5 Gbps for Gen1 implementations at 2 5 or 5 0 Gbps for Gen2 implementations and at 2 5 5 0 or 8 0 Gbps for Gen 3 implementations The Physical Layer is responsible ...

Page 80: ...ck is divided in four main sub blocks MAC Lane Both the RX and the TX path use this block On the RX side the block decodes the Physical Layer Packet and reports to the LTSSM the type and number of TS1 TS2 ordered sets received On the TX side the block multiplexes data from the DLL and the LTSTX sub block It also adds lane specific information including the lane number and the force PAD value when ...

Page 81: ...d FIFO for each lane to store symbols Each symbol includes eight data bits one disparity bit and one control bit The FIFO discards the FTS COM and SKP symbols and replaces PAD and IDL with D0 0 data When all eight FIFOs contain data a read can occur When the multilane lane deskew block is first enabled each FIFO begins writing after the first COM is detected If all lanes have not detected a COM sy...

Page 82: ...tutes a unpipelined 32 bit RX master port for the 64 or 128 bit full featured RX master port For more information about the 32 bit RX master refer to Avalon MM RX Master Block on page 5 22 Figure 5 6 shows the block diagram of a full featured PCI Express Avalon MM bridge Figure 5 6 PCI Express Avalon MM Bridge Transaction Layer PCI Express Tx Controller PCI Express Rx Controller Data Link Layer Ph...

Page 83: ...xpress Upstream Read Requests PCI Express to Avalon MM Read Completions PCI Express to Avalon MM Downstream Write Requests PCI Express to Avalon MM Downstream Read Requests Avalon MM to PCI Express Read Completions PCI Express to Avalon MM Address Translation for Endpoints Avalon MM to PCI Express Address Translation Algorithm Avalon MM to PCI Express Write Requests The Avalon MM bridge accepts Av...

Page 84: ...ased on the address and the size of the read request For Avalon MM read requests with a burst count greater than one all byte enables must be asserted There are no restrictions on byte enables for Avalon MM read requests with a burst count of one An invalid Avalon MM request can adversely affect system functionality resulting in a completion with the abort status set An example of an invalid reque...

Page 85: ...lation lookup table values The RX Avalon MM master port drives the received address to the fabric You can set up the Address Translation Table Configuration in the GUI Unsupported read requests generate a completer abort response For more information about optimizing BAR addresses refer to Minimizing BAR Sizes and the PCIe Address Space Avalon MM to PCI Express Read Completions The PCI Express Ava...

Page 86: ... Qsys interconnect fabric manages mismatched port widths transparently As Memory Request TLPs are received from the PCIe link the most significant bits are used in the BAR matching as described in the PCI specifications The least significant bits not used in the BAR match process are passed unchanged as the Avalon MM address for that BAR s RX Master port For example consider the following configur...

Page 87: ...e address space that the BARs consume For example consider a Qsys system with the following components Offchip_Data_Mem DDR3 SDRAM Controller with UniPHY controlling 256 MBytes of memory Qsys auto assigned a base address of 0x00000000 Quick_Data_Mem On Chip Memory RAM or ROM of 4 KBytes Qsys auto assigned a base address of 0x10000000 Instruction_Mem On Chip Memory RAM or ROM of 64 KBytes Qsys auto...

Page 88: ...t in the following three large BARs BAR0 is 28 bits This is the optimal size because it addresses the Offchip_Data_Mem which requires 28 address bits BAR2 is 29 bits BAR2 addresses the Quick_Data_Mem which is 4 KBytes It should only require 12 address bits however it is consuming 512 MBytes of address space BAR4 is also 29 bits BAR4 address PCIe Cra which is 16 KBytes It should only require 14 add...

Page 89: ...The Avalon MM address of a received request on the TX Avalon MM slave port is translated to the PCI Express address before the request packet is sent to the Transaction Layer You can specify up to 512 address pages and sizes ranging from 4 KByte to 4 GBytes when you customize your Avalon MM Arria V GZ Hard IP for PCI Express as described in Avalon to PCIe Address Translation Settings on page 4 13 ...

Page 90: ...butes Number of Address Pages 16 Size of Address Pages 1 MByte PCI Express Address Size 64 bits then the values in Figure 5 12 are N 20 due to the 1 MByte page size Q 16 number of pages M 24 20 4 bit page selection P 64 In this case the Avalon address is interpreted as follows Bits 31 24 select the TX slave module port from among other slaves connected to the same master by the system interconnect...

Page 91: ...the following requests Read and write requests of a single dword 32 bits from the Root Complex Completion with Completer Abort status generation for other types of non posted requests INTX or MSI support with one Avalon MM interrupt source Figure 5 12 Avalon MM to PCI Express Address Translation 1 2 3 4 5 Notes to Figure 5 12 1 N is the number of pass through bits 2 M is the number of Avalon MM ad...

Page 92: ...r bytes The RX block passes header information to the Avalon MM master which generates the corresponding transaction to the Avalon MM interface The bridge accepts no additional requests while a request is being processed While processing a read request the RX block deasserts the ready signal until the TX block sends the corresponding completion packet to the hard IP block While processing a write ...

Page 93: ...a V GZ Hard IP for PCI Express when received otherwise INTX is signaled The interrupt handler block supports a single interrupt source so that software may assume the source You can disable interrupts by leaving the interrupt signal unconnected in the IRQ column of Qsys When the MSI registers in the Configuration Space of the Completer Only Single Dword Arria V GZ Hard IP for PCI Express are updat...

Page 94: ...5 24 Chapter 5 IP Core Architecture Completer Only Single Dword Endpoint Arria V GZ Hard IP for PCI Express November 2012 Altera Corporation User Guide ...

Page 95: ...mpletion Transaction Layer Packets TLPs into standard Avalon MM read and write commands typically used by master and slave interfaces to access memories and registers Consequently you do not need a detailed understanding of the PCI Express TLPs to use the Avalon MM variants Refer to Hard IP for PCI Express Features on page 1 2 to learn about the difference in the features available for the Avalon ...

Page 96: ...2 0 rxelecidle0_ext rxstatus 2 0 txblkst0 txsychd0 1 0 txdataskip0 rxblkst0 rxsynchd0 1 0 rxdataskip0 simu_mode_pipe ltssmstate 4 0 rate 1 0 pclk_in tx_margin 2 0 txswing testin_zero 8 bit PIPE PIPE Simulation Only for Gen1 and Gen2 Gen3 Provides Serial Simulation test_in 31 0 lane_act 3 0 tl_cfg_add 3 0 tl_cfg_ctl 31 0 tl_cfg_sts 52 0 hpg_ctrler 4 0 lmi_dout 31 0 lmi_rden lmi_wren lmi_ack lmi_add...

Page 97: ...errupts for Root Ports on page 6 33 Completion Completion Side Band Signals on page 6 33 Configuration space Transaction Layer Configuration Space Signals on page 6 35 Parity Error Parity Signals on page 6 42 LMI LMI Signals on page 6 43 Hard IP reconfiguration block Hard IP Reconfiguration Interface on page 6 45 Power management Power Management Signals on page 6 47 Physical Transceiver control T...

Page 98: ... 64 When using a 128 bit Avalon ST bus the width of rx_st_data is 128 When using a 256 bit Avalon ST bus the width of rx_st_data is 256 bits rx_st_sop 1 2 O start of packet Indicates that this is the first cycle of the TLP when rx_st_valid is asserted When using a 256 bit Avalon ST bus the following correspondences apply When you turn on Enable multiple packets per cycle bit 0 indicates that a TLP...

Page 99: ...ata For 256 bit data when you turn on Enable multiple packets per cycle the following correspondences apply bit 1 applies to the eop occurring in rx_st_data 255 128 bit 0 applies to the eop occurring in rx_st_data 127 0 When the TLP ends in the lower 128bits the following equations apply rx_st_eop 0 1 rx_st_empty 0 0 rx_st_data 127 0 contains valid data rx_st_eop 0 1 rx_st_empty 0 1 rx_st_data 63 ...

Page 100: ...ndences apply bit 1 applies to rx_st_data 255 128 bit 0 applies to rx_st_data 127 0 Altera recommends resetting the Arria V GZ Hard IP for PCI Express when an uncorrectable double bit ECC error is detected Component Specific Signals rx_st_mask 1 I component specific The Application Layer asserts this signal to tell the Hard IP to stop sending non posted requests This signal can be asserted at any ...

Page 101: ... only apply to PCI Express Memory Write and I O Write TLP payload fields When using 64 bit Avalon ST bus the width of rx_st_be is 8 bits When using 128 bit Avalon ST bus the width of rx_st_be is 16 bits When using a 256 bit Avalon ST bus the width of rx_st_be is 32 bits This signal is optional You can derive the same information by decoding the FBE and LBE fields in the TLP header The byte enable ...

Page 102: ...ket TLP Header Formats for a layout of each byte of the TLP headers Figure 6 3 illustrates the mapping of Avalon ST RX packets to PCI Express TLPs for a three dword header with non qword aligned addresses with a 64 bit bus In this example the byte address is unaligned and ends with 0x4 causing the first data to correspond to rx_st_data 63 32 Figure 6 2 Qword Alignment 0x0 0x8 0x10 0x18 Header Addr...

Page 103: ...r a four dword header with qword aligned addresses with a 64 bit bus Figure 6 3 64 Bit Avalon ST rx_st_data n Cycle Definition for 3 Dword Header TLPs with Non Qword Aligned Address pld_clk rx_st_data 63 32 rx_st_data 31 0 rx_st_sop rx_st_eop rx_st_be 7 4 rx_st_be 3 0 Header1 Data0 Data2 Header0 Header2 Data1 F F F Figure 6 4 64 Bit Avalon ST rx_st_data n Cycle Definition for 3 Dword Header TLPs w...

Page 104: ...by deasserting rx_st_ready The rx_st_valid signal deasserts within three cycles after rx_st_ready is deasserted In this example rx_st_valid is deasserted in the next cycle rx_st_data is held until the Application Layer is able to accept it Figure 6 6 64 Bit Avalon ST rx_st_data n Cycle Definitions for 4 Dword Header TLPs with Non Qword Addresses 1 Note to Figure 6 6 1 rx_st_be 7 4 corresponds to r...

Page 105: ... header and qword aligned addresses The assertion of rx_st_empty in a rx_st_eop cycle indicates valid data on the lower 64 bits of rx_st_data Figure 6 8 64 Bit Avalon ST Interface Back to Back Transmission pld_clk rx_st_data 63 0 rx_st_sop rx_st_eop rx_st_ready rx_st_valid C C C C CCCC0089002 C C C C C C C C C C C C C C C C C C C C C C C C C C C C Figure 6 9 128 Bit Avalon ST rx_st_data n Cycle De...

Page 106: ...header with non qword aligned addresses In this example rx_st_empty is low because the data is valid for all 128 bits in the rx_st_eop cycle Figure 6 10 128 Bit Avalon ST rx_st_data n Cycle Definition for 3 Dword Header TLPs with non Qword Aligned Addresses rx_st_valid rx_st_data 127 96 rx_st_data 95 64 rx_st_data 63 32 rx_st_data 31 0 rx_st_sop rx_st_eop rx_st_empty Data0 Data 4 Header 2 Data 3 H...

Page 107: ...thin three cycles after rx_st_ready is deasserted In this example rx_st_valid is deasserted in the next cycle rx_st_data is held until the Application Layer is able to accept it Figure 6 12 128 Bit Avalon ST rx_st_data Cycle Definition for 4 Dword Header TLPs with Qword Aligned Addresses pld_clk rx_st_valid rx_st_data 127 96 rx_st_data 95 64 rx_st_data 63 32 rx_st_data 31 0 rx_st_sop rx_st_eop rx_...

Page 108: ...to Appendix A Transaction Layer Packet TLP Header Formats Single Packet Per Cycle In single packer per cycle mode all received TLPs start at the lower 128 bit boundary on a 256 bit Avalon ST interface Turn on Enable Multiple Packets per Cycle on the System Settings tab of the parameter editor to change multiple packets per cycle Figure 6 14 128 Bit Avalon ST Interface Back to Back Transmission pld...

Page 109: ... 16 shows the location of headers and data for the 256 bit Avalon ST packets This layout of data applies to both the TX and RX buses Figure 6 17 illustrates two single cycle 256 bit packets The first packet has two empty qword rx_st_data 191 0 is valid The second packet has two empty dwords rx_st_data 127 0 is valid Figure 6 16 Location of Headers and Data for Avalon ST 256 Bit Interface D3 255 0 ...

Page 110: ... bits of the Avalon ST interface because a new TLP can start in the lower 128 bit Avalon ST interface This mode adds complexity to the Application Layer user decode logic However it could result in higher throughput Figure 6 18 illustrates this mode for a 256 bit Avalon ST RX interface In this figure rx_st_eop 0 and rx_st_sop 1 are asserted in the same cycle Figure 6 18 256 Bit Avalon ST RX Interf...

Page 111: ... Indicates first cycle of a TLP when asserted together with tx_st_valid When using a 256 bit Avalon ST bus with Multiple packets per cycle bit 0 indicates that a TLP begins in tx_st_data 127 0 bit 1 indicates that a TLP begins in tx_st_data 255 128 tx_st_eop 1 2 I end of packet Indicates last cycle of a TLP when asserted together with tx_st_valid When using a 256 bit Avalon ST bus with Multiple pa...

Page 112: ...n the end of a packet When asserted the empty dwords are in the high order bits Valid only when tx_st_eop is asserted Not used when tx_st_data is 64 bits For 128 bit data only bit 0 applies and indicates whether the upper qword contains data For 256 bit data both bits are used to indicate the number of upper words that contain data resulting in the following encodings for the 128 and 256 bit inter...

Page 113: ... error signal Note that it must be asserted while the valid signal is asserted tx_st_parity 8 16 32 O component specific Byte parity is generated when you turn on Enable byte parity ports on Avalon ST interface on the System Settings tab of the GUI Each bit represents odd parity of the associated byte of the tx_st_data bus For example bit 0 corresponds to tx_st_data 7 0 bit 1 corresponds to tx_st_...

Page 114: ...ata 1 completion header 0 completion data tx_cred_hdrfccp 8 O component specific Header credit limit for the FC completions Each credit is 20 bytes tx_cred_hdrfcnp 8 O component specific Header limit for the non posted requests Each credit is 20 bytes tx_cred_hdrfcp 8 O component specific Header credit limit for the FC posted writes Each credit is 20 bytes ko_cpl_spc_header 8 O component specific ...

Page 115: ...r1 pcie_hdr_byte4 pcie_hdr _byte5 header pcie_hdr byte6 pcie_hdr _byte7 3 Header2 pcie_hdr _byte8 pcie_hdr _byte9 pcie_hdr _byte10 pcie_hdr _byte11 4 Data0 pcie_data_byte3 pcie_data_byte2 pcie_data_byte1 pcie_data_byte0 5 Data1 pcie_data_byte7 pcie_data_byte6 pcie_data_byte5 pcie_data_byte4 6 Data2 pcie_data_byte11 pcie_data_byte10 pcie_data_byte9 pcie_data_byte8 pld_clk tx_st_data 63 32 tx_st_dat...

Page 116: ...les after tx_st_ready is asserted Figure 6 23 illustrates back to back transmission of 64 bit packets with no idle cycles between the assertion of tx_st_eop and tx_st_sop Figure 6 21 64 Bit Avalon ST tx_st_data Cycle Definition for TLP 4 Dword Header with Non Qword Aligned Address pld_clk tx_st_data 63 32 tx_st_data 31 0 tx_st_sop tx_st_eop Header 1 Header3 Data0 Data2 Header 0 Header2 Data1 Figur...

Page 117: ...xpress TLPs for a 3 dword header with non qword aligned addresses It also shows tx_st_err assertion Figure 6 24 128 Bit Avalon ST tx_st_data Cycle Definition for 3 Dword Header TLP with Qword Aligned Address Data3 Header2 Data 2 Header1 Data1 Data n Header0 Data0 Data n 1 pld_clk tx_st_valid tx_st_data 127 96 tx_st_data 95 64 tx_st_data 63 32 tx_st_data 31 0 tx_st_sop tx_st_eop tx_st_empty Figure ...

Page 118: ...use the data ends in the upper 64 bits of tx_st_data Figure 6 26 128 Bit Avalon ST tx_st_data Cycle Definition for 4 Dword Header TLP with Qword Aligned Address pld_clk tx_st_data 127 96 tx_st_data 95 64 tx_st_data 63 32 tx_st_data 31 0 tx_st_sop tx_st_eop tx_st_empty Header 3 Data 3 Header 2 Data 2 Header 1 Data 1 Header 0 Data 0 Data 4 Figure 6 27 128 Bit Avalon ST tx_st_data Cycle Definition fo...

Page 119: ...r deasserts tx_st_valid after two cycles and holds tx_st_data until two cycles after tx_st_ready is reasserted Data Alignment and Timing for the 256 Bit Avalon ST TX Interface Refer to Figure 6 16 on page 6 15 layout of headers and data for the 256 bit Avalon ST packets with qword aligned and qword unaligned addresses Figure 6 28 128 Bit Back to Back Transmission on the Avalon ST TX Interface pld_...

Page 120: ...ts always start in the lower 128 bits of the Avalon ST interface Although this mode simplifies the Application Layer logic failure to use the full 256 bit Avalon ST may slightly reduce the throughput of a design Figure 6 30 illustrates the layout of header and data for a three dword header on a 256 bit bus with aligned and unaligned data Figure 6 30 256 Bit Avalon ST tx_st_data Cycle Definition fo...

Page 121: ...re should be no ECRC appended to the TLP and the TD bit in the TLP header should be set to 0 These packets are processed internally by the Hard IP block and are not transmitted on the PCI Express link ECRC Forwarding On the Avalon ST interface the ECRC field follows the same alignment rules as payload data For packets with payload the ECRC is appended to the data as an extra dword of payload For p...

Page 122: ...ements as listed in Table 8 3 on page 8 6 Note to Table 6 6 1 Figure 8 5 on page 8 5 illustrates these clock signals Table 6 7 Reset and Link Training Signals Part 1 of 3 Signal I O Description npor I Active low reset signal In the Altera hardware example designs npor is the OR of pin_perst and local_rstn coming from the software Application Layer If you do not drive a soft reset signal from the A...

Page 123: ...k is not 3 3V if the following 2 conditions are met The input signal meets the VIH and VIL specification for LVTTL The input signal meets the overshoot specification for 100 C operation as specified by the Maximum Allowed Overshoot and Undershoot Voltage section in volume 3 of the Arria V Device Handbook Refer to Figure 6 32 on page 6 31 for a timing diagram illustrating the use of this signal ser...

Page 124: ...cles currentspeed 1 0 O Indicates the current speed of the PCIe link The following encodings are defined 2b 00 Undefined 2b 01 Gen1 2b 10 Gen2 2b 11 Gen3 ltssmstate 4 0 O LTSSM state The LTSSM state machine encoding defines the following states 00000 Detect Quiet 00001 Detect Active 00010 Polling Active 00011 Polling Compliance 00100 Polling Configuration 00101 Polling Speed 00110 config Linkwidth...

Page 125: ... Gen2 and Gen3 capable designs to begin link initialization and ultimately to reach L0 before the FPGA is configured is pending device characterization npor IO_POF_Load PCIe_LinkTraining_Enumeration dl_ltssm 4 0 detect detect active polling active L0 100 ms Table 6 8 Error Signals for Hard IP Implementation 1 Part 1 of 2 Signal I O Description derr_cor_ext_rcv O Indicates a corrected error in the ...

Page 126: ...enerated based on the MSI configuration register values and the app_msi_tc and app_msi_num input ports app_msi_ack O Application Layer MSI acknowledge This signal acknowledges the Application Layer s request for an MSI interrupt app_msi_tc 2 0 I Application Layer MSI traffic class This signal indicates the traffic class used to send the MSI unlike INTX interrupts any traffic class can be used to s...

Page 127: ...iate status bits for Table 6 10 Exported Interrupt Signals for Endpoints when Multiple MSIMSI X Support is Enabled Signal I O Description msi_intf 81 0 O This bus provides the following MSI address data and enabled signals msi_intf 81 Master enable msi_intf 80 MSI enable msi_intf 79 64 MSI data msi_intf 63 0 MSI address msix_intf 79 0 O This bus provides the following MSI address data and enabled ...

Page 128: ...completion transaction after the 50 ms timeout period when the error is correctable The Hard IP automatically generates an advisory error message that is sent to the Root Complex cpl_err 1 Completion timeout error without recovery This signal should be asserted when a master like interface has performed a non posted request that never receives a corresponding completion transaction after the 50 ms...

Page 129: ...nsaction Layer Errors on page 13 3 cpl_err 6 Log header If header logging is required this bit must be set in the every cycle in which any of cpl_err 2 cpl_err 3 cpl_err 4 or cpl_err 5 is set The Application Layer presents the header to the Hard IP by writing the following values to the following 4 registers using LMI before asserting cpl_err 6 lmi_addr 12 h81C lmi_din err_desc_func0 127 96 lmi_ad...

Page 130: ...ed This signal should be asserted when the power controller detects a power fault for this slot If this slot has no power controller this bit should be hardwired to 0 and the Power Controller Present bit bit 1 in the Slot capability register parameter is set to 0 I 4 Power controller status This signal is used to set the command completed bit of the Slot Status register Power controller status is ...

Page 131: ...bort Bit 12 received target abort Bit 11 signalled target abort 24 Secondary Status Register 8 Master data parity error 23 6 Root Status Register 17 0 Records the following PME status information Bit 17 PME pending Bit 16 PME status Bits 15 0 PME request ID 15 0 5 1 Secondary Status Register 15 11 Records the following 5 secondary command status errors Bit 15 detected parity error Bit 14 received ...

Page 132: ..._ctrl 7 0 4 cfg_sec_ctrl 15 0 cfg_secbus 7 0 cfg_subbus 7 0 5 cfg_msi_addr 11 0 cfg_io_bas 19 0 6 cfg_msi_addr 43 32 cfg_io_lim 19 0 7 8h 00 cfg_np_bas 11 0 cfg_np_lim 11 0 8 cfg_pr_bas 31 0 9 cfg_msi_addr 31 12 cfg_pr_bas 43 32 A cfg_pr_lim 31 0 B cfg_msi_addr 63 44 cfg_pr_lim 43 32 C cfg_pmcsr 31 0 D cfg_msixcsr 15 0 cfg_msicsr 15 0 E 6 h00 tx_ecrcgen 25 3 rx_ecrccheck 24 cfg_tcvcmap 23 0 F cfg_...

Page 133: ...ink_ctrl2 15 0 the primary Link Status register contents is available on tl_cfg_sts 46 31 For Gen1 variants the link bandwidth notification bit is always set to 0 For Gen2 variants this bit is set to 1 Table 7 8 on page 7 5 0x0B0 Gen2 only cfg_prm_cmd 16 O Base Primary Control and Status register for the PCI Configuration Space Table 7 2 on page 7 2 0x004 Type 0 Table 7 3 on page 7 2 0x004 Type 1 ...

Page 134: ...e upper 44 bits of the prefetchable limit registers of the Type1 Configuration Space Available in Root Port mode Table 7 3 on page 7 2 0x024 and Table 4 3 on page 4 4 prefetchable memory cfg_pmcsr 32 O cfg_pmcsr 31 16 is Power Management Control and cfg_pmcsr 15 0 is the Power Management Status register Table 7 6 on page 7 4 0x07C cfg_msixcsr 16 O MSI X message control Table 7 5 on page 7 3 0x068 ...

Page 135: ...ge enable multiple message capable MSI enable Table 6 18 Configuration MSI Control Status Register Field Descriptions Part 1 of 2 Bit s Field Description 15 9 reserved 8 mask capability Per vector masking capable This bit is hardwired to 0 because the function does not support the optional MSI per vector masking using the Mask_Bits and Pending_Bits registers defined in the PCI Local Bus Specificat...

Page 136: ...Up to 32 parity bits are propagated to the Application Layer along with the RX Avalon ST data The RX datapath also propagates up to 32 parity bits to the Transaction Layer for Configuration TLPs On the TX datapath parity generated in the Application Layer is checked in Transaction Layer and the Data Link Layer 3 1 multiple message capable Multiple message capable This field is read by system softw...

Page 137: ...nds resetting the Arria V GZ Hard IP for PCI Express when this error is detected Contact Altera if resetting becomes unworkable rx_par_err O When asserted for a single cycle indicates that a parity error was detected in a TLP at the input of the RX buffer This error is logged as an uncorrectable internal error in the VSEC registers For more information refer to Uncorrectable Internal Error Status ...

Page 138: ...n configuration TLP accesses are no longer pending An acknowledge signal is sent back to the Application Layer when the execution is complete All LMI reads are also held and executed when no configuration TLP requests are pending The LMI interface supports two operations local read and local write The timing for these operations complies with the Avalon MM protocol described in the Avalon Interfac...

Page 139: ... address and 16 bit data You can use this bus dynamically modify the value of configuration registers that are read only at run time To ensure proper system operation Altera recommends that you reset or repeat device enumeration of the PCI Express link after changing the value of read only configuration registers of the Hard IP For a description of the registers available via this interface refer ...

Page 140: ...Read signal This interface is not pipelined You must wait for the return of the hip_reconfig_readdata 15 0 from the current read before starting another read operation hip_reconfig_readdata 15 0 O 16 bit read data hip_reconfig_readdata 15 0 is valid on the third cycle after the assertion of hip_reconfig_read hip_reconfig_write I Write signal hip_reconfig_writedata 15 0 I 16 bit write model hip_rec...

Page 141: ...s from the low power state to send the message This signal is positive edge sensitive pm_data 9 0 I Power Management Data This bus indicates power consumption of the component This bus can only be implemented if all three bits of AUX_power part of the Power Management Capabilities structure are set to 0 This bus includes the following bits pm_data 9 2 Data Register This register maintains a value ...

Page 142: ...dicates that the function would normally assert the PME message independently of the state of the PME_en bit 14 13 data_scale This field indicates the scaling factor when interpreting the value retrieved from the data register This field is read only 12 9 data_select This field indicates which data should be reported through the data register and the data_scale field 8 PME_EN 1 indicates that the ...

Page 143: ...est 32 Bit Avalon MM CRA Slave Port Optional Not available for Completer Only Single Dword 64 Bit Avalon MM TX Slave Port Not used for Completer Only Avalon MM Hard IP for PCI Express Test Interface test_in 31 0 mode rxm_bar0_write_ n rxm_bar0_address_ n 31 0 rxm_bar0_writedata_ n 63 0 or 31 0 rxm_bar0_byteenable_ n 7 0 rxm_bar0_burstcount_ n 6 0 rxm_bar0_waitrequest_ n rxm_bar0_read_ n rxm_bar0_r...

Page 144: ...aster v v RX Avalon MM Master Signals on page 6 51 Avalon MM TX Slave v 64 Bit Bursting TX Avalon MM Slave Signals on page 6 52 Clock v v Clock Signals on page 6 28 Reset and Status v v Reset Signals and Status Signals on page 6 28 Multiple MSI MSI X Interrupt Support v Interrupts for Endpoints when Multiple MSI MSI X Support Is Enabled on page 6 33 Physical and Test Transceiver Control v v Transc...

Page 145: ..._ n w 1 0 O RX data being written to slave w 64 for the full featured IP core w 32 for the completer only IP core rxm_bar0_byteenable_ n w 1 0 O Byte enable for write data rxm_bar0_burstcount_ n 6 0 O The burst count measured in qwords of the RX write or read request The width indicates the maximum data that can be requested The maximum data in a burst is 512 bytes rxm_bar0_waitrequest_ n I Assert...

Page 146: ...e interconnect fabric are translated into PCI Express request packets Incoming requests can be up to 512 bytes For better performance Altera recommends using smaller read request size a maximum of 512 bytes Figure 6 40 Simultaneous DMA Read DMA Write and Target Access RxmRead_o RxmReadDataValid_i RxmReadData_i 63 0 RxmResetRequest_o RxmAddress_o 31 0 RxmWaitRequest_i RxmWrite_o RxmBurstCount_o 9 0...

Page 147: ... This behavior is most easily implemented with a store and forward buffer in the Avalon MM master txs_writedata 63 0 I Write data sent by the external Avalon MM master to the TX slave port txs_burstcount 6 0 I Asserted by the system interconnect fabric indicating the amount of data requested The count unit is the amount of data that is transferred in a single cycle that is the width of the bus The...

Page 148: ...imulation Serial Interface Signals Table 6 31 describes the serial interface signals Table 6 29 Transceiver Control Signals Signal Name I O Description reconfig_from_xcvr n 46 1 0 reconfig_to_xcvr n 70 1 0 O These are the parallel transceiver dynamic reconfiguration buses Dynamic reconfiguration is required to compensate for variations due to process voltage and temperature PVT Among the analog se...

Page 149: ...7 0 Note to Table 6 31 1 The 1 IP core only has lane 0 The 2 IP core only has lanes 1 0 The 4 IP core only has lanes 3 0 Table 6 31 1 Bit Interface Signals Part 2 of 2 Signal I O Description Figure 6 41 Channel Placement Gen1 and Gen2 x1 and x4 Variants Gen1 and Gen 2 x1 Transceiver Bank LCD LCD Local Clock Divider Channel 0 Data Channel 1 CMU PLL Channel 2 Data Channel 4 Channel 5 PCIe Lane 0 Cha...

Page 150: ...Using CMU PLL Channel 0 Data Channel 1 Data Channel 2 Data Channel 4 CMU PLL Channel 5 Data Channel 3 Data Transceiver Bank 1 Gen1 and Gen2 x8 Transceiver Bank 0 Channel 6 Data Available for Other Protocols Channel 7 Data Channel 8 Data Channel 10 Channel 11 PCS Clock and Control Signals Channel 9 CCD CCD Central Clock Divider PCI Express Lane 0 PCI Express Lane 1 PCI Express Lane 2 PCI Express La...

Page 151: ...el Placement Gen3 8 Variants Using ATX PLL Channel 0 Data Channel 1 Data Channel 2 Data Channel 4 CMU PLL Channel 5 Data Channel 3 Data Transceiver Bank 1 Gen3 Transceiver Bank 0 Channel 6 Data Available for Other Protocols Channel 7 Data Channel 8 Data Channel 10 Channel 11 Channel 9 CCD CCD Central Clock Divider PCI Express Lane 0 PCI Express Lane 1 PCI Express Lane 2 PCI Express Lane 3 PCI Expr...

Page 152: ...L Figure 6 44 illustrates the channel placement for Gen1 and Gen2 1 and 4 variants when you select the ATX PLL Figure 6 44 Channel Placement Gen1 and Gen2 Using ATX PLL Gen1 and Gen 2 x1 Transceiver Bank LCD ATX PLL0 LCD Local Clock Divider Channel 0 Data Channel 1 CMU PLL Channel 2 Data Channel 4 Channel 5 PCI Express Lane 0 Channel 3 Other Protocols CCD Central Clock Divider Other Protocols Tran...

Page 153: ...hannel 1 Data Channel 2 Data Channel 4 CMU PLL Channel 5 Data Channel 3 Data Transceiver Bank 1 Gen1 and Gen2 x8 Transceiver Bank 0 Available for Other Protocols CCD CCD Central Clock Divider PCI Express Lane 0 PCI Express Lane 1 PCI Express Lane 2 PCI Express Lane 3 Unavailable Channel PCI Express Lane 4 PCI Express Lane 5 PCI Express Lane 6 PCI Express Lane 7 ATX PLL1 Channel 6 Data Channel 7 Da...

Page 154: ...it is not possible to use the Hard IP PIPE interface in hardware including probing these signals using SignalTap II Embedded Logic Analyzer 1 The Gen3 simulation model supports serial only simulation The Root Port BFM bypasses Phase 2 and Phase 3 Equalization You must adjust your third party Root Port BFM to terminate Equalization after Phase 0 and Phase 1 complete Figure 6 46 Channel Placement Ge...

Page 155: ...e specified state P0 P0s P1 or P2 tx_deemph0 O Transmit de emphasis selection The Arria V GZ Hard IP for PCI Express sets the value for this signal based on the indication received from the other end of the link during the Training Sequences TS You do not need to change this value rxdata0 7 0 1 2 I Receive data n 2 symbols on lane n This bus receives data on lane n rxdatak0 1 2 I Receive data n Th...

Page 156: ...ings are defined 2 b01 Ordered Set Block 2 b10 Data Block rxdataskip0 I For Gen3 operation Allows the PCS to instruct the RX interface to ignore the RX data interface for one clock cycle The following encodings are defined 1 b0 RX data is invalid 1 b1 RX data is valid ltssmstate0 4 0 LTSSM state The LTSSM state machine encoding defines the following states 5 b00000 Detect Quiet 5 b 00001 Detect Ac...

Page 157: ... b11001 L2 transmit Wake 5 b11010 Speed Recovery 5 b11011 Recovery Equalization Phase 0 5 b11100 Recovery Equalization Phase 1 5 b11101 Recovery Equalization Phase 2 5 b11110 Recovery Equalization Phase 3 5 b11111 Recovery Equalization Done rate 1 0 O The 2 bit encodings have the following meanings 2 b00 Gen1 rate 2 5 Gbps 2 b01 Gen2 rate 5 0 Gbps 2 b1X Gen3 rate 8 0 Gbps pclk_in I This clock is u...

Page 158: ...gns other required PMA analog settings including 100 ohm internal termination Table 6 33 Test Interface Signals 1 2 Signal I O Description test_in 31 0 I The bits of the test_in bus have the following definitions 0 Simulation mode This signal can be set to 1 to accelerate initialization by reducing the value of many initialization counters 4 1 Reserved Must be set to 4 b0100 5 Compliance test mode...

Page 159: ...03C PCI Type 0 Compatible Configuration Space Header Refer to Table 7 2 for details 0x000 0x03C PCI Type 1 Compatible Configuration Space Header Refer to Table 7 3 for details 0x040 0x04C Reserved 0x050 0x05C MSI Capability Structure Refer to Table 7 4 for details 0x060 0x064 Reserved 0x068 0x070 MSI X Capability Structure Refer to Table 7 5 for details 0x070 0x074 Reserved 0x078 0x07C Power Manag...

Page 160: ... Address Register BAR and Expansion ROM Settings 0x014 Base Address Register BAR and Expansion ROM Settings 0x018 Base Address Register BAR and Expansion ROM Settings 0x01C Base Address Register BAR and Expansion ROM Settings 0x020 Base Address Register BAR and Expansion ROM Settings 0x024 Base Address Register BAR and Expansion ROM Settings 0x028 Reserved 0x02C Subsystem Device ID Subsystem vendo...

Page 161: ... between the Configuration Space registers and the PCI Express Base Specification 2 0 Table 7 3 PCI Type 1 Configuration Space Header Root Ports Rev3 0 Spec Type 1 Configuration Space Header Part 2 of 2 Byte Offset 31 24 23 16 15 8 7 0 Table 7 4 MSI Capability Structure Rev3 0 Spec MSI Capability Structures Byte Offset 1 31 24 23 16 15 8 7 0 0x050 Message Control Configuration MSI Control Status R...

Page 162: ...7C Data PM Control Status Bridge Extensions Power Management Status Control Note to Table 7 6 1 Refer to Table 7 39 on page 7 22 for a comprehensive list of correspondences between the Configuration Space registers and the PCI Express Base Specification 2 0 Table 7 7 PCI Express AER Capability Structure Rev3 0 Spec AER Capability Byte Offset 31 24 23 16 15 8 7 0 0x800 PCI Express Enhanced Capabili...

Page 163: ... Slot Capabilities 0x098 Slot Status Slot Control 0x09C Root Capabilities Root Control 0x0A0 Root Status 0x0A4 Device Capabilities 2 0x0A8 Device Status 2 Device Control 2 0x0AC Link Capabilities 2 0x0B0 Link Status 2 Link Control 2 0x0B4 Slot Capabilities 2 0x0B8 Slot Status 2 Slot Control 2 Note to Table 7 8 1 Registers not applicable to a device are reserved 2 Refer to Table 7 39 on page 7 22 f...

Page 164: ...yte Offset Register Name 31 20 19 16 15 8 7 0 Table 7 10 Altera Defined VSEC Capability Header Bits Register Description Value Access 15 0 PCI Express Extended Capability ID PCIe specification defined value for VSEC Capability ID 0x000B RO 19 16 Version PCIe specification defined value for VSEC version 0x1 RO 31 20 Next Capability Offset Starting address of the next Capability Structure implemente...

Page 165: ...d type ID to specify to CvP the correct sof Variable RO Table 7 15 CvP Status Bits Register Description Reset Value Access 15 10 Reserved 0x00 RO 9 PLD_CORE_READY From FPGA fabric This status bit is provided for debug Variable RO 8 PLD_CLK_IN_USE From clock switch module to fabric This status bit is provided for debug Variable RO 7 CVP_CONFIG_DONE Indicates that the FPGA control block has complete...

Page 166: ... 0x3F corresponds to 1 to 63 clock cycles The upper bits are not used but are included in this field because they belong to the same byte enable 0x00 RW 7 4 Reserved 0x0 RO 2 CVP_FULLCONFIG Request that the FPGA control block reconfigure the entire FPGA including the Arria V GZ Hard IP for PCI Express bring the PCIe link down 1 b0 RW 1 HIP_CLK_SEL Selects between PMA and fabric clock when USER_MOD...

Page 167: ...er 1 b0 RW 0 CVP_CONFIG When asserted instructs that the FPGA control block begin a transfer via CvP 1 b0 RW Table 7 19 Uncorrectable Internal Error Status Register Bits Register Description Access 31 12 Reserved RO 11 When set indicates an RX buffer overflow condition in a posted request or Completion RW1CS 10 Reserved RO 9 When set indicates a parity error was detected on the Configuration Space...

Page 168: ...ption Reset Value Access 31 12 Reserved 1b 0 RO 11 Mask for RX buffer posted and completion overflow error 1b 1 RWS 10 Reserved 1b 0 RO 9 Mask for parity error detected on Configuration Space to TX bus interface 1b 1 RWS 8 Mask for parity error detected on the TX to Configuration Space bus interface 1b 1 RWS 7 Mask for parity error detected at TX Transaction Layer error 1b 1 RWS 6 Reserved 1b 0 RO...

Page 169: ...V GZ Hard IP for PCI Express are routed through the interconnect fabric hardware does not enforce restrictions to limit individual processor access to specific regions However the regions are designed to enable straight forward enforcement by processor software Figure 7 1 illustrates accesses to the Avalon MM control and status registers from the Host CPU and PCI Express link Table 7 22 Correctabl...

Page 170: ...PCI Express processors Avalon MM processors or both 0x2000 0x2FFF Root Port request registers An embedded processor such as the Nios II processor programs these registers to send the data to send Configuration TLPs I O TLPs single dword Memory Reads and Write request and receive interrupts from an Endpoint 0x3000 0x3FFF Registers typically intended for access by Avalon MM processors only These inc...

Page 171: ..._INT5 RW1C 1 when the A2P_MAILBOX5 is written to 20 A2P_MAILBOX_INT4 RW1C 1 when the A2P_MAILBOX4 is written to 19 A2P_MAILBOX_INT3 RW1C 1 when the A2P_MAILBOX3 is written to 18 A2P_MAILBOX_INT2 RW1C 1 when the A2P_MAILBOX2 is written to 17 A2P_MAILBOX_INT1 RW1C 1 when the A2P_MAILBOX1 is written to 16 A2P_MAILBOX_INT0 RW1C 1 when the A2P_MAILBOX0 is written to 15 0 AVL_IRQ_ASSERTED 15 0 RO Curren...

Page 172: ...LON_IRQ_VECTOR RO Stores the interrupt vector of the system interconnect fabric The host software should read this register after being interrupted and determine the servicing priority Table 7 28 PCI Express to Avalon MM Mailbox Registers 0x0800 0x081F Address Name Access Description 0x0800 P2A_MAILBOX0 RW PCI Express to Avalon MM Mailbox 0 0x0804 P2A_MAILBOX1 RW PCI Express to Avalon MM Mailbox 1...

Page 173: ... address map entry 0 0x1004 31 0 A2P_ADDR_MAP_HI0 RW Upper bits of Avalon MM to PCI Express address map entry 0 0x1008 1 0 A2P_ADDR_SPACE1 RW Address space indication for entry 1 Refer to Table 7 31 for the definition of these bits 31 2 A2P_ADDR_MAP_LO1 RW Lower bits of Avalon MM to PCI Express address map entry 1 This entry is only implemented if the number of address translation table entries is...

Page 174: ...Ps and single dword Memory Reads and Write requests The Root Port then drives the TLPs on the TLP Direct Channel to access the Configuration Space I O space or Endpoint memory Figure 7 2 illustrates these registers Figure 7 2 Root Port TLP Data Registers RX_TX_CNTL RP_RXCPL_ REG0 RP_RXCPL_ REG RP_RXCPL_ STATUS Control Register Access Slave Avalon MM Master 32 32 32 32 64 64 32 IRQ RP TX CTRL TX CT...

Page 175: ...at is aligned and unaligned to the qword Table 7 32 Root Port TLP Data Registers 0x2000 0x2FFF Root Port Request Registers Address Range 0x2800 0x2018 Address Bits Name Access Description 0x2000 31 0 RP_TX_REG0 RW Lower 32 bits of the TX TLP 0x2004 31 0 RP_TX_REG1 RW Upper 32 bits of the TX TLP 0x2008 31 2 Reserved 1 RX_TX_CNTRL SOP RW Write 1 b1 to specify the start of a packet 0 RX_TX_CNTRL EOP ...

Page 176: ...Data with 4 DWord Headers Header 1 63 32 Cycle 1 Register 1 Register 0 Register 1 Register 0 Register 1 Register 0 Register 1 Register 0 Register 1 Register 0 Data Unaligned to QWord Boundary Data Aligned to QWord Boundary Cycle 2 Header 0 31 0 Data 63 32 Header 2 31 0 Header 1 63 32 Cycle 1 Header 0 31 0 Cycle 2 Header 2 31 0 Unused but must be written Cycle 3 Data 31 0 Unused but must be written...

Page 177: ...r and subsequently loaded into RP_RXCPL registers The Application Layer performs the following sequence to retrieve the TLP 1 Polls the RP_RXCPL_STATUS SOP to determine when it is set to 1 b1 2 When RP_RXCPL_STATUS SOP 1 b 1 reads RP_RXCPL_REG0 and RP_RXCPL_REG1 to retrieve dword 0 and dword 1 of the Completion TLP 3 Read the RP_RXCPL_STATUS EOP a If RP_RXCPL_STATUS EOP 1 b0 read RP_RXCPL_REG0 and...

Page 178: ...RW1C The Root Port has received INTB from the Endpoint 0 INTA_RECEIVED RW1C The Root Port has received INTA from the Endpoint Table 7 33 Avalon MM Interrupt Status Registers for Root Ports Part 2 of 2 0x3060 Bits Name Access Mode Description Table 7 34 INT X Interrupt Enable Register for Root Ports 0x3070 Bit Name Access Mode Description 31 5 Reserved 4 RPRX_CPL_RECEIVED RW When set to 1 b1 enable...

Page 179: ...lable Table 7 35 PCI Express to Avalon MM Interrupt Status Register for Endpoints 0x3060 Bits Name Access Description 0 ERR_PCI_WRITE_ FAILURE RW1C When set to 1 indicates a PCI Express write failure This bit can also be cleared by writing a 1 to the same bit in the Avalon MM to PCI Express Interrupt Status register 1 ERR_PCI_READ_ FAILURE RW1C When set to 1 indicates the failure of a PCI Express ...

Page 180: ...A08 A2P _MAILBOX2 RW Avalon MM to PCI Express mailbox 2 0x3A0C A2P _MAILBOX3 RW Avalon MM to PCI Express mailbox 3 0x3A10 A2P _MAILBOX4 RW Avalon MM to PCI Express mailbox 4 0x3A14 A2P _MAILBOX5 RW Avalon MM to PCI Express mailbox 5 0x3A18 A2P _MAILBOX6 RW Avalon MM to PCI Express mailbox 6 0x3A1C A2P_MAILBOX7 RW Avalon MM to PCI Express mailbox 7 Table 7 38 PCI Express to Avalon MM Mailbox Regist...

Page 181: ...ved Port Arbitration Table 0x400 0x7FC Reserved PCIe spec corresponding section name 0x800 0x834 Advanced Error Reporting AER optional Advanced Error Reporting Capability 0x838 0xFFF Reserved Table 6 2 PCI Type 0 Configuration Space Header Endpoints Rev3 0 Spec Type 0 Configuration Space Header 0x000 Device ID Vendor ID Type 0 Configuration Space Header 0x004 Status Command Type 0 Configuration Sp...

Page 182: ...Type 1 Configuration Space Header 0x038 Expansion ROM Base Address Type 1 Configuration Space Header 0x03C Bridge Control Interrupt Pin Interrupt Line Bridge Control Register Offset 3Eh Table 6 4 MSI Capability Structure Rev3 0 Spec MSI Capability Structures 0x050 Message Control Next Cap Ptr Capability ID MSI and MSI X Capability Structures 0x054 Message Address MSI and MSI X Capability Structure...

Page 183: ...ble Error Mask Register 0x818 Advanced Error Capabilities and Control Register Advanced Error Capabilities and Control Register 0x81C Header Log Register Header Log Register 0x82C Root Error Command Root Error Command Register 0x830 Root Error Status Root Error Status Register 0x834 Error Source Identification Register Correctable Error Source ID Register Error Source Identification Register Table...

Page 184: ...7 26 Chapter 7 Register Descriptions Correspondence between Configuration Space Registers and the PCIe Specification Arria V GZ Hard IP for PCI Express November 2012 Altera Corporation User Guide ...

Page 185: ... 8 1 summarizes their functionality 1 Contact Altera if you want to switch between the hard and soft reset controller 1 Your Application Layer could instantiate a module similar to altpcie_rs_hip v as shown in Figure 8 1 on page 8 2 to generate app_rstn which resets the Application Layer logic Table 8 1 Use of Hard and Soft Reset Controllers Reset Controller Used Description Hard Reset Controller ...

Page 186: ...Space Non Sticky Registers reset_status pld_clk pin_perst npor refclk srst crst l2_exit hotrst_exit dlup_exit pld_clk_inuse Hard IP for PCI Express fixed_clk 100 or 125 MHz reconfig_xcvr_clk mgmt_rst_reset reconfig_busy Transceiver Reconfiguration Controller reconfig_xcvr_clk reconfig_busy reconfig_xcvr_rst pcie_reconfig_ driver_0 altpcie_ dev _hip_256_pipen1b v altpcie_rs_serdes v coreclkout_hip ...

Page 187: ...d 32 cycles after pld_clk_inuse is asserted 3 The Hard IP for PCI Express deasserts the reset_status output to the Application Layer 4 The altpcied_ device v_hwtcl sv deasserts app_rstn 32 cycles after reset_status is released Figure 8 3 illustrates the RX transceiver reset sequence Figure 8 2 Hard IP for PCI Express and Application Logic Reset Sequence pin_perst pld_clk_inuse serdes_pll_locked cr...

Page 188: ...s the npor_serdes input to the TX transceiver 2 The SERDES reset controller waits for pll_locked to be stable for a minimum of 127 cycles before deasserting tx_digitalreset Clocks The Hard IP contains a clock domain crossing CDC synchronizer at the interface between the PHY MAC and the DLL layers which allows the Data Link and Transaction Layers to run at frequencies independent of the PHY MAC and...

Page 189: ... 125 MHz 300 PPM The transitions between Gen1 Gen2 and Gen3 should be glitchless pclk can be turned off for most of the 1 ms timeout assigned for the PHY to change the clock rate however pclk should be stable before the 1 ms timeout expires Table 8 2 shows the frequency of pclk for Gen1 Gen2 and Gen3 variants Figure 8 5 Clock Domains and Clock Generation for the Application Layer Note to Figure 8 ...

Page 190: ... IP throttles the interface to achieve a lower throughput pld_clk coreclkout_hip can drive the Application Layer clock along with the pld_clk input to the Arria V GZ V Hard IP for PCI Express IP Core The pld_clk can optionally be sourced by a different clock than coreclkout_hip The pld_clk minimum frequency cannot be lower than the coreclkout_hip frequency Based on specific Application Layer const...

Page 191: ...l configuration registers that are read only at run time Use of the reconfiguration interface is optional reconfig_xcvr_clk This is a free running clock with a frequency range of 100 125 MHz This is the clock input to the Transceiver Reconfiguration Controller which performs the transceiver PHY reconfiguration functions required by Gen2 and Gen3 designs For more information refer to Transceiver PH...

Page 192: ...8 8 Chapter 8 Reset and Clocks Clocks Arria V GZ Hard IP for PCI Express November 2012 Altera Corporation User Guide ...

Page 193: ...h triggers the int_status 3 0 signals to the Application Layer int_status 0 Interrupt signal A int_status 1 Interrupt signal B int_status 2 Interrupt signal C int_status 3 Interrupt signal D Assert_INTB Receive Transmit No No No Assert_INTC Receive Transmit No No No Assert_INTD Receive Transmit No No No Deassert_INTA Receive Transmit No Yes No Deassert_INTB Receive Transmit No No No Deassert_INTC ...

Page 194: ...bility structure aer_msi_num input signal When the Implement advanced error reporting option is turned on you can set aer_msi_num to indicate which MSI is being sent to the root complex when an error is logged in the AER Capability structure ERR_NONFATAL Receive Transmit No Yes No ERR_FATAL Receive Transmit No Yes No Locked Transaction Message Unlock Message Transmit Receive Yes No No Slot Power L...

Page 195: ...dor defined Type 0 Message TLPs are passed to the Application Layer The Transaction Layer treats all other received transactions including memory or I O requests that do not match a defined BAR as Unsupported Requests The Transaction Layer sets the appropriate error bits and transmits a completion if needed These Unsupported Requests are not made visible to the Application Layer the header and dat...

Page 196: ... dedicated signals In Root Port mode the Application Layer can issue Type 0 or Type 1 Configuration TLPs on the Avalon ST TX bus The Type 0 Configuration TLPs are only routed to the Configuration Space of the Hard IP and are not sent downstream on the PCI Express link The Type 1 Configuration TLPs are sent downstream on the PCI Express link If the bus number of the Type 1 Configuration TLP matches...

Page 197: ... N N Notes to Table 9 2 1 A Memory Write or Message Request with the Relaxed Ordering Attribute bit clear b 0 must not pass any other Memory Write or Message Request 2 A Memory Write or Message Request with the Relaxed Ordering Attribute bit set b 1 is permitted to pass any other Memory Write or Message Request 3 Endpoints Switches and Root Complex may allow Memory Write and Message Requests to pa...

Page 198: ...9 6 Chapter 9 Transaction Layer Protocol TLP Details Receive Buffer Reordering Arria V GZ Hard IP for PCI Express November 2012 Altera Corporation User Guide ...

Page 199: ... the data to program the I O ring and the Hard IP for PCI Express The core bitstream contains the data to program the FPGA fabric In Arria V GZ devices the I O ring and PCI Express link are programmed first allowing the PCI Express link to reach the L0 state and begin operation independently before the rest of the core is programmed After the PCI Express link is established it can be used to progr...

Page 200: ...X port of the Application Layer When using ECRC forwarding mode the ECRC check and generation are performed in the Application Layer You must turn on Advanced error reporting AER ECRC checking ECRC generation and ECRC forwarding under the PCI Express PCI Capabilities heading using the parameter editor to enable this functionality f For more information about error handling refer to the Error Signa...

Page 201: ...LP includes an ECRC TD is the TL digest bit of the TL packet described in Appendix A Transaction Layer Packet TLP Header Formats Table 10 2 ECRC Operation on RX Path ECRC Forwarding ECRC Check Enable 1 ECRC Status Error TLP Forward to Application Layer No No none No Forwarded good No Forwarded without its ECRC bad No Forwarded without its ECRC Yes none No Forwarded good No Forwarded without its EC...

Page 202: ... assignments for normal configuration Table 10 5 summarizes the lane assignments with lane reversal Yes No TD 0 without ECRC TD 0 without ECRC Core forwards the ECRC TD 1 with ECRC TD 1 with ECRC Yes TD 0 without ECRC TD 0 without ECRC TD 1 with ECRC TD 1 with ECRC Notes to Table 10 3 1 All unspecified cases are unsupported and the behavior of the Hard IP is unknown 2 The ECRC Generation Enable fi...

Page 203: ...ort and a 4 Endpoint on the top side of the PCB Connecting the lanes without lane reversal creates routing problems Using lane reversal solves the problem Figure 10 2 Using Lane Reversal to Solve PCB Routing Problems 0 1 2 3 Root Port 3 2 1 0 Endpoint 0 1 2 3 Root Port 0 1 2 3 Endpoint No Lane Reversal Results in PCB Routing Challenge With Lane Reversal Signals Route Easily lane reversal no lane r...

Page 204: ...10 6 Chapter 10 Optional Features Lane Initialization and Reversal Arria V GZ Hard IP for PCI Express November 2012 Altera Corporation User Guide ...

Page 205: ...k starts in INTX mode after which time software decides whether to switch to MSI mode by programming the msi_enable bit of the MSI message control register bit 16 of 0x050 to 1 or to MSI X mode if you turn on Implement MSI X under the PCI Express PCI Capabilities tab using the parameter editor If you turn on the Implement MSI X option you should implement the MSI X table structures at the memory s...

Page 206: ... block with a per vector enable bit A global Application Layer interrupt enable can also be implemented instead of this per vector MSI Figure 11 1 MSI Handler Block Figure 11 2 Example Implementation of the MSI Handler Block MSI Handler Block app_msi_req app_msi_ack app_msi_tc app_msi_num pex_msi_num app_int_sts cfg_msicsr 15 0 app_int_en0 app_int_sts0 app_msi_req0 app_int_en1 app_int_sts1 app_msi...

Page 207: ...hich only 4 are allocated MSI interrupts generated for Hot Plug Power Management Events and System Errors always use TC0 MSI interrupts generated by the Application Layer can use any Traffic Class For example a DMA that generates an MSI at the end of a transmission can use the same traffic control as was used to transfer data Figure 11 4 illustrates the interactions among MSI interrupt signals for...

Page 208: ... Section 6 8 2 of the PCI Local Bus Specification Revision 3 0 Legacy Interrupts Legacy interrupts are signaled on the PCI Express link using message TLPs that are generated internally by the Arria V GZ Hard IP for PCI Express The app_int_sts input port controls interrupt generation When the input port asserts app_int_sts it causes an Assert_INTA message TLP to be generated and sent upstream Deass...

Page 209: ...structure This mechanism is an alternative to using the serr_out signal The aer_msi_num 4 0 is only used for Root Ports and you must set it to a constant value It cannot toggle during operation If the Root Port detects a Power Management Event the pex_msi_num 4 0 signal is used by Power Management or Hot Plug to determine the offset between the base message interrupt number and the message interru...

Page 210: ...ted interrupts After servicing the interrupt software must clear the appropriate serviced interrupt status bit and ensure that no other interrupts are pending For interrupts caused by Avalon MM to PCI Express Interrupt Status Register 0x0040 on page 7 13 mailbox writes the status bits should be cleared in the Avalon MM to PCI Express Interrupt Status Register 0x0040 on page 7 13 For interrupts due...

Page 211: ...ve port has an Avalon MM Interrupt output signal cra_Irq_irq A write access to an Avalon MM mailbox register sets one of the P2A_MAILBOX_INT n bits in the Avalon MM to PCI Express Interrupt Status Register 0x0040 on page 7 13 and asserts the cra_Irq_o or cra_Irq_irq output if enabled Software can enable the interrupt by writing to the INT X Interrupt Enable Register for Endpoints 0x3070 on page 7 ...

Page 212: ...SI MSI X and INTx buses 1 For more information about implementing MSI or MSI X interrupts refer to the PCI Local Bus Specification Revision 2 3 MSI X ECN Figure 11 8 Block Diagram for Custom Interrupt Handler M S MSI MSI X IRQ S MSI X Table Entries Qsys Interconnects S M PCIe Avalon MM Bridge Hard IP for PCIe PCIe Root Port MSI or MXI X Req IRQ Cntl Status Table PBA RXM Exported MSI MSI X INTX Int...

Page 213: ... the actual bandwidth of the link Figure 12 1 shows the main components of the Flow Control Update loop with two communicating PCI Express ports Write Requester Write Completer As the PCI Express Base Specification 3 0 describe each transmitter the write requester in this case maintains a credit limit register and a credits consumed register The credit limit register is the sum of all credits issu...

Page 214: ...ue by an FC Update DLLP This check is performed separately for the header and data credits a single packet consumes only a single header credit 2 After the packet is selected for transmission the credits consumed register is incremented by the number of credits consumed by this packet This increment happens for both the header and data credit consumed registers 3 The packet is received at the othe...

Page 215: ...to be the next item is transmitted In the worst case the FC Update DLLP may need to wait for a maximum sized TLP that is currently being transmitted to complete before it can be sent 7 The FC Update DLLP is received back at the original write requester and the credit limit value is updated If packets are stalled waiting for credits they can now be transmitted To allow the write requester to transm...

Page 216: ...small compared with the inaccuracy in the estimate of the external read to completion delays With multiple completions the number of available credits for completion headers must be larger than the completion data space divided by the maximum packet size Instead the credit space for headers must be the completion data space in bytes divided by 64 because this is the smallest possible read completi...

Page 217: ...includes the following sections Physical Layer Errors Data Link Layer Errors Transaction Layer Errors Error Reporting and Data Poisoning Uncorrectable and Correctable Error Status Bits Table 13 1 Error Classification Type Responsible Agent Description Correctable Hardware While correctable errors may affect system performance data integrity is maintained Uncorrectable non fatal Device software Unc...

Page 218: ...ror 101 Elastic Buffer Overflow 110 Elastic Buffer Underflow 111 Disparity Error Deskew error caused by overflow of the multilane deskew FIFO Control symbol received in wrong lane Note to Table 13 2 1 Considered optional by the PCI Express specification Table 13 3 Errors Detected by the Data Link Layer Error Type Description Bad TLP Correctable This error occurs when a LCRC verification fails or w...

Page 219: ...n Layer Unsupported Request for Endpoints Uncorrectable non fatal This error occurs whenever a component receives any of the following Unsupported Requests Type 0 Configuration Requests for a non existing function Completion transaction for which the Requester ID does not match the bus device and function number Unsupported message A Type 1 Configuration Request TLP for the TLP from the PCIe link ...

Page 220: ...t to Configuration Space In all of the above cases the TLP is not presented to the Application Layer the Hard IP block deletes it The Application Layer can detect and report other unexpected completion conditions using the cpl_err 2 signal For example the Application Layer can report cases where the total length of the received successful completions do not match the original read request length R...

Page 221: ...itions that cause parity errors Poisoned packets received by the Hard IP block are passed to the Application Layer Poisoned transmit TLPs are similarly sent to the link Malformed TLP continued Uncorrectable fatal A request specifies an address length combination that causes a memory space access to exceed a 4 KByte boundary The Hard IP block checks for this violation which is considered optional b...

Page 222: ...ition it represents has been detected Software may clear the error status by writing a 1 to the appropriate bit 0 Figure 13 1 Uncorrectable Error Status Register Rsvd Rsvd Rsvd TLP Prefix Blocked Error Status AtomicOp Egress Blocked Status MC Blocked TLP Status Uncorrectable Internal Error Status ACS Violation Status Unsupported Request Error Status ECRC Error Status Malformed TLP Status Receiver ...

Page 223: ...ncertainty once across all of the SDC files in your project Differences between Fitter timing analysis and TimeQuest timing analysis arise if these constraints are applied more than once Example 14 1 SDC Timing Constraints Required for the Arria V GZ Hard IP for PCIe and Design Example Constraints required for the Hard IP for PCI Express derive_pll_clock is used to calculate all clock derived from...

Page 224: ...figuration Controller IP Core is the example design The sdc file includes constraints for the Transceiver Reconfiguration Controller IP Core You may need to change the frequency and actual clock pin name to match your design The sdc file also specifies some false timing paths for Transceiver Reconfiguration Controller and Transceiver PHY Reset Controller IP Cores Be sure to include these constrain...

Page 225: ...e registers includes the following three steps 1 Bring down the PCI Express link by asserting the hip_reconfig_rst_n reset signal if the link is already up Reconfiguration can occur before the link has been established 2 Reprogram configuration registers using the Avalon MM slave Hard IP reconfiguration interface 3 Release the npor reset signal 1 You can use the LMI interface to change the values ...

Page 226: ...96 bytes max payload size 110 Reserved 111 Reserved 3 Surprise Down error reporting capabilities b 0 Table 7 8 on page 7 5 Link Capability register Available in PCI Express Base Specification Revision 1 1 compliant Cores only Downstream Port This bit must be set to 1 if the component supports the optional capability of detecting and reporting a Surprise Down error condition 0x91 Upstream Port For ...

Page 227: ...imum of 4 µs b 011 Maximum of 8 µs b 100 Maximum of 16 µs b 101 Maximum of 32 µs b 110 Maximum of 64 µs b 111 No limit b 000 Table 7 8 on page 7 5 Device Capability register 14 12 These bits record the presence or absence of the attention and power indicators b 000 Table 7 8 on page 7 5 Slot Capability register 0 Attention button present on the device 1 Attention indicator present for an endpoint ...

Page 228: ...ister 1 Power controller present 2 Manually Operated Retention Latch MRL sensor present 3 Attention indicator present for a root port switch or bridge 4 Power indicator present for a root port switch or bridge 5 Hot plug surprise When this bit set to1 a device can be removed from this slot without prior notification 6 0 6 Hot plug capable 9 7 Reserved b 000 15 10 Slot Power Limit Value b 00000000 ...

Page 229: ...C check b 0 Table 7 7 on page 7 4 Advanced Error Capability and Control register 10 No command completed support available only in PCI Express Base Specification Revision 1 1 compliant Cores b 0 Table 7 8 on page 7 5 Slot Capability register 13 11 Number of functions MSI capable b 010 Table 7 4 on page 7 3 Message Control register b 000 1 MSI capable b 001 2 MSI capable b 010 4 MSI capable b 011 8...

Page 230: ... 0000 Core is compliant to PCIe Specification 1 0a or 1 1 b 0001 Core is compliant to PCIe Specification 1 0a or 1 1 b 0010 Core is compliant to PCIe Specification 2 0 15 13 L0s exit latency for common clock Gen1 N_FTS of separate clock 1 for the SKIPOS 4 10 UI UI 0 4 ns Gen2 N_FTS2 of separate clock 1 for the SKIPOS 4 8 max number of received EIE 10 UI UI 0 2 ns b 110 Table 7 8 on page 7 5 Link C...

Page 231: ...A4 15 0 BAR1 63 48 b 0 0xA5 BAR2 95 64 b 0 Table 7 2 on page 7 2 0 BAR2 64 I O Space b 0 2 1 BAR2 66 65 Memory Space see bit settings for BAR0 b 0 3 BAR2 67 Prefetchable b 0 BAR2 95 68 Bar size mask b 0 15 4 BAR2 79 68 b 0 0xA6 15 0 BAR2 95 80 b 0 BAR3 127 96 b 0 Table 7 2 on page 7 2 0 BAR3 96 I O Space b 0 2 1 BAR3 98 97 Memory Space see bit settings for BAR0 b 0 3 BAR3 99 Prefetchable b 0 BAR3 ...

Page 232: ...sion BAR 207 192 b 0 0xAE 15 0 Expansion BAR 223 208 b 0 0xAF 1 0 IO b 0 Table 7 3 on page 7 2 00 no IO windows 01 IO 16 bit 11 IO 32 bit 3 2 Prefetchable b 0 00 not implemented 01 prefetchable 32 11 prefetchable 64 15 4 Reserved B0 5 0 Reserved 6 Selectable de emphasis operates as specified in the PCI Express Base Specification when operating at the 5 0GT s rate 1 3 5 dB 0 6 dB This setting has n...

Page 233: ...ace Protocols Transceiver PHY category When you instantiate your Transceiver Reconfiguration Controller IP core the Enable offset cancellation block option is On by default For Gen3 variants you should also turn on adaptive equalization Figure 15 1 shows the connections between the Transceiver Reconfiguration Controller instance and the PHY IP Core for PCI Express instance for a 4 variant Table 15...

Page 234: ...gure 15 2 illustrates the messages reported for a Gen2 4 variant The variant requires five interfaces one for each lane and one for the TX PLL When you instantiate the Transceiver Reconfiguration Controller you must specify the required Number of reconfiguration interfaces as Figure 15 3 illustrates The Transceiver Reconfiguration Controller includes an Optional interface grouping parameter Arria ...

Page 235: ...bout Transceiver PHY Reconfiguration f For more information about using the Transceiver Reconfiguration Controller refer to the Transceiver Reconfiguration Controller chapter in the Altera Transceiver PHY IP Core User Guide and to Application Note 645 Dynamic Reconfiguration of PMA Controls in Stratix V Devices Although this application note describes dynamic reconfiguration for Stratix V devices ...

Page 236: ...15 12 Chapter 15 Hard IP Reconfiguration and Transceiver Reconfiguration Transceiver PHY IP Reconfiguration Arria V GZ Hard IP for PCI Express November 2012 Altera Corporation User Guide ...

Page 237: ...tions A configuration routine that sets up all the basic configuration registers in the Root Port and the Endpoint BFM This configuration allows the Endpoint application to be the target and initiator of PCI Express transactions A Verilog HDL procedure interface to initiate PCI Express transactions to the Endpoint BFM The testbench uses a test driver module altpcietb_bfm_driver_rp to exercise the ...

Page 238: ...urned in multiple completions It always returns a single completion for every read request Some systems split completions on every 64 byte address boundary It always returns completions in the same order the read requests were issued Some systems generate the completions out of order It is unable to generate zero length read requests that some systems generate as flush requests following some writ...

Page 239: ...ules interconnect the PIPE MAC layer interfaces of the Root Port and the Endpoint The module mimics the behavior of the PIPE PHY layer to both MAC interfaces altpcietb_bfm_driver_chaining This module drives transactions to the Root Port BFM This is the module that you modify to vary the transactions sent to the example Endpoint design or your own design For more information about this module refer...

Page 240: ... mode described in the section Chaining DMA Design Examples on page 16 4 altpcietb_pipe_phy There are eight instances of this module one per lane These modules connect the PIPE MAC layer interfaces of the Root Port and the Endpoint The module mimics the behavior of the PIPE PHY layer to both MAC interfaces altpcietb_bfm_driver_rp This module drives transactions to the Root Port BFM This is the mod...

Page 241: ...es requested parameter under the PCI Express PCI Capabilities page to at least 2 The chaining DMA design example uses an architecture capable of transferring a large amount of fragmented memory without accessing the DMA registers for every memory block For each block of memory to be transferred the chaining DMA design example uses a descriptor table containing the following information Length of t...

Page 242: ... interrupts from the Hard IP block The sideband signal bus carries static information such as configuration information The descriptor tables of the DMA read and the DMA write are located in the BFM shared memory A RC CPU and associated PCI Express PHY link to the Endpoint design example using a Root Port and a north south bridge The example Endpoint design Application Layer accomplishes the follo...

Page 243: ...le is qsys_systemname synthesis submodules This module instantiates the top level module and propagates only a small sub set of the test ports to the external I Os These test ports can be used in your design variation name v or variation name vhd Because Altera provides five sample parameterizations you may have to edit one of the provided examples to create a simulation that matches your requirem...

Page 244: ...cdma_ast_msi This module converts MSI requests from the chaining DMA submodules into Avalon ST streaming data alpcierd_cdma_app_icm This module arbitrates PCI Express packets for the modules altpcierd_dma_dt read or write and altpcierd_rc_slave alpcierd_cdma_app_icm instantiates the Endpoint memory used for the DMA read and write transfer altpcierd_compliance_test v This module provides the logic ...

Page 245: ...xpress transaction layer packets altpcierd_write_dma_requester is used with the 64 bit Avalon ST IP core altpcierd_write_dma_requester_128 is used with the 128 bit Avalon ST IP core ls altpcierd_cpld_rx_buffer This modules monitors the available space of the RX Buffer It prevents RX Buffer overflow by arbitrating memory read request issued by the application altpcierd_cplerr_lmi This module transf...

Page 246: ...C DMA Wr Cntl DW3 Reserved RCLAST Idx of last descriptor to process 0x10 DMA Rd Cntl DW0 Control Field refer to Table 16 3 Number of descriptors in descriptor table 0x14 DMA Rd Cntl DW1 Base Address of the Read Descriptor Table BDT in the RC Memory Upper DWORD 0x18 DMA Rd Cntl DW2 Base Address of the Read Descriptor Table BDT in the RC Memory Lower DWORD 0x1C DMA Rd Cntl DW3 Reserved RCLAST Idx of...

Page 247: ...Hi For field definitions refer to Table 16 5 0x24 DMA Wr Status Lo Target Mem Address Width Write DMA Performance Counter Clock cycles from time DMA header programmed until last descriptor completes including time to fetch descriptors 0x28 DMA Rd Status Hi For field definitions refer to Table 16 6 0x2C DMA Rd Status Lo Max No of Tags Read DMA Performance Counter The number of clocks from the time ...

Page 248: ...r dword descriptors to start a DMA To send update status to the RP for example to record the number of descriptors completed to the descriptor header Each subsequent descriptor consists of a minimum of four dwords of data and corresponds to one DMA transfer A dword equals 32 bits Table 16 6 Fields in the DMA Read Status High Register Bit Field Description 31 24 Reserved 23 21 Max Read Request Size...

Page 249: ...PLAST_ENA bit in the control register or descriptor this location records the number of the last descriptor completed by the chaining DMA module 0x10 Descriptor 0 Control fields DMA length 0x14 Endpoint address 0x18 RC address upper dword 0x1C RC address lower dword 0x20 Descriptor 1 Control fields DMA length 0x24 Endpoint address 0x28 RC address upper dword 0x2C RC address lower dword 0x 0 Descri...

Page 250: ...dpoint design Control Register space Either BARs 2 or 3 must be at least a 256 byte memory BAR to perform the DMA channel test The find_mem_bar procedure in the altpcietb_bfm_driver_chaining does this Table 16 10 Chaining DMA Descriptor Fields Descriptor Field Endpoint Access RC Access Description Endpoint Address R R W A 32 bit field that specifies the base address of the memory transfer on the E...

Page 251: ...ate transfer completion c The chaining DMA writes the EPLast bit of the Chaining DMA Descriptor Table on page 16 13 after completing the data transfer for the first and last descriptors d The chaining DMA issues an MSI when the last descriptor has completed e The data written back to BFM is checked against the data that was read from the BFM f The driver programs the chaining DMA to perform a test...

Page 252: ...escribed in on page 16 14 DW1 0x824 0 Endpoint address DW2 0x828 0 BFM shared memory data buffer 1 upper address value DW3 0x82c 0x2800 BFM shared memory data buffer 1 lower address value Data Buffer 1 0x02800 Increment by 1 from 0x2525_0001 Data content in the BFM shared memory from address 0x02800 Table 16 13 Write Descriptor 2 Offset in BFM Shared Memory Value Description DW0 0x830 644 Transfer...

Page 253: ... by 1 from 0xAAA0_0001 Data content in the BFM shared memory from address 0x89F0 Table 16 16 Read Descriptor 1 Offset in BFM Shared Memory Value Description DW0 0x920 1 024 Transfer length in dwords and control bits as described in on page 16 14 DW1 0x924 0 Endpoint address value DW2 0x928 10 BFM shared memory data buffer 1 upper address value DW3 0x92c 0x10900 BFM shared memory data buffer 1 lowe...

Page 254: ... read transfers have completed Root Port Design Example The design example includes the following primary components Root Port variation qsys_systemname Avalon ST Interfaces altpcietb_bfm_vc_intf_ast handles the transfer of TLP requests and completions to and from the Arria V GZ Hard IP for PCI Express variation using the Avalon ST interface Root Port BFM tasks contains the high level tasks called...

Page 255: ...ce altpcietb_bfm_ep_example_chaining_pipen1b v the top level of the Root Port design example that you use for simulation This module instantiates the Root Port variation variation_name v and the Root Port application altpcietb_bfm_vc_intf_ application_width This module provides both PIPE and serial interfaces for the simulation environment This module has two debug ports named test_out_icm which i...

Page 256: ...dule Files in subdirectory qsys_systemname testbench simulation submodules altpcietb_bfm_ep_example_chaining_pipen1b v the simulation model for the chaining DMA Endpoint altpcietb_bfm_driver_rp v this file contains the functions to implement the shared memory space PCI Express reads and writes initialize the Configuration Space registers log and display simulation messages and define global consta...

Page 257: ...iver_rp v These functions provide the BFM calls to request configuration of the PCI Express link and the Endpoint Configuration Space registers For details on these procedures and functions see BFM Configuration Procedures on page 16 34 BFM Log Interface altpcietb_bfm_driver_rp v The BFM log functions provides routines for writing commonly formatted messages to the simulator standard output and op...

Page 258: ...re the Root Port and Endpoint Configuration Space registers To configure these registers call the procedure ebfm_cfg_rp_ep which is included in altpcietb_bfm_driver_rp v The ebfm_cfg_rp_ep executes the following steps to initialize the Configuration Space 1 Sets the Root Port Configuration Space to enable the Root Port to send transactions on the PCI Express link 2 Sets the Root Port and Endpoint ...

Page 259: ...is set to 0 then the 64 bit prefetchable memory BARs are assigned smallest to largest starting at the 4 GByte address assigning memory ascending above the 4 GByte limit throughout the full 64 bit memory space Refer to Figure 16 6 on page 16 26 If the addr_map_4GB_limit input to the ebfm_cfg_rp_ep is set to 1 then the 32 bit and the 64 bit prefetchable memory BARs are assigned largest to smallest s...

Page 260: ...6 19 BAR Table Structure Offset Bytes Description 0 PCI Express address in BAR0 4 PCI Express address in BAR1 8 PCI Express address in BAR2 12 PCI Express address in BAR3 16 PCI Express address in BAR4 20 PCI Express address in BAR5 24 PCI Express address in Expansion ROM BAR 28 Reserved 32 BAR0 read back value after being written with all 1 s used to compute size 36 BAR1 read back value after bei...

Page 261: ...e figures The memory space layout is dependent on the value of the addr_map_4GB_limit input parameter If addr_map_4GB_limit is 1 the resulting memory space map is shown in Figure 16 5 Figure 16 5 Memory Space Layout 4 GByte Limit Root Complex Shared Memory 0x0000 0000 Configuration Scratch Space Used by BFM routines not writable by user calls or endpoint 0x001F FF80 BAR Table Used by BFM routines ...

Page 262: ...pace Used by BFM routines not writable by user calls or endpoint 0x001F FF80 BAR Table Used by BFM routines not writable by user calls or endpoint 0x001F FFC0 Endpoint Non Prefetchable Memory Space BARs Assigned Smallest to Largest 0x0000 0001 0000 0000 Endpoint Memory Space BARs Prefetchable 32 bit Assigned Smallest to Largest Unused BAR size dependent BAR size dependent Endpoint Memory Space BAR...

Page 263: ... the request has been passed to the VC interface module for transmission ebfm_barwr_imm writes a maximum of four bytes of immediate data passed in a procedure call to an offset from a specific Endpoint BAR This procedure returns as soon as the request has been passed to the VC interface module for transmission ebfm_barrd_wait reads data from an offset of a specific Endpoint BAR and stores it in BF...

Page 264: ...ures that are specific to the chaining DMA design example BFM Read and Write Procedures This section describes the procedures used to read and write data among BFM shared memory Endpoint BARs and specified configuration registers The following procedures and functions are available in the Verilog HDL include file altpcietb_bfm_driver v These procedures and functions support issuing memory and conf...

Page 265: ...ble bar_num pcie_offset imm_data byte_len tclass Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory The bar_table structure stores the address assigned to each BAR so that the driver code does not need to be aware of the actual assigned addresses only the application specific offsets from the BAR bar_num Number of the BAR used with pcie_offset to determine PCI Exp...

Page 266: ...pcie_offset lcladdr byte_len tclass Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory The bar_table structure stores the address assigned to each BAR so that the driver code does not need to be aware of the actual assigned addresses only the application specific offsets from the BAR bar_num Number of the BAR used with pcie_offset to determine PCI Express address ...

Page 267: ...target device dev_num PCI Express device number of the target device fnc_num Function number in the target device to be accessed regb_ad Byte specific address of the register to be written regb_ln Length in bytes of the data written Maximum length is four bytes The regb_ln and the regb_ad arguments cannot cross a DWORD boundary imm_data Data to be written This argument is reg 31 0 The bits written...

Page 268: ...le 16 25 ebfm_cfgwr_imm_nowt Procedure Location altpcietb_bfm_driver_rp v Syntax ebfm_cfgwr_imm_nowt bus_num dev_num fnc_num imm_regb_adr regb_len imm_data Arguments bus_num PCI Express bus number of the target device dev_num PCI Express device number of the target device fnc_num Function number in the target device to be accessed regb_ad Byte specific address of the register to be written regb_ln...

Page 269: ...xpress device number of the target device fnc_num Function number in the target device to be accessed regb_ad Byte specific address of the register to be written regb_ln Length in bytes of the data read Maximum length is four bytes The regb_ln and the regb_ad arguments cannot cross a DWORD boundary lcladdr BFM shared memory address of where the read data should be placed compl_status Completion st...

Page 270: ... passed to all subsequent read and write procedure calls that access an offset from a particular BAR ep_bus_num PCI Express bus number of the target device This number can be any value greater than 0 The Root Port uses this as its secondary bus number ep_dev_num PCI Express device number of the target device This number can be any value The Endpoint is automatically assigned this value when it rec...

Page 271: ...emory bar_num BAR number to analyze log2_size This argument is set by the procedure to the log base 2 of the size of the BAR If the BAR is not enabled this argument will be set to 0 is_mem The procedure sets this argument to indicate if the BAR is a memory space BAR 1 or I O Space BAR 0 is_pref The procedure sets this argument to indicate if the BAR is a prefetchable BAR 1 or non prefetchable BAR ...

Page 272: ...rting address for reading data leng Length in bytes of data read Return data Data read from BFM shared memory This parameter is implemented as a 64 bit vector leng is 1 8 bytes If leng is less than 8 bytes only the corresponding least significant bits of the returned data are valid Bits 7 downto 0 are read from the location specified by addr bits 15 downto 8 are read from the addr 1 location etc T...

Page 273: ... shared memory starting address for filling data mode Data pattern used for filling the data Should be one of the constants defined in section Shared Memory Constants on page 16 35 leng Length in bytes of data to fill If the length is not a multiple of the incrementing data pattern width then the last data pattern is truncated to fit init Initial data value used for incrementing data pattern modes...

Page 274: ...ssages such as configuration register values starting and ending of tests 1 Yes No INFO EBFM_MSG_WARNING Specifies warning messages such as tests being skipped due to the specific configuration 2 Yes No WARNING EBFM_MSG_ERROR_INFO Specifies additional information for an error Use this message to display preliminary information before an error message that stops simulation 3 Yes No ERROR EBFM_MSG_E...

Page 275: ...rols which message types are suppressed Table 16 37 ebfm_display Procedure Location altpcietb_bfm_driver_rp v Syntax Verilog HDL dummy_return ebfm_display msg_type message Argument msg_type Message type for the message Should be one of the constants defined in Table 16 36 on page 16 38 message The message string is limited to a maximum of 100 characters Also because Verilog HDL does not allow vari...

Page 276: ...n Verilog HDL Formatting Functions The following procedures and functions are available in the altpcietb_bfm_driver_rp v This section outlines formatting functions that are only used by Verilog HDL All these functions take one argument of a specified length and return a vector of a specified length Table 16 40 ebfm_log_set_stop_on_msg_mask Location altpcietb_bfm_driver_rp v Syntax ebfm_log_set_sto...

Page 277: ...ger message string and passed to ebfm_display Table 16 43 himage1 Location altpcietb_bfm_driver_rp v syntax string himage vec Argument vec Input data type reg with a range of 3 0 Return range string Returns a 1 digit hexadecimal representation of the input argument Return data is type reg with a range of 8 1 Table 16 44 himage2 Location altpcietb_bfm_driver_rp v syntax string himage vec Argument r...

Page 278: ...th a range of 64 1 Table 16 46 himage8 Table 16 47 himage16 Location altpcietb_bfm_driver_rp v syntax string himage vec Argument range vec Input data type reg with a range of 63 0 Return range string Returns a 16 digit hexadecimal representation of the input argument padded with leading 0s if they are needed Return data is type reg with a range of 128 1 Table 16 48 dimage1 Location altpcietb_bfm_d...

Page 279: ...ring dimage vec Argument range vec Input data type reg with a range of 31 0 Return range string Returns a 3 digit decimal representation of the input argument that is padded with leading 0s if necessary Return data is type reg with a range of 24 1 Returns the letter U if the value cannot be represented Table 16 51 dimage4 Location altpcietb_bfm_driver_rp v syntax string dimage vec Argument range v...

Page 280: ...gument that is padded with leading 0s if necessary Return data is type reg with a range of 48 1 Returns the letter U if the value cannot be represented Table 16 53 dimage6 Table 16 54 dimage7 Location altpcietb_bfm_log v syntax string dimage vec Argument range vec Input data type reg with a range of 31 0 Return range string Returns a 7 digit decimal representation of the input argument that is pad...

Page 281: ...red memory bar_num BAR number to analyze Use_msi When set the Root Port uses native PCI express MSI to detect the DMA completion Use_eplast When set the Root Port uses BFM shared memory polling to detect the DMA completion Table 16 57 dma_wr_test Procedure Location altpcietb_bfm_driver_rp v Syntax dma_wr_test bar_table bar_num use_msi use_eplast Arguments bar_table Address of the Endpoint bar_tabl...

Page 282: ...ber to analyze Descriptor_size Number of descriptor direction When 0 the direction is read When 1 the direction is write Use_msi When set the Root Port uses native PCI Express MSI to detect the DMA completion Use_eplast When set the Root Port uses BFM shared memory polling to detect the DMA completion Bdt_msb BFM shared memory upper address value Bdt_lsb BFM shared memory lower address value Msi_n...

Page 283: ... set poll for MSI from the DMA write module Dma_read When set poll for MSI from the DMA read module Table 16 63 dma_set_msi Procedure Location altpcietb_bfm_driver_rp v Syntax dma_set_msi bar_table bar_num bus_num dev_num fun_num direction msi_address msi_data msi_number msi_traffic_class multi_message_enable msi_expected Arguments bar_table Address of the Endpoint bar_table structure in BFM share...

Page 284: ...r_table structure in BFM shared memory allowed_bars One hot 6 bits BAR selection min_log2_size Number of bit required for the specified address space sel_bar BAR number to use Table 16 65 dma_set_rclast Procedure Location altpcietb_bfm_driver_rp v Syntax Dma_set_rclast bar_table setup_bar dt_direction dt_rclast Arguments bar_table Address of the Endpoint bar_table structure in BFM shared memory se...

Page 285: ...ng bring up issues as illustrated in Figure 17 1 Link Training The Physical Layer automatically performs link training and initialization without software intervention This is a well defined process to configure and initialize the device s Physical Layer and link so that PCIe packets can be transmitted If you encounter link training issues viewing the actual data in hardware should help you determ...

Page 286: ...17 2 for possible causes Link training fails due to an incorrect reset sequence Refer to Recommended Reset Sequence to Avoid Link Training Issues for the recommended reset sequence Table 17 1 Link Training Fails to Reach L0 Part 1 of 2 Possible Causes Symptoms and Root Causes Workarounds and Solutions Link fails the Receiver Detect sequence LTSSM toggles between Detect Quiet 0 and Detect Active 1 ...

Page 287: ...rx_signaldetect bus of the active lanes is all 1 s If all active lanes are driving all 1 s the LTSSM state machine toggles between Detect Quiet 0 Detect Active 1 and Polling Active 2 states You can debug this issue using SignalTap II Refer to PIPE Interface Signals on page 17 8 for a list of the test_out bus signals This issue may be caused by mismatches between the expected power supply to RX sid...

Page 288: ...he last TLP transmitted by End Point is greater than the InitFC credit advertised by the link partner For simulation refer to the log file and simulation dump For hardware use a third party logic analyzer trace to capture PCIe transactions If the payload is greater than the initFC credit advertised you must either increase the InitFC of the posted request to be greater than the max payload size or...

Page 289: ...d indicating the presence of a TLP digest ECRC but the ECRC dword is not present at the end of TLP The payload crosses a 4KByte boundary Revise the Application Layer logic to correct the error condition Insufficient Posted credits released by Root Port If a Memory Write TLP is transmitted with a payload greater than the maximum payload size the Root Port may release an incorrect posted data credit...

Page 290: ...easserted 4 Deassert pin_perst to take the Hard IP for PCIe out of reset For plug in cards the minimum assertion time for pin_perst is 100 ms Embedded systems do not have a minimum assertion time for pin_perst 5 Wait for the reset_status output to be deasserted 6 Deassert the reset output to the Application Layer Setting Up Simulation Changing the simulation parameters reduces simulation time and ...

Page 291: ...he link by running the inverse polynomial Complete the following steps to disable the scrambler 1 Open work_dir variant testbench variant _tb simulation submodules altpcie_tbed_sv_hwtcl v 2 Search for the string test_in 3 To disable the scrambler set test_in 2 1 4 Save altpcie_tbed_sv_hwtcl v Change between the Hard and Soft Reset Controller The Hard IP for PCI Express includes both hard and soft ...

Page 292: ...ou can track the ordered sets in the link initialization and training on both sides of the link to help you diagnose link issues You can use SignalTap logic analyzer to determine the behavior Table 17 3 lists the PIPE interface signals for a two lane simulation that you can monitor on the test_out bus Table 17 3 PIPE Interface Signals Part 1 of 3 Signal Name Lane 0 Lane 1 Description reserved 57 0...

Page 293: ...l PHY requests rxvalid0 86 246 Indicates symbol lock and valid data on rxdata0 31 0 and rxdatak0 3 0 rxblkst0 85 245 For Gen3 operation indicates the start of a block rxsynchd0 1 0 84 83 244 243 For Gen3 operation specifies the block type The following encodings are defined 2 b01 Ordered Set Block 2 b10 Data Block rxdataskip0 82 242 For Gen3 operation Allows the PCS to instruct the RX interface to...

Page 294: ...dy when the OS BIOS begins enumeration of the device tree If the FPGA is not fully programmed when the OS BIOS begins its enumeration the OS does not include the Hard IP for PCI Express in its device map To eliminate this issue you can do a soft reset of the system to retain the FPGA programming while forcing the OS BIOS to repeat its enumeration txcompl0 42 202 This signal forces the running disp...

Page 295: ...0 0 TD EP Attr AT Length Byte 4 Requester ID Tag Last BE First BE Byte 8 Address 31 2 0 0 Byte12 Reserved Table A 2 Memory Read Request Locked 32 Bit Addressing 0 1 2 3 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 Byte 0 0 0 0 0 0 0 0 1 0 TC 0 0 0 0 TD EP Attr AT Length Byte 4 Requester ID Tag Last BE First BE Byte 8 Address 31 2 0 0 Byte 12 Reserved Table A 3 Memory Read Reques...

Page 296: ...1 0 7 6 5 4 3 2 1 0 Byte 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 TD EP 0 0 AT 0 0 0 0 0 0 0 0 0 1 Byte 4 Requester ID Tag 0 0 0 0 First BE Byte 8 Address 31 2 0 0 Byte 12 Reserved Table A 7 Message without Data 0 1 2 3 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 Byte 0 0 0 1 1 0 r 2 r 1 r 0 0 TC 0 0 0 0 TD EP 0 0 AT 0 0 0 0 0 0 0 0 0 0 Byte 4 Requester ID Tag Message Code Byte 8 Vend...

Page 297: ...irst BE Byte 8 Address 31 2 0 0 Byte 12 Reserved Table A 11 Memory Write Request 64 Bit Addressing 0 1 2 3 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 Byte 0 0 1 1 0 0 0 0 0 0 TC 0 0 0 0 TD EP Attr AT Length Byte 4 Requester ID Tag Last BE First BE Byte 8 Address 63 32 Byte 12 Address 31 2 0 0 Table A 12 Configuration Write Request Root Port Type 1 0 1 2 3 7 6 5 4 3 2 1 0 7 6 5...

Page 298: ...er Address Byte 12 Reserved Table A 15 Completion Locked with Data 0 1 2 3 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 Byte 0 0 1 0 0 1 0 1 1 0 TC 0 0 0 0 TD EP Attr AT Length Byte 4 Completer ID Status B Byte Count Byte 8 Requester ID Tag 0 Lower Address Byte 12 Reserved Table A 16 Message with Data 0 1 2 3 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 Byte 0...

Page 299: ...oduct literature Website www altera com literature Nontechnical support general Email nacomp altera com software licensing Email authorization altera com Note to Table 1 You can also contact your local Altera sales office or sales representative Visual Cue Meaning Bold Type with Initial Capital Letters Indicate command names dialog box titles dialog box options and other GUI labels For example Sav...

Page 300: ...ss the Enter key 1 2 3 and a b c and so on Numbered steps indicate a list of items when the sequence of the items is important such as the steps listed in a procedure Bullets indicate a list of items when the sequence of the items is not important 1 The hand points to information that requires special attention h The question mark directs you to a software help system with related information f Th...

Reviews:

No comments

Related manuals for Arria V GZ

Brand: UGREEN Pages: 24

Brand: Sunpak Pages: 8

Brand: Innes Pages: 21

Brand: Magtek Pages: 70

Brand: StarTech.com Pages: 2

Brand: Amulet Hotkey Pages: 36

Brand: Edimax Pages: 2

Brand: Access Pages: 23

Brand: Acces I/O products Pages: 31

3-in-1 Card Reader/Writer TS0MFLRD

Brand: Transcend Pages: 17

Brand: IDTECH Pages: 15

X850 - Radeon Xt Platinum Edition 256 Mb Agp

Brand: ATI Technologies Pages: 90

Brand: StarTech.com Pages: 2

Intense3D Wildcat 4105

Brand: Intergraph Pages: 22

Brand: Delkin Devices Pages: 11

Brand: Allflex Pages: 44

Brand: EMS Imaging Pages: 16

Brand: Diablotek Pages: 1

Brands by name

0 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Popular brands

Load more brands