- The Previous Chapter
- This Chapter
- The Next Chapter
- Introduction
- Two Types of Local Link Traffic
- Transaction Layer Packet Routing Basics
- Applying Routing Mechanisms
- Plug-And-Play Configuration of Routing Options
This chapter is from the book
This chapter is from the book
This chapter is from the book
Transaction Layer Packet Routing Basics
The third class of link traffic originates in the Transaction Layer of one device and targets the Transaction Layer of another device. These Transaction Layer Packets (TLPs) are forwarded from one link to another as necessary, subject to the routing mechanisms and rules described in the following sections. Note that other chapters in this book describe additional aspects of Transaction Layer Packet handling, including Flow Control, Quality Of Service, Error Handling, Ordering rules, etc. The term transaction is used here to describe the exchange of information using Transaction Layer Packets. Because Ordered Sets and DLLPs carry no routing information and are not forwarded, the routing rules described in the following sections apply only to TLPs.
TLPs Used to Access Four Address Spaces
As transactions are carried out between PCI Express requesters and completers, four separate address spaces are used: Memory, IO, Configuration, and Message. The basic use of each address space is described in Table 3-3 on page 113.
Table 3-3. PCI Express Address Space And Transaction Types
|
Address Space |
Transaction Types |
Purpose |
|---|---|---|
|
Memory |
Read, Write |
Transfer data to or from a location in the system memory map |
|
IO |
Read, Write |
Transfer data to or from a location in the system IO map |
|
Configuration |
Read, Write |
Transfer data to or from a location in the configuration space of a PCI-compatible device. |
|
Message |
Baseline, Vendor-specific |
General in-band messaging and event reporting (without consuming memory or IO address resources) |
Split Transaction Protocol Is Used
Accesses to the four address spaces in PCI Express are accomplished using split-transaction requests and completions.
Split Transactions: Better Performance, More Overhead
The split transaction protocol is an improvement over earlier bus protocols (e.g. PCI) which made extensive use of bus wait-states or delayed transactions (retries) to deal with latencies in accessing targets. In PCI Express, the completion following a request is initiated by the completer only when it has data and/or status ready for delivery. The fact that the completion is separated in time from the request which caused it also means that two separate TLPs are generated, with independent routing for the request TLP and the Completion TLP. Note that while a link is free for other activity in the time between a request and its subsequent completion, a split-transaction protocol involves some additional overhead as two complete TLPs must be generated to carry out a single transaction.
Figure 3-4 on page 115 illustrates the request-completion phases of a PCI Express split transaction. This example represents an endpoint read from system memory.
)
Figure 3-4. PCI Express Transaction Request And Completion TLPs
Write Posting: Sometimes a Completion Isn't Needed
To mitigate the penalty of the request-completion latency, messages and some write transactions in PCI Express are posted, meaning the write request (including data) is sent, and the transaction is over from the requester's perspective as soon as the request is sent out of the egress port; responsibility for delivery is now the problem of the next device. In a multi-level topology, this has the advantage of being much faster than waiting for the entire request-completion transit, but — as in all posting schemes — uncertainty exists concerning when (and if) the transaction completed successfully at the ultimate recipient.
In PCI Express, write posting to memory is considered acceptable in exchange for the higher performance. On the other hand, writes to IO and configuration space may change device behavior, and write posting is not permitted. A completion will always be sent to report status of the IO or configuration write operation.
Table 3-4 on page 116 lists PCI Express posted and non-posted transactions.
Table 3-4. PCI Express Posted and Non-Posted Transactions
|
Request |
How Request Is Handled |
|---|---|
|
Memory Write |
All Memory Write requests are posted. No completion is expected or sent. |
|
Memory Read Memory Read Lock |
All memory read requests are non-posted. A completion with data (CplD or CplDLK) will be returned by the completer with requested data and to report status of the memory read |
|
IO Write |
All IO Write requests are non-posted. A completion without data (Cpl) will be returned by the completer to report status of the IO write operation. |
|
IO Read |
All IO read requests are non-posted. A completion with data (CplD) will be returned by the completer with requested data and to report status of the IO read operation. |
|
Configuration Write Type 0 and Type 1 |
All Configuration Write requests are non-posted. A completion without data (Cpl) will be returned by the completer to report status of the configuration space write operation. |
|
Configuration Read Type 0 and Type 1 |
All configuration read requests are non-posted. A completion with data (CplD) will be returned by the completer with requested data and to report status of the read operation. |
|
Message Message With Data |
While the routing method varies, all message transactions are handled in the same manner as memory writes in that they are considered posted requests |
Three Methods of TLP Routing
All of the TLP variants, targeting any of the four address spaces, are routed using one of the three possible schemes: Address Routing, ID Routing, and Implicit Routing. Table 3-5 on page 117 summarizes the PCI Express TLP header type variants and the routing method used for each. Each of these is described in the following sections.
Table 3-5. PCI Express TLP Variants And Routing Options
|
TLP Type |
Routing Method Used |
|---|---|
|
Memory Read (MRd), Memory Read Lock (MRdLk), Memory Write (MWr) |
Address Routing |
|
IO Read (IORd), IO Write (IOWr) |
Address Routing |
|
Configuration Read Type 0 (CfgRd0), Configuration Read Type 1 (CfgRd1) Configuration Write Type 0 (CfgWr0), Configuration Write Type 1(CfgWr1) |
ID Routing |
|
Message (Msg), Message With Data (MsgD) |
Address Routing, ID Routing, or Implicit routing |
|
Completion (Cpl), Completion With Data (CplD) |
ID Routing |
PCI Express Routing Is Compatible with PCI
As indicated in Table 3-5 on page 117, memory and IO transactions are routed through the PCI Express topology using address routing to reference system memory and IO maps, while configuration cycles use ID routing to reference the completer's (target's) logical position within the PCI-compatible bus topology (using Bus Number, Device Number, Function Number in place of a linear address). Both address routing and ID routing are completely compatible with routing methods used in the PCI and PCIX protocols when performing memory, IO, or configuration transactions. PCI Express completions also use the ID routing scheme.
PCI Express Adds Implicit Routing for Messages
PCI Express adds the third routing method, implicit routing, which is an option when sending messages. In implicit routing, neither address or ID routing information applies; the packet is routed based on a code in the packet header indicating it is destined for device(s) with known, fixed locations (the Root Complex, the next receiver, etc.).
While limited in the cases it can support, implicit routing simplifies routing of messages. Note that messages may optionally use address or ID routing instead.
Why Were Messages Added to PCI Express Protocol?
PCI and PCI-X protocols support load and store memory and IO read-write transactions, which have the following features:
-
The transaction initiator drives out a memory or IO start address selecting a location within the desired target.
-
The target claims the transaction based on decoding and comparing the transaction start address with ranges it has been programmed to respond to in its configuration space Base Address Registers.
-
If the transaction involves bursting, then addresses are indexed after each data transfer.
While PCI Express also supports load and store transactions with its memory and IO transactions, it adds in-band messages. The main reason for this is that the PCI Express protocol seeks to (and does) eliminate many of the sideband signals related to interrupts, error handling, and power management which are found in PCI(X)-based systems. Elimination of signals is very important in an architecture with the scalability possible with PCI Express. It would not be efficient to design a PCI Express device with a two lane link and then saddle it with numerous additional signals to handle auxiliary functions.
The PCI Express protocol replaces most sideband signals with a variety of in-band packet types; some of these are conveyed as Data Link Layer packets (DLLPs) and some as Transaction Layer packets (TLPs).
How Implicit Routing Helps with Messages
One side effect of using in-band messages in place of hard-wired sideband signals is the problem of delivering the message to the proper recipient in a topology consisting of numerous point-to-point links. The PCI Express protocol provides maximum flexibility in routing message TLPs; they may use address routing, ID routing, or the third method, implicit routing. Implicit routing takes advantage of the fact that, due to their architecture, switches and other multi-port devices have a fundamental sense of upstream and downstream, and where the Root Complex is to be found. Because of this, a message header can be routed implicitly with a simple code indicating that it is intended for the Root Complex, a broadcast downstream message, should terminate at the next receiver, etc.
The advantage of implicit routing is that it eliminates the need to assign a set of memory mapped addresses for all of the possible message variants and program all of the devices to use them.
Header Fields Define Packet Format and Routing
As depicted in Figure 3-5 on page 119, each Transaction Layer Packet contains a three or four double word (12 or 16 byte) header. Included in the 3DW or 4DW header are two fields, Type and Format (Fmt), which define the format of the remainder of the header and the routing method to be used on the entire TLP as it moves between devices in the PCI Express topology.
)
Figure 3-5. Transaction Layer Packet Generic 3DW And 4DW Headers
Using TLP Header Information: Overview
General
As TLPs arrive at an ingress port, they are first checked for errors at both the physical and data link layers of the receiver. Assuming there are no errors, TLP routing is performed; basic steps include:
-
The TLP header Type and Format fields in the first DWord are examined to determine the size and format of the remainder of the packet.
-
Depending on the routing method associated with the packet, the device will determine if it is the intended recipient; if so, it will accept (consume) the TLP. If it is not the recipient, and it is a multi-port device, it will forward the TLP to the appropriate egress port--subject to the rules for ordering and flow control for that egress port.
-
If it is neither the intended recipient nor a device in the path to it, it will generally reject the packet as an Unsupported Request (UR).
Header Type/Format Field Encodings
Table 3-6 on page 120 below summarizes the encodings used in TLP header Type and Format fields. These two fields, used together, indicate TLP format and routing to the receiver.
Table 3-6. TLP Header Type and Format Field Encodings
|
TLP |
FMT[1:0] |
TYPE [4:0] |
|---|---|---|
|
Memory Read Request (MRd) |
00 = 3DW, no data 01 = 4DW, no data |
0 0000 |
|
Memory Read Lock Request (MRdLk) |
00 = 3DW, no data 01 = 4DW, no data |
0 0001 |
|
Memory Write Request (MWr) |
10 = 3DW, w/ data 11 = 4DW, w/ data |
0 0000 |
|
IO Read Request (IORd) |
00 = 3DW, no data |
00010 |
|
IO Write Request (IOWr) |
10 = 3DW, w/ data |
0 0010 |
|
Config Type 0 Read Request (CfgRd0) |
00 = 3DW, no data |
0 0100 |
|
Config Type 0 Write Request (CfgWr0) |
10 = 3DW, w/ data |
0 0100 |
|
Config Type 1 Read Request (CfgRd1) |
00 = 3DW, no data |
0 0101 |
|
Config Type 1 Write Request (CfgWr1) |
10 = 3DW, w/ data |
0 0101 |
|
Message Request (Msg) |
01 = 4DW, no data |
1 0 RRR* (for RRR, see routing subfield) |
|
Message Request W/Data (MsgD) |
11 = 4DW, w/ data |
1 0 RRR* (for RRR, see routing subfield) |
|
Completion (Cpl) |
00 = 3DW, no data |
0 1010 |
|
Completion W/Data (CplD) |
10 = 3DW, w/ data |
0 1010 |
|
Completion-Locked (CplLk) |
00 = 3DW, no data |
0 1011 |
|
Completion W/Data (CplDLk) |
10 = 3DW, w/ data |
0 1011 |