3.2. RDMA Path¶

3.2.1. IB Paths¶

In the IBA a path is unidirectional, but it may have the special property of being reversible. For reversible paths the return path is computed by calling rdma.path.IBPath.reverse(). Further, the path may be a ‘forward’ or ‘reverse’ path relative to the associated rdma.devices.EndPort - the exact direction will depend on context - send paths are ‘forward’ while receive paths are ‘reverse’.

The attributes for a IBPath as associated with packet headers are:

The rule for this class is that it represents the packet headers in the context it is used/created. Eg when sending a packet it represents the packet header to be sent. When receiving a packet it represents the packet headers as received.

There are additional fields for use by verbs, review Verbs and rdma.path.IBPath.

To make the use of paths simpler in some common cases there are get/set properties that access end port GID indexes, PKey indexes, and LID bits. When used as a setattr the property will update the full version. When used as a getattr the property will use the end port to compute the index. The properties are able to cache the results of these searches. For example:

# Use pkey index 0
IBPath(end_port,pkey_index=0);
IBPath(end_port,pkey=end_port.pkeys[0]);

# Use the default PKEY
IBPath(end_port,pkey=IBA.PKEY_DEFAULT);
IBPath(end_port,end_port.pkeys.index(IBA.PKEY_DEFAULT));

Do the same things.

Directed route paths are supported through the rdma.path.IBDRPath subclass that includes additional information for use in the rdma.IBA.SMPFormatDirected MAD.

3.2.2. rdma.path module¶

class rdma.path.IBDRPath(end_port, **kwargs)¶

Bases: rdma.path.IBPath

Describe a directed route path in an IB network using a VL15 QP0 packet, a LRH and rdma.IBA.SMPFormatDirected MADs.

end_port is the rdma.devices.EndPort this path is associated with. kwargs is applied to set attributes of the instance during initialization.

By default this class construct a DR path to the local port.

SGID_index¶

raises AttributeError, GID addressing is not possible for DR paths.

complete()¶

drDLID = 65535¶

Holds rdma.IBA.SMPFormatDirected.drDLID

drPath = None¶

Holds rdma.IBA.SMPFormatDirected.drPath. len(drPath) is used to set hopCount.

drSLID = 65535¶

Holds rdma.IBA.SMPFormatDirected.drSLID. Should be the same as SLID.

has_grh¶

Returns False, GID addressing is not possible for DR paths.

class rdma.path.IBPath(end_port, **kwargs)¶

Bases: rdma.path.Path

Describe a path in an IB network with a LRH and GRH and BTH

Our convention is that a path describes the LRH/GRH/BTH header fields on the wire. So a packet being sent from here should have a path with SGID == self and the path for a received packet should have DGID == self. Ditto for [ds]qpn and [DS]LID. Paths for packets received will thus have reversed semantics, SGID == remote.

This includes all relevant addressing information as well as any method specific information (ie a UMAD agent_id) required to use it with the method in question. Since this includes the BTH information it must also include the DQPN if that is required (eg for unconnected communication)

end_port is the rdma.devices.EndPort this path is associated with. kwargs is applied to set attributes of the instance during initialization.

DGID = None¶

Holds rdma.IBA.HdrGRH.DGID

DLID = 0¶

Holds rdma.IBA.HdrLRH.DLID

DLID_bits¶

Cache and return the LMC portion of the DLID. Assignment updates the DLID using the value from rdma.devices.EndPort.lid.

MTU = 1¶

Maximum MTU to use on this path

SGID = None¶

Holds rdma.IBA.HdrGRH.SGID

SGID_index¶

Cache and return the index of the SGID for the associated EndPort. Assignment updates the SGID value.

SL = 0¶

Holds rdma.IBA.HdrLRH.SL

SLID = 0¶

Holds rdma.IBA.HdrLRH.SLID

SLID_bits¶

Cache and return the LMC portion of the SLID. Assignment updates the SLID using the value from rdma.devices.EndPort.lid.

complete()¶

copy_qp_params(rhs)¶

Copy the QP related parameters from rhs

dack_resp_time = 20¶

Destination ack response time. Used to compute qp_timeout

dqpn = None¶

Holds rdma.IBA.HdrBTH.destQP

dqpsn = 0¶

Destination queue PSN, holds the initial value for the remote sqpsn.

drdatomic = 255¶

Destination issuable RD atomic

flow_label = 0¶

Holds rdma.IBA.HdrGRH.flowLabel

forward_path¶

Return a forwards direction path

has_grh = False¶

True if a rdma.IBA.HdrGRH is present

hop_limit = 0¶

Holds rdma.IBA.HdrGRH.hopLmt

mad_timeout¶

The timeout to use for MADs on this path. Expressed as float seconds.

min_rnr_timer = 0¶

Minimum value for rdma.IBA.HdrAETH.syndrome for RNR NAK. (FIXME: Where does this come from??)

packet_life_time¶

The packet lifetime value for this path. The lifetime value for the path is the expected uni-directional transit time. If a value has not been provided then the port’s rdma.devices.EndPort.subnet_timeout is used. To convert to seconds use 4.096 uS * 2**(packet_life_time)

pkey = 65535¶

Holds rdma.IBA.HdrBTH.PKey

pkey_index¶

Cache and return the index of the PKey for the associated EndPort. Assignment updates the pkey value.

qkey = None¶

Holds rdma.IBA.HdrDETH.QKey

qp_timeout¶

The timeout to use for RC/RD connections. This is 2 * packet_life_time + target_ack_delay. Expressed as float seconds.

rate = 2¶

Maximum injection rate for this path

resp_time = 20¶

Used to compute mad_timeout

return_path¶

Return a return direction path

reverse(for_reply=True)¶

Reverse this path in-place according to IBA 13.5.4. If for_reply is True then the reversed path will be usable as a MAD reply, otherwise it is simply reversed (the only difference is that reply paths have the hop_limit set to 255). Returns self

sack_resp_time = 20¶

Source RC ack response time. Value is from rdma.ibverbs.Context.query_device().

set_end_port(node)¶

Set self.end_port to the end port on node that matches the source description in this path

sqpn = None¶

Holds rdma.IBA.HdrDETH.srcQP

sqpsn = 0¶

Source queue PSN, holds the initial value for rdma.IBA.HdrBTH.PSN

srdatomic = 255¶

Source issuable RD atomic

traffic_class = 0¶

Holds rdma.IBA.HdrGRH.TClass

umad_agent_id = None¶

Method specific - override the agent_id for rdma.umad.UMAD

class rdma.path.LazyIBPath(end_port, **kwargs)¶

Bases: rdma.path.IBPath

Similar to rdma.path.IBPath but the unpack of the actual data deferred until necessary since most of the time we do not care.

end_port is the rdma.devices.EndPort this path is associated with. kwargs is applied to set attributes of the instance during initialization.

class rdma.path.Path(end_port, **kwargs)¶

Bases: object

Describe an RDMA path. This also serves to cache the path for cases that need it in an AH or other format. Paths should be considered final, once construction is finished their content must never change. This is to prevent cached data in the path from becoming stale.

The repr() format will produce a string that is valid Python and is also compatible with from_spec_string().

end_port is the rdma.devices.EndPort this path is associated with. kwargs is applied to set attributes of the instance during initialization.

complete()¶

Return True if the path is complete and usable.

copy(**kwargs)¶

Return a new path object that is a copy of this one. This takes care of the internal caching mechanism. kwargs is applied to set attributes of the instance after copying.

drop_cache()¶

Release any cached information.

end_port = None¶

End Port this path is associated with

retries = 0¶

Number of times a MAD will be resent, or the value of retry_cnt for a RC QP.

exception rdma.path.SAPathNotFoundError(fmt, err=None)¶

Bases: rdma.MADClassError

Thrown when a path record query fails with a no records error from the SM.

rdma.path.fill_path(qp, path, max_rd_atomic=255)¶

Fill in fields in path assuming path will be used with a QP. The filled fields are used to establish the QP parameters.

If max_rd_atomic is set then that at most that many responder resources for RDMA READ and ATOMICs will be provisioned. Since HCAs have limited responder resources this value should always be limited if possible. If RDMA READ and ATOMICs will not be used then it should be set to 0. Otherwise at least set to the sendq depth.

rdma.path.from_spec_string(s)¶

Construct a Path (or derived) instance from it’s repr string.

This parser is safe to use with untrusted data.

Raises:	ValueError – If the string can not be parsed.

rdma.path.from_string(s, default_end_port=None, require_dev=None, require_ep=None)¶

Convert the string s into an instance of Path or derived.

Supported formats for s are:

Format	Example	Creates
Port GID	fe80::2:c903:0:1491	IBPath.DGID = s
Scope’d GID	fe80::2:c903:0:1491%mlx4_0/1	IBPath.DGID = s
Port GUID	0002:c903:0000:1491	IBPath.DGID = fe80:: + s
LID	12	IBPath.DLID = 12
Hex LID	0xc	IBPath.DLID = 12
DR Path	0,1,	IBDRPath.drPath = ‘\0\1’
Path Spec	IBPath(DLID=2,SL=2)	IBPath.{DLID=2,SL=2}

If the format unambiguously specifies an end port, eg due to a provided scope or by specifying the subnet prefix then the result will have end_port set appropriately. Otherwise end_port is set to default_end_port.

require_dev and require_ep will restrict the lookup to returning a path for those conditions. If a scoped address is given that doesn’t match then ValueError is raised. These options should be used when a path is being parsed for use with an existing bound resource (eg a rdma.ibverbs.Context or rdma.ibverbs.)

FUTURE: This may return paths other than IB for other technologies.

Raises:	ValueError – If the string can not be parsed.

rdma.path.get_mad_path(mad, ep_addr, **kwargs)¶

Query the SA and return a path for ep_addr.

This is a simplified query function to return MAD paths from the end port associated with mad to the destination ep_addr. If ep_addr is a string then from_string() is called automatically, otherwise rdma.IBA.conv_ep_addr() is used. Thus this will accept a destination address string, an integer (DLID), GID (DGID) and GUID (DGID).

This returns a single reversible path.

If mad is an async instance then this routine returns a coroutine that will do the resolution, otherwise the new path is returned.

Raises:	ValueError – If dest is not appropriate. rdma.path.SAPathNotFoundError – If ep_addr was not found at the SA. rdma.MADError – If the RPC failed in some way.

rdma.path.resolve_path(mad, path, reversible=True, properties=None)¶

Resolve path to a full path for use with a QP. path must have at least a DGID or DLID set.

properties is a dictionary of additional PR fields to set in the query.

If mad is an async instance then this routine returns a coroutine that will do the resolution, otherwise the new path is returned.

FUTURE: This routine may populate path with up to 3 full path records, one for GMPs, one for the forward direction and one for the return direction. If the path is being used for UD then it should probably set the reversible argument to True.

Raises:	rdma.path.SAPathNotFoundError – If ep_addr was not found at the SA. rdma.MADError – If the RPC failed in some way.