6. IB Subnet Database

The library has a general storage database for holding information for an IB subnet. Support is provided for partial population, eg the database can sometimes store a SMPPortInfo without first having a SMPNodeInfo and the like.

The design is careful to separate the notion of an end port compared to an arbitrary port. Only switches can have arbitrary ports! Most functions work with and return end ports which will always correspond to switch port 0.

6.1. rdma.subnet Store IB Subnet Data

class rdma.subnet.CA

Bases: rdma.subnet.Node

Hold onto information about a single CA node in the network.

class rdma.subnet.Node

Bases: object

Hold onto information about a single node in the network. A node is a switch, *CA, or router with multiple end ports. A node has a single nodeGUID and there can not be duplicate nodeGUID’s. The port information in the ninf stores a random port.

desc = None

Result of rdma.IBA_describe.description() on the nodeString.

get_desc(sched, path)

Coroutine to fetch the node description

get_port(portIdx)

Return the port for index portIdx.

get_port_nc(portIdx)

Return the port for index portIdx, or None if it does not exist.

iterend_ports()

Iterate over all end ports. This only returns ports that are in the network, ie ports on a CA that are not reachable are not returned.

Return type:generator of Port
iterports()

Iterate over all ports. This only returns ports that are in the network, ie ports on a CA that are not reachable are not returned.

Return type:generator of tuple(Port,portIdx)
ninf = None

Instance of rdma.IBA.SMPNodeInfo

ports = None

Array of :class`Port`. Note: CA port 1 is stored in index 1.

set_desc(nodeString)

Set the description from a nodeString type value.

set_port(portIdx, port)

Store port in portIdx for ports.

to_end_port(port)

Return the end port that is associated with port. For switches this returns port 0, otherwise it returns port.

class rdma.subnet.Port(parent)

Bases: object

LID = None

Base LID if it is an end port

parent = None

Node this port belongs to

pinf = None

Instance of rdma.IBA.SMPPortInfo

portGUID = None

GUID for the port

port_id

The port number for this port.

to_end_port()

Return the end port that is associated with this port. For switches this returns port 0, otherwise it returns itself.

class rdma.subnet.Router

Bases: rdma.subnet.Node

Hold onto information about a single router node in the network.

class rdma.subnet.Subnet

Bases: object

Stores information about an entire IB subnet.

The database is organized around Port and Node objects. The database can be sparse, a node may not have any associated Port objects and Port objects may be missing nodes. Thus it is important to use the accessors to get new objects which properly join things together. The storage system is intended to support automatically filling in data no matter what the order is.

To support the discovery module and caching the loaded attribute contains a listing of what discovery actions have been performed.

This class can be efficiently pickled.

class DRCacher(sbn, end_port, start)

Bases: object

Instances of this are returned by rdma.subnet.Subnet.get_dr_cache().

Iterates over the ports from target to start (eg reversed). The ports selected are the egress ports on the path from start.

Return type:generator of Port
Raises:ValueError – If there is no path.
get_path(target)

Return a DR path from start to target.

Raises:ValueError – If there is no path.
Subnet.advance_dr(path, portIdx)

Create a new IBDRPath that goes to the device connected to port_idx of path.

Subnet.all_nodes = None

set of all Node

Subnet.get_dr_cache(end_port, start=None)

Return a DRCacher instance with a DRCacher.get_path() method that will return a DR path from start to target. If start is not specified then it defaults to the port described by end_port. Computing DR paths is very expensive and this API returns an object that can be used to compute multiple DR paths.

Subnet.get_node(type_, **kwargs)

Return an existing or new Node and Port instance associated with the end port described by kwargs. kwargs is the same signature as for search_port(). kwargs must include enough information to link a Port to the Node.

Return type:tuple(Node, Port)
Subnet.get_node_ninf(ninf, path=None, LID=None)

Return the Node object that holds the associated ninf. If none exists then one is created. If path or LID are specified then the appropriate information from both is integrated into the database.

Return type:tuple(Node, Port)
Subnet.get_path_smp(sched, end_port)

Return a VL15 SMP path to end_port. If directed routing is being used then this must be used to get paths. sched is an object with an end_port attribute.

Subnet.get_port(port_select=None, localPortNum=None, portIdx=None, **kwargs)

Return the Port object for an arbitrary port (ie a non-end port on a switch). This is designed to be used with information available during MAD processing, the main purpose of this function is to disambiguate what requested port 0 means.

If at all possible call this with portIdx set correctly and nothing else. Otherwise set localPortNum to the value returned by the MAD.

Return type:Port
Raises:ValueError – If the node type is needed but not known.
Subnet.get_port_pinf(pinf, port_select=None, portIdx=None, path=None, LID=None)

Return the Port object that holds the associated pinf. This function requires a correct portIdx if the node type is not known.

Return type:Port
Subnet.iterbfs(start, priors=None)

Iterate over all end ports in a BFS order. If priors is set to a dict then it will be populated with a mapping of end port to ingress port.

Return type:generator of Port
Subnet.iterend_ports()

Iterate over all end ports. This only returns ports that are in the network, ie ports on a CA that are not reachable are not returned.

Return type:generator of Port
Subnet.iternodes()

Iterate over all nodes.

Return type:generator of Node
Subnet.iterpeers(start)

Iterate over all ports connected to start.

Return type:generator of (Port,:class:Port) (peer,prior)
Subnet.iterports()

Iterate over all ports. This only returns ports that are in the network, ie ports on a CA that are not reachable are not returned.

Return type:generator of tuple(Port,portIdx)
Subnet.iterswitches()

Iterate over all switches.

Return type:generator of Node
Subnet.lid_routed = True

True if routes are done via LID not DR

Subnet.lids = None

list of unicast LID to Port objects

Use the provided information about port to update the database.

Note: For switches portIdx must be 0.

Subnet.loaded = None

set of strings indicating what discovery has been performed.

Subnet.nodes = None

dict of nodeGUID to Node objects

Subnet.path_to_port(path)

Return a Port instance for path or None if one does not exist.

Subnet.paths = None

dict of Port to rdma.path.IBPath indicating paths. This should be accessed via get_path_smp as it will probably be converted into a DR route cache.

Subnet.ports = None

dict of portGUID to Port objects

Subnet.search_end_port(portIdx=None, portGUID=None, nodeGUID=None, path=None, LID=None, LMC=0)

Return a Port instance associated with the supplied information if it exists or None.

Note: For switches portIdx must be 0.

Subnet.set_max_lid(max_lid)

Make lids sufficiently big to store max_lid.

Subnet.topology = None

dict of Port to Port indicating links.

class rdma.subnet.Switch

Bases: rdma.subnet.Node

Hold onto information about a single switch node in the network. Switches have several unique bits of information.

get_switch_fdb(sched, do_lfdb, do_mfdb, path)

Generator to fetch switch forwarding database.

get_switch_inf(sched, path)

Coroutine to fetch a switch info and then can schedual a LFDB/MFDB load.

Returns:via sched a contex t
iterend_ports()

Iterate over all end ports.

Return type:generator of Port
iterports()

Iterate over all ports. This only returns ports that are in the network, ie ports on a CA that are not reachable are not returned. For switches all ports are always in the network.

Return type:generator of tuple(Port,portIdx)
lfdb = None

list starting at LID 0 holding the linear forwarding databse.

mfdb = None

class:list starting at LID rmda.IBA.LID_MULTICAST holding the multicast forwarding database.

swinf = None

Instance of rdma.IBA.SMPSwitchInfo

to_end_port(port)

Return the end port that is associated with port. For switches this returns port 0, otherwise it returns port.

top_unicast_lid

Top unicast lid in the switch forwarding database.

trim_db()

Remove unnecessary entries from the end of the LFDB and MFDB.

6.2. rdma.discovery Retrieve IB Subnet Data

Functions that end in _SA collect information using SA SubnAdmGetTable() RPCs, while _SMP functions collect information using VL15 SubnGet() RPCs. These are all helper data collection functions to fill in a rdma.subnet.Subnet instance.

topo_ functions collect information on the subnet by walking it using directed route, while subnet_ functions collect information from LID routing.

Generally the rdma.discovery.load() function should be used as the entry point for this module.

Note

The functions have some assumption about the state of the rdma.subnet.Subnet instance. Often they will not re-fetch data they already have. Some care must be taken to wipe out existing information before doing a discovery if the desire is to get new information.

rdma.discovery.load(sched, sbn, stuff)

Fill sbn with the discovery items in stuff. stuff may be a list of strings where each string is one of:

  • all_LIDs fill in sbn.lids completely.
  • all_NodeInfo may be followed by a space and then a node type number.
  • all_PortInfo
  • all_NodeDescription may be followed by a space and then a node type number.
  • all_SwitchInfo
  • all_topology
rdma.discovery.subnet_fill_LIDs_SA(sched, sbn)

Coroutine to fill in the LID mapping in sbn to compensate for LMC. This should be called after subnet_ninf_SA() if desired.

rdma.discovery.subnet_fill_port(sched, sbn, port, path=None, get_desc=True)

Generator to fill in the pinf, ninf, and desc for port. This will also collect the end port pinf as well as the pinf for port if they are different.

This does nothing if the information is already loaded.

rdma.discovery.subnet_get_port(sched, sbn, path, get_desc=True)

Coroutine to associate path with a rdma.subnet.Port structure with a filled in pinf, ninf and desc. The Port can be retrieved through rdma.subnet.Subnet.path_to_port().

This does nothing if the information is already loaded.

rdma.discovery.subnet_ninf_GUID(sched, sbn, node_guid)

Coroutine to fetch a SMPNodeInfo record from the SA for a specific GUID and store it in sbn.

rdma.discovery.subnet_ninf_LIDS_SMP(sched, sbn, LIDs, get_desc=True)

Generator to fetch SMPNodeInfo and SMPNodeDescription for every LID in LIDs and store them in sbn. This does not refetch existing info.

rdma.discovery.subnet_ninf_SA(sched, sbn, node_type=None)

Coroutine to fetch all SMPNodeInfo records from the SA and store them in sbn.

rdma.discovery.subnet_ninf_SMP(sched, sbn, path, get_desc=True, use_sa=None, done_desc=None)

Coroutine to send SMPs to get SMPNodeInfo and SMPNodeDescription. The result of the co-routine is the same as rdma.subnet.Subnet.get_node_ninf(). If sched is a rdma.satransactor.SATransactor then the query is optimized into a combined node info/node description SA query.

rdma.discovery.subnet_pinf_SA(sched, sbn)

Coroutine to ask the SA for all SMPPortInfo.

rdma.discovery.subnet_pinf_SMP(sched, sbn, sel, path)

Coroutine to send SMPs to get SMPPortInfo.

rdma.discovery.subnet_swinf_SA(sched, sbn)

Coroutine to fetch all SMPSwitchInfo records from the SA and store them in sbn.

rdma.discovery.subnet_swinf_SMP(sched, sbn)

Coroutine to fetch all rdma.IBA.SMPSwitchInfo.

rdma.discovery.subnet_topology_SA(sched, sbn)

Coroutine to fill in the topology in sbn.

rdma.discovery.topo_SMP(sched, sbn, get_desc=True)

Generator to fetch an entire subnet topology using SMPs.

rdma.discovery.topo_peer_SMP(sched, sbn, port, get_desc=True, path=None, peer_path=None)

Coroutine to fetch a single connected peer. This updates rdma.subnet.Subnet.topology. It also fetches a port info to setup LID routing.

peer_path is the path out port, created by
rdma.subnet.Subnet.advance_dr().

This does nothing if the information is already loaded.

rdma.discovery.topo_surround_SMP(sched, sbn, node, get_desc=True)

Coroutine to fetch everything connected to all end ports on node. This updates rdma.subnet.Subnet.topology. It also fetches a port info to setup LID routing.

This does nothing if the information is already loaded.