- Protecting LAN Users from Router Loss
- Dial Backup by One Router for a Line on Another
- Protecting against LAN Segmentation
- Wrap-up
Dial Backup by One Router for a Line on Another
Another challenge introduced by having multiple routers at a site, particularly a small site, is using dial backup to adequately protect against line outages. While it is easy to set up dial backup for a single router using the backup interface approach, this approach only works when the primary link and the backup link are on the same router. It is generally not useful for dial backup when there are two or more routers at a site.
Backup interface is triggered by a router detecting that an interface has failed. There is no provision for one router to detect the failure of an interface on a second router so that the first router can initiate a backup call. Consider the minimum-cost, physically diverse site configuration in Figure 6-6. We put one router at each end of a large building and connect the two 10BaseT concentrators to create a single extended LAN.
){kind=link}
Figure
6-6.
Physical configuration of a two-router site with physical diversity and dial
backup
In a production network, we would replace the single link with physically diverse, redundant links using IEEE 802.1d transparent Spanning-Tree (TST) bridge protocol. But for now, we want to keep the configuration simple. For the same reason, we connect this site to a single router at a single data center site, as shown in Figure 6-7. We can generalize the concepts we will use to handle this basic configuration to support a site with an arbitrary number of routers connecting to an arbitrary mesh of other sites.
){kind=link}
Figure
6-7.
Logical connectivity and IP addressing for two-router site example
The key to success is to trigger the dialing of the backup link based on the inability of the South Router to reach the data center LAN rather than on the state of the Frame Relay interface on the South Router. In other words, we want to trigger the dial backup based on whether or not an IP address is in the routing tables of the North and South routers. Based on our survey of dial backup approaches in Chapter 4, we know that either dialer watch or dial-on-demand should be usable, so we will discuss examples of both approaches.
Configuration Example: Using Dialer Watch
The Cisco dialer watch feature introduced in IOS 12.0 is ideal for this application, provided we are comfortable with using EIGRP as our routing protocol.1 Dialer watch works by taking a list of IP addresses (the dialer watch-list) and checking if any of them is still in the local routing table any time the routing table is modified to remove a watched IP address. If no valid route remains, the primary link is considered down and the backup link is forced to dial, regardless of the presence or absence of "interesting" traffic defined by the dialer-list. For example, if we enabled dialer watch on the address of the other end of the Frame Relay link, any time the Frame Relay link went down, the route to it would be removed from the routing table. We would immediately initiate a dial backup call.
Operation is more reliable if we do not depend upon the Frame Relay link to fail before initiating a dial backup call. Rather than watch the other end of the Frame Relay link, which will always be considered reachable—unless the Frame Relay link has failed to the extent that the local interface is down—we can watch the Ethernet address of the data center LAN. That way, regardless of the state of the serial interface on the South Router, we will dial as soon as any failure occurs that makes the data center LAN unreachable.
We will start our examination of the router configurations in Listing 6-11 with the South Router. The South Router supports the Frame Relay link from the remote site to the data center. From the viewpoint of dialer watch, it makes no difference what technology is used for this primary link. All that matters is that we use EIGRP for the routing protocol between this router and the North Router. If desired, we could use another routing protocol between this router and the data center. However, configuration is much easier if EIGRP is used between the North Router running dialer watch and the data center router, so we use EIGRP everywhere.
Listing 6-11. South Router with primary (Frame Relay) link to data center
version 11.0 ! hostname South ! Router with primary link being backed up interface Ethernet0 ip address 10.101.5.1 255.255.255.0 ! interface Serial0 no ip address encapsulation frame-relay ! interface Serial0.17 point-to-point description Primary link to Data Center ip address 10.201.5.2 255.255.255.252 frame-relay interface-dlci 17 broadcast ! router eigrp 1 network 10.0.0.0 no auto-summary ! end
Note that although dialer watch requires running IOS 12.0 or better, that requirement applies only to the North Router. Other routers in the network only need to support EIGRP and any interfaces and protocols used. While here we show them running IOS 11.0—as is the convention when the IOS release is not significant—most practitioners would prefer to run the same IOS release on all routers in the network.
Running the same software release on all systems minimizes problems with incompatible changes to default parameters and other inconsistencies. Still, some prefer the more conservative route of deliberately mixing software releases and router vendors to reduce the number of systems affected when a major software defect surfaces. As we discussed in Chapter 1, the latter is a better approach for maximum availability, but the cost of the testing effort required is too high for most organizations. Most are better served by adequately testing a single release and restricting their configurations to use only well-tested features.
Turning our attention to the North Router, which is running the dialer watch code in Listing 6-12, we see a configuration very similar to the dialer watch example explored in Chapter 4. If anything, this example is simpler because there are no WAN links to configure other than the ISDN support. Note that we also support dial-on-demand to simplify testing of ISDN link functionality. This requires us to define "interesting" traffic in access-list 100 so that EIGRP neighbor multicasts will not bring up or keep up the ISDN link.
Listing 6-12. North Router using dialer watch to control backup ISDN link
version 12.0 ! hostname North ! Router with ISDN backup link username DataCenter password SharedSecret ! interface Ethernet0 ip address 10.101.5.2 255.255.255.0 ! interface BRI0 ! Secondary backup line ip address 10.2.1.5 255.255.0.0 encapsulation ppp ! Dialer map for the IP network which is being watched dialer map ip 10.1.0.0 name DataCenter speed 56 broadcast18005551212 ! Dialer map for the actual remote end address for routing dialer map ip 10.2.255.254 name DataCenter speed
Remember that for dialer watch to function, we must assign unique numbers across all dialer-lists and dialer watch-lists. That is, we must not use the same number to identify a dialer watch-list and a dialer-list on the same router. Since this is an undocumented limitation, it may or may not exist in any particular IOS release.
We also can use dialer watch with OSPF or some other routing protocol as our primary routing protocol, as long as we run EIGRP on this router and redistribute the watched routes from the primary routing protocol into EIGRP. However, we will then need to ensure that the watched routes cannot be learned across the dial link by any protocol other than EIGRP, or the dial backup will be unstable.
At the data center router in Listing 6-13, we have a standard dial-in router configuration. Again, there are no traces of dialer watch on this router, so any IOS release will work. We could even use a non-Cisco router if we chose to work around the EIGRP requirement by using route redistribution on the North Router.
Listing 6-13. Router backup using dialer watch; data center router with Frame Relay and ISDN links
version 11.0 ! hostname DataCenter ! username North password SharedSecret ! Names and passwords for other callers go here ! interface Ethernet0 description Data center LAN ip address 10.1.255.254 255.255.0.0 ! interface Serial1 description Frame Relay to remote locations no ip address encapsulation frame-relay ! interface Serial1.105 point-to-point description Primary link to South Router ip address 10.201.5.1 255.255.255.252 frame-relay interface-dlci 105 broadcast ! ! Subinterfaces for other sites not shown... ! interface BRI0 description This is where backups dial in. ip address 10.2.255.254 255.255.0.0 encapsulation ppp dialer idle-timeout 300 dialer map ip 10.2.1.5 name North speed 56 broadcast ! Dialer maps for other callers go here dialer-group 1 ppp authentication chap ! router eigrp 1 network 10.0.0.0 no auto-summary ! access-list 100 deny eigrp any any access-list 100 permit ip any any dialer-list 1 protocol ip list 100 ! end
Even though this end of the ISDN link will never place a call, we still configure it with a 5-minute idle time-out and ignore the routing protocol hello packets that would otherwise keep the link up forever. We do this to catch some errors that would not be detected by a simple ping test of the ISDN test address using dial-on-demand. A series of ISDN calls in the router log, each exactly 5 minutes long, then becomes an immediate red flag. It indicates that there is a serious problem with ISDN dial backup that is not being detected by the router placing the call. Without the idle time-out at this end of the link, the remote would keep the link up even though no responses were coming back. The inability to communicate might not be noticed until users complained.
Clearly there are many improvements we would make to this example design before putting it into a production environment. We would dial into a different router in order to avoid that single point of failure, and we would use an ISDN PRI interface at the data center to allow more than two locations to be on ISDN at the same time. We might also want to consider enhancements such as strengthening authentication through the use of caller ID information or using multilink PPP for higher bandwidth. The exact configuration will obviously be highly dependent upon user and organizational requirements.
There is another possible improvement that is not as obvious. If there are multiple LANs at the data center site, we could "watch'' a loopback interface address defined on the data center router so that dial backup would not be triggered unnecessarily by an inconsequential LAN failure. Similarly, if there were only one data center LAN (not very common in a high availability data center, but quite possible in a random mesh topology), we could "watch" a loopback address on a different router at the site. That way, we could not only recover from a primary link failure, but also attempt to dial around a LAN failure at the site. The key is to think through what we are trying to accomplish and look for ways to adapt our tools to enable us to get there.
Configuration Example: Using Dial-on-Demand
While dialer watch provides an easy solution to multiple router site dial backup, it is not a solution for everyone. The following reiterates the three critical limitations to be considered in any potential application of dialer watch.
The only routing protocols supported are IGRP and EIGRP.
The only network protocol supported is IP.
There is no support on IOS releases prior to 12.0.
Cisco may address the first two shortcomings in subsequent releases, but there are no plans to retrofit the dialer watch capabilities into prior IOS releases. It is also not clear when other router vendors will provide similar capabilities on their systems. So we need to look at how we can achieve the same functionality provided by dialer watch using any router supporting dial-on-demand and floating static routes. The trick is to use judiciously defined floating static routes rather than dialer watch to trigger the dial event.
We will start with the network design in Figures 6-6 and 6-7. Our discussion will use Cisco semantics, but, unlike dialer watch, our approach is not Cisco proprietary and could be implemented in a mixed-vendor environment. The key is to define floating static routes for the target destinations that will force the DDR activation anytime the primary link is unavailable. Then we must redistribute those routes to other local routers to attract the traffic necessary to activate the dial-on-demand.
In this example, we define the default route on both the North and South routers to be via the data center LAN. We then add to the North Router with the backup link, a floating static route that associates the address of the data center LAN with the BRI interface. When the primary link is connected, the route to the data center LAN is learned from the dynamic routing protocol, and both routers use the primary link as their best path to everywhere.
When the primary link fails, the route to the data center LAN via the primary link is removed from the routing tables. This allows the route to the data center via the backup link to float into action on the North Router. We configure the routing protocol on the North Router to redistribute the new route to the data center. The South Router then learns about it and forwards traffic to the North Router to be delivered to the data center.
This extra level of redirection is essential. We need the South Router to send all outbound traffic to the North Router in order to ensure that "interesting" traffic will be available to force up the dial link. But we must not put a floating static route on the South Router to do the job. Otherwise, we introduce a routing loop: the South Router would discover the primary link down and start to send traffic to the North Router for delivery. In the meantime, until the North Router learned of the failure through the routing protocol, it would continue to send all traffic to the South Router for delivery over the primary link.
Instead, the North Router activates the backup route only when it learns from the South Router of the loss of connectivity. At this point, the South Router has no route to the data center, and any traffic is discarded. As soon as the North Router activates the route, it informs the South Router; all traffic for the data center is then forwarded to the North Router for delivery.
The configuration of the South Router in Listing 6-14 is identical to the dialer watch configuration except for three added lines (in bold type). Two lines specify the bandwidth and delay of the primary link for use in routing metrics, and the third adds a floating static route defining a default route via an address on the data center LAN. In a real configuration, these lines probably would have been part of the dialer watch configuration as well—but now that we are using dial-on-demand, they become critical to efficient operation. Note also that while we continue to use EIGRP as the production routing protocol, to minimize the number of changes, our choice of routing protocol is no longer constrained.
Listing 6-14. South Router with primary (Frame Relay) link to data center, with dial-on-demand additions in bold
version 11.0 ! hostname South ! Router with primary link being backed up interface Ethernet0 ip address 10.101.5.1 255.255.255.0 ! interface Serial0 no ip address encapsulation frame-relay ! interface Serial0.17 point-to-point description Primary link to Data Center ip address 10.201.5.2 255.255.255.252 bandwidth 56 delay 2000 frame-relay interface-dlci 17 broadcast ! router eigrp 1 network 10.0.0.0 no auto-summary ! ! Default route is via learned path to data center LAN ip route 0.0.0.0 0.0.0.0 10.1.255.254 150 ! end
The default route via the data center LAN (ip route 0.0.0.0 0.0.0.0 10.1.255.254) is required even if a default route is being advertised by the production routing protocol. The target of this default route must not be on any subnetwork physically attached to the local router. It must be an address that can only be learned through the production routing protocol so that it will disappear from the local routing tables when no links are up between this site and the data center.
The values used in the bandwidth and delay statements on the Frame Relay interface are arbitrary. The only requirement is that when the Frame Relay link and the ISDN link are both up, all traffic between the two sites be routed via Frame Relay, even if that means an extra hop across the LAN at each end. Otherwise the ISDN link could stay up forever even though it was no longer needed. With routing protocols other than IP, it is necessary to add statements guiding the path selection for any packets considered "interesting" in the dial-on-demand filter—such as ipx delay for IPX. This is true both here and on the backup and target routers.
Moving over to the North Router in Listing 6-15, note that all the extra lines for dialer watch are gone. In their place, we find a bandwidth 56 and delay 5000 under the BRI interface to ensure that the ISDN link not carry traffic when the primary link is available. There is also a redistribute static command under router eigrp 1 (controlled by distribute-list 11 out static), and a pair of floating static routes to activate dial-on-demand.
Listing 6-15. North Router with ISDN backup link to data center, with dial-on-demand additions in bold
version 11.0 ! hostname North ! Router with ISDN backup link username DataCenter password SharedSecret ! interface Ethernet0 ip address 10.101.5.2 255.255.255.0 ! interface BRI0 ! Secondary backup line ip address 10.2.1.5 255.255.0.0 encapsulation ppp bandwidth 56 delay 5000 dialer map ip 10.2.255.254 name DataCenter speed 56 broadcast
As long as the primary link is up, the routing protocol running will announce the data center LAN, and the floating static route (ip route 10.1.0.0 255.255.0.0 10.2.255.254 155) will remain inactive. The floating static route defining the default route (ip route 0.0.0.0 0.0.0.0 10.1.255.254 150) may or may not be active, depending upon whether the routing protocol is distributing a default route. We block redistribution of the static route defining the default route to prevent its distribution outside the local site. Depending on other connectivity that may be in place, we may also find it beneficial to filter routing advertisements from the North and South routers on links leaving the site. This would remove the backup path to the data center LAN.
When the primary link fails, all routes learned through the link disappear and both of the North router's floating static routes are activated. The floating static route to the data center LAN activates the default route on the South Router when it arrives there via the production routing protocol. As soon as the dial backup link comes up, the routing protocol learns where everything really is, and the floating static routes float out of action.
At the data center in Listing 6-16, no configuration changes are required. However, if filtering of external routes is required to prevent the introduction of incorrect routing information from the redistributed static routes, it may be necessary to install the filters here. What can be filtered and where will depend upon the production routing protocol in use and the version of the router software.
Listing 6-16. Router backup using dial-on-demand; data center router with Frame Relay and ISDN links
version 11.0 ! hostname DataCenter ! username North password SharedSecret ! Names and passwords for other callers go here ! interface Ethernet0 description Data center LAN ip address 10.1.255.254 255.255.0.0 ! interface Serial1 description Frame Relay to remote locations no ip address encapsulation frame-relay ! interface Serial1.105 point-to-point description Primary link to South Router ip address 10.201.5.1 255.255.255.252 frame-relay interface-dlci 105 broadcast ! ! Subinterfaces for other sites not shown... ! interface BRI0 description This is where backups dial in. ip address 10.2.255.254 255.255.0.0 encapsulation ppp dialer idle-timeout 300 dialer map ip 10.2.1.5 name North speed 56 broadcast ! Dialer maps for other dial ins go here dialer-group 1 ppp authentication chap ! router eigrp 1 network 10.0.0.0 no auto-summary ! access-list 100 deny eigrp any any access-list 100 permit ip any any ! dialer-list 1 protocol ip list 100 ! end
Configuration Example: Using BGP-Driven Dial-on-Demand
We have mentioned numerous times that a critical requirement for successful use of dial-on-demand for backup purposes is having "interesting" traffic present at the router with the backup line to force the dial. When there is only one dial link, it is usually easy enough to ensure that some normal traffic (often the syslog event reporting that the primary link has gone down) will force the link up. If we want to provide a second dial backup link to improve the probability of a successful connection, it becomes much more difficult to ensure that there will be appropriate traffic to trigger the dialing.
The easy solution is to use IOS 12.0 and dialer watch. First, two unique loopback port addresses are defined at the target site. Next, route advertisement filters are configured so that a route to both addresses can be learned over the primary link, a route to only one can be learned over the preferred dial link, and no routes to either can be learned via the secondary dial link. We can then set up a dialer watch on the preferred dial link for the address that is only advertised over the primary link, and a dialer watch on the secondary dial link for the address that is advertised over both the primary and the preferred dial links. This is a trivial extension to the dialer watch example discussed at the beginning of this section.
Even if we are not in a position to use dialer watch, we can still get two levels of backup. To do so, we set up a pair of loopback addresses with restricted advertising—the same as if we were using dialer watch to do the job. But instead of dialer watch, we use the TCP keepalive packets between BGP neighbors to force up dial-on-demand. This is the same thing we did in the BGP-driven dial backup example in Chapter 5 to provide two levels of backup for a single router.
In this example, we build on the configuration we have been using so far and add a second backup link to the mix—an asynchronous modem off the Auxiliary port of the South Router with the primary link. Then, just to keep it interesting, we also add support for Novell NetWare IPX and configure the ISDN backup to use unnumbered addresses to conserve IP address space. We also get more realistic for a large site at the target by configuring a T1 for Frame Relay and a PRI for ISDN back at the data center. The resultant network is shown in Figure 6-8.
){kind=link}
Figure
6-8.
BGP-driven dial-on-demand routing example
The theory of operation here is identical to that in the single router example in Chapter 5. So we can move straight to the Cisco configurations—starting with the South Router with the primary link and asynchronous modem on its AUX port, from Listing 6-17—to see how the theory can be applied to practice.
Listing 6-17. IP and IPX with BGP-driven ISDN, South Router
version 11.0 ! hostname South ! username DataCenter password SouthSecret ! ipx routing 0000.0c01.0105 ipx internal-network A10105 chat-script courier ABORT ERROR ABORT "NO " ABORT BUSY ""
The configuration of the South Router is standard point-to-point Frame Relay with Async dial backup through the AUX port. Refer to the discussion in Chapter 5 on the use of the AUX port for dial backup for operational considerations. Although this is an example of using BGP to drive dial-on-demand in a multiple router site, there is no BGP on this router. There is, however, one extra floating static default route, activated by knowledge of an address that can only be learned via the routing protocol running across the ISDN link. Otherwise, the only influence of the North Router's ISDN link is in the ip host and ip route definitions to support ISDN link testing.
Each pair of routers can have a unique shared secret. Here, data center and South use the shared password SouthSecret, while data center and North use the password NorthSecret. The ipx routing command turns on IPX so that Novell NetWare can be supported on the network. The IPX internal network assignment is critical, as it becomes part of the IPX host address identifying this router when IPXWAN is used. Note that in IOS 11.2 and later, the IPX definition of "interesting" traffic would read as follows:
|
access-list |
902 |
deny |
rip |
|
access-list |
902 |
deny |
sap |
|
access-list |
902 |
permit a |
ny |
Keep in mind that this configuration is the minimum needed to get ISDN and asynchronous dial support of IP and IPX, not a tuned production configuration. Typical commands useful when supporting IPX are not shown. These include custom queuing to provide better IPX responsiveness, use of NLSP rather than IPXRIP, inclusion of all the filters appropriate for keeping the link from coming up unnecessarily, suppression of advertisement of unnecessary addresses and services, and spacing out of RIP and SAP updates over slow WAN links. In particular, this IPX configuration will not function without the support of IP to bring up the dial links. If there is no outbound IP traffic present at the primary router when the Frame Relay link dies, it will never even attempt to bring up the asynchronous link. Indeed, as you can see from the dialer maps, IPX traffic is incapable of bringing up the dial link on this router, as only the IP dialer map has the necessary chat script specified.
The definition and weighting of the floating static routes is essential to successful multiple levels of backup operation. The static routes must have a higher distance specified than that used by the interior routing protocol. The routing advertisements from the data center router must filter its defined loopback addresses so that only the desired target addresses are advertised over each link. If multiple data center routers are used, all that is required is that the appropriate loopback interface addresses be defined on each. There is no problem with defining the same target addresses on multiple routers, as long as they are filtered out of any direct advertisements, such as over a common Ethernet (hence the advertisement filter on the Ethernet port in the data center router).
The forwarding of traffic via the ISDN link, when required, is handled automatically by the routing protocol. Under normal conditions, all traffic—including BGP keepalives—flows via Frame Relay. When the Frame Relay link goes down, the 10.0.0.1 target is no longer reachable and the router looks for an alternative route. Since the target address 10.0.0.3 is only advertised over ISDN, it will not be available. The only available route will be the default route via 10.0.0.4, the asynchronous link. Under the rules of dial-on-demand routing, dial links are always connected (whether they are physically connected or not), so the floating static route through the asynchronous port is activated, and a call is initiated.
At first glance, the North Router in Listing 6-18 also appears to be a standard IP and IPX dial-on-demand configuration. However, when we look at the hierarchy of floating static default routes, we find that all the traffic that would normally bring up the ISDN line is routed to the Async interface on the South Router. We need to take a closer look at the configuration of the North Router to see how ISDN calls are forced up by BGP.
Listing 6-18. IP and IPX with BGP-driven ISDN, North Router
version 11.0 ! hostname North ! username DataCenter password NorthSecret ipx routing 0000.0c02.0105 ipx internal-network A20105 isdn switch-type basic-ni1 ! interface Loopback0 description Source address for ISDN calls ip address 10.201.5.2 255.255.255.255 ! interface Ethernet0 description Building LAN ip address 10.101.5.2 255.255.255.0 ipx network A00105 encapsulation SAP ! interface BRI0 description Primary backup path for this site ip unnumbered Loopback0 encapsulation ppp bandwidth 56 delay 5000 no keepalive ipx ipxwan 0 unnumbered North ipx delay 30 isdn spid1 80055512120101 isdn spid2 80055512130101 dialer idle-timeout 170 dialer map ipx 0.0000.1234.0000 name DataCenter speed
Since BGP will only establish a connection to a neighbor that is explicitly in the routing tables, and the address 10.0.0.10 is not advertised over any links from the target router, we define two explicit static routes just for BGP. The preferred route is via the Frame Relay target address so that when Frame Relay is up, BGP will exchange keepalive packets using the Frame Relay link. However, whenever the primary route disappears, we fall back to the floating static route through the "always available" target address used for ISDN. This is why we need two ISDN target addresses. One is for routing production traffic in both routers when the ISDN is up and running; the other is for use by the BGP on the North Router to force up the ISDN link when needed. So even if we are currently carrying all production traffic through the asynchronous link on the primary router, the secondary router will continue to try to bring up the ISDN link, due solely to BGP's exchange of keepalive messages.
Although it increases our overhead slightly, we adjust the timers on BGP to send a keepalive every 10 seconds rather than the default 60 seconds. This reduces the maximum delay before an ISDN dial attempt is forced. We need to make sure any modified timer values are matched on the remote-peer BGP configuration.
One other convenience feature that is easy to overlook is the definition of a static route to the asynchronous activation address. This allows for testing of the asynchronous port by pinging the activation address from any system on the LAN, eliminating the need for a telnet session to the South Router. We include a static route on the South Router pointing the ISDN activation address to the North Router for exactly the same reason.
Before we discuss the individual functions configured on the data center router in Listing 6-19, keep in mind that normally these functions would be distributed across two or more routers to eliminate the data center router as a single point of failure. Since routers that are the target of dial backup calls like this one will often be serving many callers, it is quite possible that all functions will be on each data center router. This is true even though each caller may use one data center router for its primary link and a different data center router for each mode of dial backup call. In Chapter 7 we will explore ways to make multiple independent call-accepting routers look and behave like a single router for dial backup purposes. This will further enhance redundancy without adding significant complexity to the remote router configurations.
Note the inclusion of an advertisement filter on the local Ethernet port in addition to those on the links to the remotes. This filter allows us to reuse the same loopback address targets on all routers in the data center without the routers' confusing each other. Just be careful when setting up this filter not to block a loopback address used to identify the real router to network management. The safest approach is to dedicate a block of addresses to these artificial needs and allocate router IDs from a different block of addresses.
Listing 6-19. IP and IPX with BGP-driven ISDN, data center router
version 11.0 ! hostname DataCenter ! username North password NorthSecret username South password SouthSecret ipx routing 0000.0C00.F001 ipx internal-network 1234 isdn switch-type primary-dms100 chat-script courier ABORT ERROR ABORT "NO " ABORT BUSY "" "at"
The key to success is to map out what connectivity is available and which backups should be active under what conditions. Once the priorities are identified, the appropriate loopback addresses can be defined at the required target locations, and the required floating static routes sorted by administrative distance. In addition to checking that the operation will be what we want under the conditions we are trying to protect against, we should look for any failure modes that could cause an undesirable set of connections. We can then iterate, fixing our design by including the "bad scenario" in the set of protected conditions.