CE-PE: Static Routes:

First, let’s configure the customer’s remote sites (CEs):

R15
interface Loopback15
ip address 15.15.15.1 255.255.255.0
!
interface FastEthernet0/0
ip address 172.16.158.2 255.255.255.0
!
ip route 0.0.0.0 0.0.0.0 172.16.158.1
!

R16
interface Loopback16
ip address 16.16.16.1 255.255.255.0
!
interface FastEthernet0/0
ip address 172.16.36.2 255.255.255.0
!
ip route 0.0.0.0 0.0.0.0 172.16.36.1
!

As you can see from the above config, The CEs are pushing all traffic to the next hop IP addresses which are the PEs.

Next, we should configure the provider edges (PEs facing the remote sites) for both sites:

R8
router bgp 100
 address-family ipv4 vrf CUSTA100
  redistribute static
  no synchronization
  network 172.16.158.0 mask 255.255.255.0
  exit-address-family
 !
ip route vrf CUSTA100 15.15.15.0 255.255.255.0 172.16.158.2
!

R13
router bgp 100
 address-family ipv4 vrf CUSTA111
  redistribute static
  no synchronization
  network 172.16.36.0 mask 255.255.255.0
  exit-address-family
!
ip route vrf CUSTA111 16.16.16.0 255.255.255.0 172.16.36.2
!

PEs are pushing the traffic destined to the remote site networks through VRF aware static routes. Of course, you will need to redistribute the static route under the related address family sub-command to advertise it to the other PE.

An important trick regarding the PE-CE link. As you noticed, I have advertised the subnet of the link under the address-family sub-command. Another way to do it is to use “redistribute connected” command.

NOTE: In this lab environment, we can ignore advertising/redistributing the PE-CE links and source the ping from the loopback interface to ping the loopback interface of the other CE. That should work. Remember if you don’t specify a source for your ping, it will be sourced from the exit interface which is the PE-CE link and since you don’t have this advertised, the ping will fail. To avoid all this, it is best to advertise the network or better yet, just redistribute connected.

Finally, we need to establish point-to-point connectivity throughout the ISP clouds (PEs to PEs)

R5:
router bgp 100
 address-family ipv4 vrf CUSTA100
  neighbor 172.16.151.1 remote-as 500
  neighbor 172.16.151.1 activate
  neighbor 172.16.151.1 as-override
  neighbor 172.16.151.1 soft-reconfiguration inbound
  no synchronization
  exit-address-family
!

R1:
router bgp 500
 address-family ipv4 vrf CUSTA500
  neighbor 172.16.151.2 remote-as 100
  neighbor 172.16.151.2 activate
  neighbor 172.16.151.2 as-override
  neighbor 172.16.151.2 soft-reconfiguration inbound
  no synchronization
  exit-address-family
!

R4:
router bgp 500
 address-family ipv4 vrf CUSTA500
  neighbor 172.16.141.2 remote-as 100
  neighbor 172.16.141.2 activate
  neighbor 172.16.141.2 as-override
  neighbor 172.16.141.2 soft-reconfiguration inbound
  no synchronization
  exit-address-family
!

R10:
router bgp 100
 address-family ipv4 vrf CUSTA111
  neighbor 172.16.141.1 remote-as 500
  neighbor 172.16.141.1 activate
  neighbor 172.16.141.1 soft-reconfiguration inbound
  no synchronization
  exit-address-family
!

Going back to the point mentioned above on how to handle the PE-CE links. A quick ping test shows that we have point to point connectivity between R15 and R16 across the MPLS cloud:

Both R8 and R13 are learning the customer remote and link networks across the MPLS cloud:

I removed the network command under the VRF process which advertised the PE-CE link network on R8 and R13. Show commands clearly demonstrate that, also how the ping is failing now. However, if you source from the loopback interface which IS being advertised, the ping will work because the remote networks are still being advertised. Notice how I used the RD in the show command in the second example instead of the VRF name.

The other command that I think it is very important to touch upon at this point is “neighbor x.x.x.x as-override”. I went under the customer A VRF for both R1 and R4 and removed this command. This obviously broke the site to site connectivity:

Let’s dig deeper to see why this is. As shown below, both R1 and R4 are learning the remote networks 15.15.15.0/24 and 16.16.16.0/24. Notice the commands to show the routing table for a VRF:

Moreover, R4 is indeed advertising 15.15.15.0/24 to router R10 and R1 is also advertising 16.16.16.0/24 to R5:

This is the interesting part. R10 and R5 are NOT receiving what R4 and R1 are advertising:

The reason behind that is the loop prevention mechanism in BGP. When R4 advertises 15.15.15.0/24 to R10 the AS path will be (100 500), when R10 checks the AS path and sees its own BGP AS, it will consider this as a loop and will drop this advertisement. I mentioned in the first post that I designed this network with this point in my mind, to show you what will happen when you are using the same AS number throughout the cloud.

There are two commands to fix this issue, you either use “neighbor x.x.x.x as-override” on the advertising router or the command “neighbor x.x.x.x allowas-in” In the first command, the advertising router will replace the AS numbers in the path with its own AS number while keeping the AS path. While the second command simply instructs the receiving router to disable the loop preventing mechanism and accept any advertisement regardless of the AS path.