Category Archives: Cisco

Cisco Category

VRRP – start build redundant network

Redundant in the network is critical for service availability. Everyone wanted to have five "9" – 99,999% of available service. In order to get this we are buying expensive equipement, learn how works STP, RSTP, MSTP and other redundant protocols. Often we forget to make redundant obvious things. How many gateways could you configure on the PC? Always one.. You know that gateway is our exit point from the network in the big Internet. So in case this point is failing we are losing our way out.

VRRP is based on the RFC 3768 and RFC 5798. RFC 3768 is already version 2 of VRRP and is only for IPv4. RFC 5798 is already for IPv6 support. You could see that they are pretty new (2004 and 2010 year). To understand very good VRRP you could read RFCs, but I don't know too many people who like text without images :). This is why I decided to create some images and sample which could help you understand how RFC 3768 recommend to implement this standard.

VRRP goal is to provide redundancy for the gateway IP address. We will not change the way how we are configuring default gateway on the PC. So we have to do something on the gateways. VRRP will force routers to talk between them in order to see who owns Virtual IP address. Virtual IP address will be assigned as default gateway on the PCs in our network. Routers will talk via multicast ip address 224.0.0.18.

VRRP has 3 states: Init, Slave and Master. Init state is saying that VRRP is enabled on the Router but for some reason is irrelevant to start communication. For example we have to send VRRP packets via Fa0/0 interface, but this interface is down. VRRP slave means that router at the moment is backup. VRRP Master means that VRRP router owns Virtual IP. Routers have to select VRRP master. They could this based on the VRRP priority. Higher priority is better. We could configure VRRP priority from 1 to 254.

Do you remember that in ethernet LANs we have to encapsulate packets and add IP source, IP destination, Mac Source, MAC destination ? Here we have another interesting thing for VRRP. VRRP enter not only VIrtual IP but also Virtual MAC which will have following format: 00-01-5E-00-01- <GROUP_ID>. Group id value is limited from 1 to 255 (from 01 until FF in hex). Virtual IP and VIrtual MAC will be used only for the traffic which is going through routers, all other traffic will use Physical MAC and physical IP. Let's see the sample how ill work ARP protocol with VRRP:

We must have same values for 3 parameters on both routers:

  • Group ID
  • Group IP address
  • Advertisement interval

What for we need advertisement interval and how fast backup router will become VRRP master in case of failure? Here is the formula: 3*Adv.timer + skew time. Skew time formula = (256-Priority)/ 256. Skew time goal is to add time required for the message propagation in the LAN. So in case we have advertisement timer 1 sec, switchover will be done in maximum = 3*1 + (256-100)/256= 3.6 s. Pretty good result. We could decrease this value under 1 sec. I will not recommend to do that for beginners. VRRP is working with other protocol in the networks and adjustments in this area could create another issues for other protocols. 

Who is the VRRP Master?

VRRP master is router with biggest priority when VRRP preemption is enabled on all routers. Let's see one example:

When our Routers are up VRRP Master will be always R2. When R2 is down or link between R2 and switch is down VRRP master will be R1. Simple until now.

Let's suppose that we have network with big fluctuations. Routers are going down often. VRRP switchover is too often and sometimes without any reason. Let's suppose that R2 has preemption on and he is rebooting each 5 minutes. Our default gateway will go up/down every 5 minutes. R1 at the same time is stable and ok. For this case we could configure VRRP preemption off on R2. When R2 will go up after reboot he will stay in the VRRP Slave state. He lost Master state and the only way to recover it is to shutdown R1.

When our Routers are up VRRP Master will be always R2. When R2 is down or link between R2 and switch is down VRRP master will be R1. Simple until now.

Let's suppose that we have network with big fluctuations. Routers are going down often. VRRP switchover is too often and sometimes without any reason. Let's suppose that R2 has preemption on and he is reabooting each 5 minutes. Our default gateway will go up/down every 5 minutes. R1 at the same time is stable and ok. For this case we could configure VRRP preemption off on R2. When R2 will go up after reboot he will stay in the VRRP Slave state. He lost Master state and the only way to recover it is to shutdown R1.

VRRP Priority 0 and 255

RFC 3768 reserved VRRP priority 0 and 255 for two special cases. R2 is VRRP master for the network. Based on some events R2 decided to leave VRRP Master role. So he will send VRRP priority 0 which means that backup Router has to become VRRP Master immediately.

VRRP allows us to configure the same ip address for the Virtual IP and physical only one one router in the network. In this case Router where we configured the same IP address for the VIP will send VRRP packets with priority 255.

Decrease VRRP priority in some conditions

Last thing what I want to say about VRRP is related to dynamic priority. Network developers decided to adjust VRRP priority based on some important metrics. For example R2 router will monitor link to the ISP and will decrease VRRP priority when this link is down. More than that you could use Cisco IP SLA (when you have Cisco equipment) and monitor quality of the link (delay, drops…). Quality will be trigger for value of VRRP priority in your case.

In the network with big numbers of VLANs it is recommended to balance VRRP master roles between R1 and R2. For Example R1 is VRRP master for VLAN 10,20 and 30, when R2 is VRRP Master for VLAN 15 and 25. In this way you will use all links in your network. VRRP has to work very close with protocols like ARP, Proxy-ARP and STP. When you set VRRP on L3 switches and have redundant links you have to think how traffic will flow through network at the L2 and L3. In other case suboptimal routing will deprecate al benefits of protocols. STP and VRRP timers also have to be synced and this is why I recommended you not to change default timers.

During the switchover from one VRRP router to another, new master will send gratuitous ARP and announce him as new owner of the IP. This will converge network faster.

Links:

  1. https://www.ietf.org/rfc/rfc3768.txt
  2. https://tools.ietf.org/html/rfc5798

Configuring Cisco ASA 8.4(2) on GNS3 1.3.13

In one of my previous post (link here) I did a description on how to configure and run an ASAv instance on GNS3 1.4.5. In this post I will describe the steps required to configure and run a Cisco ASA 8.4(2) image on GN3 1.3.13.

Why in this case I insisted to use the previous version suite 1.3.x of GNS3 (1.3.13 at the moment of this writing) instead of the latest available 1.4.x suite ? The answer is because for ASA 8.4(2) to run successful, a QEMU emulator of version 0.11.0 needs to be in place and functional or as it seems, the QEMU 0.11.0 categorically refuse to work in GNS3 1.4.x suite (even it is installed here). Somewhere in forums found that GNS team will no longer offer support for QEMU 0.11.0 in its product. Maybe in one of the future releases but certainly not now …

Someone might ask, why not just use the 2.4.0 version of QEMU present and functional in both GNS3 versions suites ? The truth is that it works, but with a little issue, a very slow speed through device interfaces. In my testings, a simple ASDM image copying can take several hours with a high chance for device crashing. Types and number of NICs, license or whatever else doesn’t change the situation.

So, because of unavailability of QEMU 0.11.0 in GNS3 1.4.x I switched back to 1.3.x version suite. Why then, I didn’t use the 1.3.x suite in my previous article for an ASAv instance configuration ? It is because we needed the VNC console for ASAv initial configuration or that is present only in newer 1.4.x suite.

To conclude: for ASAv you will need GNS3 1.4.x (because of VNC console) and respectively for ASA 8.4(2) you will need the GNS3 1.3.11 (because of QEMU 0.11.0). At least in my testings, this offer me a stable and workable setup.

Note0: do not mix the GNS3 1.3.13 and GNS3 1.4.x on the same machine, simply because they wasn’t designed to work together, configuration that most probably lead to complete nonfunctional setup.

Why bothering also with ASA in addition to ASAv ? Doesn’t ASAv being sufficient ?

Well, the ASAv is a software designed to run in virtual infrastructure and such that, some features are no longer needed here. Take simple, what ASA doing by clustering, in virtual infrastructure with ASAv is accomplished by hypervisor’s High Availability features, multiple contexts are replaced by multiple standalone ASAv instances and so on. To be more precise, that’s the unsupported features that that the official documentation (link here) states: The ASAv does not support the following features: clustering, multiple context mode, active/active failover, EtherChannels, Shared AnyConnect Premium Licenses.

So in case you want to play with ASA clustering or multiple context mode you will need a classic ASA instance running in GNS3. ASAv is good but not always sufficient. Even so, ASAv should be your standard, especially since it is somewhat closer to VIRL style.

What images are needed to run ASA in GNS3 ?

Compared to ASAv where an original qcow (KVM) image was sufficient to configure the device in GNS3, for ASA the original bin image are not sufficient. The truth is that this original image needs to be unpacked, then some files modifications needs to be done and after that repack the content in a way suitable for QEMU. All of this can be done manually or scripted (see GNS3 forums for Cisco Image Unpacker for Windows or repack.v4.sh shell script for Linux) but imho, if you not search for a specific ASA version just do a search on the Internet for ASA 8.4(2) GNS3 files. Other versions should be also be available to. In any case, you should be left with two files: asa-vmlinuz (a Linux kernel) and asa-initrd.gz (a RAM disk file). These are the files used by QEMU in GNS3.

How to configure Cisco ASA 8.4(2) in GNS3 1.3.13 ?

  1. Download the latest version for 1.3.x GNS3 version suite. To do that, go to GNS3 web site – avoid the download button witch will direct you only to GNS3 1.4.x download, instead, click on software  (top bar menu) – on the left, written with small font size, click on To download Version 1.3.13 of GNS3 Click Here (see screenshot below). After, authentication a download for GNS3-1.3.13-all-in-one.exe file should start.

art - configuring Cisco ASA 8.4(2) on GNS3 1.3.13 - download page

  1. Install the 1.3.13 GNS3. No rocket science here, just follow the installation wizard. Setup will install one by one all the components needed: Dynamips, QEMU, GNS3, WinPcap, Wireshark and many others parts.
  2. Initial GNS3 configuration. At this step, I usualy configure some general, non-essential, GNS3 preferences:

     

    1. General – Local paths – projects/binary images – redirected to a shorted path, e.g. C:\GNS3\projects and respectively C:\GNS3\images
    2. In Topology View change the default label text style to a less accentuated one.
    3. In Miscellaneous – disable the Automatically check for update and Automatically send crash reports options.
  3. Create a new Cisco ASA device by starting New QEMU VM template from Edit – Preferences – QEMU VMs – New menu. Use the following parameters:

     

    1. Type: ASA 8.4(2)
    2. Name: ASA5520-8.4(2) or any meaningful title you choose
    3. Qemu binary: leave the default qemu.exe (v0.11.0)
    4. RAM: 1024MB or more, 1024 is the minimum
    5. Initial RAM disk (initrd): select RAM disk file, e.g. asa8420-initrd.gz
    6. Kernel image (vmlinuz): select Kernel image file, e.g. asa842-vmlinuz

I will recommend to store original OS images in other folder than that used by GNS3 for image storage. When you specify an image to be used by GNS3 a copy of that original file would be automatically copied to GNS3 binary image folder location.

After device creation, edit VM configuration and add up to 6 network adapters (network section). Leave the rest parameters untouched.

art - configuring Cisco ASA 8.4(2) on GNS3 1.3.13 - ASA device summary settings

art - configuring Cisco ASA 8.4(2) on GNS3 1.3.13 - ASA device general settings    art - configuring Cisco ASA 8.4(2) on GNS3 1.3.13 - ASA device HDD

 art - configuring Cisco ASA 8.4(2) on GNS3 1.3.13 - ASA device network    art - configuring Cisco ASA 8.4(2) on GNS3 1.3.13 - ASA device advanced settings

Note1: The steps above was successfully tested inside a Virtual Machine with Windows Server 2012R2 as a guest OS. The VM was provisioned with 2x vCPU and 8GB RAM and run on an ESXi host. Physical server equipped with Intel Xeon E5320 1.86GHz CPU.

Note2: There is no need for: Expose hardware assisted virtualization to the guest OS …No VT-x needed inside the Virtual Machine.

Use the newly created ASA device template

Now, you can use the newly created ASA device template, just drag the device to the map pane, build the topology and start it. Open console, if everything is ok you should see a typical ASA OS loading progress. After a minute, you will be faced with long awaited Cisco ASA CLI (use empty password for privileged exec mode).

You can use as many Cisco ASA instances in your projects as you want, sure no more than your hardware permit. Every time you instantiate a new Cisco ASA device, a flash disk device is created and mounted automatically for each Cisco ASA instance thus no additional steps for virtual disk creation needed. You can find the qcow virtual disk file (flash.qcow2) associated to your ASA device in project folder/project-files/qemu/dev-uiid/ folder.

Usualy, the first step before using ASA is to put a license key. If you do a simple google search you will find several freely flying license keys. Just use one of them. For me, it turned out to be ok the following activation key: activation-key 0x7212d04a 0xe041d3fe 0x1d22f820 0xea5440e4 0x8231dd9f … which unlike other keys does not hung loading progress for several minutes after activation.

A curious case for FTP passive mode over NAT on Cisco IOS router

Recently, I came across an interesting feature on Cisco IOS that mysteriously do something similar to an application inspection for FTP protocol and more that that, make modifications into replay messages from FTP server. You know, a router is a Layer 3 device that has no rights to do payload modifications, in some cases at most just several fields in IP header (e.g. TTL, src/dst IP addresses and port numbers in case of NAT). The behavior observed by me looks like oposite: a Cisco IOS router, when configured to do NAT, make specific changes on some FTP control messages. It’s like a router had borrowed something from ASA firewall inspection features.

Let’s do now some testings. For our scenario I built the following simple topology in GNS3.

a curios case for FTP passive mode over NAT on Cisco IOS router - lab topolgy

The three components used are:

  • the FTP client is a Windows 7 VM running in VMware Workstation
  • the FTP Server is a Windows Server 2008R2 VM running in VMware Workstation
  • the R1 is a Cisco IOS 7200 router with Cisco IOS version 12.4(24)T5

Note0: In my testings I used the FileZila 0.9.56 as an FTP server and Total Commander of version 8.51a with his embedded FTP client.

Note1: FTP server run in Passive Mode. My focus was only for FTP passive mode simply because this is the use case I have in my production scenario. I will not talk about FTP active mode.

On Cisco router applied the following simple configuration (interface & static inside NAT conf):

After NAT, the FTP server is seen by the FTP client by 192.168.3.20 IP address.

As you can observe an additional ACL that deny any un-NAT-ed communications is applied inside on FastEthernet0/0 (from FTP client side) – just to be sure that all sessions pass through NAT.

Now, I will start two packet capture sessions in Wireshark for both router’s interfaces and initiate a new FTP passive mode connection to FTP server.

In Passive Mode the client request Passive mode operation by issuing the PASV command over the control connection on TCP port 21. The server suggest a data port and IP address, to which the client must connect. The port number are somewhat encoded in several message fields, see the formula in the illustration below.

a curios case for FTP passive mode over NAT on Cisco IOS router - passive mode ilustration

The following screen show the capture of Passive mode acknowledge message (Passive OK, 227) packet with most important fields highlighted (src/dst ports, pasv response args).

a curios case for FTP passive mode over NAT on Cisco IOS router - packet capture at source_c

Where: 1-passive mode request, 2-passive mode acknowledge, 3-acknowledge message arguments, 4-passive IP address (server IP), 5-passive port (listener for data connection), 6-source port for data connection (ephemeral), 7-control connection server source port, 8-control connection server destination port (a value different from 6).

If we look now at the same packet but this time after it passed the router/NAT we would see, besides the destination IP change in IP header (because of NAT), also the FTP message payload modifications: specifically, in message arguments, values from 192,168,2,20 changed to 192,168,3,20. It is obviously an effect of NAT FTP protocol inspection and corresponding changes in message made by router.

a curios case for FTP passive mode over NAT on Cisco IOS router - packet capture at destination_b

A more in depth information about how this all works you can find here: ciscopress.com – Routing TCP/IP, Volume II (CCIE Professional Development) – Network Address Translation (link here), btw, one the few sources available on the net.

What if the FTP server will be set to function on a non-standard port ?

The answer is that the inspection embedded in NAT would ignore such a server. It simply do just the standard NAT section but not also the payload inspection and modification. Ironically, that was the reason why, initially, I couldn’t understand why this works perfect in lab but not also in production. It appeared to be that the non-standard port applied on prod. FTP server was the culprit.

The next question then arise: What needs to be done for NAT FTP inspection work for non-standard FTP ports ?

More by accident than intentionally I found this cool Cisco article: cisco.com – Using Non-Standard FTP Port Numbers with NAT (link here), that explain what to do – not so much, just two additional configurations. In my case these would be:

First, we create a standard ACL that would include the FTP server’s IP address (before NAT), and second, we would teach the IOS NAT to inspect on non-standard port number (in our case 5555) for that IP. If more than one FTP servers are used then add those to ACL too and if more than one tcp port numbers are used for control connections then specify those one by one separated by comma in second configuration command.

Bellow, you can see the ftp connection logs before (first section) and after (second section) the above configuration wad applied and FTP server configured to work with non-standard port number.

a curios case for FTP passive mode over NAT on Cisco IOS router - ftp client logs

As can be observed, after applying configuration, the FTP server answer message arrive already modified with NATed (global inside) server IP address in payload.

How to deliberately disable NAT inspection?

It can be done only per NAT entry (no global configuration), by specifying no-payload keyword at the end of NAT entry definition command.

Why is so important for IP addr. in pasv answer to be fixed to FTP server’s NATed (global inside) value?

Well, actually the importance depends on how FTP client are configured/build. The truth is that an FTP client can work either in restrictive or in less restrictive mode (can’t remember from where I got the terms, they may be wrong) and depending on that, ignore or not the value provided in pasv answer message. An FTP client configured for restrictive mode will try a data connection only to IP address suggested in pasv answer, if the de connection could not be established the FTP session will fail. At the oposite side, an FTP client in less restrictive mode would simply ignore the IP address suggested in pasv answer and would try instead the IP address used initially for data connection (the IP address that you configure in FTP client. Obviously, the global inside address).

So, if you have an FTP client configured/build for restrictive mode then is important to have NAT inspection doing IP address exchange in pasv answer message. Again, absolutely by accident, I found that older versions of Total Commander use the restrictive mode for their FTP client (7.04a in my particular case). It was an opportunity for a little test with such a client. Bellow, you can see the FTP client connection logs for (a) with NAT inspection and (b) without it and the final FTP session status. 

a curios case for FTP passive mode over NAT on Cisco IOS router - ftp client in restrictive mode

Clearly, for FTP session to succeed, a NAT FTP inspection that will do also translations in pasv answer message payload, needs to be in place.

TCP Performance. Why I couldn’t reach expected speed?

After you read this article you will find answers or suggestions for following questions:

  • I have 1Gbps NIC on PC and Server. They are on the same network, all links between are 1Gbps but our throughput is low.
  • TCP header should have 20 bytes, but I saw headers with 32 or 40 bytes. What means those values?
  • I heard about different congestion control mechanisms. What is the difference between them?

TCP connection is established between two end hosts. Doesn't matter how many routers, networks, switches you have between. TCP will ask remote host application to get/transmit data. This means that TCP interact directly with application on local and remote host. For example we open browser on our PC and enter in the address field www.google.com. This means that we are asking web page located on the google's server hosted by some hosting application, like apache2 or something else.

Let's take a look on the TCP header:

Source: https://ciscoskills.net/2011/02/25/understanding-tcp/tcpheader/

In the TCP header we could find answer why sometimes we could see TCP header with more than 20 bytes. Options allow us to extend header up to 52 bytes. Usually we could see bigger header in the TCP three way handshake, after that communication has standard header.

TCP is called reliable protocol and connection oriented. Let's review what means reliable protocol. TCP has on the header fields called Sequence number (seq) and Acknowledgement number (ack). Sequence numbers are generated in the random way at the beginning of TCP communication. Acknowledgement number is used to confirm that we get data send by sender. In case we don't have confirmation data will be retransmitted. Let's see how combination of seq and ack works as concept in following example:

First 3 packets in the communication are related tp three way handshake. You could note that start sequence number on the PC and Server are different (they could be the same). We set them different just to avoid confusion. This example shows us download of example.zip file from server. Server has to divide file on segments to send them via Internet. How many segments will be? This Depends on MSS (Maximum segment size) and size of the file.

Acknowledgement number has to confirm that we get data. For example server sends us two segments with total size 1500 bytes. Server start sequence number was 401, so 401 + 1500 bytes will be 1901. This means that server expect to get segment with ack 1901+1=1902. Server will keep sent segments without acknowledgement in his tx buffer. When server get ack about received data he release segments from tx buffer (keep in mind tx buffer is limited in size). In the same way we will get Ack number for the last sent segment.

Why PC acknowledge first two segments in one hit and third in separate ack? Server task is to send as much as he could segments, PC task is to advertise as fast as he could already received segments. So you could have 1 ACK segments per 50 segments and after that 1 ACK after next 5 segments.

TCP Performance

TCP could adapt speed of transfer to the link condition

Let's go back to the TCP header. There we have very important field called Window. This field has crucial impact on performance and speed transfer. Let's see why. We have a transfer sample between server and client. Let's suppose that client announce that his window is 14600 bytes (Window = Receive buffer size). This means that server could send segments with total size up to 14600 bytes until he has to stop and wait for ack message. Usually client will send ack faster and window will not become full. In case window become full server stop communication until client will say that he is ready to get new data. So in other words bigger window require less ack packets and increase throughput. Window field is limited to 16 bits, which means 2^16 =65535 bytes.

Why window field vary from packet to packet?

When we establish TCP connection window field has low value. This value is linked to default mss of OS multiplied to 2,3 or 10. In short time window value increase very fast up to saturation. Window is increased after each ACK sent by receiver. This procedure is called TCP slow start.

Here we have a formula which shows us what is maximum speed:

MaxSpeed = Window * 8 / Delay in sec.

65535 * 8 / 0.001 sec = 524 Mbps.

Most interesting in this formula is dependency of delay. When your server is far and between you is present slow slow link this means that speed will dramatically decrease.

Window field will decrease in case of lost packets, this will be done very fast. In case of too many lost packets TCP connection will be simply closed with error.

TCP Congestion control mechanism

Window, sequence number and acknowledgement number fields are responsible for the speed control. Server cannot push too much data on the client, this could overhead RX buffer and all new packets will be dropped (unacceptable situation for the TCP communication). All those specification were collected in the TCP congestion control mechanism called TCP Reno. In our days we have links, NICs, OS with much more capacity this is why it is required to do changes in the TCP stack inside of operating systems. See here list of new proposed mechanism. For example starting Linux kernel 2.6.19 up to kernel 3.1 default congestion mechanism inside of Linux is CUBIC.

Are the new requirements for the speed control change TCP header? No, just few new options will be added and TCP stack inside of operating system will be changed accordingly.

We said that TCP is connection oriented which means that before to start exchange of data we have to negotiate and answer following questions:

  • Are both hosts ready to exchange data between their applications?
  • Which additional options support both hosts?
  • At the end of communication we have to close connection. Operating systems limit maximum number of connections to avoid overhead of the system.

Which options could be interesting for us:

  • Selective Acknowledgement
  • Windows Scaling
  • Timestamp options

Let's look inside of packet captured by wireshark (good bless people who are working on this application).

Please pay attention that information placed between brackets in the wireshark is not present in real packet, it is info added by wireshark after analysis to help us understand communication.Different wireshark version will treat in different way this information.

Selective Acknowledgement

We said that Server should keep in his TX buffer sent segments until he would get Ack for data.Next question it is how data will de retransmitted in case of lost packet? Let's suppose that PC announced Window 14000 bytes. Server is sending 14 segments each by 1000 bytes. We lost segment #5 and #8. Classical TCP have to retransmit full window (all 14 segments). Selective ACK suport allow sender to send only #5 and #8. Also this allow server to release ack packets. I will give an example how it works little bit latter.

Window Scaling

Let's go back to the speed formula MaxSpeed = Window * 8 / Delay in sec. When we have Long Fat Networks (networks with big delay, like satellite communication) we could reach 600ms delay. This means that maximum speed through this network could be 65535 * 8 /0.6= 0.8 Mbps :). It could be strange to pay huge amount of money to get less than 1 Mbps just because of TCP limitation. Window Scaling option allow us to multiply announced window with scalar factor and get calculated window size. I will not explain how is calculated at which value multiply window, see RFC 1323 for more details. I just want to add that window scaling allow us to get up to 1 GB of RX Window without ack :).

Timestamp

Sequence and acknowledgement number are limited by in the TCP header by 32 bits. Together with ip source and destination those allow us to distinguish which application has to receive specific segment in the network. In High loaded servers this is not enough. It could be that segment will be assigned to wrong TCP stream when we reach 1 Gbps traffic. Timestamp added to the packet header allow us to get in game one more unique parameter to distinguish flow between them.

TCP Fast Retransmission

TCP is reliable protocol. He has to retransmit lost segments. How much time has to wait host until resend lost segment? Regular timer could be 2 sec. It is too much for high performance application. How to deal with that? We have ack segments which inform us how much data receiver got. Let's suppose that Server sent 14 segments. Segment #7 was lost. Receiver will inform Server that he got data up to segment #6 even in his buffer more data are present. Receiver will send again and again that he received up to segment #6. All new segments will be received and buffered. After third duplicate ACK server will think: "Hmmm.. Maybe he indeed didn't get 7th segment. Let me send it one more time.." (of course only in case selective ack is supported). In case of 1 ms delay between Server and PC retransmission could take 1.5 ms (regular timer 2 sec). Really impressive, isn't ? When delay between server and PC is more we could attest hundreds of DUP ACK messages in the capture (it should be like this!). For example 80 ms delay creates in the capture 80 DUP ack messages (it was around 80 on different tests).

Why not to adjust regular TCP timer for retransmission to relevant value instead of fast retransmission?

RFC 6582 is trying to adjust RTO (Recovery time Objective). We have different network types (Wi-Fi, Satellite, Fiber…). They have different delay and RTO couldn't be equal for all them. Right, let's measure RTT of each TCP segment and adjust RTO based on that measurement. How to do it ? Take a look in the RFC :)

Now we are ready to see full picture of TCP Congestion Control Mechanism:

Note: In this sample I did one mistake. After receiver notify lost packet he has to decarese Window Size.

What for information described in this article could be useful for us?

We still don't have answer for one objective of this article:

  • I have 1Gbps NIC on PC and Server. They are on the same network, all links between are 1Gbps but our throughput is low.

We have checked everything on the network, but issue is still present. Let's take a look on the traffic sniffer:

So in this graph we could see that our PC announce very low window size together with latency this creates average speed ~21Mbps. I observed that speed depends on the application used in the communication. For example built-in FTP client in Windows 7 got 35 Mbps, at the same conditions Filezilla FTP client got 140 Mbps. Try to use another application or adjust parameter is the operating system (do it carefully, could damage another things). Following image shows available parameters for the TCP in Centos6.5.

More often TCP performance could be affected by CPU utilization spike on the sender/receiver, buffer overload.. Traffic sniffer could help us to find when it happened.

Wireshark graph shows us that communication stops for several seconds. Let's explain what happened in the capture on that time:

Our PC for some reason decreases his window from 5840 bytes to 2920 bytes. Server sends last two segments (each 1460 bytes) and has to stop until ack from the PC. PC is sending last segment where window value is 0. This means stop sending traffic I couldn't get more. Server will wait and try from time to time asking PC if we could continue send traffic. Only after 6 seconds client will open again window and get remain info. Why PC closed window? You have to analyze logs on the PC to find reason.

Links:

  1. TCP congestion control – https://en.wikipedia.org/wiki/TCP_congestion_control
  2. Wireshark application –  www.wireshark.org/
  3. RFC 1323 TCP Extensions for High Performance – www.ietf.org/rfc/rfc1323.txt
  4. RFC 6582  The NewReno Modification to TCP's Fast Recovery Algorithm – https://tools.ietf.org/html/rfc6582

How to setup Cisco ASDM in Demo mode

Today, I’ve encountered some issues during installing Cisco ASDM in Demo mode. In this post I will address this issues and show a step by step instruction on how to successfully setup ASDM for Demo mode.

In my attempts, I started by installing the lattest available versions for ASDM Demo (ASDM Demo 7.3.1) and Java JRE (Java 8 update 91) but finally got an unworkable setup. Every time trying to start demo mode a generic error that state that Demo software is not installed popping up (screen below).

How To setup ASDM demo mode - error mesage

Furthermore, if you go to application folder in Program Files (x86) you will see an empty ASDM\Demo folder, as like Demo mode not even installed.

After several attempts, I haven’t found a better solution than to downgrade my Java JRE (8u91) to the previous major release (lattest update): Java 7 update 72. Also, at least when you start setup process you must have a 32 bit version of Java installed.

To complete a Cisco ASDM setup in Demo mode:

  1. Download the lattest available Cisco ASDM Demo setup file. For this, go to Cisco download page at Products – Security – Firewalls – Firewall Management – Adaptive Security Device Manager – Adaptive Security Appliance (ASA) Device Manager and search through the ASDM versions available the latest one that have the word demo in setup (msi) file title. The release policy for ASDM demo don’t coincide with that for ASDM. At the moment of this writing the lattest available ASDM Demo was: ASDM Demo 7.3.1. For download to succeed you will need a service contract associated with your cisco.com login, otherwise a simple googling will reveal a leaked image somewhere in Internet.
  2. Download the latest available Java JRE 7 release (Java 7 update 72), both for 32 and 64 bit with 32 bit being mandatory (setup files are jre-7u72-windows-i586.exe and respectively jre-7u72-windows-x64.exe). Install both versions, these will function perfect together.
  3. Launch ASDM Demo setup and go through a banal installation wizard. The ASDM Demo 7.3.1 setup will install also the ASDM-IDM Launcher of version 1.5(73) so if you have a newer Launcher already installed it will be overlapped. If you later try to connect with this older Launcher to an updated ASA ASDM you will prompt for Launcher update. To avoid this version swapping back and forward I will recommend to setup DEMO mode somewhere on another PC, perhaps on a Virtual Box/VMware Player VM.

Note0: The steps above was successfully tested in a Windows Server 2012R2 OS Virtual Machine.

Note1: For a guide on how to disable Java Update to proceed automatically you can read here.  Simple unchecking the Automatically Updates from Java Control Panel is not enough you will need edit specific registry key.

If everything succeeded, your ASDM\Demo folder in Program Files (x86) should be full with plenty of files:

How To setup ASDM demo mode - demo folder

Now, we can start using Cisco ASDM in DEMO mode: start ASDM Launcher (icon on your desktop) – check Run in Demo Mode:

How To setup ASDM demo mode - launcher for demo mode

Select the preferred configuration, and click OK, ASDM Demo mode should start. In the above screen note the Device IP Address/Name field automatically filled with a localhost address (not appear on first run).

How To setup ASDM demo mode - asdm demo started

Now, you can start gamming with an imaginary topology with configured ASA devices.

 

How to configure ASA for ASDM access

It this short post I will go through the steps of configuring ASDM access on an ASA device. I will use the ASAv 9.5.2 appliance just configured for GNS3 in previous post.

Copy the ASDM image to ASAv appliance

First, we need to copy a compatible ASDM image to ASAv internal storage. Therefor:

  1. Go to Cisco Download Software portal at Products > Security > Firewalls > Firewall Management > Adaptive Security Device Manager and download a compatible ASDM image for your ASA device. For download to success you will need a service contract associated with your cisco.com profile otherwise try a simple Internet search for a leaked image. Verify compatibility by consulting the Cisco ASA compatibility (link) article. For my ASAv version 9.5.2 an ASDM version 7.5.2.153 will compatible and sufficient.
  1. In GNS3, build a simple topology that will connect ASAv to some external network. To do that, connect one interface from ASAv to a cloud object configured to be linked to one of the host interface – for this purpose I usualy use a simple loopback adapter (for how to install such a one, read this technet article. reboot required). Because the ASA can’t connect directly to a cloud object a transit synthetic switch needs to be added. At this step, your topology should look like this:

How To configure ASA for ASDM access - topology view

Note0: Ethernet0 on ASA as presented by GNS3 correspond to Mangement0/0 intf seen from inside the device.

Note1: For a better look, changed the symbol/hostname used for cloud representation.

  1. On host computer start your favorite TFTP daemon (for this purpose I use tftpd32 from tftpd32.jounin.net. Configure the daemon directory and listening interface, additionally verify you host firewall to allow tftp protocol.

How To configure ASA for ASDM access - tftpd32 config

  1. Start the ASAv device and open the serial console. Configure interface IP settings, verify connectivity and copy the ASDM image to ASAv internal storage:

How To configure ASA for ASDM access - intf. configuration

A copy process should now begin. The progress seems to be less rapid than expected (in my case a top was the 60kbps) which could be because of unlicensed state of ASAv. In essence not a big problem, just wait for 3-5 minute for operation to complete. For confirmation do a dir command:

How To configure ASA for ASDM access - dir flash

Configure ASAv for ASDM access

Now it’s time to configure ASAv for ASDM access. Execute the following commands:

First two lines configure authentication, in this particular case against the local user database, second group of two commands enable HTTPS server and access from 192.168.49.0/24 network via mgmt interface (Management0/0) an the last command tell the firewall to use asdm-752-153.bin image for ASDM access.

Next, switch to your browser and try to open https for management interface https://192.168.49.100. If everything is ok, a security certificate error should appear in your browser, confirm the certificate exception to go forward. You should see a page like this:

How To configure ASA for ASDM access - ASDM welcome page

From this point you have two options: (a) via Java Plugins or (b) through ASDM Launcher. My preference is to use the ASDM Launcher. First install the ASDM Launcher – after click Install ASDM Launcher and successfully authentication a setup file will be made available for download, second start ASDM Launcher (icon on your desktop should be already present).

In ASDM Launcher authentication window, put the ASAv IP address and the authentication credentials.

How To configure ASA for ASDM access - ASDM launcher

Finally, after loading ASAv configuration, ASDM application should start:

How To configure ASA for ASDM access - ASDM view

Configuring Cisco ASAv 9.x on GNS3 1.4.x

Recently I went through an interesting experience of Cisco ASA setup in GNS3. I must say it was a real challenge, but finally, not an impossible task. There is a lot of particularities you must take into account, all depending from ASA version to GNS3 release. In this post, I will focus on how to configure an ASAv firewall to run as a QEMU VM in new GNS3 version suite 1.4.x. As of date of this writing I was able to access ASAv image version 9.5(2.204) and the GNS3 1.4.5 setup.

The 1.4.x suite of GNS3 is a relatively new appearing into the scene, and how is typical with new software releases, surely expected to have some bugs and/or incompatibilities. That’s why my first attempts was made in previous software suite of 1.3.x version (actually 1.3.13 version). It was a wrong way, because how I realized soon, for ASAv to be configured, a VNC console should be attached, or in 1.3.13 I didn’t found how to do that (is not excluded that I missed something). As a consequence, I quickly switched to newest GNS3 1.4.5 version. The following text assume a newly setup of GNS3 version 1.4.5 with no additional settings, except maybe only for the default location for project and binary images folder – I prefer to reconfigure these on C:\GNS3\Projects and respectively C:\GNS3\Images.

Cisco ASA virtual appliance (ASAv)

Cisco ASAv is a re-imaged version of Cisco ASA specifically designed to run as a VM on top of some hypervisor. In fact, the same ASA code is running, but in different form factor. There are versions for vSphere, Hyper-V and KVM. Just because GNS3 use QEMU as a VM emulator we will employ the KVM image of ASAv. By the way, ASAv is the image Cisco use in their notable virtual labs VIRL. Not all ASA versions are available in a VM format – I suppose only those starting with 9.x, thereby if you want to try some older versions, e.g. popular ASA 8.4(2), you will need to experience another approach (a new article devoted to this subject should come). It’s worth noting that the ASAv have some limitations compared to classical ASA, in particular you wouldn’t be able to build firewall clusters (failover or A/A), test multiple context mode feature or play with Etherchannel. For this scenarios, I usualy use an 8.4(2) ASA setup – which, by the way should run only in QEMU 0.11.0 which in turn can’t be started in GNS 1.4.x, only in previous suite 1.3.x !!!.

So, before we start we need to obtain somewhat the ASAv image. If you are fortunate enough to have access to Cisco downloads (a service contract associated with your profile is needed) then just go to cisco.com – All downloads – Products – Security – Firewalls – Adaptive Security Appliances (ASA) – Adaptive Security Virtual Appliance (ASAv) and download the qcow2 (KVM) image of ASAv for your preferred version.

configuring Cisco ASAv 9.x on GNS3 1.4.x - download page for Cisco ASAv

In case you do not have access to official Cisco downloads, yet I recommend to try a simple Internet search, good chances are to find somewhere a leaked image (usualy on some China resources). To be honest, I can’t understand why Cisco restrict downloads to this type of software, anyway, next after setup you will need a license key to go over the limitations of unlicensed state of appliance (bandwith limitation to 100kbps). It would be fine if Cisco would allow download and free use of appliance in unlicensed state, respectively for production usage a suitable license should be bought.

Configuring ASAv template on GNS3

A step by step guide follow:

  1. Start new QEMU VM Template wizard with following parameters:

    1. Type: Default
    2. Name: ASAv-8.5(2.204) or any meaningful title
    3. Qemu binary: qemu-system-x86_64w.exe (v2.4.0)
    4. RAM: 2048 MB
    5. Disk Image (hda): C:\GNS3\images\QEMU\asav952-204.qcow2

Note: I will recommend to store original OS images in other folder than that used by GNS3 for image storage. When you specify an image to be used by GNS3 a copy of that original file would be automatically copied to GNS3 binary image folder location.

  1. Edit newly created QEMU Template:

    1. General settings – Symbol :/symbols/asa.svg
    2. General settings – Category: Security Devices
    3. General settings – Console Type: VNC

Note0: in my testing, I tried to change vCPUs from 1 to 4, but nothing more than 1514 Illegal Instruction (core dumped) … error message got in ASAv, hence don’t touch that value, we will set the number of vCPUs in other place for ASAv to be an SMP virtual machine.

Note1: Switching the console to VNC type one it’s like directly connect with a keyboard and a monitor to the virtual machine. Initial ASAv configuration don’t allow access to the serial console port so at least at this stage, the only possible option is VNC. Don’t forget, the ASAv was designed to play in a VM with a full console. Even so, we will configure serial console port to ASAv as well.

  1. Network – Adapters: 6x (default e1000 type)
  2. Advanced Settings – Additional settings – Options: -cpu Haswell -smp 4,sockets=4,cores=1,threads=1

Note0: I successful used this string for all my Intel CPU. The microarchitecture (Haswell, Nehalem and so on) seems to no matter – successfully ran on different CPU generation with no problems. For AMD CPUs, community recommend to use (haven’t tested): -cpu Opteron_G5 -smp 4,sockets=4,cores=1,threads=1

Note1: the default option’s value: –nographic, should be cleared. This will be guarantee an automatic VNC console opening (for non-linked mode VM operation).

  1. (Optional) Activate CPU throttling – Percentage of CPU allowed: 80%
  2. Advanced Settings – uncheck: Use as a linked base VM.

configuring Cisco ASAv 9.x on GNS3 1.4.x - qemu ASAv template - summary

configuring Cisco ASAv 9.x on GNS3 1.4.x - qemu ASAv template - general settings    configuring Cisco ASAv 9.x on GNS3 1.4.x - qemu ASAv template - hdd

configuring Cisco ASAv 9.x on GNS3 1.4.x - qemu ASAv template - network    configuring Cisco ASAv 9.x on GNS3 1.4.x - qemu ASAv template - advanced settings

I think I will provide some additional inputs about the setting named: Use as a linked base VM. By default, QEMU VMs works as a linked VM which means that every time you create a new QEMU VM (in our case ASAv) in your project, a linked virtual disk is created to the original qcow2 image. All the modifications are thus recorded in that new file but yet unmodified block are read from original image. Through this, we can create hundreds of new QEMU VMs without needing to clone the virtual disk (that’s the similar to the technology used in VDI). Given the fact that during the life of an ASAv VM, disk modifications are really very few, results that the disk overhead created by each new ASAv are truly negligible. If you disable linked VM mode (uncheck the: Use as a linked base VM) the QEMU VM will interact directly with original qcow2 virtual disk (all writes will be recorded here). As a consequence a single QEMU VMs from this template can be started (just try to drag and drop a second ASAv to workspace and you will see an error message).

Why then we intentionally disabled linked base VM mode? First off, we need this only during ASAv template making and after this we will switch back to linked mode. Our interest is to do a series of configuration changes (first boot, serial console, ASDM image upload) in the original image file which we want to keep in all new ASAv instances created from this template.

Surely, the same results can be achieved by making the template in linked mode (linked qcow2 virtual disk) and then committing all the changes to the original qcow2 image via qemu-img.exe tool, but, I think it is harder. Just disabling and then re-enabling the VM’s linked mode settings seems to be much easier … the choice is yours.

To check the virtual disk that is mounted to QEMU VM just drag a new ASAv to an empty project, right click ASAv device – and choose show in file manager. An explorer window to qcow2 image opens – with linked mode disabled this would be the template image asav952-204.qcow2 located in binary image folder, whereas for linked mode this would be a qcow2 image (somewhere in project’s folder) linked to the original template – base virtual disk image. Also, additionaly you can check what qcow2 images are involved via Windows resource monitor – CPU – Associated Handles – filter by QEMU string.

  1. Drag a new instance of ASAv 9.5(2.204) to the working space on an empty project in GNS3. No topology are needed to continue, just single, unconnected ASAv device.
  2. Power-ON newly instantiated ASAv device (right-click – start) and immediately open the console (right-click – console).  In opened VNC terminal a loading progress (Linux) can pe observed.
  3. On Boot Loader phase choose the option: bootflash:/asa952-204-smp-k8.bin with no configuration load (anyway no configuration yet exists).

configuring Cisco ASAv 9.x on GNS3 1.4.x - bootloader

  1. In the meantime, it would be interesting to do some analyzing in Resource Monitor. First, to confirm the SMP nature of started QEMU VM look at the number of threads/CPU associated with qemu-system-x86_64w.exe process (CPU – Processes) – should be more than 4x thread/CPU in use, and second, to confirm the non-Linked mode of operation for the ASAv VM do a search in Associated Handles for a qemu key (CPU-Associated Handles) – in non-Linked mode, the VM should interact directly with the original qcow2 image: asav952-204.qcow2 (a screen is inserted below).

screen resource monitor - qemu treads plus handler not linked mode - 2

At the command prompt the number of vCPU can be checked by the show cpu usage commmad:

ciscoasa# sh cpu usage
CPU utilization for 5 seconds = 1%; 1 minute: 1%; 5 minutes: 0%

Virtual platform CPU resources
------------------------------
Number of vCPUs              :     4
Number of allowed vCPUs      :     0
vCPU Status                  :  Noncompliant: Over-provisioned

  1. If you carefully track the booting progress you will see that the appliance will discover that it starts for the first time (Initial bootup detected …) and for the system variables to be applied an automatic reboot will come.  So first time booting will end up with an automatic reboot. On the second boot, also choose the option with no configuration load in Bootloader Dialog. First and second time booting could take some time to progress so be patient and wait them to complete – sometimes it may seem that the appliance hung, try to wait several minutes before doing a forced powering off.
  2. If everything goes smoothly, after the second boot, you should reach the traditional Cisco command line prompter (empty password for privileged mode). At this stage, we will enable the serial console for the appliance. By default, the ASAv works only with traditional VM console (monitor/keyboard directly connected to x86 hardware) and additional steps needed to enable console via serial ports. More about that you can read here ASAv Quick Start Guide, 9.5, section Configure a Network Serial Console Port.

For serial console to be on, a file named use_ttyS0 should exist in root of disk0. It doesn’t matter the content, just to be present. The simplest mode to create such a file is to make a copy of an existing file – the documentation suggest to clone from coredump.cfg file, like shown below:

ciscoasa(config)# cd coredumpinfo

ciscoasa(config)# copy coredump.cfg disk0:/use_ttyS0

  1. Theoretically, here, we can do also some additional configurations, one that we want to keep in all the ASAv instances derived from this template. For example, we can copy here the ASDM image to disk0 to not be bothered with that in the future. Anyway, I will skip this step.
  2. Reload de appliance (type reload in privileged mode). You will see that the command prompt can’t anymore be accessed via de VNC console. I mean, the console will open, but, at one moment the interaction will be handover to the serial console and no more activity going to be possible by VNC. The last message recorded in VNC confirm that: Lina to use serial port /dev/ttyS0 for console IO.

configuring Cisco ASAv 9.x on GNS3 1.4.x - qemu ASAv template - console handover

  1. Now, after all the modifications to the ASAv image, we can switch the template back to his original Linked mode of operation. Also, we will switch the console settings to telnet type. Do the configuration changes in template settings, not in ASAv instance. The ASAv device from our temporary project can be safety removed, it has already done his job.

Using newly created ASAv template

To use the newly created ASAv template, just drag the template icon to the workspace, do your connections and power-on the device. You can use multiple ASAv devices running simultaneous with no problem, on my PC (i7-4970s CPU with12GB RAM) I ran five concurrent instances, all started ok and became usable shortly (less than 1 min).

Just because we don’t mention –nographic in template’s Advanced Settings – Additional Settings the VNC console will automatically open every time you start the device. If you close that window, the appliance will power-off automatically. The VNC console don’t interfere with serial console which you can open via context menu. If you add the -nographic option, the VM will start silent without a VNC console. Anyway, my preference is to leave the VNC console to open automatically, at least for the begging, just to have an additional visibility of the process.

After you load the ASAv device, you will periodically be announced by a missing license warning message: Warning: ASAv platform license state is Unlicensed … It is because the appliance don’t have a license key applied and it works in unlicensed state. As mentioned above, for lab and test scenarios, an unlicensed state are more than sufficient. In this state, you will get all the ASAv features but at the same time be limited to 100 Kbps interface bandwith.

It is interesting to see what virtual disks files are involved for an ASAv device started from our completed template. Beacause the template was configured as a Linked Mode VM, a linked virtual disk plus the base disk should be used, a fact confirmet by the screen below: 

configuring Cisco ASAv 9.x on GNS3 1.4.x - qemu ASAv template - linked mode virtual disks

To complete the story,  bellow I insterted a screencast for the process described above (youtube link). Enjoy.

What can happen if unexpected switch is inserted into network ?

It’s interesting to notice how one of the most used protocol in LANs STP (Spanning Tree Protocol) is so weak from a security standpoint. What I want today is to put a simple test scenario build in a Packet Tracer lab that illustrate what can happen if by mistake (or by an informed hacker) and some specific switch configuration a new switch is introduced into an existing network. Let assume the network topology illustrated in picture below (a screen taken from a running simulation in Packet Tracer apps). As can be seen, the topology is based on Cisco’s layered model with dedicated switches for network core and distribution/access (actually a particular case with distribution and access layers collapsed in a single one). Usually, switches used in core are more powerful in term of performance than other – these should have sufficient capacity to switch all aggregated traffic between network modules interconnected by core (in our simple topology the network core connect user’s access switch with data center distribution/access module). The network is fully redundant and protected from single-point-of-failure – there are redundant paths as well as spare switches. As a result of redundancy the network topology inevitably forms physical loops, which without a proper handling are translated in a logical loops. As is known, logical loops ca be detrimental for network performance and operations (if not completely destructive). In such a condition, once caught, a broadcast frame will spin endless in closed loop, more, they will tend to accumulate and form what is known as a broadcast storm (in typical size network is a matter of seconds, even just by ARP requests). As a result of storm, broadcast traffic will eat precious switch/link bandwidth resources leaving less (or nothing) for legitimate traffic. Above that, other negative effects of loops are: MAC address table entries flapping, occurrence of duplicate unicast frames, flooding connected end-nodes with looped broadcast and others. 

What can happen if unexpected switch is inserted to network-networkdiagram

The role of STP protocol is to prevent logical loops creations by ensuring one logical path in a network with multiple physical paths. The STP protocol will intentionally block all off the redundant paths that could cause a loop and in case of active path failure will dynamically react by re-activating (put in forwarding state) of some previously blocked paths. The STP is a distributed protocol, which run on all switches and that use a special format frames named BPDU (Bridge Protocol Data Units) for inter-switch coordination. The logic of STP is based on following two steps: (a) a root bridge (master switch) election and (b) a spanning tree algorithm (STA) running in determination what paths should be open for forwarding and what should be left blocked. Election of Root Bridge take place by comparing bridge priority together with switch MAC address of each switch involved in a specific STP instance (values are exchanged by BPDU messages). The switch with lower value Priority/MAC will win the election. The bridge priority, is a configurable parameter and by default has the value of 32768 on all switches, if not changed in advance the only factor that influence the root bridge election remain switch MAC address. Usually, an administrator will involve and change bridge priority in such a way that a specific switch can be designated as a Root Bridge – normally one of the core switches (more on why those will come). In second stage, the STA on every switch will calculate the cost of each logical path to the root bridge, select the path with least cost and use that path for traffic forwarding, other paths that do not form a best way to Root Bridge for other switches are left in blocking state. Path costs are calculated as a sum port cost consecutive to the root bridge. Port costs is based on values predefined by IEEE standard for each possible Ethernet bandwidth (e.g. 100Mbps have a port cost of 19, 1GbE cost of 4, lower speed higher cost).

In our topology, we can already observe the results of STP protocol actions: some links are active (ports on both ends are in forwarding state, shown by green circle) and some other links are blocked (one port at one of the end of link are leaved in blocked state shown by orange color circle). In that state, there are no any of L2 logical loops. Let’s try to explain how STP did that. If briefly:

  • One of the core switches (in picture from left) are designated as a Root Bridge. That was done by setting a lower (245776) bridge priority than the default value of 32768. Without that, Root Bridge can become any of switches from the topology (actually that with lower switch MAC address table). For backup purposes, the other core switch is configured with a bridge priority (28672) greater than first switch but lower than default value. In case of primary core switch failure the second core switch will gain (trough STP election) the Root Bridge role. The Root Bridge role assignment and configured bridge priority value can be checked from the output of show spanning-tree privileged mode command (see picture below, show command on SW1).
  • Every switch in topology will calculate the cost of every path leading to the Root Bridge switch and choose the least cost one as a forwarding path. It can be seen by one active path from every switch to root bridge (e.g. for SW3 F0/2 to F0/2 on SW1). Not for this topology, but in case of multiple paths with the same cost the STP algorithm will choose the path that begin with a port whose port_priority/port_ID is lower. The path cost can be viewed from the output of show spanning-tree command (in picture below see at show command for SW3).
  • For the remaining physical links, switches at both ends will negotiate who will leave the port in blocked state (switch with higher Bridge ID). On a link, only one end is leave blocked while the other end is always is passed in forwarding state. See proof on topology screen, here, none of physical links blocked at both ends at the same time. Let’s see for example what happens on the segment between SW3 and SW4, here, SW4 have a lower Bridge ID than SW3 therefore this one will pass his port ]n forwarding state (F0/4). Given that bridge priority of both switches are the same (default value of 32768) result that the factor that make difference when comparing Bridge Priority is MAC addresses – SW4 have a MAC address lower than that of SW3.

What can happen if unexpected switch is inserted to network-show spanning tree output

If now, we will re-draw the topology so that only active paths are shown then a nice tree (hence the term Spanning Tree) can be observed (picture below, left side). As can be seen all traffic flows are aggregated in one of core switch. 

What can happen if unexpected switch is inserted to network-spanning trees

If now, intentionally or by mistake, a new switch with a lower than existed bridge priority is inserted then a suboptimal STP topology can form. Let’s suppose that the new switch with pre-configured bridge priority of 20480 (lowest in topology) is connected directly to SW3. In that case, after STP re-calculation, a new root bridge will be elected (that new switch) and another active/blocked links topology will be established. If compare this topology (shown in picture below) to initial one then differences are obvious. As show in previous picture (right side) all the traffic are now aggregated in SW3 – a switch that is less powerful than the core switches, so a inevitable bottleneck point will occur in this network.

What can happen if unexpected switch is inserted to network-networkdiagram after new switch added

If to look at cost of path from SW3 to root bridge then, now, it become more expensive, which means that path is crossing along more ports. In show spanning-tree command, the value of 57 added by 3 time cost of FastEthernet port (19 as of IEEE standard).

Obviously such scenarios should be avoided. There are techniques like BPDU guard/filter that once configured can automatically block the ports or cancel any BPDU swapping toward such unexpected switches, but this is a subject for another post so enough for today.

Spanning Tree PVID- and Type-Inconsistencies

M-am ciocnit de curând cu o problemă in care două switch-uri Ethernet refuzau să se înțeleagă unul pe celălalt. Imaginați-vă scenariul in care aveți un switch în companie, pe care îl administrați și pe care încercați sa-l interconectați cu un alt switch, sa-i zicem al unui furnizor de servicii, pe a cărui configurație nu o cunoașteți și nu aveți cum s-o schimbați. Ambele switch-uri sunt Cisco. Problema e că în anumite configurații pe switch-ul companiei comunicațiile prin respectivul interconect se întrerup, în plus uneori consola switch-ului mai și afișează mesaje cu STP inconsistencies. Până la urmă problema sa dovedit a fi de fapt un regim normal de funcționare a switch-urilor și reprezintă o funcție specifică a STP-ului prin care porturile între switch-uri sunt blocate în caz că există diferențe in configurația acestora. De exemplu în cazurile când pe un trunk dot1q, pe capete, nu coincid VLAN-urile native sau un port în trunk este interconectat cu un altul configurat ca acces mode. Ca să înțeleg mai bine respectivele procese am încercat să modelez situația intr-un laborator pe GNS3. Mai jos vedeți un intro teoretic și observațiile din lab.

Despre STP și tip-uri de STP

Fără a intra în discuții lungi legate de protocolul STP cred ar fi interesant să reamintesc doar că acesta este implementat pe toate switch-urile Ethernet și că are grijă să evite formarea buclelor logice într-o rețea cu bucle fizice, or orice rețea cu redundanță formează inevitabil bucle fizice. O buclă într-o rețea e nocivă pentru performanța acesteia dacă nu chiar distructivă și trebuie evitată cu orice preț. STP-ul e un protocol în continuă evoluție așa că în timp au existat mai multe variații, unele standarde IEEE, altele versiuni Cisco proprietary. Să revizuim cele mai importante din ele: 

  • STP-ul tradițional, descris in standardul IEEE 802.1D, minimum pe un switch Ethernet, o singură instanță pentru toate VLAN-urile pe switch (MST – Mono Spanning Tree). Pe un switch Cisco, regim implicit pe porturile configurate in mode acces.
  • Per-VLAN STP (PVST), Cisco proprietary, cu instanțe STP separate pe fiecare VLAN activ, funcționează doar pe trunk-uri Cisco ISL. Pe un switch Cisco, regim implicit pe porturile configurate ca trunk-uri ISL.
  • Per-VLAN STP (PVST+), identic cu PVST dar funcționează și pe trunk-uri 802.1q. Pe un switch Cisco, regim implicit pe porturile configurate ca trunk-uri 802.1q.
  • Rapid STP (RSTP), standard IEEE, varianta îmbunătățită pentru STP-ul tradițional 802.1D, oferă un timp de convergență mai rapid. Incorporat în 802.1D.
  • Rapid-PVST+, Cisco proprietary, varianta rapidă pentru instanțele PVST+
  • Multiple STP (MSTP), standard IEEE, evoluție STP/RSTP, permite crearea mai multor instanțe STP pe același switch, mai multe VLAN-uri pot fi asociate cu aceeași instanță MSTP.

Despre VLAN 1 și native VLAN pe un trunk dot1q

VLAN-ul cu ID-ul 1 există pre-creat pe orice switch Cisco Catalyst, nu poate fi șters și are o semnificație aparte. Pe switch-urile Cisco, VLAN 1 e folosit pentru transportul mesajelor unor protocoale de control și ca VLAN nativ in configurația implicită a unui trunk dot1q. Următoarele protocoale folosesc VLAN 1 pentru transportul propriilor mesaje: (1) CDP, (2) VTP, (3) DTP, (4) PAgP. Mesajele DTP au specificul că sunt transportate întotdeauna untagged, prin VLAN-ul nativ, care pe un trunk dot1q in configurația sa implicită coincide cu VLAN-ul 1. Celelalte protocoale folosesc VLAN 1 indiferent dacă e nativ sau nu pe trunk. În caz că nu e nativ, acestea vor fi încapsulate in frame-uri tagged (marcate cu VLAN 1). Pe un switch Cisco Catalyst, STP-ul schimbă mesaje pe toate VLAN-urile admise în trunk, inclusiv și pe VLAN-ul 1.

În acest context, de obicei, recomand studenților mei următoarele două practici: (a) să nu folosească în nici un fel VLAN-ul 1 pentru transportul de date și (b) să schimbe VLAN-ul nativ implicit din VLAN 1 într-un oarecare alt VLAN ID. VLAN-ul 1 cât și noul VLAN nativ să rămână rezervate exclusiv pentru protocoale de control.

Despre mesaje PVST+ BPDU

Funcționarea protocolului STP (PVST+) se bazează pe un schimb continuu de mesaje intre switch-urile topologiei de rețea. Mesajele STP sunt numite mesaje BPDU (Bridge Protocol Data Unit) și sunt folosite de switch-uri pentru stabilirea arborelui de Spanning Tree. Din arbore se deduc porturile pentru a fi deconectate astfel încât să se excludă formarea buclelor L2. În variantele per-VLAN a STP-ului (PVST/PVST+), grupuri de mesaje BPDU sunt transmise pentru fiecare instanță/VLAN de STP.

Următoarele reguli se respectă la transmiterea mesajelor BPDU PVST+ pe un trunk 802.1q: 

  • Pentru VLAN-ul nativ, switch-ul va transmite mesaje BPDU in formatul STP-ului clasic, untagged și pe o adresa MAC rezervată STP-ului: 0180.c200.0000 (IEEE STP MAC).
  • Pentru VLAN-ul nativ, switch-ul va dubla fiecare mesaj din punctul precedent cu BPDU-uri in format PVST+, transmise untagged și pe o adresă MAC rezervată protocolului PVST+ 0100.0ccc.cccd. Acestea mai sunt numite mesaje SSTP (shared spanning tree). Diferența dintre un mesaj clasic STP și PVST+ este într-un field adițional de tip TLV (PVID) adăugat la capătului BPDU-ului. Field-ul TLV indică numărului VLAN-ului căruia îi aparține mesajul BPDU. TLV-ul e în format pe 3x octeți: (a) type – întotdeauna zero (b) length – întotdeauna 2 și (c) value – respectiv numărul VLAN-ului.
  • Pentru restul VLAN-urilor active, switch-ul va transmite mesaje in format PVST+, tagged cu numărul VLAN-ului și cu destinație pe adresa MAC rezervată protocolului PVST+ 0100.0ccc.cccd.

BPDU-urile menționate în primul punct sunt de format STP și pe o adresa destinație STP tocmai pentru a asigura compatibilitatea intre un switch Cisco cu PVST+ și un switch clasic IEEE ce funcționează pe o singură instanță STP. O instanță STP (MST) pe un switch IEEE se asociază cu instanța STP a VLAN-ului nativ pe un switch PVST+ într-un așa numit domain CST (Common Spanning Tree).

BPDU-urile SSTP au rolul de a detecta diferențele dintre configurațiile porturilor pe capetele unui trunk, mai precis deosebiri dintre numărul VLAN-ului nativ. Diferențele intre VLAN-urile native la capetele unui trunk duc pe deoparte la apariția fenomenului de VLAN hopping (frame-urile dintr-un VLAN sar intr-un alt VLAN) și pe de altă parte împiedică funcționarea corectă a STP-ului pe acele VLAN-uri, fapt ce poate provoca instabilități și bucle. Din moment ce un switch Cisco  recepționează un mesaj SSTP pe unul din porturile sale trunk, acesta va compara valoarea din TLV (PVID) cu VLAN-ul nativ configurat pe respectivul port, daca acestea nu coincid portul va fi trecut in statut inconsistent și blocat pentru forwarding de frame-uri.

BPDU-urile SSTP sunt intenționat asamblate în frame-uri cu o adresă destinație rezervată PVST-ului. Scopul e ca într-o topologie mixtă cu switch-uri IEEE și switch-uri Cisco cu PVST+ mesajele BPDU SSTP să treacă neconsumate de STP-ul din switch-ul IEEE. Dacă SSTP-ul ar avea o destinație IEEE STP MAC atunci acesta s-ar fi oprit pe primul switch IEEE fiind consumat de procesul lui STP (de fapt interpretat ca o dublură la BPDU-ul STP inițial).

STP_inconsistencies_stp_sstp_pvst_diagram

Spanning Tree PVID- and Type-Inconsistencies

Să încercăm acum să confirmăm afirmațiile de mai sus într-o topologie pe lab. Pentru simplitate vom folosi GNS-ul in care vom porni routere Cisco configurate cu module Etherswitch. Spre norocul nostru, Etherswitch-urile pe routere Cisco in GNS suportă Spanning Tree-ul și încă în varianta PVST+.

Așadar, pe un router Cisco 3600 cu Etherswitch (NM-16ESW) configuram VLAN-urile și una din interfețe în regim de trunk 802.1q. Pe trunk, re-configurăm VLAN-ul nativ din 1 într-un alt VLAN, de exemplu in VLAN 5. Din lista de dispozitive, aducem un switch sintetic (Ethernet Switch) pe care il conectam cu un port al său la trunk-ul pe Etherswitch – o facem pentru a ține interfața trunk întotdeauna up/up. Vedeți mai jos, topologia și configurația pe router.

STP_inconsistencies_lab_diagram

Acum e cazul să pornim wireshark-ul pentru o captură de pachete pe segmentul R1-Fa0/0 – SW1-1. Vedeți mai jos screen-ul cu rezultate. Din captură se observă clar cum Etherswitch-ul transmite un BPDU STP, untagged pe adresa STP-ului (pachet 1), după care dublează mesajul cu un BPDU PVST+, untagged pe adresa PVST+ și PVID=5 (pachet 3, marcat și desfășurat) și câte un BPDU pentru fiecare VLAN activ pe switch, inclusiv VLAN=1, in format PVST+ cu PVID-ul și tagg-ul 802.1q corespunzător VLAN-ului. 

STP_inconsistencies_stp_sstp_pvst_wireshark_captured

Să încercăm acum să extindem topologia din GNS3 prin a adăuga încă un router Cisco cu modul Etherswitch pe care îl vom interconecta cu primul printr-un link. Vom verifica acum reacția switch-urilor (Etherswitch) la următoarele divergențe în configurația porturilor ce le interconectează (de la R1 la R2):

  • Trunk (native 1) to Acces (VLAN 1)
  • Trunk (native 1) to Acces (VLAN 5)
  • Trunk (native 1) to Trunk (native 5)
  • Trunk (native 5) to Trunk (native 5)

Primele două cazuri dau același rezultat, port-ul access pe router-ul R2 este blocat iar in consola se afișează mesajul cu Inconsistent port type ..

STP_inconsistencies_access_mode_syslog_message_case_a

sau pentru cazul doi (diferența e doar in numărul VLAN-ului)

STP_inconsistencies_access_mode_syslog_message_case_b

din acest moment port-ul e blocat pentru comutație de frame-uri. Statutul poate fi verificat cu show spanning-tree summary din care se aflăm numărul de port-uri blocate per VLAN și show spanning-tree inconsistent ports din care găsim portul blocat.

STP_inconsistencies_access_mode_blocked_status

Rezultatul ne re-confirmă un regim normal de funcționare prin care port-urile configurate in mod acces sunt blocate automat în cazul în care primesc mesaje BPDU pe adresă SSTP or un așa mesaj poate fi primit doar dacă la capătul opus se află un port configurat in mod trunk. Porturile se blochează doar pe partea acces nu și pe trunk.

De menționat că există o excepție la regulă și poate fi observată doar pe switch-urile Catalyst – testat personal pe switch-uri reale. Pentru cazul când VLAN-ul nativ pe portul trunk coincide cu VLAN-ul pe interfața acces și ID-ul VLAN-ului este 1, portul totuși nu este blocat.

Să trecem la al treilea caz cu interfețe in trunk pe ambele capete dar cu configurații diferite pentru VLAN-ul nativ. Din moment ce unul port-uri este pornit, STP-ul pe ambele switch-uri reacționează și blochează porturile pentru amândouă VLAN-uri native. 

STP_inconsistencies_access_mode_syslog_message_case_trunk_to_trunk

ambele port-uri pe amândouă VLAN-uri sunt blocate:

STP_inconsistencies_access_mode_syslog_message_case_trunk_to_trunk_inconsisten_ports

Acum, dacă e să modificăm VLAN-urile native pe porturile trunk astfel încât să coincidă (al patrulea caz) vom observa cum porturile pe VLAN-urile respective se vor debloca automat.

Pe final, câteva referințe care pentru mine s-au dovedit a fi extrem de utile:

  1. blog.ine.com – PVST+ Explained (link)
  2. cisco.com – Troubleshooting Spanning Tree PVID- and Type-Inconsistencies (link)
  3. cciethebeginning – Differences between PVST and PVST+ (link)
  4. PVST+ BPDU frame format (link)
  5. InformIT – CCNP Practical Studies (link)

Configuration Logger pe routere Cisco

Uneori e bine să ai la îndemână istoricul instrucțiunilor ce au fost aplicate pentru modificarea configurației pe un router Cisco. Comanda show history poate să afișeze un așa istoric dar are neajunsul că e legat de sesiunea de consola/linie VTY așa că odată ieșit din terminal se pierde și istoricul. Începând cu Cisco IOS 12.3 .. se introduce funcționalitatea Configuration Change Notification and Logging, care configurată poate înregistra istoricul modificărilor de configurație independent de sesiunile in care au fost executate. Pe deasupra, aceasta poate fi configurat în așa fel încât istoricul să reziste la reload-ul de echipament. Funcția de logger de configurație poate fi utilă în troubleshooting atunci când ai de exemplu nevoie să identifici ultimele modificări de configurație ce au dus la o problemă sau să urmărești comenzile executate de mai mulți administratori în diferite sesiuni ș.a.m.d. Până la configuration logger identificarea schimbărilor de configurație era posibilă doar prin a compara linie cu linie running-config-ul curent cu o copie din trecut a configurației – incomod și fără detalii legate de ordinea instrucțiunilor și autorul lor.

Informațiile reținute de configuration logger includ:

  • comandă IOS executată
  • modul de configurație specific in care a fost executată comanda
  • numele utilizatorului (autentificat in sesiunea terminal) ce a executat comanda
  • un număr de secvență consecutiv a comenzii 

Logger-ul înregistrează doar comenzile ce au dus la modificarea configurației nu și cele de afișare sau comenzi incomplete/eronate. Comenzile sunt înregistrate într-un log circular a cărui lungime poate fi setată din timp.

Să încercăm acum o configurație, cu un simplu router Cisco cu logger de configurație setat. Vom executa câteva comenzi din contexte de utilizatori diferite (în sesiuni de consolă), după care vom afișa istoricul instrucțiunilor stocat de configuration logger. Testele de mai jos sunt executate în GNS3 pe un router 3700 cu IOS 12.4(15)T7 (C3725-ADVENTERPRISEK9-M).

configuration_logger_cisco_network_map

Așadar, pe routerul din schemă aplicăm următoarele configurații:

  1. utilizatori definiți local (mark.smith/jane.doe) și autentificare pentru linia de consola (line con 0)

  1. activare configuration logger, executată în modul de configurare archive – log config prin logging enable. Mai jos, logging size pentru lungime log și hidekeys pentru a ascunde (sub asterisc) password-urile in log.

Continue reading