Monitor Temperature and Humidity using Raspberry Pi

Today I would like to share you the way to monitor via web page temperature and humidity using raspberry Pi and Sensor for humidity and temperature. You could find a lot of similar articles in the Internet. Mine solution is just combination of several and small adjustment.

Level of moisture in our house is too high and this requires intervention from our side to reduce it. Recommended value of moisture should be between 35% and 55%. High level of moisture increase probability to get ingress on our things, clothes, walls..

Moisture level depends on air flows and temperature in your house. If in the winter temperature is below 20 Celsius degree the level of humidity will be higher than recommended value. Minimum recommended level of temperature should be 21 Celsius degrees.  In other words you need to have smart heating system and good ventilation to reduce fast humidity when you are cooking or after you used bathroom. So let’s see how to get data which we need for decisions.

We need equipment:

  • DHT22 AM2302 Digital Temperature and Humidity Sensor module Replace SHT11 SHT15, Price – 5.97 USD
  • Raspberry Pi-2 + Acrylic Enclosure Case + Heat sink, Price – 42.27 USD 

Let’s connect sensor to the raspberry. I did everything like in this link http://www.home-automation-community.com/temperature-and-humidity-from-am2302-dht22-sensor-displayed-as-chart/ except the thing that I didn’t used resistor and development plate. My sensor has resistor built in. Here is my equipment and connections:

Before to collect data let’s update our raspberry:

sudo aptget update

Note: Many steps in this tutorial require internet acces on raspberry PI (If you don't know how to do it see my previous article how to connect raspbery to the internet via Wi-Fi)

Raspberry collect data from sensor on the GPIO pin 4. To read data we will use python library called Adafruit_Python_DHT.

To install library, get the dependencies with:

sudo aptget install y buildessential pythondev git

and then download and install the library with:

mkdir -p /home/pi/sources 
cd /home/pi/sources 

git clone https://github.com/adafruit/Adafruit_Python_DHT.git 

cd Adafruit_Python_DHT 

sudo python setup.py install 

 Here we can do fast check to see that Pi is ready to collect data from sensor:

sudo /home/pi/sources/Adafruit_Python_DHT/examples/AdafruitDHT.py 2302 4

The first argument is the sensor type, it can be 11 or 22 or 2302. The second argument is the RPi GPIO pin which is connected to the sensor data pin.

Output will be like:

sudo /home/pi/sources/Adafruit_Python_DHT/examples/AdafruitDHT.py 2302 4

Temp = 24.2*C Humidity=26.3%

Let’s create sql database to collect our data

Install sqlite

sudo apt-get install sqlite3

  Create DB schema

BEGIN;

CREATE TABLE temp_hum (timestamp DATETIME, temp NUMERIC, hum NUMERIC);

COMMIT;

Great, now we have DB where we could write data. We created table called temp_hum. This table has 3 columns: timestamp, temp and hum. First column will contain actual date and time. So let’s assure that time and data is right. Use date command to check it.

If something is wrong let’s install ntpdate packet and adjust time:

sudo apt-get install ntpdate

sudo service ntpd stop

sudo ntpdate time.nist.gov

sudo service ntpd start

Last step is to adjust timezone. For me timezone is GMT+3. Find your on the internet if you don’t know which one is yours.

sudo cp /usr/share/zoneinfo/Etc/GMT+3 /etc/timezone

Let’s install and configure web server

sudo apt-get install apache2

We will use cgi-bin scripts to display collected data on web page. So let's say apache web server to use them. This step could be optional for you:

a2enmod cgi

a2enmod cgid

service apache2 restart

Move created DB to the /var/www/ folder and change owner of DB:

sudo cp templog.db /var/www/
sudo chown www-data:www-data /var/www/templog.db 

Now we have to do little bit programming or just copy ready solution from the internet (as I did, thanks this author for help). I adjusted liitle bit found code in the internet in order to get expected results.

nano /root/monitor.py

and insert following text inside:

#!/usr/bin/env python

import sqlite3
import Adafruit_DHT
import os
import time
import glob

# global variables
dbname='/var/www/templog.db'

# store the temperature and hummidity in the database
def log_temperature(temp,humm):

    conn=sqlite3.connect(dbname)
    curs=conn.cursor()

    curs.execute("INSERT INTO temp_hum values(datetime('now'), (?), (?))", (temp,humm))
    #curs.execute("INSERT INTO temp_hum values(datetime('now'), (?))", (humm,))
    # commit the changes
    conn.commit()

    conn.close()


# display the contents of the database
def display_data():

    conn=sqlite3.connect(dbname)
    curs=conn.cursor()

    for row in curs.execute("SELECT * FROM temp_hum"):
        print str(row[0])+”     “+str(row[1])+”        “+str(row[2])

    conn.close()

# get temerature and humidity
# returns None on error, or the temperature,humidity as a float
def get_temp_hum():
        try:
                humm, temp = Adafruit_DHT.read_retry(Adafruit_DHT.DHT22, 4)
                humm = round (humm, 2)
                temp = round (temp, 2)
                return humm, temp
        except:
                return None, None

# main function
# This is where the program starts
def main():
        # get the temperature from the device file
        humm, temp = get_temp_hum()
        # Store data to DB in case we have values
        if (humm != None and temp !='None'):
                log_temperature(humm, temp)
#       display_data()

if __name__=="__main__":
    main()

Let me add several comments to the code. First of all we defined database location dbname='/var/www/templog.db'. After that we defined function log_temperature to insert data in our DB. We have function called get_temp_hum to read data from the sensor using python library installed at the beginning of this article.The most important part is main function where we are calling get_temp_hum, check that we have valid data and insert them to DB using log_temperature. One more function display_data() is commented and we need it only for debug purposes. Uncomment this line and execute the script to assure that you have inserted data in the DB.

Don't forget to make file monitor.py executable:

chmod 755 /root/monitor.py

Let's collect data every 15 minutes. For this print:

crontab -e

Insert this line:

*/15 * * * * /root/monitor.py

Note: If execution of script fail for some reason try to excute script manually with sudo /root/monitor.py. Also you could monitor script execution in crontab using postfix mail service. To install it use sudo apt-get install postfix.

Last step is create script which will display data on the web page.

nano /usr/lib/cgi-bin/webgui.py

Insert there following text:

#!/usr/bin/env python
import sqlite3
import os
import sys
import cgi
import cgitb

# global variables
dbname='/var/www/templog.db'

# print the HTTP header
def printHTTPheader():
    print "Content-type: text/html\n\n"

# print the HTML head section
# arguments are the page title and the table for the chart
def printHTMLHead(title, table):
    print "<head>"
    print "    <title>"
    print title
    print "    </title>"

    print_graph_script(table)

    print "</head>"


# get data from the database
# if an interval is passed,
# return a list of records from the database
def get_data(interval):

    conn=sqlite3.connect(dbname)
    curs=conn.cursor()

    if interval == None:
        curs.execute("SELECT * FROM temp_hum")
    else:
        curs.execute("SELECT * FROM temp_hum WHERE timestamp>datetime('now','-%s hours') AND timestamp<=datetime('now')" % interval)

    rows=curs.fetchall()

    conn.close()
    return rows


# convert rows from database into a javascript table
def create_table(rows):
    chart_table=""

    for row in rows[:-1]:
        rowstr="[‘{0}’, {1}, {2}],\n”.format(str(row[0]),str(row[1]),str(row[2]))
        chart_table+=rowstr

    row=rows[-1]
    rowstr="[‘{0}’, {1}, {2}]\n”.format(str(row[0]),str(row[1]),str(row[2]))
    chart_table+=rowstr

    return chart_table


# print the javascript to generate the chart
# pass the table generated from the database info
def print_graph_script(table):

    # google chart snippet
    chart_code="""
    <script type="text/javascript" src="https://www.google.com/jsapi"></script>
    <script type="text/javascript">
      google.load("visualization", "1", {packages:[“corechart”]});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data = google.visualization.arrayToDataTable([
          [‘Time’, ‘Humidity’, ‘Temperature’],
%s
        ]);
        var options = {
          title: 'Temperature and Humidity'
        };
        var chart = new google.visualization.LineChart(document.getElementById('chart_div'));
        chart.draw(data, options);
      }
    </script>"""

    print chart_code % (table)

 


# print the div that contains the graph
def show_graph():
    print "<h2>Temperature Chart</h2>"
    print '<div id="chart_div" style="width: 900px; height: 500px;"></div>'

# connect to the db and show some stats
# argument option is the number of hours
def show_stats(option):

    conn=sqlite3.connect(dbname)
    curs=conn.cursor()

    if option is None:
        option = str(24)

    curs.execute("SELECT timestamp,max(temp) FROM temp_hum WHERE timestamp>datetime('now','-%s hour') AND timestamp<=datetime('now')" % option)
    tempmax=curs.fetchone()
    tempmax="{0}&nbsp&nbsp&nbsp{1} %".format(str(tempmax[0]),str(tempmax[1]))

    curs.execute("SELECT timestamp,min(temp) FROM temp_hum WHERE timestamp>datetime('now','-%s hour') AND timestamp<=datetime('now')" % option)
    tempmin=curs.fetchone()
    tempmin="{0}&nbsp&nbsp&nbsp{1} %".format(str(tempmin[0]),str(tempmin[1]))

    curs.execute("SELECT avg(temp) FROM temp_hum WHERE timestamp>datetime('now','-%s hour') AND timestamp<=datetime('now')" % option)
    tempavg=curs.fetchone()

    curs.execute("SELECT timestamp,max(humm) FROM temp_hum WHERE timestamp>datetime('now','-%s hour') AND timestamp<=datetime('now')" % option)
    hummmax=curs.fetchone()
    hummmax="{0}&nbsp&nbsp&nbsp{1} C".format(str(hummmax[0]),str(hummmax[1]))

    curs.execute("SELECT timestamp,min(humm) FROM temp_hum WHERE timestamp>datetime('now','-%s hour') AND timestamp<=datetime('now')" % option)
    hummmin=curs.fetchone()
    hummmin="{0}&nbsp&nbsp&nbsp{1} C".format(str(hummmin[0]),str(hummmin[1]))

    curs.execute("SELECT avg(humm) FROM temp_hum WHERE timestamp>datetime('now','-%s hour') AND timestamp<=datetime('now')" % option)
    hummavg=curs.fetchone()


    print "<hr>"


    print "<h2>Minumum Value&nbsp</h2>"
    print tempmin
    print hummmin
    print "<h2>Maximum Value</h2>"
    print tempmax
    print hummmax
    print "<h2>Average Value</h2>"
    print "%.1f" % tempavg+"%"
    print "%.1f" % hummavg+"C"
    print "<hr>"

    print "<h2>In the last hour:</h2>"
    print "<table>"
    print "<tr><td><strong>Date/Time</strong></td><td><strong>Temperature</strong><td><strong>Humidity</strong></td></td></tr>"

    rows=curs.execute("SELECT * FROM temp_hum WHERE timestamp>datetime('now','-1 hour') AND timestamp<=datetime('now')")
    for row in rows:
        tempstr="<tr><td>{0}&emsp;&emsp;</td><td>{1} %</td><td>{2} C</td></tr>".format(str(row[0]),str(row[1]),str(row[2]))
        print tempstr
    print "</table>"

    print "<hr>"

    conn.close()

def print_time_selector(option):

    print """<form action="/cgi-bin/webgui.py" method="POST">
        Show the temperature logs for
        <select name="timeinterval">"""


    if option is not None:

        if option == "6":
            print "<option value=\"6\" selected=\"selected\">the last 6 hours</option>"
        else:
            print "<option value=\"6\">the last 6 hours</option>"

        if option == "12":
            print "<option value=\"12\" selected=\"selected\">the last 12 hours</option>"
        else:
            print "<option value=\"12\">the last 12 hours</option>"

        if option == "24":
            print "<option value=\"24\" selected=\"selected\">the last 24 hours</option>"
        else:
            print "<option value=\"24\">the last 24 hours</option>"

        if option == "168":
            print "<option value=\"168\" selected=\"selected\">the last 168 hours</option>"
        else:
            print "<option value=\"168\">the last 168 hours</option>"

    else:
        print """<option value="6">the last 6 hours</option>
            <option value="12">the last 12 hours</option>
            <option value="24">the last 24 hours</option>
            <option value="168">the last 168 hours</option>"""

    print """        </select>
        <input type="submit" value="Display">
    </form>"""


# check that the option is valid
# and not an SQL injection
def validate_input(option_str):
    # check that the option string represents a number
    if option_str.isalnum():
        # check that the option is within a specific range
        if int(option_str) > 0 and int(option_str) <= 24:
            return option_str
        else:
            return None
    else:
        return None


#return the option passed to the script
def get_option():
    form=cgi.FieldStorage()
    if "timeinterval" in form:
        option = form[“timeinterval”].value
        return validate_input (option)
    else:
        return None

# main function
# This is where the program starts
def main():

    cgitb.enable()

    # get options that may have been passed to this script
    option=get_option()

    if option is None:
        option = str(24)

    # get data from the database
    records=get_data(option)

    # print the HTTP header
    printHTTPheader()

    if len(records) != 0:
        # convert the data into a table
        table=create_table(records)
    else:
        print "No data found"
        return

    # start printing the page
    print "<html>"
    # print the head section including the table
    # used by the javascript for the chart
    printHTMLHead("Raspberry Pi Temperature and Humidity Logger", table)

    # print the page body
    print "<body>"
    print "<h1>Raspberry Pi Temperature and Humidity Logger</h1>"
    print "<hr>"
    print_time_selector(option)
    show_graph()
    show_stats(option)
    print "</body>"
    print "</html>"

    sys.stdout.flush()

if __name__=="__main__":
    main()

Let me again comment the script. First of all we need to know how much hours to display in graph using get_option function. By default 24 hours will be displayed. Function get_data will collect dates from DB. If you have at least one string in the DB data will be converted create_table function to display everything on the web. Actual string is not converting data for 1 week. This is what I have to fix after.

Don't forget to make file webgui.py executable:

chmod 755 /usr/lib/cgi-bin/webgui.py

To see data in the web page open web browser on the PC in the same network with raspberry and insert in the address field:

192.168.1.10/cgi-bin/webgui.py

192.168.1.10 is the ip address of Pi

Final result look like this:

After 10 AM I moved sensor closer to the heating system. This is why we have so big changes in the graph.

Links

  1. http://www.home-automation-community.com/temperature-and-humidity-from-am2302-dht22-sensor-displayed-as-chart/
  2. https://github.com/adafruit/Adafruit_Python_DHT
  3. http://raspberrywebserver.com/cgiscripting/rpi-temperature-logger/building-a-web-user-interface-for-the-temperature-monitor.html

Connect raspberry pi to Wi-FI

Some time ago I decided to become more familiar with Internet of things. I thought about several projects, but I didn't decide what to do this is why I bought several things and forgot about them for a while.

Few days ago I decided to take a look inside. Newbies (like me) think that pieces for the project could be expensive, actually they are not so. I decided to publish what I bought and how much it cost.

  • Electronic Parts Pack KIT for ARDUINO component Resistors Switch Button WT, Price – 3.70 USD
  • Soil Hygrometer Humidity Detection Module Soil Moisture Water Sensor, Price –  1.16 USD
  • MB102 Power Supply Module 3.3V 5V+Breadboard Board 830 Point+65PCS Jumper cable, Price – 5.08 USD
  • DHT22 AM2302 Digital Temperature and Humidity Sensor module Replace SHT11 SHT15, Price – 5.97 USD
  • UNO R3 ATmega328P CH340 Mini USB Board for Compatible-Arduino NEW, Price – 3.95 USD
  • Raspberry Pi-2 + Acrylic Enclosure Case + Heat sink, Price – 42.27 USD 
  • Micro USB Charger 5V,  2A, Price – 7.64 USD
  • Temperature sensor DS18b20 in iron case, Price – 1.25 USD
  • Micro SD memory card Class 10 Size 8GB , Price – 5.17 USD
  • Wi-Fi USD Adapter EDUP EP-N8508GS, Price – 5.66 USD

Let's start with something. I went to the https://www.raspberrypi.org/ and found that I have to copy operating system to the SD card and insert SD card to the raspberry. Ok I decided to load NOOBS OS and after that select raspbian OS. I bought Class 10 SD card (fast SD Card :) ). After I inserted card in the raspberry and connect via HDMI raspberry I found that nothing is printed on screen. First attempt fail, the good idea was to consult here the list of compatible SD card before to buy something. Ok I found at home old SD card at 8 Gb and I decided to try it just for proof of concept. Second attempt was good.

To configure raspberry you have to get USB mouse and Keyboard. Nothing of those I have in the house. After several hours I found only mouse. Ok let's start with this, especially that on first stage I have only to select OS and boot raspberry. After boot I got the screen with GUI and I still don't have keyboard. Probably I could connect via ssh to OS was my idea. I could do it in two ways via LAN or WLAN. My Home router has password, so even wi-fi card is up raspberry doesn't know my password. I connected laptop to device via LAN, bring up on laptop DHCP server (tftpf32 could do it very fast), found which ip address was leased to raspberry and connects to him via putty (default user: pi, password: raspberry). Everything is easy here I thought. Let's connect raspberry to wi-fi. I have never been connect linux to wi-fi before. It's shouldn't be too difficult, especially that I am connecting linux via LAN in a day by day activities. Spent some time on google I found a lot of step-by-step instructions and different options, but no one work for me.

My Wi-Fi Card is:

pi@raspberrypi:~ $ lsusb
Bus 001 Device 004: ID 0bda:8176 Realtek Semiconductor Corp. RTL8188CUS 802.11n WLAN Adapter

My OS is:

pi@raspberrypi:~ $ cat /etc/*release
PRETTY_NAME="Raspbian GNU/Linux 8 (jessie)"
NAME="Raspbian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

After several hours I start to hate Linux and I saw that I am not single in this feeling. How can so easy task take so much time? I started my attempts with wireless-tools installed on raspbian. Command iwlist wlan0 scan provides info about my router. This means that driver for wlan0 interface was installed and interface is up. Let's try connect manually to router using: iwconfig wlan0 essid <YOUR_SSID> key s:<Your password>.  Several attemps fail. Ok let's try to use another option. I found that wpa_supplicant was created especially for people who want to use Wi-FI in secure way and has WPA2 encryption. Command wpa_passphrase <Your_SSID> <Your_Password> >> /etc/wpa_supplicant/wpa_supplicant.conf will create config file for your Network and save password in the safe way. If select wpa_supplicant method you have to fill /etc/network/interfaces with something like this:

allow-hotplug wlan0
iface wlan0 inet manual
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

Inet manual is required by wpa_supplicant even you have to use dhcp. If you want dhcp do this:

allow-hotplug wlan0
iface wlan0 inet manual
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

iface default inet dhcp

I created wpa_supplicant config for me, checked /etc/network/interfaces but I still couldn't associate with Wi-Fi. After several attempts I tried to run wpa_supplicant daemoon manually. This part I want to share with you in more details. After network service restart (/etc/init.d/networking restart) in the background is started wpa_supplicant daemoon. Command ps aux | grep wpa* provides me the following:

/sbin/wpa_supplicant -s -B -P /run/wpa_supplicant.wlan0.pid -i wlan0 -D nl80211,wext -c /etc/wpa_supplicant/wpa_supplicant.conf

This still doesn't help me. Somewhere should be the log files. Yes they are in /var/log/syslog or /var/log/messages or /var/log/wpa_supplicant.log. For me it was /var/log/syslog. Inside I found that:

nl80211: Driver does not support authentication/association or connect commands

Hmm, let's kill the process and play manually with different options, drivers of wpa_supplicant.

Sometimes I could get another error:

wlan0: CTRL-EVENT-SSID-TEMP-DISABLED id=0 ssid="Home" auth_failures=11 duration=120 reason=CONN_FAILED

I edited several times wpa_supplicant.conf file with different options and at the end I got something like this:

root@raspberrypi:/home/pi# cat /etc/wpa_supplicant/wpa_supplicant.conf
ap_scan=0
country=GB
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
        ssid="<Your_SSID>"
        proto=RSN
        scan_ssid=1
        key_mgmt=WPA-PSK
        pairwise=CCMP TKIP
        group=CCMP TKIP
        psk="<Your_Password>"
        #psk=8d0526f31a39e78dfgk;srs61ae5a1e9d0c30e8e48cc838f8541d4a8e7af259 #password generated by wpa_passphrase
}

At the beginning I had only ssid and psk key generated by wpa_passphrase. Information about key_mgmt, pairwise you could get using iwlist wlan0 scan command. Even everything seems to be logic I still couldn’t connect to the Router and get ip address from him. This is why I tried to launch manually supplicant using debug options:

/sbin/wpa_supplicant -s -B -P /run/wpa_supplicant.wlan0.pid -i wlan0 -D nl80211,wext -c /etc/wpa_supplicant/wpa_supplicant.conf -dd

Debug options and /var/log/syslog will give you information what failed for you. Each new option enabled/disabled in wpa_supplicant.conf file and drivers used in command provide new error to my investigation. It's interesting, but didn't help. Try to see in your case this.

Some instructions proposed to change /etc/network/interfaces in different way to connect raspberry to wi-fi. I have new way to investigate. I saw that someone is saying to use wpa-ssid, someone wpa-essid and different things. Why options are different? Let's try all of them :). It's good idea but this still didn't solve the issue. Windows seems to be better :) was my idea. Trying to find solution for me I found very interesting article written by mrEngman. Inside he shares his script how to configure wi-fi cards based on RTL8188CUS. I decided to read script instead to launch him (thanks author for very good comments inside). And there I found that wpa-ssid is for Router without authentication. For those with autehntication should be wpa-essid. wpa-psk should be used when you have WPA2 (most used), in case you have WEP you shold use wpa-key. Thanks to mrEngman that he decided to explain what means different options. After I edited /etc/network/interfaces like below everything started to work for me.

auto wlan0
iface wlan0 inet dhcp
wpa-essid <Your_SSID>
wpa-psk <Your_Password>

Conclusion: "Never give up!". Linux could be awful at first meet, but when you start to understand and think what is the impact of command, where to see logs, what means your options (use man pages), how to restart process manually and apply changes it becomes logic.

Important: Don't use at the same time iwconfig, wpa_supplicant. Select one method or script and debug it. Remember that processes are running in background and applied commands could affect actual results.

Links:

  1. https://www.raspberrypi.org/ 
  2. http://elinux.org/RPi_SD_cards
  3. http://tftpd32.jounin.net/
  4. http://www.raspberrypi.org/forums/viewtopic.php?t=6256

 

VRRP – start build redundant network

Redundant in the network is critical for service availability. Everyone wanted to have five "9" – 99,999% of available service. In order to get this we are buying expensive equipement, learn how works STP, RSTP, MSTP and other redundant protocols. Often we forget to make redundant obvious things. How many gateways could you configure on the PC? Always one.. You know that gateway is our exit point from the network in the big Internet. So in case this point is failing we are losing our way out.

VRRP is based on the RFC 3768 and RFC 5798. RFC 3768 is already version 2 of VRRP and is only for IPv4. RFC 5798 is already for IPv6 support. You could see that they are pretty new (2004 and 2010 year). To understand very good VRRP you could read RFCs, but I don't know too many people who like text without images :). This is why I decided to create some images and sample which could help you understand how RFC 3768 recommend to implement this standard.

VRRP goal is to provide redundancy for the gateway IP address. We will not change the way how we are configuring default gateway on the PC. So we have to do something on the gateways. VRRP will force routers to talk between them in order to see who owns Virtual IP address. Virtual IP address will be assigned as default gateway on the PCs in our network. Routers will talk via multicast ip address 224.0.0.18.

VRRP has 3 states: Init, Slave and Master. Init state is saying that VRRP is enabled on the Router but for some reason is irrelevant to start communication. For example we have to send VRRP packets via Fa0/0 interface, but this interface is down. VRRP slave means that router at the moment is backup. VRRP Master means that VRRP router owns Virtual IP. Routers have to select VRRP master. They could this based on the VRRP priority. Higher priority is better. We could configure VRRP priority from 1 to 254.

Do you remember that in ethernet LANs we have to encapsulate packets and add IP source, IP destination, Mac Source, MAC destination ? Here we have another interesting thing for VRRP. VRRP enter not only VIrtual IP but also Virtual MAC which will have following format: 00-01-5E-00-01- <GROUP_ID>. Group id value is limited from 1 to 255 (from 01 until FF in hex). Virtual IP and VIrtual MAC will be used only for the traffic which is going through routers, all other traffic will use Physical MAC and physical IP. Let's see the sample how ill work ARP protocol with VRRP:

We must have same values for 3 parameters on both routers:

  • Group ID
  • Group IP address
  • Advertisement interval

What for we need advertisement interval and how fast backup router will become VRRP master in case of failure? Here is the formula: 3*Adv.timer + skew time. Skew time formula = (256-Priority)/ 256. Skew time goal is to add time required for the message propagation in the LAN. So in case we have advertisement timer 1 sec, switchover will be done in maximum = 3*1 + (256-100)/256= 3.6 s. Pretty good result. We could decrease this value under 1 sec. I will not recommend to do that for beginners. VRRP is working with other protocol in the networks and adjustments in this area could create another issues for other protocols. 

Who is the VRRP Master?

VRRP master is router with biggest priority when VRRP preemption is enabled on all routers. Let's see one example:

When our Routers are up VRRP Master will be always R2. When R2 is down or link between R2 and switch is down VRRP master will be R1. Simple until now.

Let's suppose that we have network with big fluctuations. Routers are going down often. VRRP switchover is too often and sometimes without any reason. Let's suppose that R2 has preemption on and he is rebooting each 5 minutes. Our default gateway will go up/down every 5 minutes. R1 at the same time is stable and ok. For this case we could configure VRRP preemption off on R2. When R2 will go up after reboot he will stay in the VRRP Slave state. He lost Master state and the only way to recover it is to shutdown R1.

When our Routers are up VRRP Master will be always R2. When R2 is down or link between R2 and switch is down VRRP master will be R1. Simple until now.

Let's suppose that we have network with big fluctuations. Routers are going down often. VRRP switchover is too often and sometimes without any reason. Let's suppose that R2 has preemption on and he is reabooting each 5 minutes. Our default gateway will go up/down every 5 minutes. R1 at the same time is stable and ok. For this case we could configure VRRP preemption off on R2. When R2 will go up after reboot he will stay in the VRRP Slave state. He lost Master state and the only way to recover it is to shutdown R1.

VRRP Priority 0 and 255

RFC 3768 reserved VRRP priority 0 and 255 for two special cases. R2 is VRRP master for the network. Based on some events R2 decided to leave VRRP Master role. So he will send VRRP priority 0 which means that backup Router has to become VRRP Master immediately.

VRRP allows us to configure the same ip address for the Virtual IP and physical only one one router in the network. In this case Router where we configured the same IP address for the VIP will send VRRP packets with priority 255.

Decrease VRRP priority in some conditions

Last thing what I want to say about VRRP is related to dynamic priority. Network developers decided to adjust VRRP priority based on some important metrics. For example R2 router will monitor link to the ISP and will decrease VRRP priority when this link is down. More than that you could use Cisco IP SLA (when you have Cisco equipment) and monitor quality of the link (delay, drops…). Quality will be trigger for value of VRRP priority in your case.

In the network with big numbers of VLANs it is recommended to balance VRRP master roles between R1 and R2. For Example R1 is VRRP master for VLAN 10,20 and 30, when R2 is VRRP Master for VLAN 15 and 25. In this way you will use all links in your network. VRRP has to work very close with protocols like ARP, Proxy-ARP and STP. When you set VRRP on L3 switches and have redundant links you have to think how traffic will flow through network at the L2 and L3. In other case suboptimal routing will deprecate al benefits of protocols. STP and VRRP timers also have to be synced and this is why I recommended you not to change default timers.

During the switchover from one VRRP router to another, new master will send gratuitous ARP and announce him as new owner of the IP. This will converge network faster.

Links:

  1. https://www.ietf.org/rfc/rfc3768.txt
  2. https://tools.ietf.org/html/rfc5798

Configuring Cisco ASA 8.4(2) on GNS3 1.3.13

In one of my previous post (link here) I did a description on how to configure and run an ASAv instance on GNS3 1.4.5. In this post I will describe the steps required to configure and run a Cisco ASA 8.4(2) image on GN3 1.3.13.

Why in this case I insisted to use the previous version suite 1.3.x of GNS3 (1.3.13 at the moment of this writing) instead of the latest available 1.4.x suite ? The answer is because for ASA 8.4(2) to run successful, a QEMU emulator of version 0.11.0 needs to be in place and functional or as it seems, the QEMU 0.11.0 categorically refuse to work in GNS3 1.4.x suite (even it is installed here). Somewhere in forums found that GNS team will no longer offer support for QEMU 0.11.0 in its product. Maybe in one of the future releases but certainly not now …

Someone might ask, why not just use the 2.4.0 version of QEMU present and functional in both GNS3 versions suites ? The truth is that it works, but with a little issue, a very slow speed through device interfaces. In my testings, a simple ASDM image copying can take several hours with a high chance for device crashing. Types and number of NICs, license or whatever else doesn’t change the situation.

So, because of unavailability of QEMU 0.11.0 in GNS3 1.4.x I switched back to 1.3.x version suite. Why then, I didn’t use the 1.3.x suite in my previous article for an ASAv instance configuration ? It is because we needed the VNC console for ASAv initial configuration or that is present only in newer 1.4.x suite.

To conclude: for ASAv you will need GNS3 1.4.x (because of VNC console) and respectively for ASA 8.4(2) you will need the GNS3 1.3.11 (because of QEMU 0.11.0). At least in my testings, this offer me a stable and workable setup.

Note0: do not mix the GNS3 1.3.13 and GNS3 1.4.x on the same machine, simply because they wasn’t designed to work together, configuration that most probably lead to complete nonfunctional setup.

Why bothering also with ASA in addition to ASAv ? Doesn’t ASAv being sufficient ?

Well, the ASAv is a software designed to run in virtual infrastructure and such that, some features are no longer needed here. Take simple, what ASA doing by clustering, in virtual infrastructure with ASAv is accomplished by hypervisor’s High Availability features, multiple contexts are replaced by multiple standalone ASAv instances and so on. To be more precise, that’s the unsupported features that that the official documentation (link here) states: The ASAv does not support the following features: clustering, multiple context mode, active/active failover, EtherChannels, Shared AnyConnect Premium Licenses.

So in case you want to play with ASA clustering or multiple context mode you will need a classic ASA instance running in GNS3. ASAv is good but not always sufficient. Even so, ASAv should be your standard, especially since it is somewhat closer to VIRL style.

What images are needed to run ASA in GNS3 ?

Compared to ASAv where an original qcow (KVM) image was sufficient to configure the device in GNS3, for ASA the original bin image are not sufficient. The truth is that this original image needs to be unpacked, then some files modifications needs to be done and after that repack the content in a way suitable for QEMU. All of this can be done manually or scripted (see GNS3 forums for Cisco Image Unpacker for Windows or repack.v4.sh shell script for Linux) but imho, if you not search for a specific ASA version just do a search on the Internet for ASA 8.4(2) GNS3 files. Other versions should be also be available to. In any case, you should be left with two files: asa-vmlinuz (a Linux kernel) and asa-initrd.gz (a RAM disk file). These are the files used by QEMU in GNS3.

How to configure Cisco ASA 8.4(2) in GNS3 1.3.13 ?

  1. Download the latest version for 1.3.x GNS3 version suite. To do that, go to GNS3 web site – avoid the download button witch will direct you only to GNS3 1.4.x download, instead, click on software  (top bar menu) – on the left, written with small font size, click on To download Version 1.3.13 of GNS3 Click Here (see screenshot below). After, authentication a download for GNS3-1.3.13-all-in-one.exe file should start.

art - configuring Cisco ASA 8.4(2) on GNS3 1.3.13 - download page

  1. Install the 1.3.13 GNS3. No rocket science here, just follow the installation wizard. Setup will install one by one all the components needed: Dynamips, QEMU, GNS3, WinPcap, Wireshark and many others parts.
  2. Initial GNS3 configuration. At this step, I usualy configure some general, non-essential, GNS3 preferences:

     

    1. General – Local paths – projects/binary images – redirected to a shorted path, e.g. C:\GNS3\projects and respectively C:\GNS3\images
    2. In Topology View change the default label text style to a less accentuated one.
    3. In Miscellaneous – disable the Automatically check for update and Automatically send crash reports options.
  3. Create a new Cisco ASA device by starting New QEMU VM template from Edit – Preferences – QEMU VMs – New menu. Use the following parameters:

     

    1. Type: ASA 8.4(2)
    2. Name: ASA5520-8.4(2) or any meaningful title you choose
    3. Qemu binary: leave the default qemu.exe (v0.11.0)
    4. RAM: 1024MB or more, 1024 is the minimum
    5. Initial RAM disk (initrd): select RAM disk file, e.g. asa8420-initrd.gz
    6. Kernel image (vmlinuz): select Kernel image file, e.g. asa842-vmlinuz

I will recommend to store original OS images in other folder than that used by GNS3 for image storage. When you specify an image to be used by GNS3 a copy of that original file would be automatically copied to GNS3 binary image folder location.

After device creation, edit VM configuration and add up to 6 network adapters (network section). Leave the rest parameters untouched.

art - configuring Cisco ASA 8.4(2) on GNS3 1.3.13 - ASA device summary settings

art - configuring Cisco ASA 8.4(2) on GNS3 1.3.13 - ASA device general settings    art - configuring Cisco ASA 8.4(2) on GNS3 1.3.13 - ASA device HDD

 art - configuring Cisco ASA 8.4(2) on GNS3 1.3.13 - ASA device network    art - configuring Cisco ASA 8.4(2) on GNS3 1.3.13 - ASA device advanced settings

Note1: The steps above was successfully tested inside a Virtual Machine with Windows Server 2012R2 as a guest OS. The VM was provisioned with 2x vCPU and 8GB RAM and run on an ESXi host. Physical server equipped with Intel Xeon E5320 1.86GHz CPU.

Note2: There is no need for: Expose hardware assisted virtualization to the guest OS …No VT-x needed inside the Virtual Machine.

Use the newly created ASA device template

Now, you can use the newly created ASA device template, just drag the device to the map pane, build the topology and start it. Open console, if everything is ok you should see a typical ASA OS loading progress. After a minute, you will be faced with long awaited Cisco ASA CLI (use empty password for privileged exec mode).

You can use as many Cisco ASA instances in your projects as you want, sure no more than your hardware permit. Every time you instantiate a new Cisco ASA device, a flash disk device is created and mounted automatically for each Cisco ASA instance thus no additional steps for virtual disk creation needed. You can find the qcow virtual disk file (flash.qcow2) associated to your ASA device in project folder/project-files/qemu/dev-uiid/ folder.

Usualy, the first step before using ASA is to put a license key. If you do a simple google search you will find several freely flying license keys. Just use one of them. For me, it turned out to be ok the following activation key: activation-key 0x7212d04a 0xe041d3fe 0x1d22f820 0xea5440e4 0x8231dd9f … which unlike other keys does not hung loading progress for several minutes after activation.

A curious case for FTP passive mode over NAT on Cisco IOS router

Recently, I came across an interesting feature on Cisco IOS that mysteriously do something similar to an application inspection for FTP protocol and more that that, make modifications into replay messages from FTP server. You know, a router is a Layer 3 device that has no rights to do payload modifications, in some cases at most just several fields in IP header (e.g. TTL, src/dst IP addresses and port numbers in case of NAT). The behavior observed by me looks like oposite: a Cisco IOS router, when configured to do NAT, make specific changes on some FTP control messages. It’s like a router had borrowed something from ASA firewall inspection features.

Let’s do now some testings. For our scenario I built the following simple topology in GNS3.

a curios case for FTP passive mode over NAT on Cisco IOS router - lab topolgy

The three components used are:

  • the FTP client is a Windows 7 VM running in VMware Workstation
  • the FTP Server is a Windows Server 2008R2 VM running in VMware Workstation
  • the R1 is a Cisco IOS 7200 router with Cisco IOS version 12.4(24)T5

Note0: In my testings I used the FileZila 0.9.56 as an FTP server and Total Commander of version 8.51a with his embedded FTP client.

Note1: FTP server run in Passive Mode. My focus was only for FTP passive mode simply because this is the use case I have in my production scenario. I will not talk about FTP active mode.

On Cisco router applied the following simple configuration (interface & static inside NAT conf):

After NAT, the FTP server is seen by the FTP client by 192.168.3.20 IP address.

As you can observe an additional ACL that deny any un-NAT-ed communications is applied inside on FastEthernet0/0 (from FTP client side) – just to be sure that all sessions pass through NAT.

Now, I will start two packet capture sessions in Wireshark for both router’s interfaces and initiate a new FTP passive mode connection to FTP server.

In Passive Mode the client request Passive mode operation by issuing the PASV command over the control connection on TCP port 21. The server suggest a data port and IP address, to which the client must connect. The port number are somewhat encoded in several message fields, see the formula in the illustration below.

a curios case for FTP passive mode over NAT on Cisco IOS router - passive mode ilustration

The following screen show the capture of Passive mode acknowledge message (Passive OK, 227) packet with most important fields highlighted (src/dst ports, pasv response args).

a curios case for FTP passive mode over NAT on Cisco IOS router - packet capture at source_c

Where: 1-passive mode request, 2-passive mode acknowledge, 3-acknowledge message arguments, 4-passive IP address (server IP), 5-passive port (listener for data connection), 6-source port for data connection (ephemeral), 7-control connection server source port, 8-control connection server destination port (a value different from 6).

If we look now at the same packet but this time after it passed the router/NAT we would see, besides the destination IP change in IP header (because of NAT), also the FTP message payload modifications: specifically, in message arguments, values from 192,168,2,20 changed to 192,168,3,20. It is obviously an effect of NAT FTP protocol inspection and corresponding changes in message made by router.

a curios case for FTP passive mode over NAT on Cisco IOS router - packet capture at destination_b

A more in depth information about how this all works you can find here: ciscopress.com – Routing TCP/IP, Volume II (CCIE Professional Development) – Network Address Translation (link here), btw, one the few sources available on the net.

What if the FTP server will be set to function on a non-standard port ?

The answer is that the inspection embedded in NAT would ignore such a server. It simply do just the standard NAT section but not also the payload inspection and modification. Ironically, that was the reason why, initially, I couldn’t understand why this works perfect in lab but not also in production. It appeared to be that the non-standard port applied on prod. FTP server was the culprit.

The next question then arise: What needs to be done for NAT FTP inspection work for non-standard FTP ports ?

More by accident than intentionally I found this cool Cisco article: cisco.com – Using Non-Standard FTP Port Numbers with NAT (link here), that explain what to do – not so much, just two additional configurations. In my case these would be:

First, we create a standard ACL that would include the FTP server’s IP address (before NAT), and second, we would teach the IOS NAT to inspect on non-standard port number (in our case 5555) for that IP. If more than one FTP servers are used then add those to ACL too and if more than one tcp port numbers are used for control connections then specify those one by one separated by comma in second configuration command.

Bellow, you can see the ftp connection logs before (first section) and after (second section) the above configuration wad applied and FTP server configured to work with non-standard port number.

a curios case for FTP passive mode over NAT on Cisco IOS router - ftp client logs

As can be observed, after applying configuration, the FTP server answer message arrive already modified with NATed (global inside) server IP address in payload.

How to deliberately disable NAT inspection?

It can be done only per NAT entry (no global configuration), by specifying no-payload keyword at the end of NAT entry definition command.

Why is so important for IP addr. in pasv answer to be fixed to FTP server’s NATed (global inside) value?

Well, actually the importance depends on how FTP client are configured/build. The truth is that an FTP client can work either in restrictive or in less restrictive mode (can’t remember from where I got the terms, they may be wrong) and depending on that, ignore or not the value provided in pasv answer message. An FTP client configured for restrictive mode will try a data connection only to IP address suggested in pasv answer, if the de connection could not be established the FTP session will fail. At the oposite side, an FTP client in less restrictive mode would simply ignore the IP address suggested in pasv answer and would try instead the IP address used initially for data connection (the IP address that you configure in FTP client. Obviously, the global inside address).

So, if you have an FTP client configured/build for restrictive mode then is important to have NAT inspection doing IP address exchange in pasv answer message. Again, absolutely by accident, I found that older versions of Total Commander use the restrictive mode for their FTP client (7.04a in my particular case). It was an opportunity for a little test with such a client. Bellow, you can see the FTP client connection logs for (a) with NAT inspection and (b) without it and the final FTP session status. 

a curios case for FTP passive mode over NAT on Cisco IOS router - ftp client in restrictive mode

Clearly, for FTP session to succeed, a NAT FTP inspection that will do also translations in pasv answer message payload, needs to be in place.

TCP Performance. Why I couldn’t reach expected speed?

After you read this article you will find answers or suggestions for following questions:

  • I have 1Gbps NIC on PC and Server. They are on the same network, all links between are 1Gbps but our throughput is low.
  • TCP header should have 20 bytes, but I saw headers with 32 or 40 bytes. What means those values?
  • I heard about different congestion control mechanisms. What is the difference between them?

TCP connection is established between two end hosts. Doesn't matter how many routers, networks, switches you have between. TCP will ask remote host application to get/transmit data. This means that TCP interact directly with application on local and remote host. For example we open browser on our PC and enter in the address field www.google.com. This means that we are asking web page located on the google's server hosted by some hosting application, like apache2 or something else.

Let's take a look on the TCP header:

Source: https://ciscoskills.net/2011/02/25/understanding-tcp/tcpheader/

In the TCP header we could find answer why sometimes we could see TCP header with more than 20 bytes. Options allow us to extend header up to 52 bytes. Usually we could see bigger header in the TCP three way handshake, after that communication has standard header.

TCP is called reliable protocol and connection oriented. Let's review what means reliable protocol. TCP has on the header fields called Sequence number (seq) and Acknowledgement number (ack). Sequence numbers are generated in the random way at the beginning of TCP communication. Acknowledgement number is used to confirm that we get data send by sender. In case we don't have confirmation data will be retransmitted. Let's see how combination of seq and ack works as concept in following example:

First 3 packets in the communication are related tp three way handshake. You could note that start sequence number on the PC and Server are different (they could be the same). We set them different just to avoid confusion. This example shows us download of example.zip file from server. Server has to divide file on segments to send them via Internet. How many segments will be? This Depends on MSS (Maximum segment size) and size of the file.

Acknowledgement number has to confirm that we get data. For example server sends us two segments with total size 1500 bytes. Server start sequence number was 401, so 401 + 1500 bytes will be 1901. This means that server expect to get segment with ack 1901+1=1902. Server will keep sent segments without acknowledgement in his tx buffer. When server get ack about received data he release segments from tx buffer (keep in mind tx buffer is limited in size). In the same way we will get Ack number for the last sent segment.

Why PC acknowledge first two segments in one hit and third in separate ack? Server task is to send as much as he could segments, PC task is to advertise as fast as he could already received segments. So you could have 1 ACK segments per 50 segments and after that 1 ACK after next 5 segments.

TCP Performance

TCP could adapt speed of transfer to the link condition

Let's go back to the TCP header. There we have very important field called Window. This field has crucial impact on performance and speed transfer. Let's see why. We have a transfer sample between server and client. Let's suppose that client announce that his window is 14600 bytes (Window = Receive buffer size). This means that server could send segments with total size up to 14600 bytes until he has to stop and wait for ack message. Usually client will send ack faster and window will not become full. In case window become full server stop communication until client will say that he is ready to get new data. So in other words bigger window require less ack packets and increase throughput. Window field is limited to 16 bits, which means 2^16 =65535 bytes.

Why window field vary from packet to packet?

When we establish TCP connection window field has low value. This value is linked to default mss of OS multiplied to 2,3 or 10. In short time window value increase very fast up to saturation. Window is increased after each ACK sent by receiver. This procedure is called TCP slow start.

Here we have a formula which shows us what is maximum speed:

MaxSpeed = Window * 8 / Delay in sec.

65535 * 8 / 0.001 sec = 524 Mbps.

Most interesting in this formula is dependency of delay. When your server is far and between you is present slow slow link this means that speed will dramatically decrease.

Window field will decrease in case of lost packets, this will be done very fast. In case of too many lost packets TCP connection will be simply closed with error.

TCP Congestion control mechanism

Window, sequence number and acknowledgement number fields are responsible for the speed control. Server cannot push too much data on the client, this could overhead RX buffer and all new packets will be dropped (unacceptable situation for the TCP communication). All those specification were collected in the TCP congestion control mechanism called TCP Reno. In our days we have links, NICs, OS with much more capacity this is why it is required to do changes in the TCP stack inside of operating systems. See here list of new proposed mechanism. For example starting Linux kernel 2.6.19 up to kernel 3.1 default congestion mechanism inside of Linux is CUBIC.

Are the new requirements for the speed control change TCP header? No, just few new options will be added and TCP stack inside of operating system will be changed accordingly.

We said that TCP is connection oriented which means that before to start exchange of data we have to negotiate and answer following questions:

  • Are both hosts ready to exchange data between their applications?
  • Which additional options support both hosts?
  • At the end of communication we have to close connection. Operating systems limit maximum number of connections to avoid overhead of the system.

Which options could be interesting for us:

  • Selective Acknowledgement
  • Windows Scaling
  • Timestamp options

Let's look inside of packet captured by wireshark (good bless people who are working on this application).

Please pay attention that information placed between brackets in the wireshark is not present in real packet, it is info added by wireshark after analysis to help us understand communication.Different wireshark version will treat in different way this information.

Selective Acknowledgement

We said that Server should keep in his TX buffer sent segments until he would get Ack for data.Next question it is how data will de retransmitted in case of lost packet? Let's suppose that PC announced Window 14000 bytes. Server is sending 14 segments each by 1000 bytes. We lost segment #5 and #8. Classical TCP have to retransmit full window (all 14 segments). Selective ACK suport allow sender to send only #5 and #8. Also this allow server to release ack packets. I will give an example how it works little bit latter.

Window Scaling

Let's go back to the speed formula MaxSpeed = Window * 8 / Delay in sec. When we have Long Fat Networks (networks with big delay, like satellite communication) we could reach 600ms delay. This means that maximum speed through this network could be 65535 * 8 /0.6= 0.8 Mbps :). It could be strange to pay huge amount of money to get less than 1 Mbps just because of TCP limitation. Window Scaling option allow us to multiply announced window with scalar factor and get calculated window size. I will not explain how is calculated at which value multiply window, see RFC 1323 for more details. I just want to add that window scaling allow us to get up to 1 GB of RX Window without ack :).

Timestamp

Sequence and acknowledgement number are limited by in the TCP header by 32 bits. Together with ip source and destination those allow us to distinguish which application has to receive specific segment in the network. In High loaded servers this is not enough. It could be that segment will be assigned to wrong TCP stream when we reach 1 Gbps traffic. Timestamp added to the packet header allow us to get in game one more unique parameter to distinguish flow between them.

TCP Fast Retransmission

TCP is reliable protocol. He has to retransmit lost segments. How much time has to wait host until resend lost segment? Regular timer could be 2 sec. It is too much for high performance application. How to deal with that? We have ack segments which inform us how much data receiver got. Let's suppose that Server sent 14 segments. Segment #7 was lost. Receiver will inform Server that he got data up to segment #6 even in his buffer more data are present. Receiver will send again and again that he received up to segment #6. All new segments will be received and buffered. After third duplicate ACK server will think: "Hmmm.. Maybe he indeed didn't get 7th segment. Let me send it one more time.." (of course only in case selective ack is supported). In case of 1 ms delay between Server and PC retransmission could take 1.5 ms (regular timer 2 sec). Really impressive, isn't ? When delay between server and PC is more we could attest hundreds of DUP ACK messages in the capture (it should be like this!). For example 80 ms delay creates in the capture 80 DUP ack messages (it was around 80 on different tests).

Why not to adjust regular TCP timer for retransmission to relevant value instead of fast retransmission?

RFC 6582 is trying to adjust RTO (Recovery time Objective). We have different network types (Wi-Fi, Satellite, Fiber…). They have different delay and RTO couldn't be equal for all them. Right, let's measure RTT of each TCP segment and adjust RTO based on that measurement. How to do it ? Take a look in the RFC :)

Now we are ready to see full picture of TCP Congestion Control Mechanism:

Note: In this sample I did one mistake. After receiver notify lost packet he has to decarese Window Size.

What for information described in this article could be useful for us?

We still don't have answer for one objective of this article:

  • I have 1Gbps NIC on PC and Server. They are on the same network, all links between are 1Gbps but our throughput is low.

We have checked everything on the network, but issue is still present. Let's take a look on the traffic sniffer:

So in this graph we could see that our PC announce very low window size together with latency this creates average speed ~21Mbps. I observed that speed depends on the application used in the communication. For example built-in FTP client in Windows 7 got 35 Mbps, at the same conditions Filezilla FTP client got 140 Mbps. Try to use another application or adjust parameter is the operating system (do it carefully, could damage another things). Following image shows available parameters for the TCP in Centos6.5.

More often TCP performance could be affected by CPU utilization spike on the sender/receiver, buffer overload.. Traffic sniffer could help us to find when it happened.

Wireshark graph shows us that communication stops for several seconds. Let's explain what happened in the capture on that time:

Our PC for some reason decreases his window from 5840 bytes to 2920 bytes. Server sends last two segments (each 1460 bytes) and has to stop until ack from the PC. PC is sending last segment where window value is 0. This means stop sending traffic I couldn't get more. Server will wait and try from time to time asking PC if we could continue send traffic. Only after 6 seconds client will open again window and get remain info. Why PC closed window? You have to analyze logs on the PC to find reason.

Links:

  1. TCP congestion control – https://en.wikipedia.org/wiki/TCP_congestion_control
  2. Wireshark application –  www.wireshark.org/
  3. RFC 1323 TCP Extensions for High Performance – www.ietf.org/rfc/rfc1323.txt
  4. RFC 6582  The NewReno Modification to TCP's Fast Recovery Algorithm – https://tools.ietf.org/html/rfc6582

How to setup Cisco ASDM in Demo mode

Today, I’ve encountered some issues during installing Cisco ASDM in Demo mode. In this post I will address this issues and show a step by step instruction on how to successfully setup ASDM for Demo mode.

In my attempts, I started by installing the lattest available versions for ASDM Demo (ASDM Demo 7.3.1) and Java JRE (Java 8 update 91) but finally got an unworkable setup. Every time trying to start demo mode a generic error that state that Demo software is not installed popping up (screen below).

How To setup ASDM demo mode - error mesage

Furthermore, if you go to application folder in Program Files (x86) you will see an empty ASDM\Demo folder, as like Demo mode not even installed.

After several attempts, I haven’t found a better solution than to downgrade my Java JRE (8u91) to the previous major release (lattest update): Java 7 update 72. Also, at least when you start setup process you must have a 32 bit version of Java installed.

To complete a Cisco ASDM setup in Demo mode:

  1. Download the lattest available Cisco ASDM Demo setup file. For this, go to Cisco download page at Products – Security – Firewalls – Firewall Management – Adaptive Security Device Manager – Adaptive Security Appliance (ASA) Device Manager and search through the ASDM versions available the latest one that have the word demo in setup (msi) file title. The release policy for ASDM demo don’t coincide with that for ASDM. At the moment of this writing the lattest available ASDM Demo was: ASDM Demo 7.3.1. For download to succeed you will need a service contract associated with your cisco.com login, otherwise a simple googling will reveal a leaked image somewhere in Internet.
  2. Download the latest available Java JRE 7 release (Java 7 update 72), both for 32 and 64 bit with 32 bit being mandatory (setup files are jre-7u72-windows-i586.exe and respectively jre-7u72-windows-x64.exe). Install both versions, these will function perfect together.
  3. Launch ASDM Demo setup and go through a banal installation wizard. The ASDM Demo 7.3.1 setup will install also the ASDM-IDM Launcher of version 1.5(73) so if you have a newer Launcher already installed it will be overlapped. If you later try to connect with this older Launcher to an updated ASA ASDM you will prompt for Launcher update. To avoid this version swapping back and forward I will recommend to setup DEMO mode somewhere on another PC, perhaps on a Virtual Box/VMware Player VM.

Note0: The steps above was successfully tested in a Windows Server 2012R2 OS Virtual Machine.

Note1: For a guide on how to disable Java Update to proceed automatically you can read here.  Simple unchecking the Automatically Updates from Java Control Panel is not enough you will need edit specific registry key.

If everything succeeded, your ASDM\Demo folder in Program Files (x86) should be full with plenty of files:

How To setup ASDM demo mode - demo folder

Now, we can start using Cisco ASDM in DEMO mode: start ASDM Launcher (icon on your desktop) – check Run in Demo Mode:

How To setup ASDM demo mode - launcher for demo mode

Select the preferred configuration, and click OK, ASDM Demo mode should start. In the above screen note the Device IP Address/Name field automatically filled with a localhost address (not appear on first run).

How To setup ASDM demo mode - asdm demo started

Now, you can start gamming with an imaginary topology with configured ASA devices.

 

How to configure ASA for ASDM access

It this short post I will go through the steps of configuring ASDM access on an ASA device. I will use the ASAv 9.5.2 appliance just configured for GNS3 in previous post.

Copy the ASDM image to ASAv appliance

First, we need to copy a compatible ASDM image to ASAv internal storage. Therefor:

  1. Go to Cisco Download Software portal at Products > Security > Firewalls > Firewall Management > Adaptive Security Device Manager and download a compatible ASDM image for your ASA device. For download to success you will need a service contract associated with your cisco.com profile otherwise try a simple Internet search for a leaked image. Verify compatibility by consulting the Cisco ASA compatibility (link) article. For my ASAv version 9.5.2 an ASDM version 7.5.2.153 will compatible and sufficient.
  1. In GNS3, build a simple topology that will connect ASAv to some external network. To do that, connect one interface from ASAv to a cloud object configured to be linked to one of the host interface – for this purpose I usualy use a simple loopback adapter (for how to install such a one, read this technet article. reboot required). Because the ASA can’t connect directly to a cloud object a transit synthetic switch needs to be added. At this step, your topology should look like this:

How To configure ASA for ASDM access - topology view

Note0: Ethernet0 on ASA as presented by GNS3 correspond to Mangement0/0 intf seen from inside the device.

Note1: For a better look, changed the symbol/hostname used for cloud representation.

  1. On host computer start your favorite TFTP daemon (for this purpose I use tftpd32 from tftpd32.jounin.net. Configure the daemon directory and listening interface, additionally verify you host firewall to allow tftp protocol.

How To configure ASA for ASDM access - tftpd32 config

  1. Start the ASAv device and open the serial console. Configure interface IP settings, verify connectivity and copy the ASDM image to ASAv internal storage:

How To configure ASA for ASDM access - intf. configuration

A copy process should now begin. The progress seems to be less rapid than expected (in my case a top was the 60kbps) which could be because of unlicensed state of ASAv. In essence not a big problem, just wait for 3-5 minute for operation to complete. For confirmation do a dir command:

How To configure ASA for ASDM access - dir flash

Configure ASAv for ASDM access

Now it’s time to configure ASAv for ASDM access. Execute the following commands:

First two lines configure authentication, in this particular case against the local user database, second group of two commands enable HTTPS server and access from 192.168.49.0/24 network via mgmt interface (Management0/0) an the last command tell the firewall to use asdm-752-153.bin image for ASDM access.

Next, switch to your browser and try to open https for management interface https://192.168.49.100. If everything is ok, a security certificate error should appear in your browser, confirm the certificate exception to go forward. You should see a page like this:

How To configure ASA for ASDM access - ASDM welcome page

From this point you have two options: (a) via Java Plugins or (b) through ASDM Launcher. My preference is to use the ASDM Launcher. First install the ASDM Launcher – after click Install ASDM Launcher and successfully authentication a setup file will be made available for download, second start ASDM Launcher (icon on your desktop should be already present).

In ASDM Launcher authentication window, put the ASAv IP address and the authentication credentials.

How To configure ASA for ASDM access - ASDM launcher

Finally, after loading ASAv configuration, ASDM application should start:

How To configure ASA for ASDM access - ASDM view

Configuring Cisco ASAv 9.x on GNS3 1.4.x

Recently I went through an interesting experience of Cisco ASA setup in GNS3. I must say it was a real challenge, but finally, not an impossible task. There is a lot of particularities you must take into account, all depending from ASA version to GNS3 release. In this post, I will focus on how to configure an ASAv firewall to run as a QEMU VM in new GNS3 version suite 1.4.x. As of date of this writing I was able to access ASAv image version 9.5(2.204) and the GNS3 1.4.5 setup.

The 1.4.x suite of GNS3 is a relatively new appearing into the scene, and how is typical with new software releases, surely expected to have some bugs and/or incompatibilities. That’s why my first attempts was made in previous software suite of 1.3.x version (actually 1.3.13 version). It was a wrong way, because how I realized soon, for ASAv to be configured, a VNC console should be attached, or in 1.3.13 I didn’t found how to do that (is not excluded that I missed something). As a consequence, I quickly switched to newest GNS3 1.4.5 version. The following text assume a newly setup of GNS3 version 1.4.5 with no additional settings, except maybe only for the default location for project and binary images folder – I prefer to reconfigure these on C:\GNS3\Projects and respectively C:\GNS3\Images.

Cisco ASA virtual appliance (ASAv)

Cisco ASAv is a re-imaged version of Cisco ASA specifically designed to run as a VM on top of some hypervisor. In fact, the same ASA code is running, but in different form factor. There are versions for vSphere, Hyper-V and KVM. Just because GNS3 use QEMU as a VM emulator we will employ the KVM image of ASAv. By the way, ASAv is the image Cisco use in their notable virtual labs VIRL. Not all ASA versions are available in a VM format – I suppose only those starting with 9.x, thereby if you want to try some older versions, e.g. popular ASA 8.4(2), you will need to experience another approach (a new article devoted to this subject should come). It’s worth noting that the ASAv have some limitations compared to classical ASA, in particular you wouldn’t be able to build firewall clusters (failover or A/A), test multiple context mode feature or play with Etherchannel. For this scenarios, I usualy use an 8.4(2) ASA setup – which, by the way should run only in QEMU 0.11.0 which in turn can’t be started in GNS 1.4.x, only in previous suite 1.3.x !!!.

So, before we start we need to obtain somewhat the ASAv image. If you are fortunate enough to have access to Cisco downloads (a service contract associated with your profile is needed) then just go to cisco.com – All downloads – Products – Security – Firewalls – Adaptive Security Appliances (ASA) – Adaptive Security Virtual Appliance (ASAv) and download the qcow2 (KVM) image of ASAv for your preferred version.

configuring Cisco ASAv 9.x on GNS3 1.4.x - download page for Cisco ASAv

In case you do not have access to official Cisco downloads, yet I recommend to try a simple Internet search, good chances are to find somewhere a leaked image (usualy on some China resources). To be honest, I can’t understand why Cisco restrict downloads to this type of software, anyway, next after setup you will need a license key to go over the limitations of unlicensed state of appliance (bandwith limitation to 100kbps). It would be fine if Cisco would allow download and free use of appliance in unlicensed state, respectively for production usage a suitable license should be bought.

Configuring ASAv template on GNS3

A step by step guide follow:

  1. Start new QEMU VM Template wizard with following parameters:

    1. Type: Default
    2. Name: ASAv-8.5(2.204) or any meaningful title
    3. Qemu binary: qemu-system-x86_64w.exe (v2.4.0)
    4. RAM: 2048 MB
    5. Disk Image (hda): C:\GNS3\images\QEMU\asav952-204.qcow2

Note: I will recommend to store original OS images in other folder than that used by GNS3 for image storage. When you specify an image to be used by GNS3 a copy of that original file would be automatically copied to GNS3 binary image folder location.

  1. Edit newly created QEMU Template:

    1. General settings – Symbol :/symbols/asa.svg
    2. General settings – Category: Security Devices
    3. General settings – Console Type: VNC

Note0: in my testing, I tried to change vCPUs from 1 to 4, but nothing more than 1514 Illegal Instruction (core dumped) … error message got in ASAv, hence don’t touch that value, we will set the number of vCPUs in other place for ASAv to be an SMP virtual machine.

Note1: Switching the console to VNC type one it’s like directly connect with a keyboard and a monitor to the virtual machine. Initial ASAv configuration don’t allow access to the serial console port so at least at this stage, the only possible option is VNC. Don’t forget, the ASAv was designed to play in a VM with a full console. Even so, we will configure serial console port to ASAv as well.

  1. Network – Adapters: 6x (default e1000 type)
  2. Advanced Settings – Additional settings – Options: -cpu Haswell -smp 4,sockets=4,cores=1,threads=1

Note0: I successful used this string for all my Intel CPU. The microarchitecture (Haswell, Nehalem and so on) seems to no matter – successfully ran on different CPU generation with no problems. For AMD CPUs, community recommend to use (haven’t tested): -cpu Opteron_G5 -smp 4,sockets=4,cores=1,threads=1

Note1: the default option’s value: –nographic, should be cleared. This will be guarantee an automatic VNC console opening (for non-linked mode VM operation).

  1. (Optional) Activate CPU throttling – Percentage of CPU allowed: 80%
  2. Advanced Settings – uncheck: Use as a linked base VM.

configuring Cisco ASAv 9.x on GNS3 1.4.x - qemu ASAv template - summary

configuring Cisco ASAv 9.x on GNS3 1.4.x - qemu ASAv template - general settings    configuring Cisco ASAv 9.x on GNS3 1.4.x - qemu ASAv template - hdd

configuring Cisco ASAv 9.x on GNS3 1.4.x - qemu ASAv template - network    configuring Cisco ASAv 9.x on GNS3 1.4.x - qemu ASAv template - advanced settings

I think I will provide some additional inputs about the setting named: Use as a linked base VM. By default, QEMU VMs works as a linked VM which means that every time you create a new QEMU VM (in our case ASAv) in your project, a linked virtual disk is created to the original qcow2 image. All the modifications are thus recorded in that new file but yet unmodified block are read from original image. Through this, we can create hundreds of new QEMU VMs without needing to clone the virtual disk (that’s the similar to the technology used in VDI). Given the fact that during the life of an ASAv VM, disk modifications are really very few, results that the disk overhead created by each new ASAv are truly negligible. If you disable linked VM mode (uncheck the: Use as a linked base VM) the QEMU VM will interact directly with original qcow2 virtual disk (all writes will be recorded here). As a consequence a single QEMU VMs from this template can be started (just try to drag and drop a second ASAv to workspace and you will see an error message).

Why then we intentionally disabled linked base VM mode? First off, we need this only during ASAv template making and after this we will switch back to linked mode. Our interest is to do a series of configuration changes (first boot, serial console, ASDM image upload) in the original image file which we want to keep in all new ASAv instances created from this template.

Surely, the same results can be achieved by making the template in linked mode (linked qcow2 virtual disk) and then committing all the changes to the original qcow2 image via qemu-img.exe tool, but, I think it is harder. Just disabling and then re-enabling the VM’s linked mode settings seems to be much easier … the choice is yours.

To check the virtual disk that is mounted to QEMU VM just drag a new ASAv to an empty project, right click ASAv device – and choose show in file manager. An explorer window to qcow2 image opens – with linked mode disabled this would be the template image asav952-204.qcow2 located in binary image folder, whereas for linked mode this would be a qcow2 image (somewhere in project’s folder) linked to the original template – base virtual disk image. Also, additionaly you can check what qcow2 images are involved via Windows resource monitor – CPU – Associated Handles – filter by QEMU string.

  1. Drag a new instance of ASAv 9.5(2.204) to the working space on an empty project in GNS3. No topology are needed to continue, just single, unconnected ASAv device.
  2. Power-ON newly instantiated ASAv device (right-click – start) and immediately open the console (right-click – console).  In opened VNC terminal a loading progress (Linux) can pe observed.
  3. On Boot Loader phase choose the option: bootflash:/asa952-204-smp-k8.bin with no configuration load (anyway no configuration yet exists).

configuring Cisco ASAv 9.x on GNS3 1.4.x - bootloader

  1. In the meantime, it would be interesting to do some analyzing in Resource Monitor. First, to confirm the SMP nature of started QEMU VM look at the number of threads/CPU associated with qemu-system-x86_64w.exe process (CPU – Processes) – should be more than 4x thread/CPU in use, and second, to confirm the non-Linked mode of operation for the ASAv VM do a search in Associated Handles for a qemu key (CPU-Associated Handles) – in non-Linked mode, the VM should interact directly with the original qcow2 image: asav952-204.qcow2 (a screen is inserted below).

screen resource monitor - qemu treads plus handler not linked mode - 2

At the command prompt the number of vCPU can be checked by the show cpu usage commmad:

ciscoasa# sh cpu usage
CPU utilization for 5 seconds = 1%; 1 minute: 1%; 5 minutes: 0%

Virtual platform CPU resources
------------------------------
Number of vCPUs              :     4
Number of allowed vCPUs      :     0
vCPU Status                  :  Noncompliant: Over-provisioned

  1. If you carefully track the booting progress you will see that the appliance will discover that it starts for the first time (Initial bootup detected …) and for the system variables to be applied an automatic reboot will come.  So first time booting will end up with an automatic reboot. On the second boot, also choose the option with no configuration load in Bootloader Dialog. First and second time booting could take some time to progress so be patient and wait them to complete – sometimes it may seem that the appliance hung, try to wait several minutes before doing a forced powering off.
  2. If everything goes smoothly, after the second boot, you should reach the traditional Cisco command line prompter (empty password for privileged mode). At this stage, we will enable the serial console for the appliance. By default, the ASAv works only with traditional VM console (monitor/keyboard directly connected to x86 hardware) and additional steps needed to enable console via serial ports. More about that you can read here ASAv Quick Start Guide, 9.5, section Configure a Network Serial Console Port.

For serial console to be on, a file named use_ttyS0 should exist in root of disk0. It doesn’t matter the content, just to be present. The simplest mode to create such a file is to make a copy of an existing file – the documentation suggest to clone from coredump.cfg file, like shown below:

ciscoasa(config)# cd coredumpinfo

ciscoasa(config)# copy coredump.cfg disk0:/use_ttyS0

  1. Theoretically, here, we can do also some additional configurations, one that we want to keep in all the ASAv instances derived from this template. For example, we can copy here the ASDM image to disk0 to not be bothered with that in the future. Anyway, I will skip this step.
  2. Reload de appliance (type reload in privileged mode). You will see that the command prompt can’t anymore be accessed via de VNC console. I mean, the console will open, but, at one moment the interaction will be handover to the serial console and no more activity going to be possible by VNC. The last message recorded in VNC confirm that: Lina to use serial port /dev/ttyS0 for console IO.

configuring Cisco ASAv 9.x on GNS3 1.4.x - qemu ASAv template - console handover

  1. Now, after all the modifications to the ASAv image, we can switch the template back to his original Linked mode of operation. Also, we will switch the console settings to telnet type. Do the configuration changes in template settings, not in ASAv instance. The ASAv device from our temporary project can be safety removed, it has already done his job.

Using newly created ASAv template

To use the newly created ASAv template, just drag the template icon to the workspace, do your connections and power-on the device. You can use multiple ASAv devices running simultaneous with no problem, on my PC (i7-4970s CPU with12GB RAM) I ran five concurrent instances, all started ok and became usable shortly (less than 1 min).

Just because we don’t mention –nographic in template’s Advanced Settings – Additional Settings the VNC console will automatically open every time you start the device. If you close that window, the appliance will power-off automatically. The VNC console don’t interfere with serial console which you can open via context menu. If you add the -nographic option, the VM will start silent without a VNC console. Anyway, my preference is to leave the VNC console to open automatically, at least for the begging, just to have an additional visibility of the process.

After you load the ASAv device, you will periodically be announced by a missing license warning message: Warning: ASAv platform license state is Unlicensed … It is because the appliance don’t have a license key applied and it works in unlicensed state. As mentioned above, for lab and test scenarios, an unlicensed state are more than sufficient. In this state, you will get all the ASAv features but at the same time be limited to 100 Kbps interface bandwith.

It is interesting to see what virtual disks files are involved for an ASAv device started from our completed template. Beacause the template was configured as a Linked Mode VM, a linked virtual disk plus the base disk should be used, a fact confirmet by the screen below: 

configuring Cisco ASAv 9.x on GNS3 1.4.x - qemu ASAv template - linked mode virtual disks

To complete the story,  bellow I insterted a screencast for the process described above (youtube link). Enjoy.

VMware NSX logical routing LAB

Following the subject of previous post, I will continue with NSX routing LAB topology used by me in my NSX learning. LAB topology diagram are inserted below. 

VMware NSX logical routing LAB diagram

(click on picture for hi resolution image)

A brief description for lab topology:

  • There are tree VXLAN logical switches: 5001, 5002 and 5003 corresponding to segments in a formal three-tier application (WEB/APP/DB). Each segment have a related subnet network: 10.128.20.0/24 for LS5003 (WEB), 10.128.21.0/24 for LS5002 (APP) and 10.128.22.0/24 for LS5001 (DB).
  • Inter VXLAN routing are provided by a Distributed Logical Router (DLR) and an Edge Services Gateway (ESG). DLR act as a default gateway for DB and APP segments and ESG provide routing for WEB network segment. First IP from each subnet are reserved for default gateway.
  • Both DLR and ESG are configured in HA configuration mode (active-standby) – that is why these services are represented by two VMs each (vShield Edge for ESG and Control VM for DLR) – x per Edge ESXi host.
  • A group of two ESG appliances configured for Equal Cost Multi Path (ECMP) routing provides upstream communications. ECMP are enabled as well for DLR and ESG routers.
  • All routers are interconnected trough a transit logical switch 5004. Routers in this segment will use IP addresses from 10.128.10.0/24 subnet: .5 for ESG, .254 and .253 for each of upstream routers. The DLR router use two IP addresses: first (.2) as a forwarding address and second (.3) as a control plane address (protocol address).
  • DLR, ESG and upstream routers exchange routing information via OSPF. All routers forms OSPF neighbor relationships.
  • ESG router are configured with both interfaces (internal and uplink) as part of the OSPF process (mapped to OSPF Area 0). DLR router have OSPF activated only on uplink interface and configured to redistribute in OSPF directly connected networks (APP and DB).
  • Upstream routers use OSPF on its internal interfaces and form BGP peering with external physical routers via its uplink interfaces.
  • External communications take place trough a dedicated dvPG (EDGE) with uplinks via physical NICs – DirectPath IO pass-through from physical host to EDGE nested ESXi.
  • Upstream ESG routers will redistribute BGP routes in OSPF and conversely OSPF to BGP.  

Presented below, are the configurations applied on physical routers (RTR1/2). VLAN 100 are used to interconnect with ESG upstream routers (EDGE/10.128.30.0/24 subnet) and VLAN 102 to reach next hop on Mikrotik router (subnet 192.168.1.0/24). For Internet access, a NAT overload function is configured on Mikrotik network appliance.

VMware NSX logical routing - physical router configuration

After all the configurations applied, we can check the routing protocol adjacencies and routing table information. For instance, here’s what DLR routers showing us:

OSPF adjacencies with ESG router (.5) and upstream routers (.253 and .254)

VMware NSX logical routing - DLR ospf adjacencies

Respectively, the routing table: 

VMware NSX logical routing - DLR routing table

We can observe two routes learned via OSPF: one to 10.128.20.0/24 (WEB) learned from ESG routers (.5) and a default route 0.0.0.0/0 as external redistribute route. All other routers are directly connected type. Btw, 172.16.22.121 is the IP address of the DLR’s router management interface and 169.254.1.8/30 are the automatically assigned network used for heartbeating between DLR router failover peers.

BGP peering can be checked on one of the upstream ESG router:

VMware NSX logical routing - upstream ESG BGP peering

and corresponding, all known networks: 

VMware NSX logical routing - upstream ESG known networks

It’s interesting to verify the routes installed in routing table on physical routers. Bellow, are shown the routes on RTR1 physical routers.

VMware NSX logical routing - physical router routing table

Finally, we can check the connectivity to some internet resources from one of the VM connected to internal logical switch (5002/APP): 

VMware NSX logical routing - traceroute test