How to find qbr,qvo interfaces of the particular Instance in OpenStack

locate qbr,qvo interfaces of the particular Instance in OpenStack
OpenStack networking topology is pretty complicated structure, even with it’s, let’s say, default configuration based on OVS (OpenVSwitch), not even mentioning about additional overlays, like OpenDaylight or Nuage.
If you are troubleshooting network issues in regard to a particular Instance (VM), it’s good to locate network overlay interfaces (qbr, qvo) assigned to that Instance to have a starting point to begin your network analysis from. You can then analyze inbound/outbound traffic on those interfaces using network protocol analyzers, like tcpdump/wireshark.

Below you can find few steps, how to locate qbr,qvo interfaces belonging to a particular Instance:

1. Locate Nova Compute host including faulty Instance

Login to your controller node and source keystone file (import admin credentials):

[root@controller ~]# source /root/keystonerc_admin 
[root@controller ~(keystone_admin)]#

Display Instances for all tenants:

[root@controller ~(keystone_admin)]# openstack server list --all-projects --long
+--------------------+-------------------+---------+------------+-------------+--------------------+-------------+--------------------+-------------------+---------+------------+
| ID                 | Name              | Status  | Task State | Power State | Networks           | Image Name  | Image ID           | Availability Zone | Host    | Properties |
+--------------------+-------------------+---------+------------+-------------+--------------------+-------------+--------------------+-------------------+---------+------------+
| 94b233cb-c377-4b94 | kali              | SHUTOFF | None       | Shutdown    | priv_net=192.168.2 | kali_2017.3 | 60602fe5-caaa-4d0f | nova              | compute |            |
| -ab5f-f4982ffc0e23 |                   |         |            |             | 2.13, 192.168.2.57 |             | -a681-a9648e5e3f6b |                   |         |            |
| a3164da8-9af8-4000 | cirros_instance-2 | SHUTOFF | None       | Shutdown    | priv_net=192.168.2 |             |                    | nova              | compute |            |
| -90d2-9d9d35880ed1 |                   |         |            |             | 2.8, 192.168.2.60  |             |                    |                   |         |            |
| c022869a-acf0      | cirros_instance-1 | ACTIVE  | None       | Running     | priv_net=192.168.2 |             |                    | nova              | compute |            |
| -432f-8e5a-        |                   |         |            |             | 2.5, 192.168.2.51  |             |                    |                   |         |            |
| b4439ca7f5d0       |                   |         |            |             |                    |             |                    |                   |         |            |
+--------------------+-------------------+---------+------------+-------------+--------------------+-------------+--------------------+-------------------+---------+------------+

Let’s say cirros_instance-1 is having networking issues and we need to find qbr,qvo interfaces that belong to this Instance.
From the above example we can see, that Instance is located on compute node, so we need to log in to the compute node.

In our case compute node is a KVM vHost (hypervisor), so all the Instances running on compute node are in fact KVM Virtual Machines.

Login to the compute node, which hosts faulty Instance and list KVM VMs:

[root@compute ~]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 2     instance-00000001              running
 -     instance-00000002              shut off
 -     instance-00000009              shut off

2. Locate faulty Instance

VM names at the end of the string include IDs of corresponding Instances, that means instance-00000001 has ID=1 in OpenStack database.

Let’s check Instance’s ID in MariaDB database on controller node:

[root@controller ~(keystone_admin)]# mysql -u root
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 293
Server version: 10.1.17-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use nova;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [nova]> select * from instance_id_mappings where id='1';
+---------------------+------------+------------+----+--------------------------------------+---------+
| created_at          | updated_at | deleted_at | id | uuid                                 | deleted |
+---------------------+------------+------------+----+--------------------------------------+---------+
| 2016-10-22 22:28:12 | NULL       | NULL       |  1 | c022869a-acf0-432f-8e5a-b4439ca7f5d0 |       0 |
+---------------------+------------+------------+----+--------------------------------------+---------+
1 row in set (0.00 sec)

Now on controller node verify Instance using above uuid:

[root@controller ~(keystone_admin)]# openstack server show c022869a-acf0-432f-8e5a-b4439ca7f5d0
+--------------------------------------+----------------------------------------------------------+
| Field                                | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                    | AUTO                                                     |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | compute                                                  |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | compute                                                  |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000001                                        |
| OS-EXT-STS:power_state               | Running                                                  |
| OS-EXT-STS:task_state                | None                                                     |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2016-10-22T22:29:19.000000                               |
| OS-SRV-USG:terminated_at             | None                                                     |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| addresses                            | priv_net=192.168.22.5, 192.168.2.51                      |
| config_drive                         |                                                          |
| created                              | 2016-10-22T22:28:12Z                                     |
| flavor                               | m2.tiny (31e77a52-e81e-46e0-989f-f95044ee1a0a)           |
| hostId                               | ea2e42e26aac9ab745b35dcbc01120db9630e29e16fb212cb3ce5a52 |
| id                                   | c022869a-acf0-432f-8e5a-b4439ca7f5d0                     |
| image                                |                                                          |
| key_name                             | None                                                     |
| name                                 | cirros_instance-1                                        |
| os-extended-volumes:volumes_attached | [{u'id': u'7c70f9cb-9936-4f1e-8c4e-bb6731cde0cb'}]       |
| progress                             | 0                                                        |
| project_id                           | d8b0f52bdb2741b6a4e1cf3d8082a8f2                         |
| properties                           |                                                          |
| security_groups                      | [{u'name': u'default'}]                                  |
| status                               | ACTIVE                                                   |
| updated                              | 2017-11-22T10:21:32Z                                     |
| user_id                              | 2c540c9c6ba541d1be6ef02afba5b004                         |
+--------------------------------------+----------------------------------------------------------+

The above listing proves that instance-00000001 in KVM is a cirros_instance-1 in OpenStack database with id=c022869a-acf0-432f-8e5a-b4439ca7f5d0.

3. Find qbr,qvo interfaces of the faulty Instance

Once we are sure that instance-00000001 is the faulty Instance, we can try to locate its interfaces on compute host:

[root@compute ~]# virsh domiflist instance-00000001
Interface  Type       Source     Model       MAC
-------------------------------------------------------
tapc5dafda7-1f bridge     qbrc5dafda7-1f virtio      fa:16:3e:68:8f:30

The output shows, that instance-00000001 is assigned to qbrc5dafda7-1f interface.

Knowing qbr interface, it’s fairly easy to find corresponding qvo interface, which in fact has the same suffix c5dafda7-1f, so the corresponding qvo interface is qvoc5dafda7-1f.

qvoc5dafda7-1f interface is connected, as a port, to OVS integration bridge br-int on Nova Compute host, just like other qvo interfaces from other existing Instances:

[root@compute ~]# ovs-vsctl list-ports br-int
int-br-eth1
patch-tun
qvo489f9dbe-7c
qvoc14c83e0-c0
qvoc5dafda7-1f
[root@compute ~]# ovs-vsctl show
6b8ed57a-4e2a-4ddd-ae5f-cc6872a83067
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "qvoc5dafda7-1f"
            tag: 1
            Interface "qvoc5dafda7-1f"
        Port "qvo489f9dbe-7c"
            tag: 1
            Interface "qvo489f9dbe-7c"
        Port br-int
            Interface br-int
                type: internal
        Port "qvoc14c83e0-c0"
            tag: 1
            Interface "qvoc14c83e0-c0"
        Port "int-br-eth1"
            Interface "int-br-eth1"
                type: patch
                options: {peer="phy-br-eth1"}
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
...
..
.

Once we know both qbr and qvo interface, we can start analyzing traffic of the Instance using tcpdump/wireshark:

[root@compute ~]# tcpdump -i qbrc5dafda7-1f
[root@compute ~]# tcpdump -i qvoc5dafda7-1f

I think this is a good starting point in troubleshooting networking issues of Instances in OpenStack environment.


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.