KVM OpenStack Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]

Packstack Error: Command exceeded timeout
Few times during KVM based OpenStack (Mitaka, Newton) automated installations using packstack we encountered DB synchronization errors.

It turned out, that these installation errors appeared due to slow network and/or poor hardware performance of KVM virtualized hardware used to build OpenStack virtual nodes (Controller, Network, Compute).

After running OpenStack installation using packstack:

[root@controller ~]# packstack --answer-file=/root/answers.txt

…the installation failed with the following timeout errors:

ERROR : Error appeared during Puppet run:
Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]: Failed to call refresh: Command exceeded timeout
ERROR : Error appeared during Puppet run:
Error: /Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]: Failed to call refresh: Command exceeded timeout

To fix db sync errors during KVM based OpenStack installations, you can perform the following steps:

1. Execute packstack automated installer with extended timeout

Packstack uses Puppet for OpenStack automated node deployment. Packstack, by default, sets timeout for Puppet’s manifests execution to 300 seconds. If you install OpenStack on old hardware, which has poor performance or the nodes are connected to the network with poor throughput, Puppet may encounter execution timeouts when applying it’s manifests across all OpenStack nodes. In other words, Openstack nodes are not able to respond to Puppet (running on Controller node) within the default timeout, this causes Packstack installation to fail.

We can execute packstack installation with timeout parameter to extend Puppet’s manifests execution timeout (i.e.: up to 600 seconds):

[root@controller ~]# packstack --answer-file=/root/answers.txt --timeout=600

2. Set CPU pinning for OpenStack KVM nodes (KVM performance tuning for OpenStack)

CPU pinning significantly improves performance of KVM virtual machines and all KVM virtual/isolated networks where virtual machines are hooked up together. KVM CPU pinning reserves particular KVM hypervisor’s CPU(s) for particular VM to secure it’s demand for CPU resources.

We can use virt-manager to pin hypervisor’s CPUs to particular VMs.

Example pinning scheme:
Hypervisor’s total CPU resources: 4 CPUs (CPU0, CPU1, CPU2, CPU3)
– Compute node: 1 CPU (CPU0)
– Controller node: 2 CPUs (CPU1, CPU2)
– Network node: 1 CPU (CPU3)

Compute node CPU pinning:
cpu pinning compute node openstack kvm installation

Controller node CPU pinning:
cpu pinning controller node openstack kvm installation

Network node CPU pinning:
cpu pinning network node openstack kvm installation

3. Provide sufficient amount of RAM for OpenStack nodes

If your nodes (especially Controller node) run out of RAM during packstack installation, they will start swapping and their performance will significantly decrease. Equip your nodes with the appropriate amount of RAM to avoid performance drops.

Below we propose the minimum RAM requirements for OpenStack nodes:
– Controller node: 3GB RAM
– Network node: 2GB RAM
– Compute node: 3GB+ RAM (depends on the amount of Instances)

Share on: Share on FacebookShare on Google+Tweet about this on TwitterShare on StumbleUponShare on LinkedInPin on PinterestBuffer this pageShare on TumblrDigg thisFlattr the authorShare on RedditShare on VKShare on Yummly
Liked it? Take a second to support tuxfixer.com on Patreon!
Hadoop Developer Training

One thought on “KVM OpenStack Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]

  1. ghost July 23, 2017 at 20:44

    Got the same error on Hyper-V.
    Have increased VM configuration to 4 cores and 8 GB of Ram but – still the same. Any ideas?

Leave a Reply

Name *
Email *