Home > OpenStack > Deployment Considerations for nova-conductor Service in OpenStack Grizzly

Deployment Considerations for nova-conductor Service in OpenStack Grizzly

The Grizzly release of OpenStack Nova includes a new service, nova-conductor. Some previous posts about this service can be found here and here. This post is intended to provide some additional insight into how this service should be deployed and how the service should be scaled as load increases.

Smaller OpenStack Compute deployments typically consist of a single controller node and one or more compute (hypervisor) nodes. The nova-conductor service fits in the category of controller services. In this style of deployment you would run nova-conductor on the controller node and not on the compute nodes. Note that most of what nova-conductor does in the Grizzly release is doing database operations on behalf of compute nodes. This means that the controller node will have more work to do than in previous releases. Load should be monitored and the controller services should be scaled out if necessary.

Here is a model of what this size of deployment might look like:

nova-simple

As Compute deployments get larger, the controller services are scaled horizontally. For example, there would be multiple instances of the nova-api service on multiple nodes sitting behind a load balancer. You may also be running multiple instances of the nova-scheduler service across multiple nodes. Load balancing is done automatically for the scheduler by the AMQP message broker, RabbitMQ or Qpid. The nova-conductor service should be scaled out in this same way. It can be run multiple times across multiple nodes and the load will be balanced automatically by the message broker.

Here is a second deployment model. This gives an idea of how a deployment grows beyond a single controller node.

nova-complex

There are a couple of ways to monitor the performance of nova-conductor to see if it needs to be scaled out. The first is by monitoring CPU load. The second is by monitoring message queue size. If the queues are getting backed up, it is likely time to scale out services.  Both messaging systems provide at least one way to look at the state of message queues. For Qpid, try the qpid-stat command. For RabbitMQ, see the rabbitmqctl list_queues command.

About these ads
Categories: OpenStack
  1. February 19, 2013 at 11:06 am | #1

    It would be interesting to see some reference configurations for cells and nova controller as we scale to more hypervisors/VMs.

  2. February 19, 2013 at 2:00 pm | #2

    Being a proponent of the nova-no-db-compute effort, I think this is a step in the right direction. I hope the conductor is only a refactoring step in resolving the interaction between controller and compute nodes.

    Did you test the “scale out” setup regarding actual scaling? Depending on the usage and interaction patterns I suspect that you might actually see an increase in message queue size and api response times because of blocking transactions in the db if you add more schedulers/conductors, because in the end these services coordinate ressources (like available memory etc.) which are synchronised via the db.

    When I tested the folsom version (single scheduler) I already noticed an increase in api response times with higher concurrency in requests. If my reasoning is correct, this should get worse with more schedulers and conductors.

  3. February 19, 2013 at 2:58 pm | #3

    Yes, this is indeed a first step in refactoring the controller/compute interaction. There are others already working hard on design proposals for further significant refactoring to make the nova-compute service more of a slave to logic that has been moved up. Details are still being worked out though …

    I haven’t personally tested out the scaling. It’s somewhat theoretical, but also based on what I learn from talking to people deploying things. I would definitely be interested in hearing things that work well, or don’t work well, based on real usage.

    You’re right that there is some resource coordination, but much of it is actually fairly optimistic. A scheduler will pick a node for an instance to run on, but it doesn’t completely assume that it will work. The request will go down to the compute node and resources get reserved there. If it doesn’t work out, the request gets kicked back to the scheduler for it to retry somewhere else. So, I don’t think there should be a lot of blocking and contention in the scheduler on the db.

  4. August 8, 2013 at 5:05 pm | #4

    Hi Russel, i’ve deployed grizzle like yo mention on the first image, but i can’t launch any VMs. I think it’s related to the nova-conductor being misconfigured, but i haven’t found any doc about it. Where should i tell nova-compute it has to use nova-conductor? Thanks

  1. June 15, 2013 at 11:58 pm | #1
  2. August 7, 2013 at 4:14 am | #2
  3. September 2, 2013 at 4:02 am | #3

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 1,189 other followers

%d bloggers like this: