Categories
Tech

Auto-scale Application with vRealize Automation and vRealize Operations

Automation and operations help each other, applications’ reliability is guaranteed

I was requested to implement an auto-scale use case for a web application. Personally, I am not a fan of this auto-scale use case for some reasons, but elaborating on this could be the topic for a nice 9:30 chat with my friend Luca and possibly the subject for a new blog post. This entry in our blog is about how to configure auto-scale with vRealize solution family. What I like of this use case is that it gave me the chance to experiment with something I really like (vRA REST API) and dip my toes with something I am less familiar (vROps Alerts/Policies and vRO Scriptable Tasks). For this specific lab I used version 8.2 of vRA/vRO and vROps, but I am quite sure the same applies to 8.0 and 8.1 as well.

The use case is about using vRA to deploy a web application that, in addition to other resources, presents one or more web server behind a load balancer. vROps monitors web server(s) load and if the load gets too higher for the current number of web servers it fires an alert that triggers a vRA deployment update day 2 action that increases the number of web servers. This is a scale-out scenario, a similar approach applies to the scale-in scenario that is triggered when load on web servers is lower than a given thresholds. Of course, in this second scenario the deployment update action reduces the number of web servers.

This is a summary of the configurations I had to put together to implement the use case:

  1. A VMware Cloud Template that deploys a web app with a configuration suitable for scale out/in (load balancer and a clustered frontend web servers);
  2. A vROps alert definition with a recommendation that automatically runs an action (vRO Workflow) when the alert is triggered;
  3. A vRO Workflow that executes a vRA ABX action (this is basically a wrapper for a vRA REST API request);
  4. A vRA ABX action that determines the new size of the cluster and update the right deployment accordingly.

This could be implemented in many ways and certainly my approach is inefficient as points 3 and 4 in the list above could be implemented into a single vRO workflow.

OK, let’s get started.

First things I needed was a VMware Cloud Template that deploys a web app with one or more web servers behind a load balancer. I am reusing a Cloud Template created by Chris McClanahan and his team, this template deploys a web app named OpenCart (an open source ecommerce platform https://www.opencart.com/).

(Click on the image to enlarge)

Here the important point is that the template allows to have multiple VMs named frontend behind a NSX-T load balancer. We control the number of fronted VMs by means of an input parameters named clusterSize that can have the following values: small, medium and large.

clusterSize:
  type: string
  enum:
    - small
    - medium
    - large
  default: small
  title: Frontend Cluster Size
  description: Frontend Cluster Size

Values of the clusterSize parameter are mapped as follows: small is 1 VM, medium 2 VMs and large 4 VMs.

count: '${input.clusterSize == "small" ? 1 : (input.clusterSize == "medium" ? 2 : 4)}'

Next I had to create an ABX action on VRA, I didn’t started from scratch as along with the Cloud Template Chris passed to me a python script that does 60% of the work I need. Here is the logic of final script that lays behind the ABX action:

  • The action expects 2 inputs: vrops_moid is the MOID of the VM with the load too high/low and the direction of the scale action: out (we need to add web servers) in (we need to remove web servers);
  • First thing the script retrieves the vRA deployment the vrops_moid belongs to;
  • It retrieves the current size of the frontend cluster;
  • It determines the new size of the cluster according to the current size and the direction input;
  • It updates the deployment with the new cluster size.

Then I turned to vRO to build a simple vRO workflows that accepts a VC:VirtualMachine object as input, it authenticates on vRA to retrieve a bearer token and use it to invoke the vRA REST API that execute the ABX action we created before. To automatically run this workflow when a vROps alert is fired (auto-remediation of an alert according to the VMware terminology) the workflow is required to have only one input parameter and this input parameter has be of type VC:<VC datatype>. Actually, I created two vRO workflows named AutoScaleOut and AutoScaleIn, they are identical with the exception of the value assigned to the direction input parameter of the AutoScale ABX Action. As last step I created a package that includes workflows and their dependencies.

As a final step I had to configure the vROps alert. In order to execute vRO workflow as alert auto remediation, it is required to have installed and configured the Management Pack for vRO. I am not covering this in this blog post. Here is the detailed procedure to configure and execute auto remediation. Hereafter I just go through the key steps I went through.

First thing I had to import my vRO package into vROps. I had to be patient and wait for a collection cycle before my package including my workflows appeared along with those previously listed.

Second, I mapped the my workflows to a vCenter resource of type VC:VirtualMachine (this have to match the vRO workflow input type).

Then I turned my attention to create the custom alerts, but before that I created a new policy named AutoScale that applies only to frontend VMs (for this purpose I created a custom group that includes only frontend VMs). This policy was created to host my new custom alert definitions. I am pretty sure this is not the most appropriate approach, but it was just fine for my purpose.

The remainder of this post just cover the creation of the custom alert definition for the scale out use case, the same procedure applies to the scale in alert definition with few minor differences (e.g. scale symptom threshold definition). First step to create my custom alert was to create a custom recommendation with action configured with the AutoScaleOut workflow.

Then I created a symptom which looks for VMs that have sustained CPU Usage (e.g. greater than or equal to 70%). I could add one or more symptoms related to Load Balancer Sessions or HTTP Requests (these and other metrics are available in vROps). This can be part of a monitoring tuning activity.

At this point I was ready to create my custom alert based on the symptom and recommendation created before. This alert will invoke the vRO workflow/action automatically when CPU Usage of frontend Virtual Machine is higher or equal to 70%. Oh, by the way the new UI for alert definition and policy management is simply awesome and allows a person with little experience like me to get the job done quickly.

Finally I had to check that my alert had both State and Automate properties set to Enabled.

Time to test it!

I have a OpenCart Cloud Template based deployment with clusterSize = small (1 VM). On the only frontend VM I generated high CPU utilization long enough to trigger an alert, I used the stress Linux tool.

vROps triggered an alert according to my symptom definition

The alert automatically started the vRO workflow that through the ABX action triggered the deployment update.

And finally my deployment ended up with 2 fronted VMs