#vCops 6.0 Beta

Finally one might say. Its been long awaited.

The vCops 6.0 beta has been released, and I’ve been lucky enough to have been included in the beta group testers. I’ve asked VMware if I’m allowed to blog about the product itself, but has as of yet, not been given a permission to talk about the features. As it is still in Beta some features might not make it to the final version. A version I do hope will be announced at #vmworld in San Francisco in August.

One thing they’ve kinda promised, is the ability to run different licensed versions against different clusters or vdc’s. I might not be all customers who want to pay for the full features. Right now even though you might have different licenses in your environment vCenter only allows you to map 1. Meaning people usually run a smaller license all over the place, than the one they might actually have paid for (in the vCloud suite). In the environment I work with daily we have a mix of Standard and Advanced licenses depending on the requirement of the clusters. We don’t really have the need for the Advanced features on our development clusters, but it’s very nice to have on production. Especially since the Advanced license gives you access to 3rd party metrics, like SAN and Network equipment directly.

#VMworld – My expectations

Summer is approaching fast, and soon this years VMworlds in San Francisco and Barcelona will be upon us. This brings with it, a lot of speculations on what will be unveiled and what will happen with old products.


As stated in an earlier post, I’ve submitted a Session to VMworld about vCops, and are right now waiting on June the 2nd, to hear whether or not that session has been accepted. Regardless of the outcome, I will attend the US VMworld in San Francisco again. Hotel, Flight and VMworld ticket has already been purchased, but a short trip to Barcelona might be required if the session is accepted. This will be my 5th VMworld.

This post is about my views on VMworld and what I think will happen there, it’s not based on any facts so don’t put any bets down, based upon what I write here.

Last year nearly 23.000 VMworld fans attended the US conference. That is a huge number, but whats more amazing was the fact that, if I recall correctly, nearly 70% of the attendees were first timers. If even more people attend this year, I fear the Moscone Center will be too small 🙂

So what can we expect from VMware in august. They have often announced the new version of vSphere, so that should suggest we would see them unveil vSphere 6.0 at the US conference. I however have heard talk that suggests otherwise. It suggests that VMware will focus more on stability when it comes to vSphere and lengthen the release cycles to 18-24 months, from the current 12. This in my eyes is great news, the base Virtualization layer must be very stable, as it is the foundation that all their other products build upon. So I don’t expect vSphere 6 till sometime early 2015.

I expect to see version 6 of vCops however. There has been a closed beta for a while now, and there are some improvements in there that are really needed. Like different licenses for different vCenters or Clusters. You might not want the biggest license for your development cluster, or a customer might not want to pay for the biggest license in your hosting center. I’m told that Custom reports has gotten a big lift.

NSX was the big talking point last VMworld, but it hasn’t really hit it off like VSAN has. I think this might be due to the lengthy beta phase of VSAN, and I’m hoping VMware will try to let people have a go at NSX before buying it. One of the features I would love to see in NSX was to switch the built-in firewall and load balancers to something else. But I’m guessing that a lot of at least financial and pharmaceutical customers would have to qualify the built-in versions. A faster adoption could be that the firewall could be a Checkpoint or Juniper. The same goes for the load balancing part an F5 or a Citrix Netscaler plug-able version would help with a lot of the customers.

Last year we had Train supported by Imagine Dragon at the US conference and Taio Cruz in Barcelona. That’s two widely different styles of music. But I’m actually out of guesses on which bands will be selected this year, guess we just have to wait and see 🙂

Food, drink and Snacks
One thing I’ve always missed at VMworld, was better catering. It seems like its being kept to the bare minimum and last year in SF I only tried the lunch once, it was really bad. Barcelona 2 years ago was also bad, which kinda baffles me considering the amount of great food you can find in Barcelona. Looking at TechEd or CiscoLive my colleges who come home from these, are actually amazed at the amount of food, drink and snacks that are available to them during the day, both healthy and no-so-healthy stuff. I remember having to bring my own Coca Cola to VMworld last year. VMware please step up the effort on this part, pretty much the only bad thing I have to say about VMworld.

My goal this year at VMworld is to try to collect as many T-shirts I can grab, and review them during VMworld, so hopefully if you find my reviews nice you can go grab the same shirts. If it really hits off, I’m hoping people would suggest where I could find great shirts too.

Hmm writing about this makes me hope that august comes much quicker! 🙂 I’m really looking forward to VMworld and hope to see a lot of you there!.



Using vCenter Operations Manager to help in performance testing Part II

Just a quick follow-up post on how things went after the last post about this topic

So after finding that scaling the webservers up wasn’t the solution to the problems we experienced with the webservers, we started to look elsewhere, and actually found that a load balancer that was set to do round-robin load balancing instead of least used, caused all the hiccups. Once that was changed we started hitting the limits on bandwidth instead.

And may 14th the website was actually launched, and there hasn’t been any stress at all on the infrastructure since launch.

Here is a shot of how everything looks after launch.

Demand - Usage - Core Util 22 maj

The spikes around may 1st. is due to us stress testing the system before launch. And there is a tiny little spike on may 14th when the website went live. And there hasn’t even been anything that vCops found to be outside normal (no yellow boxes) after the website was launched.

Overall I’m really pleased with the fact that vCenter Operations Manager stores your performance stats for you to review at your pleasure, and the amounts of metrics you can pull out of the system is really amazing.


Using vCenter Operations Manager to help in performance testing

I’ve recently been involved in performance testing a SharePoint 2013 farm. This has led to some discoveries on what you can use vCops for when you do performance testing, including what metrics you should look at.


The setup we used for testing,  was what Microsoft calls a Medium Farm topology. It consists of 2 Web Frontends, 2 application servers for search and index and 1 database server. In front of the web frontends we’ve placed a Citrix Netscaler for load balancing, SSL off load and such. Each server runs Windows 2012, the SharePoint is 2013 and Microsoft SQL is 2012. The SharePoint has a few webparts on the front page that’s been built for the webpage.

Initial configuration:

  • Webservers: 16 vCpu’s  16 Gb ram
  • Application Servers: 8 vCpu’s 8 Gb ram
  • Database Server: 8 vCpu’s and 16 Gb ram

All these were run on a single vSphere 5.0 host with 4 CPU’s and 8 cores each, and 196 Gb of Ram, giving us 32 cores. So at this point we were overcomitting on CPU somewhat, having 56 vCpu’s provisioned. These were not the final destination hosts of these servers, at the final destination hosts we wouldn’t overcommit. However we were certain that if we saw bottlenecks here then they would also be present at the final destination for the farm.

One of the requirements was that the front page should load in less than 2 seconds, with up to 2000 users in 5 minutes. The test runs were run in Visual Studio and simulated users viewing the front page, loading all the javascripts and stylesheets and stuff. To make sure we would get data into vCops that was a full load for the 5 minutes intervals, we ran the test for a full 15 minutes.

Inside Windows we could see the CPU maxing out, to a point where system processes were complaining, and we saw load times of over 30 seconds for the front page.
First thing that happened was someone was looking inside windows and saying more CPU is needed. I, however was looking at vCenter performance and only seeing a max of roughly 40% CPU usage. When looking at vCops i saw a Usage maxing out at 35.63% but a demand of 54.52%.

Demand - Usage 20 februar

I interpreted that result as, the VM actually requested more CPU than vmware was giving it. However it was decided to try and give 8 more vCpu’s to the Webservers, instead of scaling them down to 8 vCpu’s

Running the test again with 24 vCpu webservers yielded the exact same result, more than 30 second load times and CPU inside windows maxing out. Looking at vCops we saw this:

Demand - Usage 21 februar

An even lower % Usage and Demand. I’m thinking we’re overcomitting too much now, and going back into vCops i pulled out the %Ready counters as well, for both feb 20 and 21.

Demand - Usage - Ready 20 februar

Demand - Usage - Ready 21 februar

What that told us was that at 16 vCpu’s %READY maxed out at around 14%, which is kinda bad. And adding 8 more vCpu’s that jumps to 28.75%, meaning close 1/3 of the time the machine was ready, but couldn’t get access to a CPU on the host. One funny thing you can tell from the first picture, is that you can tell when the 8 vCpu’s were added, since that moved the “idle” %READY from around 6% to 14.87%.

The following Monday we scaled the machines down from 24 to 8 vCpus’s and ran the test again. Load times were still at around 30 seconds, so we didn’t really solve the problem, but looking at vCops we saw a completely different picture

Demand - Usage - Ready 24 februar

The graphs shows the %READY dropping from roughly 12% to around 1%, and funny thing here is that, while we’re running the test the %READY drops even further.

At this time we decided to give Michael Monberg from Vmware a call, to ask about what to look for in vCops. He showed us this neat trick, when looking at performance metrics for a given VM, and have some graphs from the VM showing, like the ones above,  you can do this:

Click on the + sign near the health tree

health tree It expands and shows you:

klik server

The host, the VM and the Datastore, if you then single click on the host, you get a new Metric Selector

metric selector

But one from the Host, so expanding the CPU Usage i could select the Core Utilization metric and add that to the graph page. That now gave us graphs from the VM and the Host and the same time. Showing the graphs from Feb 20, 21 and 24 but with the new metric added:

Feb 20, 16 vCpu’s

Core util 20 feb

Feb 21th 24 vCpu’s

Core util 21 feb

Feb 24th 8 vCpu’sCore util 24 feb

From that we could tell that with 16 and 24 vCpu’s the host was totally maxed out on its physical cores, where on 8 vCpu’s we only used around 66%. So when the VM metric only shows 35% CPU usage the Host was maxed out, and thus adding the 8 extra vCpu’s had no positive effect on the VM.

When we ran the test with 8 vCpu’s and got the exact same results, we actually weren’t crossing NUMA nodes, which is what Microsoft recommends. See my blog post on NUMA and vNUMA here

This blog post was written to show some of the nice things you can do with vCenter Operations, and i hope you found it useful.