VMTN: I/O Performance in vSphere, Block Sizes and Disk Alignment

Yes folks, it rears its ugly head again…Disk Alignment… If you have not read it yet, check out the whitepaper on disk alignment from VMware.

First, chethan from VMware posted a great thread on VMTN about I/O performance in vSphere. The start of the thread talks about I/O, then leads into anice discussion about block size. A couple of weeks ago, Duncan Epping posted a very informative article about block sizes. It convinced me to use 8MB blocks in VMFS designs.

Finally, the thread kicked into a discussion about disk alignment. As you know, the VMFS partitions created using the VI Client will aoutmatically be aligned. This is why I advocate NOT putting VMFS partitioning into a kcikstart script. The whitepaper demonstrates how to create aligned patrtitions on winders and Linux guests as well. The process is highly recommended for any intensive app. But I have always questioned the need to do this for system drives (C:) on guests. To do it requires a multi step process or the use of a tool, like mbrscan and mbralign, And I have wondered if it was worth the effort. Well, Jason Boche gave me a reason why it should be done across the board. And it makes sense: “This is an example of where the value of the savings is greater than the sum of all of its parts.”

Jas also outlined a very nice process for aligning Linux VMs and fixing a common Grub issue. Thanks for the tip Jas!

I should also thank everyone else involved: Chethan, Duncan and Gabe!

Differences between vSwitches and dvSwitches

There are not huge differences between a vNetwork Standard Switch (vSwitch, vSS) and a vNetwork Distributed Switch (dvSwitch, vDS). The big thing is the concept of dvSwitches being centralized in vCenter and using the concept of compliance to assign a dvSwitch to a host.

Both types of switches provide the following:

  • can forward L2 frames
  • can segment traffic into VLANs
  • can use and understand 802.1q VLAN encapsulation
  • can have more than one uplink (NIC Teaming)
  • can have traffic shaping for the outbound (TX) traffic

Other “features” of dvSwitches are the following:

  • can shape inbound (RX) traffic
  • has a central unified management interface through vCenter
  • supports Private VLANs (PVLANs)
  • provides potential customisation of Data and Control Planes
    • supports using the Nexxus 1000v

For more information on the concepts of dvSwitches and the differences between vSwitches and dvSwitches, check out this VMware KB Article.

VMware ESX Configuration Maximums Comparison Matrix

Have you ever needed an easy to reference way to see what the configuration maximums are for different versions of VMware ESX.  I know I seem to need this all the time.  I find it a huge pain to keep referring to each of the individual VMware documents to get the answers.  Sometimes I also want to see what the changes are between versions and I can’t seem to memorize this information in my tiny little brain.  So I went ahead and created a “Configuration Maximums Comparison Matrix” based on the VMware Configuration Maximums for each version.

You’ll notice some settings don’t have values for each version.  This is because they were not published in the VMware documents.  As I go through some additional documents and extract these values I will update the document to reflect.  For no the document does include everything from the VMware Configuration maximums published for each of these Versions:

VMware ESX 3
VMware ESX 3.5 & ESX 3.5 Update 1
VMware ESX 3.5 Update 2, Update3, & Update 4
VMware vSphere 4.0 (ESX 4)

You can find the document in our downloads section or you can click here. Hope you find this useful I know I will.

vSphere Install and Upgrade Best Practices KB Articles and Links

So, I use NewsGator to aggregate a BAZILLION feeds from several sources, blogs, like this one, actual news feeds and a bunch of VMware feeds. The VMware feeds are from the VI:OPS and VMTN forums. The VMTN forums allow you to create a custom feed by selecting the RSS link at the bottom right of each page or you can get a feed from a specific section of the forum by clicking the link on the bottom left of a list. On of the custom feed options is to get a feed of the new KB articles.

VMware has released quite a lot of new KB articles surrounding vSphere. They just released nice best practice guidelines for installing or upgrading to ESX 4 and vCenter 4. They are short and to the point. There is also a nice article covering best practices for upgrading an ESX 3.x virtual machine to ESX 4.0. One thing I noticed, but never thought about is this :

“Note: If you are using dynamic DNS, some Windows versions require ipconfig/reregister to be run.”

Eric Seibert over at vSphere-Land posted a nice set of “missing links” for everything vSphere. This is a nice, comprehensive set of links to evetrything you need for vSphere upgrades or installs.So, go check that out as well.

SPLUNK! Goes the Syslog Server…

The use of a “syslog” server is important in today’s data center. Most network and SAN switches, along with Unix and Linux servers are capable of sending logging information to a syslog server. The obvious reason for a syslog server is to centralize all of your logs. This enables you to troubleshoot issues more efficiently. Most syslog servers allow you to do a time-line based analysis of log data so that you have an enterprise – wide view of all activity. This allows you to see how different devices interact.

An less obvious reason for a syslog server is for security purposes. The theory is that an attacker will attempt to elevate to root privileges and then try to delete or alter logs to hide evidence of the attack. If all log information is relayed to a syslog server, the hope is that this data is secured for forensic study, if needed.

I have tried a few different “free” and non-free syslog servers. I didn’t do extensive research into all available syslog servers, but I have to say that I like Splunk the best. It starts with a free server with a limited amount of data. This may be fine for smaller shops. There is also a paid version that allows for more data collection. The fully “free” syslog server that came close was the combination of syslogd and phplogcon on a Linux server. I also tried Kiwi syslog, which also has a “free” version and a paid version. But it only installs on winders. Most of the syslog servers are great. There were a few capabilities I felt made Splunk a nice syslog server:

  • Act as a standard syslog server.
  • The ability to “scrape” directories.
  • Monitor Windows logs.
  • Allow for upload of log data.
  • Provide Time line Analysis.

Acting as a standard syslog server is really a no-brainer. All of the packages that I tested worked fine in this respect. You set up pointers to the syslog server in the *nix /etc/syslog.conf file and all logs are automatically sent.

When dealing with collecting logs on an ESX server, the standard syslog.conf settings may not cut it. The HA logs reside in a different location and should be “scraped”. In this context, “scraping” is the process of reading all of the text files in a specified directory and compiling them into the syslog database.

Monitoring Windows logs is also a key ingredient in the datacenter stew. If you are going to do centralized collection of logs, collect everything. Splunk uses WMI to gather this information.

The ability to upload log data manually is also a nice option. I was recently troubleshooting an issue with VMware Consolidated Backup and I was able to manually upload all of the related VCB logs right into a Splunk server VM. I exported the Windows system and application logs to .csv files and copied them to a directory on the Splunk server. I also copied the VCB logs and ESX logs to the same directory. After a few minutes, the data was assimilated into the database and ready for analysis. I was able to look at a specific point in time and look at errors across the entire environment. I could see errors in the VCB logs and relate them to errors in the Windows system and application logs. I was also able to track all of the ESX and VM logs for the time period.

The Splunk server offers WAY more than the logging functions described here. It is also a great tool for compliance, change control, security, server management, etc. It has install packages for winders, Linux, Solaris (x86, x64 AND Sparc), Mac OSX, FreeBSD and AIX.

As you can see, the Splunk server is very useful for capturing all kinds of logs for security and troubleshooting purposes. In part two, I will dig deeper into setting up a Splunk server and configuring *nix, ESX, ESXi and winders machines to send their logs. As with the VCB Proven Practice Guide, there will be a companion doc on the VI:OPS site.

Go Virtual, Get Green, Save Money, & Be a Hero!

In this tough economy many companies are scrambling to virtualize to cut costs and promote a green environment.  Others would like to but with budgetary constraints and freezes they just can’t get the funding to take on a VMware virtualization initiative.  Even with the promises of ROI on their horizon once they do so it just doesn’t seem enough for some organizations.  Well I’m here to say so what, don’t let a light budget get in your way.

Start with the low hanging fruit, development, and staging machines.  Take the initiative to take advantage of these free hypervisors and save your organization some money.  Once you do reap the reward.  Keep track of how much money you just saved your organization, let everyone know, and not only will you get recognized, but you will just have gained some job security.  I’m surprised more organizations are offering cost savings initiative to hep try and curb the expenses and cut back.  Think about it, if your company offered a bonus based on savings everyone would be looking for ways to cut costs and cash in on the savings.

I heard of a company that did just this.  Employees were offered bonus equal to a percentage of what they saved.  How great is that…..it is a win win for everyone.  The company saves money, the employee who drove the initiative makes more money, jobs are saved and everyone is a hero.  So what are you waiting for?  If your company doesn’t offer something like this why not drop a note in the suggestion box.  Then go get yourself some by getting virtual.

How you say, it’s very simple.  Develop a plan.  Identify the servers in your environment that would be easy candidate, nothing production or at least nothing in the lime light.  Determine how many you could virtualize on to how many hosts and calculate the savings that could be had with nothing but a little bit of elbow grease.  Submit your finding and  cash in.  I know I make it sound so easy, but the reality is that it’s not that hard.  I never could understand why it is not a much larger initiative.

Even if you don’t feel you could tackle the job yourself or with the IT staff in your organization, no problem.  Factor in consulting costs, yes I know spend money, but you can still save considerably with this approach.  Consulting fees are fixed term costs.  There are no licensing, support, etc….and once your project is complete you will immediately start saving and be able to start getting that outlay of capitol back in the bank.  Many consulting companies offer Resident Consultants that you can work out a fixed fee to perform the work, no need to buy licensing or a paid product for these guys, just pay for their time.

I know I know all these companies want to sell you hardware and licensing and they push you into quoting you extremely high prices for this end to end.  Forget those guys I can tell you that there are companies that are not like that and focus on the services and the customer.  I can tell you this because I work for one.

Citrix XEN Server a Free Hypervisors

So everyone knows that the big three all have freely available versions of their hypervisor.  The obvious reasons are to gain market share.  Get you hooked for free and then charge you for the all the extras.  Just because that is their plan, doesn’t mean it has to be yours.  You can go virtual now and later when everything levels off with the economy you can choose to get the more advanced features, or not if you feel you don’t need them.

Citrix is including XenMotion and the management console for free.  This alone gives them a huge advantage over VMware and Microsoft in the free hypervisor market.  I frequently hear people stating the XEN is not an enterprise solution yet, well get your head out of the sand because it most certainly is.  Not more than a year ago I was saying the same thing when I got certified on XEN 4, but now with the release of XEN 5 I totally believe that XEN is an enterprise solution and a damn good one at that.

If you haven’t kicked the tires around on yet, what are you waiting for?  It’s free!  You have nothing to loose except a little time, even then you will at least gane some knowledge that you didn’t have before.  So what are you waiting for go get virtual.

Setitng up Hyper-V Linux Integration Components on SLES 10

After finally setting up a SLES 10 server on Hyper-V I thought I write a little how to on getting the Linux Integration components working. Microsoft includes an install doc that doesn’t really tell the whole story.

I had already installed SLES before I read the manual and found out that XEN Virtualization support needed to be installed as well and Microsoft documentation also states that you need the C++ compiler. What they really mean is the gcc compiler. What they fail to mention is that you also need he kernel-sources installed as well.

Read More of this post

Microsoft Hyper-V and Linux…is it a match made in heaven?

Microsoft has recently launched an initiative through it’s interoperability division to prove Hyper-V can compete in the Linux realm. Microsoft has partnered with Novell to provide Hyper-V support for SUSE Linux (SLES 10). Expanding on this support Novell has built a Systems Center Configuration manager Management Pack for SLES 10. This management pack is compatible with the SCOM 2007 R2 Release Candidate with hopes to be GA by the end of this year.

Ream More of this post

VMware vSphere Upgrade Path Overview

Many of you are wonder how you will go about upgrading to VMware vSphere when it is release. Well I’m here to say don’t worry. The upgrade path from ESX 2.x & 3.x is very painless and fairly simple. A lot of you will remember all the phone you have had in the past performing upgrades and scripting installs, well VMware is quickly trying to make all of that a thing from the past with new features available in vSphere.

Read More here

Just some more vSphere information

Here is some information about vSphere that I thought would be good to share with the world. As with everything else this is just a drop in the bucket. I’m currently working on putting together some upgrade videos and screenshots so take a look back and hopefully I will have them done by the end of the week.

Here is some interesting information about vSpehere and what it supports keep in mind these are just some notes I jotted down:

ESX 4 Hosts (vSphere Host)
256VM’s per host
64 Cores per host
512GB Ram per host

vSphere VMs (Hardware version 7)
8 vCPUs
256GB Ram
VMDirectPath I/O
Hot Plug Support (Supports CPU’s and Memory)
ESX 2.x and 3.x VM support
Paravirtual SCSI adapter
MSCS 2008
Persistent Reservations in vmkernel
LSI Logic SAS (Virtual SAS controller)

Networking Improvements
New iSCSI stack with 10-30% improved performance
TCPIP 2 Support (Based on FreeBSD 6.1 / IPv6 / locking and threading capabilties)
VMXNet3
MSI/MSI-X
Receive Side Scaling
VLAN offloading
VMware Directpath I/O

Storage Improvements
SCSI-3 Compliant
VMFS still SCSI-2
Target PortGroup Support (TPGS)
Asymmetric Logical Unit (ALUA)
Pluggable Storage Architecture (PSA)
Updated iSCSI stack
Native SATA

Service Console
64-bit, 2.6 based Linux kernel compatible with RHEL 5.2
Supports for both 32bit and 64bit applications
root file system stored in VMDK
vmkernel runs and owns device drivers only 64-bit
Address Space layout Randomization (ASLR)
No Linux dev packagers and libraries

CPU
Enhanced Intel step down
Enhanced AMD Power Now

Security
Trusted Platform Module (TPM)
Digitally signed and validated modules
Memory integrity techniques with microprocessor capabilities to protect against buffer-overflow

Guided Consolidation
500 Simultaneous Physical Machines
Modular Plug-in can be installed on different machine

Coverter

Physical / Virtual / 3rd party
Server 2008 Support
Covert Hyper-V Machines to VM’s

Update Manager

ESX / ESXi and Virtual Appliance Upgrades
Upgrade Virtual Hardware
VMware Tools
Base Line Groups

Upgrade vcenter steps

No SQL 2000 Support
2.x & 3.x Upgade Path
Upgrade vCenter
Upgrade Update Manager
Use Upgrade Manager to Upgrade Hosts
Upgrade VMware Tools, then the VMware Hardware.

vSphere Host Update Utility
3.x to 4.x
Dosn’t Upgrade VMFS Datastores or VMs
Installs with vSphere client
Support Rollback for ESX only
Can be used to install patch releases to standalone hosts
Copies Script and ISO to ESX hosts reboots and installs