April 2009 – Dailyhypervisor

April 30, 2009

Installing VMware ESX 4 in Text Mode

There are many reasons to install VMware ESX in text mode. The main reasons I use text mode are that it seems quicker for me and text mode responds better when using remote console connections, such as iLo, DRAC or console over IP. Previous versions of VMware used a text mode that incorporated Anaconda and was very similar to the text mode for RPM based Linux distributions. The new text mode in ESX 4 is VERY rudimentary when compared to the earlier versions. Hoever, it performs very well and is fairly straight forward to use.

The text mode installer uses simple lists of choices. Usually, 1 is for continue or to answer yes. Some items will have more than one choice. Here is a screenshot:

The console OS truly appears as a VM in this version. You must create a datastore and then a VMDK that represents the COS. A disk of sufficient size will be required for this. My first attempt, using an 8GB disk failed. My second attempt, using a 10GB disk was successful.

You can download a doc outlining text mode installation HERE.

April 30, 2009

VI4 Certification News

From the VMware Certification Site:

Certification News:
VMware Certified Professional on vSphere™ 4
With the launch of vSphere™ 4, a new certification will be available. The VMware Certified Professional (VCP) on vSphere™4 beta exam will be available 30 days post GA. Candidates eligible for the beta exam will be contacted directly by VMware.
There are four possible paths to acheive VCP on vSphere™ 4
If you are NEW to VMware
Attend the VMware vSphere™ 4: Install, Configure, Manage course (first courses available in late June 2009) OR attend the VMware vSphere 4: Fast Track (available in Q3)
Take and pass the VCP on vSphere™ 4 exam
If you are currently a VCP on VMware Infrastructure 3
Take and pass the VCP on vSphere™ 4 exam. This option will only be available until December 31, 2009. Beginning in 2010, VCPs on VI3 must attend the VMware vSphere 4: What’s New class in order to upgrade.
If you are currently a VCP on ESX 2.x
Take and pass the VCP on VMware Infrastructure 3 exam
Take and pass the VCP on vSphere™ 4 Exam. This option will only be available until December 31, 2009. Beginning in 2010, VCPs on VI3 must attend the VMware vSphere™ 4: What’s New class in order to upgrade.
If you are not a VCP on VI3, but have attended one of the prerequisite classes (Install & Configure; Deploy Secure & Analyze; or Fast Track).
Take and pass the VCP on VMware Infrastructure 3 exam OR attend the VMware vSphere™ 4: What’s New course.
Take and pass the VCP on vSphere™ 4 Exam.
…more info

April 30, 2009

FLASH: ESX 4 Console OS is REALLY a VM this time!

While I was setting up ESX in text mode for my next blog post, I discovered that the installation sequence first creates a VMFS file system and then creates a VMDK file for the console OS. I confirmed it in the VIC. Here is a screen shot:

Click to enlarge image

I also noticed that the logs are now in a separate directory:

Click to enlarge image.

April 29, 2009

Running VMware ESX 4 RC in a VMware 6.5.2 VM.

I just set up another quick VI4 lab on my laptop for the purposes of capturing screen shots and testing some things out. I was worried because I was not able to start VMs in this lab using ESX 4 Beta 2, but everything is fine again! Here is a screen shot of a Winders 2003 VM running inside an ESX 4 RC VM which is running inside of Workstation 6.5.2 on an Ubuntu machine.

Click on the image for a full-size view.

My VMX settings were from a post on VMTN when I was trying to get ESX 3.0.x to run on a WS 6.0. Actually, XTraVirt came up with the solution originally.

Well, my VMX has not changed MUCH since then. I only added some parameters for sharing SCSI disks so I don’t need an iSCSI server. I found THAT information on Duncan’s Blog.

##################################################
# Start DAC Customization
guestOS = “other-64”
monitor_control.restrict_backdoor = “true”
# monitor_control.virtual_exec = “hardware”
monitor.virtual_exec = “hardware”
monitor_control.vt32 = “true”
# REQUIRD FOR USING NTFS DRIVES WITH LINUX HOSTS
mainMem.useNamedFile=FALSE
# For SCSI disk sharing
disk.locking = “FALSE”
diskLib.dataCacheMaxSize = “0”
diskLib.dataCacheMaxReadAheadSize = “0”
diskLib.dataCacheMinReadAheadSize = “0”
diskLib.dataCachePageSize = “4096”
diskLib.maxUnsyncedWrites = “0”
bios.bootDelay = “5000”
ethernet0.present = “TRUE”
ethernet0.connectionType = “custom”
ethernet0.wakeOnPcktRcv = “FALSE”
ethernet0.vnet = “/dev/vmnet3”
ethernet0.virtualdev = “e1000”
ethernet1.present = “TRUE”
ethernet1.connectionType = “custom”
ethernet1.vnet = “/dev/vmnet3”
ethernet1.virtualDev = “e1000”
ethernet1.wakeOnPcktRcv = “FALSE”
ethernet2.present = “TRUE”
ethernet2.connectionType = “custom”
ethernet2.vnet = “/dev/vmnet3”
ethernet2.virtualDev = “e1000”
ethernet2.wakeOnPcktRcv = “FALSE”
ethernet3.present = “TRUE”
ethernet3.connectionType = “custom”
ethernet3.vnet = “/dev/vmnet3”
ethernet3.virtualDev = “e1000”
ethernet3.wakeOnPcktRcv = “FALSE”
ethernet4.present = “TRUE”
ethernet4.connectionType = “custom”
ethernet4.vnet = “/dev/vmnet3”
ethernet4.virtualDev = “e1000”
ethernet4.wakeOnPcktRcv = “FALSE”
ethernet5.present = “TRUE”
ethernet5.connectionType = “custom”
ethernet5.vnet = “/dev/vmnet3”
ethernet5.virtualDev = “e1000”
ethernet5.wakeOnPcktRcv = “FALSE”
ethernet0.addressType = “generated”
ethernet1.addressType = “generated”
ethernet2.addressType = “generated”
ethernet3.addressType = “generated”
ethernet4.addressType = “generated”
ethernet5.addressType = “generated”
# End DAC Customization
#####################################################

My next posts will involve installing ESX 4 in text mode and some very interesting findings during that install….

April 23, 2009

VMware vSphere 4 under the covers – First Look

Tuesday April 21st VMware announced they will be releasing vSphere 4 by the end of 2nd quarter. This is exciting news for many looking to take advantage of some of the new features available with this release. In this post I’m going to walk through a handful of some of these new features. There are over 100 new features in vSphere 4 and this post doesn’t come close to covering them all but I will be touching on some really exciting ones with more to come in my next few posts.

Let’s start with the new home screen. It’s a handy way to navigate all the configuration areas of vSphere.

Next let’s take a look at the new “Hardware Status” screen. In this screen shot there is limit hardware information which is due to my hardware. On actual server grade hardware you can view just about anything you want regarding your hardware. If you remember from my other vSphere post you can also trigger alarms based on most of these sensors as well.

Next lets take a look at the changes made to the Host Summary screen. Notice the arrow pointing to the Datastore with the alert. One of the alarms we can set is based on storage usage so we no longer have to manually verify that the free storage is within the acceptable levels.

With ESX 3.5 we had to manually make sure we didn’t overuse resources to maintain and N+1 configuration for HA. I wrote an article on VMware HA and how to size your environment to maintain N+1. Well when you determine that you need to have 37.5% overhead available on each server you can now specify that in your HA configuration rather than manually making sure you don’t exceed.

Take a look at the arrow in the above screen shot it’s showing the reserved capacity of the hosts to ensure the proper failover capacity. The screen shot below show the HA setting to configure this functionality.

There are some exciting and new interesting features surrounding networking. Once new feature is the distributed switch. The distributed switch is a really exciting improvement as it comes with support for private vlans, network vMotion, and of course support for 3rd party switches such as the Cisco Nexus v1000 switch.

There are many other enhancements to the networking such as the new VMXNET Generation 3 driver and other features like IP pools similar to whats available in Lab manager.

In my last post on vSphere I showed configuration items available as part of host profiles. Below is a screen shot showing the hosts compliance similar to that of Update manager.

One exciting change to the virtual machines is the ability to hot add memory and vCPU’s to the virtual machines. This gives even more flexibility to make changes to servers without having to schedule downtime.

The new resource view is very handy. Giving you a snapshot of what is really going on with your vm. It gives you insight into how much memory is private to the vm and how much is shared memory with other vm’s in the environment. This allows you to see how much memory is being “de-duplicated” It also allows you to see the host overhead, memory being swapped, and if ballooning is taking place.

VMware finally moved aware form the annoying license server and now has integrated licensing into vCenter itself. This should make alot of users very happy to not have to deal with managing those license files any longer.

Update manager now has built in support for a shared repository. So if you have a large deployment you can easily manage your update repositories across multiple Update Manager servers.

Have you had those annoying issues where some of your services would crash o stop running that vCenter is dependent on? Well it’s easy to kep track of your vCenter service status now with the new vCenter Service Status information.

One thing that I think is still lacking is the scheduled tasks. That have added a few new options that can be scheduled, but i would have expected some additional improvements in this area.

I hope you enjoyed this preview and be sure to check back as I will be covering some additional new features including “Fault Tolerance”. Over the next few weeks and month I will be putting together more overview posts, best practice articles as well as video tutorials.

April 20, 2009

VMware Partner Exchange Technical Notes

Below are some notes that I took from the VMware Partner Exchange technical sessions that I attended. I left off the BCDR Workshop because it will become a separate post.

Upgrade and Migration Tips:
This session was conducted by Mustafa Kahlil, who is probably THE senior SE at VMware. He has over 10 years with the company and started around ESX 0.9. The session centered around a group of flow charts that followed a nice decision tree for upgrade or migration to VI4. The flowchart will provide everything you need for a upgrade / migration engagement. Some highlights: “Upgrade VMotion” will perform a combination VMotion from ESX 2.5 to ESX/ESXi 4 and a Storage VMotion from VMFS 2.x to VMFS 3.x. A VMotion licence will be required. This will allow direct MIGRATION from ESX 2.5 to ESX/ESXi 4. In order to UPGRADE, the latest update build of ESX/ESXi 3.5 will be required. A “vSphere Update Utility” (AKA VMware Infrastructure Update Utility) will be used to update ESXi if VUM is not used. VUM will be the easiest path. The utility will only be able to update one host at a time, but could be scripted to perform a chain of updates. THERE WILL BE NO UPGRADE PATH FOR VMs ON NFS! Only migration will be used for NFS VMs. The most interesting thing Mustafa said was

“Eventually, no service console”

Remember this in your Plan and Design engagements for VI3. I have always been a supporter of ESXi over ESX.

Performance Best Practices for ESX:
This was, by far, THE BEST session. It was two hours of drinking from a fire hose. I took three pages of notes and they didn’t get through all of the slides. Next week, they are expecting to release an updated performance whitepaper on the VROOM! site. Here are the main things that should be considered for performance:

First and foremost, VM performance has little to do with the ESX configuration tweaks, but has a LOT to do with VM configuration tweaks. The main points were this – Understand the application profile, choose the platform wisely and tune the VM settings. All of the performance information below assumes enough CPU and RAM are provided to an app.

Most CPU intensive apps virtualize very well. Most memory intensive apps virtualize very well. As stated before VMware demonstrated a RHEL/ORA VM that could handle 8x of Visa’s online transactions.

Network usage – between 1 and 16 Gbps will virtualize with proper VM tuning, Over 16Gbps will not. (40Gbps in VI4)
Storage I/O – between 10-100k will virtualize with proper VM tuning. Over 100k will not (200k in VI4)
Anything with more than 8CPUs or 255GB RAM will also not virtualize.

See the doc -> http://www.vmware.com/pdf/asplos235_adams.pdf

Use Intel VT / AMD RVI. The newer chips also include hardware based paging for memory, which takes a load off of the hypervisor.

Use TSO and Jumbo Frames for networks. JF is disabled by default and must be enabled on all devices involved – NIC BIOS, vSwitch, VM OS, pSwitch. VI4 will support JF in iSCSI as well. pNICs should also handle 64bit DMA and multiple scatter/gather elements for the frame. Also, force speed and duplex, separate traffic, use teamin, etc.

Configure storage properly – spindle count, hot spots, LUN layout, etc. Use VMFS instead of RDMs. Use 4k I/O for best performance over 16k and 64k.

VM setting:
Choose the proper OS. This determines the proper monitor type, optimal devices, etc.
Use 64bit OSes for high memory usage
Enable large pages in OS and app (default is disabled)
most apps do not scale well beyond 4/8 CPUs with the exception of RHEL/ORA

See the doc http://www.vmware.com/files/pdf/consolidating_webapps_vi3_wp.pdf

“STORAGE QUEUE SETTINGS ARE ALREADY OPTIMIZED”

They should not need to be tweaked! Kernel latency MAY indicate it is necessary to tweak queues.

Perform adminitsrative tasks (new VMs, clones, etc.) during off hours. These produce storage performance hits.

April 20, 2009

VMware Partner Exchange Notes – Keynotes

Here are some breif notes from VMware Partner Exchange. I will also post about some of the technical sessions. I am not going to regurgitate the keynotes. The content will be available soon on Partner Central and there are several blogs that have plenty of information from the keynotes. I will however provide some highlights:

Partner Central and Partner University will soon be revamped. The accreditations will be changing and will require a certain number of accredited VCPs before the company can get an accreditation. The categories will be similar to our practices, such as infrastructure virtualization, desktop virtualization and BCDR. If you go to partner central now and click on the partner university link, you will see a little bit of what the changes will be. There is also plenty of web-based, self-paced training. On line tests are available so you can receive accreditations for many different products, most are jumpstarts and plan and design related.

VMware’s obvious desire is “100% virtualized.” Their primary focus will center around cloud computing with an initial push for the internal cloud as many see challenges with getting acceptance for the external cloud. Private clouds will eventually bridge the gaps between the internal and external clouds. Much of this information is already available on VMware’s main site.

The software surrounding VI4 took around 3 million engineering hours to develop. It includes great improvements on resources that will be available to the VMs. The resources will be increased to 8 vCPUs per VM, 256GB RAM per VM, 40 Gbps network throughput per VM, and 200,000 storage IOPS per VM. vCenter maximums will increase to 3000 VMs / 300 Hosts. There will also be a capability for linking up to 10 vCenter servers with a centralized search function.

A new function centers around host profiles, which works similarly to VUM. It establishes configuration baselines for an ESX / ESXi host that includes such things as network, security, storage and NTP settings. A host can be scanned for compliance and remediated with the baseline. The BIG “however” is that it will require “Enterprise Plus” to enable host configuration controls and distributed switches. This will carry a $600 price tag and is not ala carte.

Using ESX4 allowed for 85% native performance on 8-way RHEL/Oracle servers in spec performance tests. The amount of transactions (I forget how many) was 8x the number of Visa’s current transactions.

Out of the gate, vSphere will offer optional components surrounding security, BCDR and networking. Additional vSpere components will become available “over the summer.”

April 17, 2009

VMware vSphere 4 (ESX 4.0, vCenter 4.0) Alarms and Host Profiles

Some are speculating that next Tuesday VMware is going to announce the release of VMware vSphere which is what essentially is Virtual Infrastructure 4.0 which would include ESX 4.0. I can’t say what VMware is going to do but over the next few weeks I will be publishing information on vSphere as well as some instructional videos. For now I have some teasers for you.

Here is a screen shot of the alarms available in vSphere. A you can see they have expanded the alarm feature from what was available in VI3.

I’m sure most of you have heard of the new host profiles. If you haven’t had the fortune of checking out this cool new feature here are some screenshots to show you what options are available to you as part of a host profile. If you are not much for scripting and just can’t stand those pesky automated build scripts then you will love this feature. It gives you the ability to configure just about every aspect of the ESX host without having to deal with any scripting.

As you can see in this screenshot all these settings are very easy to set via the GUI.

So stay tuned as there is much more to come. I’m currently working on making videos covering installing, and configuring vSphere from the ground up and plan on getting into all of the new feature available in this release.

April 14, 2009

VMware Virtual Center – Physical or Virtual?

Over the years there have been some controversy over this topic. Should Virtual Center (vCenter) be physical or virtual? There is the argument that it should be physical to ensure consistent management of the virtual environment. Of course there is also the fact that Virtual Center requires a good amount of resources to handle the logging and performance information.

I’m a big proponent for virtualizing Virtual Center. With the hardware available today there is no reason not to. Even in large environments that really tax the Virtual Center server you can just throw more resources at it.

Many companies are virtualizing mission critical application to leverage VMware HA to protect these applications. How is Virtual Center any different. So what do you do if Virtual Center crashes? How do you find and restart Virtual Center when DRS is enabled on the cluster?

You ave a few options here.

Override the DRS setting for the Virtual Center vm and set it to manual. Now you will always know where your virtual center server is if you need to resolve issues with it.

Utilize Powershell to track the location of your virtual machines. I wrote an article that included a simple script to do this which I will include on our downloads page for easy access.

Run an isolated 2 node ESX cluster for infrastructure machines.

So my last option warrants a little explaining. Why would you want to run a dedicated 2 node cluster just for infrastructure vms? The real question is why wouldn’t you? Think about it. Virtual Center is a small part of the equation. VC and your ESX hosts depend on DNS, NTP, AD, and other services. What happens if you loose DNS? You loose your ability to manage your ESX hosts through VC if you follow best practice and add them by FQDN. Now if AD goes down you have much larger issues, but if your AD domain controllers are virtual and you somehow loose them both that’s a problem. It’s a problem that could affect your ability to access Virtual Center. So why not build an isolated two node cluster that houses your infrastructure servers. You’ll always know where they will be, you can use affinity rules to keep servers that back each other up on separate hosts, and you can always have a cold spare available for Virtual Center.

Obviously this is not a good option for small environments, but if you have 10,30, 40, 80, 100 ESX hosts and upwards of a few hundred VM’s I believe this is not only a great design solution but a much needed one. If you are managing this many ESX hosts it’s important to know for sure where your essential infrastructure virtual machines reside that affect a large part of your environment if not all.

April 14, 2009

Business Continuity and Disaster Recovery with Virtualization

In the previous years Business Continuity and Disaster Recovery have been big buzz words. All companies small and large vowed to launch initiatives to implement either or both in their current IT strategies. My question is what happened? Why is it that I rarely see organizations that have implemented or even have a plan to implement Disaster Recovery?

Is it a lack of understanding? Is it that most companies believe it is to expensive or complicated to implement? Well it doesn’t have to be either. Most companies that are undergoing virtualization initiatives already have half if not more of what they need to implement Disaster Recovery. The simple fact is if you already have at least two data centers and are virtualizing you are a prime candidate. Here are some common question and my answers regarding this subject:

1.) Do I need to utilize SAN replication to implement Disaster Recovery in a virtualized environment?

No! There are other option to achieve Disaster Recovery without SAN replication. If you are running VMware you can utilize some of what you already have. VMware VCB in conjunction with VMware converter can be used to implement Disaster Recovery. Now this wouldn’t be as elegant as doing SAN replication but you could implement scheduled V2V’s of your Virtual Machines from one site to another and it’s a very simple solution to implement.

What about the hardware right….where do we get the additional hardware? The answer is simple reuse what you already have. Take those old servers you just freed up and put them to some good use. Beef them up! Need more ram in them tear ram out of some and add it to other, do the same with CPU’s to make a number of more power servers that you can use for DR. Granted you may need more of the reused servers to host all the vm’s needed but at the end of the day you would have a disaster recovery plan.

2.) What if I can’t do SAN replication but want synchronous and asynchronous replication?

This can still be achieved using software based replication in your virtual machines. Software like NSI Doubletake and Replistor provide this functionality at a a relatively low cost. With virtualization you can cut cost even more. With physical servers you traditionally needed to have a 1 to 1 mapping for replication which required a license for each host. With virtualization you can take a many to one approace cutting down on the licenses you need to replicate your data.

With this approach I would still use VCB or VMware converter to make weekly copies of your virtual machine OS drives. You can then utilize one of the mentioned applications (Doubletake or Replisor) to synchronous replication of your data volumes. You can achieve this and save licenses by installing say Doubletake on each of the source systems. The you would create a virtual machine at the DR site and add a drive to it for each of the source systems data volumes and replicate each source data to a different data volume on the destination vm. If you ever need to fail over just dismount the volumes from the destination vm and attach each one it’s respective vm that was created through the use of VCB or VMware converter.

3.) These methods are great but what would it take to bring an environment back up using them?

That’s rather hard to say because it depends on the size of your environment and how many vm’s you are relocating to your DR site. If your environment is large and you have specific SLA’s to adhere to regarding RTO (Recovery Time Objective’s) and RPO (Recovery Point Objectives) then you should consider SAN to SAN replication and utilizing something like VMware SRM which does an outstanding job of handling this. VMware SRM also allows you to run disaster recovery simulations to determine the effectiveness of your DR strategy that allows you to determine if you are meeting your SLA.

If you are doing DR on the cheap the real answer is to this question is you will be able to recover your systems a heck of a lot quicker than if had to restore via backups of rebuild your systems.

4.) This is great but where do we begin?

Don’t know where to begin, the answer is easy. Start small and grow into it. Find at least 2 servers that you can reuse beef’em up determine a configuration for them and deploy ESX to the servers. You need to have some infrastructure in place at your DR location to make DR work so that is a good place to start. You need to add the following service at your DR location:

Active Directory Servers
DNS Servers
NTP Servers
Virtual Center Server

It may be required to to deploy additional servers for your specific environment but I think you get the idea.

Next pick a few development machines or test machines that you can replicate to the DR site. Develop a plan and schedule down time and perform a test fail over to the remote site. Once you have work out the kinks and have a written DR plan determine your first phase of servers to incorporate into your DR site. Generally at this point you would want to pick some of your most valuable servers to ensure they are protected.

You can then break all your servers that need to be replicated into phases and determine the host requirements at the DR site and develop a plan for each phase of your DR implementation. It would be a good idea to have a remote replication vm for every 20 or so source vm’s. This really would depend on the data chance rate of your servers but 20 is a good starting point.

This article is obviously not all inclusive and is very high level but hopefully it inspires some of you to start developing a DR strategy and at least start testing some of these solutions in your environment because data is a terrible thing to waste.