TIP: Linux Virtual Machine Performance Tweaks

The advent of virtualization, changed many rules of technology -- what may apply to the physical world could be detrimental to its virtual equivalent. This is likewise true in terms of performance. Optimizing the performance of one virtual machine impacts the performance of others. In the end, tweaking for performance not only pays off big time, but does that in magnitudes of two-, three-fold or even more.

The above statements are true due to sharing of resources in the hypervisor level. If resources are freed by performance tweaks, it makes them available to others waiting for the same resource to be available for use.

The first of these tweaks is to ensure that the disk used by the virtual machine is aligned. Why is this important? To better understand the underlying infrastructure of the virtual filesystem, refer to the screenshot. (Illustration from http://www.yellow-bricks.com)

As seen from the diagram, the misalignment of the array, to the virtual filesystem, and, to the guest disk is very, very significant. A big performance degradation will occur due to the misalignment. What happens here is that, what could be seen as a single I/O operation on the guest operating system could end up as multiple I/O operations on the physical disk or array.

Translate the above performance degradation to multiple virtual machines and you see the drift. According to VMware (Reference):
The degree of improvement from alignment is highly dependent on workloads and array types. You might want to refer to the alignment recommendations from your array vendor for further information. 

Still, I/O could be a potential performance bottleneck if the disk alignment is not kept in check.

While on the same topic of I/O bottlenecks, Linux virtual machines could benefit from the proper I/O scheduler. Again, this is another arena where the physical rules and virtual rules differ.

Modern Linux kernels default to the "completely fair queuing" (or CFQ) scheduler. This is how the kernel deals with optimizing I/O. It re-orders I/O in such a manner that when the head moves across the platter, it does that in an orderly, sequential manner rather than moving back and forth to complete requests. The best illustration to this is an elevator, where it drops people to floors in an orderly sequential manner. Instead of dropping passengers as they push the corresponding floor buttons, what happens is the elevator drops people off at 2, then 5, then 6, before going to 11. The kernel has different methods of doing this.

Imagine that the hypervisor has its own elevator, the guest has its own elevator and the array has its own. When the guest performs that optimization, the hypervisor does the same, and the array does the same. Who then decides what is the best re-ordering or optimization? The hypervisor has no insight on the guest operating system (this time a Linux machine), meaning it can't tell the method used for optimization, or how they got re-ordered, if they indeed got re-ordered or why it came to that decision.

This kernel optimization has no impact on the virtual environment and in some situations could have a negative effect. Why? The hypervisor has no way of telling how long an I/O has been outstanding or how it got re-ordered and why it was that way on the guest level. Add the hypervisor elevator and that adds further latency to the already outstanding I/O request.

It would be best to leave all optimizations on the hypervisor level and turn off the elevator on the kernel level. This is done by having the string "elevator=noop" on the kernel line of the grub.conf file of Linux. The other way of turning it off is by choosing the noop scheduler, as recommended by VMware.

This is the default setting.
# cat /sys/block/[DEVICE]/queue/scheduler
noop deadline anticipatory [cfq]
Change it for every [DEVICE].
# echo noop > /sys/block/[DEVICE]/queue/scheduler
# cat /sys/block/[DEVICE]/queue/scheduler
[noop] deadline anticipatory cfq

If you have screensavers running, those are detrimental to the performance of virtual machines. Disable them at once. I have seen a virtual machine with a load of 5.0 reduced to 0.0 simply by disabling the screensaver!

The other tweak you could perform is to change to runlevel 3. Running on a GUI (runlevel 5), consumes a lot of resources that would otherwise be free if running at runlevel 3.

Permanently change it on the file /etc/inittab.

By tweaking, you squeeze the best performace out of your current infrastructure and free up resources otherwise sapped by useless unnecessary processes or simple misconfigurations. The combination of the tweaks above may yield better results than applying one tweak alone.


FAQ: Cloned Linux Virtual Machine with Boot Errors

Q. While working on a few cloned RHEL virtual machines just a few days ago, I encountered a weird error that almost stopped me dead on my tracks. I was expecting the tasks to be a breeze since cloning via virtual machine templates is as easy as click, click, click.

As it turned out, what was supposedly a few clicks was more complex than what I thought. The clone virtual machines simply refused to boot. To give you a better understanding of the error see the screenshot below.

Opening the console of the virtual machine, the repeated errors prompted are:
vmsvc[xxx]: [ warning] [guestinfo] RecordRoutingInfo: Unable to collect IPV4 routing table.
vmsvc[xxx]: [ warning] [guestinfo] RecordRoutingInfo: Unable to collect IPV4 routing table.

The messages above are repeated at about every minute.. And, worse, the boot sequence is stuck at this phase and will not proceed.

After exhausting my patience waiting for the host to boot up, I gave up and resorted to shutting down the virtual machines to figure out what was wrong. The VMware logs didn't give a clue what was wrong. Even RedHat forums didn't have information regarding the error.

What was striking was the "vmsvc" string, indicating this was somewhat VMware configuration related or a service related to VMware. This was happening to a RHEL Linux virtual machine. So I tried a different flavor of Linux -- CentOS. Same result. But for Windows Server 2003, it was not.

Google search came out with this same exact error message on the VMware Knowledge Base (KB #2048572). Executing the procedure in the resolution part did not help either.

It only applied when the guest operating system's time is ahead of the host. But that gave me a further clue -- how about if the time of the cloned guest is too far off from the host ESXi host? (Afterall it was converted to a template a while back. And it was not selective, as every cloned machine displayed this manifestation.)

It turned out this was the case. I decided to disable time synchronization completely between the host ESXi and guest RHEL virtual machine.. Bingo error gone.

A. To completely disable the time sync between the host ESXi hypervisor and the guest RHEL virtual machine, simply shutdown or power off the guest virtual machine and edit the .vmx configuration file. Add the following lines, if they don't exist:
tools.syncTime = "0"
time.synchronize.continue = "0"
time.synchronize.restore = "0"
time.synchronize.resume.disk = "0"
time.synchronize.shrink = "0"
time.synchronize.tools.startup = "0"
time.synchronize.tools.enable = "0"
time.synchronize.resume.host = "0"

It helps to enable SSH on the ESXi server and use vi to make changes to the guest .vmx file. After adding the above lines, problem was solved and subsequent clones were easier.

The only task left to be done was to convert the template to a VM, make the same changes and re-template. This was a very fruitful learning experience.


HOW-TO: Migrate Sickbeard to its Bricky Fork

The demise and take down of most Usenet premium providers left most of its followers with lesser and lesser options by the day. Not only its followers feel the grunt as a result, but its service dependent software as well. One of the end services I use myself is Sickbeard.

Sickbeard is my choice in automated download of TV series that I follow and watch. The best feature is the automatic download of episodes that you program it to automatigically download for you. Another neat feature is it notifies you of this download through your PC, phone or twitter (and other social media) services. But Sickbeard relies heavily on Usenet as it has limited torrent support.

Due to the above limitation, it forked to its bricky counterpart -- Sickbeard with better torrent support. The screenshot below is for the original version of Sickbeard.

I have a Popcorn Hour C-200 (touted as one of the best media players out in the market during its time). The outstanding feature of this media player is it could run Sickbeard -- meaning not only could it play the TV series, it could download and manage them automatically as well. This part was discussed in detail on our previous article on "Install Sickbeard on Popcorn Hour C-200".

The above link is used to install the original main Sickbeard version, the one with limited torrent support. To install the bricky fork, the above "HOW-TO" still applies. Just choose not to start Sickbeard after the install.

It is important (or should I say very important) to have a shell to the C-200 for the next steps to work. This could be done via BusyBox or LTU (Linux Terminal Utilities). Checkout the links on how that is done then follow the procedure below:

[1] Open a terminal connection via PuTTY or another utility. Browse the directory /share/Apps/sickbeard. All the other steps below will discuss about changes to files and directories relative to this path.

[2] Remove the directory .git. Do this so the bricky fork will not conflict with the original midgetspy version of sickbeard. I also removed the file .gitignore but this seems irrelevant. You may opt to remove it in this step.

[3] On the same directory /share/Apps/sickbeard, is the file setup.sh. Look for the line:
git clone git://github.com/midgetspy/Sick-Beard.git SB
.. change it to this line:
git clone git://github.com/bricky/Sick-Beard.git SB

[4] Remove the directory autoProcessTV under /share/Apps/sickbeard. If you don't want to remove it, you may choose to rename it instead. Any arbitrary name aside from the original name will do.

[5] Execute setup.sh on the command line. Either of these commands will work:
/share/Apps/sickbeard/setup.sh install
.. or
cd /share/Apps/sickbeard; ./setup.sh install

[6] Once the git clone is complete, run this:
/share/Apps/AppInit/appinit.cgi start sickbeard

The obvious difference between the two versions is the availability of more torrent search engines for the bricky fork. If you browse to the "Search Providers" portion of the configuration you will see more options for bricky fork than the original.

As seen from the above screenshot, there are more providers available for this fork of Sickbeard. I hope this post helps you get more from Sickbeard.


Subscribe for Latest Update

Popular Posts

Post Labels

100gb (1) acceleration (1) acrobat (1) adblock (1) advanced (1) ahci (1) airdrop (2) aix (14) angry birds (1) article (21) aster (1) audiodg.exe (1) automatic (2) autorun.inf (1) bartpe (1) battery (2) bigboss (1) binance (1) biometrics (1) bitcoin (3) blackberry (1) book (1) boot-repair (2) calendar (1) ccleaner (3) chrome (5) cloud (1) cluster (1) compatibility (3) CPAN (1) crypto (3) cydia (1) data (3) ddos (1) disable (1) discount (1) DLNA (1) dmidecode (1) dns (7) dracut (1) driver (1) error (10) esxi5 (2) excel (1) facebook (1) faq (36) faucet (1) firefox (17) firewall (2) flash (5) free (3) fun (1) gadgets (4) games (1) garmin (5) gmail (3) google (4) google+ (2) gps (5) grub (2) guide (1) hardware (6) how (1) how-to (45) huawei (1) icloud (1) info (4) iphone (7) IPMP (2) IPV6 (1) iscsi (1) jailbreak (1) java (3) kodi (1) linux (28) locate (1) lshw (1) luci (1) mafia wars (1) malware (1) mapsource (1) memory (2) mikrotik (5) missing (1) mods (10) mouse (1) multipath (1) multitasking (1) NAT (1) netapp (1) nouveau (1) nvidia (1) osmc (1) outlook (2) p2v (2) patch (1) performance (19) perl (1) philippines (1) php (1) pimp-my-rig (9) pldthomedsl (1) plugin (1) popcorn hour (10) power shell (1) process (1) proxy (2) pyspark (1) python (13) qos (1) raspberry pi (7) readyboost (2) reboot (2) recall (1) recovery mode (1) registry (2) rename (1) repository (1) rescue mode (1) review (15) right-click (1) RSS (2) s3cmd (1) salary (1) sanity check (1) security (15) sendmail (1) sickgear (3) software (10) solaris (17) squid (3) SSD (3) SSH (9) swap (1) tip (4) tips (42) top list (3) torrent (5) transmission (1) treewalk (2) tunnel (1) tweak (4) tweaks (41) ubuntu (4) udemy (6) unknown device (1) updates (12) upgrade (1) usb (12) utf8 (1) utility (2) V2V (1) virtual machine (4) VirtualBox (1) vmware (14) vsphere (1) wannacry (1) wifi (4) windows (54) winpe (2) xymon (1) yum (1) zombie (1)