SUBSCRIBE via RSS

HOW-TO: TaiG JailBreak for iOS 8.1.1

It's been a couple of months back since Apple released iOS8. I have not made the transition to it and stayed with iOS7 since my iPhone4S is rooted. I'd like to term a successful jailbreak as rooted -- simpler and easier to say (there is just a nicer ring to it). I just like the functionality of a "rooted" phone and the availability of Control Center button customizations that are only available from a "rooted" phone.

Last time I did this was way back to the early 2014 and used the evasi0n utility. This time TaiG is the utility that is able to jailbreak iOS8, particularly the iOS8.1.1 version.

The TaiG jailbreak utility requires the "Find My iPhone" feature to be turned off first. And although optional, turn off the passcode security feature of the phone (turn it back on once jailbreak is a success).

Plug the phone via USB and launch the TaiG utility executable. You may uncheck the 3K option if you wish, then hit the big green button on the screen. My initial attempt at this hit a snag -- Error -1004 -- which according to the TaiG website is an issue with the computer running the jailbreak.

TaiG JailBreak Utility

This really doesn't have anything to do with the computer running the utility. I tried two (2) things that "might" have resolved the issue for me. [1] I moved the USB cable to a USB3 port; and, [2] I turned off the passcode. I'm not exactly sure which solved it for me but either way I got the jailbreak that I need. Please note that I mentioned *might* and your mileage may vary.

Once successful, the device will restart twice. The first is mid-way through the spinning green circle at about 43% and the last when the injection is installed. You will then see the Cydia button on the second screen of the device signifying success.

iPhone 4S Rooted

As seen from the screenshot, my iPhone is running iOS8.1.1 and Cydia 1.1.16 is installed. It's way past midnight by the time I was done. But the TaiG utility got me the jailbreak I needed.


HOW-TO: P2V (Physical to Virtual) Prep Work for Ubuntu

This post is supplementary to the previously written article Prep Work for Linux P2V (Physical to Virtual). The previous post covered mostly the tasks for an RPM based host (RedHat, Fedora and CentOS). This time it will cover more of the Debian based distribution.

It was just recently that I worked on several Ubuntu hosts. These presented a different perspective on the P2V process, especially since I had been tasked to convert them with little time to spare -- it is a spur of the moment, unexpected, unplanned need. With the added adjectives, you probably get the idea.

Although Linux-based, Ubuntu is a bit different than its RPM based brothers. For one, Ubuntu disk devices are mounted based on UUID. That is just the primer. The end of the P2V of VMware vCenter Standalone Converter prompted an error installing grub. To give you an idea of the error, refer to the screenshot below.

P2V Grub Install Error

FAILED: An error occurred during the conversion: 'GrubInstaller::InstallGrub: /usr/lib/vmware-converter/installGrub.sh failed with return code: 127, and message: Installing GRUB1 on (hd0)... /vmware-updateGrub.sh: 52: /vmware-updateGrub.sh: grub: not found Error installing GRUB Command: grub --no-floppy --batch --device-map="/vmware-device.map" root (hd0,0) setup (hd0) Error running vmware-updateGrub.sh through chroot into /mnt/p2v-src-root /usr/lib/vmware-converter/installGrub.sh: line 143: /mnt/p2v-src-root:
Is a directory '

Let's tackle these prep work one at a time.

For the UUID based filesystems, if you take a look at /etc/fstab you would see something similar to the below entries:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#                
# / was on /dev/sda1 during installation
UUID=a4fc19b3-6df9-4fcb-a565-12a9bd59fff1 /               ext4    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=1061e56a-f69d-4d44-b3d1-566c960719b1 none            swap    sw              0       0

The operating system is good enough to note the disk slices and its corresponding UUID unique identifiers. However, if this is not present on your system, the command "blkid" will show the same information.

[admin@vhost ~] sudo blkid
/dev/sda1: UUID="a4fc19b3-6df9-4fcb-a565-12a9bd59fff1" TYPE="ext4" 
/dev/sda5: UUID="1061e56a-f69d-4d44-b3d1-566c960719b1" TYPE="swap" 

The above entries need to be converted to its /dev/sdX counterpart prior to P2V. This is a simple change of removing the UUID=xxxxx-xxx-xxx-xxx and replacing it with its /dev/sdX equivalent. Exercise prudence, and make a backup of the file /etc/fstab prior to making changes.

Next, let's deal with the GrubInstall error. The grub install happens when the P2V is almost complete, meaning, the copying of the physical host files are done. The only challenge is the virtual machine made will not boot. This was a tough one to resolve at first.

As it turns out, you require the Linux Security Remix ISO file. Boot off the ISO and execute Boot Repair (recommended Boot Repair).

Once Boot Repair is done, reboot and the virtual machine is up and running.

The only hiccup that delayed the activity for me was to download the Linux Security Remix ISO. It is close to 1GB in size so if you are doing a lot of P2V conversions this file will come in handy. There are two (2) versions of this file -- the 32-bit and the 64-bit versions. If your system is 64-bit, the 32-bit version will not work. Having both versions is recommended.

As consolation, the only similarity I found with other Linux flavors was on the file /etc/udev/rules.d/70-persistent-net.rules, where the MAC addresses of the physical machine are hard-coded. If this file exists, comment out the entries or remove the file completely prior to P2V. The udev process will re-create this file on boot-up.

ERROR: "The operation is not allowed in the current state of the host"

I experienced the above error while powering on virtual machines after a scheduled shutdown. Just so you can replicate my experience, I was administering vCenter v5.5 Update 2b and ESXi v5.5 Update 2 hosts. The cluster's HA was turned off, DRS was set to partially automated from its fully automated setting, and all ESXi hosts were put into maintenance mode prior to them shutting down.

Powering up there were no issues -- all VMFS datastores were mounted and there are no alarms nor alerts. So after checks and everything was all set, ESXi hosts were exited from maintenance mode and virtual machines were started.

Most of the virtual machines booted back. But some of those that were powered up did not proceed on booting but instead prompted the error: "The operation is not allowed in the current state of the host."

This was completely new to me. And VMware forums and public knowledge base had an almost similar error: "The operation is not allowed in the current connection state of the host." Which was not quite what I was facing.

I was using the vSphere Web Client, so I tried using the old graphical user interface. Things did not seem to change -- same error.

I restarted the VMware VirtualCenter Server service but I still ended up with the same error. So I started to scratch my head. Of the several virtual machines that manifested this error, one thing was common: they belonged to similar ESXi hosts.

I went back to reading the posts with the almost identical error and it suggested to restart the management agents on the ESXi hosts. So I tried that but instead of just the management agents, I executed a stop and start of the entire services.sh.

Below are the traces of the execution.

~ # services.sh stop
Running hp-ams.sh stop
Terminating hp-ams-watchdog process with PID 35513 50567 50568
Stopping process 35479 35477 ...
hpHelper process is now stopped.
Running vmware-fdm stop
Stopping vmware-fdm:success
Running hp-mst.init stop
Unable to unload module mst: Module not found
Running xorg stop
Running sfcbd stop
This operation is not supported.
Please use /etc/init.d/sfcbd-watchdog stop
Running wsman stop
Stopping openwsmand
Running snmpd stop
Running sfcbd-watchdog stop
Running hpcru.init stop
Running hpilo.init stop
Running hpsmx.init stop
Running vpxa stop
watchdog-vpxa: Terminating watchdog process with PID 34561
vpxa stopped.
Running vobd stop
watchdog-vobd: Terminating watchdog process with PID 33332
vobd stopped
Running lacp stop
watchdog-net-lacp: Terminating watchdog process with PID 33542
Running memscrubd stop
memscrubd is not running
Running smartd stop
watchdog-smartd: Terminating watchdog process with PID 34425
smartd stopped
Running dcbd stop
watchdog-dcbd: Terminating watchdog process with PID 34370
Running cdp stop
watchdog-cdp: Terminating watchdog process with PID 34328
Running nscd stop
watchdog-nscd: Terminating watchdog process with PID 34283
Running slpd stop
Stopping slpd
Running storageRM stop
watchdog-storageRM: Terminating watchdog process with PID 34231
storageRM stopped
Running hostd stop
watchdog-hostd: Terminating watchdog process with PID 34191
hostd stopped.
Running vmfstraced stop
watchdog-vmfstracegd: PID file /var/run/vmware/watchdog-vmfstracegd.PID does not exist
watchdog-vmfstracegd: Unable to terminate watchdog: No running watchdog process for vmfstracegd
vmfstracegd is not running
Failed to clear vmfstracegd memory reservation
Running lbtd stop
watchdog-net-lbt: Terminating watchdog process with PID 34130
net-lbt stopped
Running sdrsInjector stop
watchdog-sdrsInjector: Terminating watchdog process with PID 34094
sdrsInjector stopped
Running rhttpproxy stop
watchdog-rhttpproxy: Terminating watchdog process with PID 34053
rhttpproxy stopped.
Running sensord stop
sensord is not running
Running hp-oem.init stop
Running usbarbitrator stop
watchdog-usbarbitrator: Terminating watchdog process with PID 33941
usbarbitrator stopped
Running DCUI stop
Disabling DCUI logins
VobUserLib_Init failed with -1
Running ntpd stop
Stopping ntpd
watchdog-ntpd: Terminating watchdog process with PID 33876
Connect to localhost failed: Connection failure
Running vsantraced stop
watchdog-vsantraced: Terminating watchdog process with PID 33807
vsantraced stopped
watchdog-vsantracedUrgen: Terminating watchdog process with PID 33836
vsantracedUrgen stopped
Persisting traces to /locker/vsantraces
~ #
~ # services.sh start
Running vsantraced start
Scratch partition is not backed by persistent storage
Storing traces to /vmfs/volumes/4dc4866f-040de4b6-1f88-0017a477f825/vsantraces
vsantraced started
vsantracedUrgen started
Running ntpd start
Connect to localhost failed: Connection failure
Starting ntpd
Running DCUI start
Enabling DCUI login: runlevel = 
VobUserLib_Init failed with -1
Running usbarbitrator start
usbarbitrator started
Running hp-oem.init start
Running sensord start
sensord started
Running rhttpproxy start
rhttpproxy started.
Running sdrsInjector start
sdrsInjector started
Running lbtd start
net-lbt started
Running vmfstraced start
VMFS Global Tracing is not enabled.
Running hostd start
Ramdisk 'hostd' with estimated size of 803MB already exists
hostd started.
Running storageRM start
storageRM started
Running slpd start
Starting slpd
Running nscd start
nscd started
Running cdp start
cdp started
Running dcbd start
dcbd started
Running smartd start
smartd started
Running memscrubd start
The checkPages boot option is FALSE, hence memscrubd could not be started.
Running lacp start
LACP daemon started
Running vobd start
vobd started
Running vpxa start
Connect to localhost failed: Connection failure
Running hpsmx.init start
Running hpilo.init start
Running hpcru.init start
Running sfcbd-watchdog start
Connect to localhost failed: Connection failure
Connect to localhost failed: Connection failure
Running snmpd start
Running wsman start
Starting openwsmand
Running sfcbd start
This operation is not supported.
Please use /etc/init.d/sfcbd-watchdog start
Running xorg start
Running hp-mst.init start
Running vmware-fdm start
Starting vmware-fdm:success

Running hp-ams.sh start
Starting hpHelper service...
~ # 

One thing to note here is that during the stop and start of services.sh vCenter server will complain about the host being disconnected. This is normal as the daemons handshaking and talking to each other lose connectivity.

After the stop and start was done, powering up of the virtual machines that had errors were successfully executed. Another point chalked to experience.

HOW-TO: Online Resizing of a MultiPath LUN

Online resizing of a multipath LUN is challenging. Try to wrap your head around the task at hand for a bit and understand the underlying principles behind this complex and challenging task. It is overwhelming! I have been in this particular scenario a lot and I know how it is to be on your shoes the first time.

But trust me, it may not be as daunting a task as you may think. Bear with me for a bit as I outline the steps needed to perform this task, successfully and without complications.

The previous article, Add LUNs to a Linux Host Without Reboot suggested the use of a well working script "rescan-scsi-bus.sh" via installation of the sg3_utils RPM. This task will do the same thing. So if this package is not installed, the script will probably be missing from your system as well. It would be a good idea to install it (yum -y install sg3_utils).

The first task on the list is to expand the LUN on the storage level. If you have an EMC storage, get to Unisphere and expand the LUN or the OnCommand System Manager for NetApp. On other storage systems, use the tools native to the storage to expand. Once that is done use rescan-scsi-bus.sh to rescan the LUNs.

For me (and some of you), "rescan-scsi-bus.sh -l" may already work. If not, use "rescan-scsi-bus.sh --forcerescan" to detect the new LUN sizes.

NOTE: The examples below will use the multipath device /dev/mpath/mpath165; and, /dev/sdat, /dev/sdh, /dev/sdap and /dev/sdd as its underlying devices. Its size is 190GB and we will add 100GB to it.

Next, check the devices relevant to the LUN. In this case -- mpath165. Change this with the multipath alias of your LUNs. Execute:

[root@vhost ~#] multipath -l mpath165
mpath165 (36006016098402e007cb4e6580ef1e311) dm-17 DGC,VRAID
[size=190G][features=1 queue_if_no_path][hwhandler=1 alua][rw]
  \_ round-robin 0 [prio=0][enabled]
   \_ 1:0:1:3 sdat 66:208 [active][undef]
   \_ 0:0:1:3 sdh  8:112  [active][undef]
  \_ round-robin 0 [prio=0][active]
   \_ 1:0:0:3 sdap 66:144 [active][undef]
   \_ 0:0:0:3 sdd  8:48   [active][undef]

The above output means there are four (4) paths composing the multipath device mpath165. They are /dev/sdat, /dev/sdh, /dev/sdap and /dev/sdd. The outputs on your system may vary.

Then, let the system detect the new size of the LUNs.

[root@vhost ~#] blockdev --rereadpt /dev/sdat
[root@vhost ~#] blockdev --rereadpt /dev/sdh
[root@vhost ~#] blockdev --rereadpt /dev/sdap
[root@vhost ~#] blockdev --rereadpt /dev/sdd
[root@vhost ~#] blockdev --getsz /dev/sdat
  608174080
[root@vhost ~#] blockdev --getsz /dev/sdh
  608174080
[root@vhost ~#] blockdev --getsz /dev/sdap
  608174080
[root@vhost ~#] blockdev --getsz /dev/sdd
  608174080

From the above commands, only getsz will provide an output. The outputs will be the same (due to the multipathing) and note them down for the commands we will execute later. If the outputs of "getsz" are not the same, something is wrong. Re-execute "blockdev --rereadpt" for that particular device.

Provided the outputs are all the same, proceed with the instructions below.

Next, dump the current multipath configuration and write the output to files -- mpath165.bak.
[root@vhost ~#] dmsetup table mpath165 | tee mpath165.bak
  0 398458880 multipath 1 queue_if_no_path 1 alua 2 2 round-robin 0 2 1 66:208 1000 8:112 1000 round-robin 0 2 1 66:144 1000 8:48 1000

Create a copy of the file "mpath165.bak" -- say "mpath165.new". Edit this file and replace the second number (398458880, which is the current size of the LUN) with the output of "blockdev --getsz" above.

(NOTE: Double check that the file mpath165.new has new values, where what used to be 398458880 should now be 608174080.)

This is the part where the LUN gets to be offline for a split second. It is imperative to stick these commands together. Or else..
[root@vhost ~#] dmsetup suspend mpath165; \
  dmsetup reload mpath165 mpath165.new ; \
  dmsetup resume mpath165

What actually happens in the above command is the multipath device mpath165 got suspended. Its configuration was replaced with the configuration contained in the file mpath165.new (or a resize was made). Then, the I/O operations were resumed. This was done quickly enough as if nothing happened.

Notify the kernel of a change of partition tables. Try partprobe first.
[root@vhost ~#] partprobe /dev/mpath/mpath165
-- or --
[root@vhost ~#] kpartx -a /dev/mpath/mpath165
[root@vhost ~#] kpartx -l /dev/mpath/mpath165

Now its time for LVM to take care of the rest of the resizing. When done right, this procedure can save you from unnecessary downtime due to multipath LUN resizing. Having known this procedure, resizing multipath LUNs will be just a routine for you.


HOW-TO: Add LUNs to a Linux Host Without Reboot

Abstraction between the operating system and the hardware is key to being able to modify running systems without having to reboot. Reboot is often associated with the dreaded word "downtime". This short downtime could cost some hundreds of dollars while for some, it could be in the magnitude of hundreds of thousands of dollars.

One of the major reasons for downtime is adding hardware to the host. And, more often than not, it is storage that is being added. Abstraction between the operating system and the hardware layer is key to planning and playing it out without causing or even costing the company downtime.

On the hardware layer, adding LUNs is a simple introduction or mapping of the drive to the host. This is the conventional way of doing it. Sad thing, however, it doesn't always work.

By conventional, I mean this method of rescanning the SCSI bus for a particular host.
echo "- - -" > /sys/class/scsi_host/host0/scan
The triple dashes "- - -" are wildcards that rescan every channel, every target, and every LUN on the specificed SCSI host. The above command scans the SCSI bus for new devices and map those drives to new device nodes that could be used by the Linux host. This works about 60-70% of the time. For the remaining 30-40%, I follow a better route that has flawlessly worked for me so far.

The utility or tool I use doesn't get installed by default. It is imperative that you have the original installation media or an internet connection for this to work.

For RHEL, Fedora and CentOS hosts, the command to run is:
yum -y install sg3_utils
The above command will likewise install dependent packages that it requires. The sg3_utils package contains a script named rescan-scsi-bus.sh. This is what is needed for the addition of LUNs or drives.

To detect or rescan new devices, invoke the script via "rescan-scsi-bus.sh -l". For SAN LUNs, what I found to have worked perfectly is "rescan-scsi-bus.sh --forcerescan". To get colored output the addition of "--color" makes all that possible.

This works for me whether the host is bare-metal or virtual.

RedHat warns about caveats for using sg3_utils scripts.

SG3 Known Issues

So far this tool has helped me avoid reboots and incurring the corresponding downtime. I hope it helps you as well.