到底功夫熊貓(Xen)踢不踢的動大象(Hadoop)呢

到底功夫熊貓(Xen)踢不踢的動大象(Hadoop)呢？ 

這幾天我在Facebook - Taiwan Hadoop User Group 拋磚引玉貼了 "要使用大象，真的得養頭大象嗎？為何不使用AWS EMR"的文章，意外引起熱烈的討論，把許多潛水的高手和同好都吸了出來~XD (你自己不是也都在潛水)，因為大家都有遇到虛擬化效能的問題。

一般來說，要養Hadoop (或是大型分散式系統)，如果不是用實體機器來養 (除非你像google一樣有錢，不然一定養不多，也養不起)，不然通常就是用虛擬化的技術來玩。
如果只是玩玩和測試安裝與練習寫寫程式，那可能還沒什麼大礙，但是一到要玩到真實案例，甚至要上Production 就會遇到許多效能瓶頸的問題需要去解決。像在 Taiwan Hadoop User Group 的討論串裡， James 大大就有提到：如果同樣都是 100TB 的 terasort 好了, 如果把一個實體的hadoop cluster搬到AWS，要多多少個 node 才能有相同的效能 ?

老實說，目前我們也還沒有玩到這麼大的資料量，所以還真的沒對於這麼大量的資料做過benckmark，希望將來我們系統長到那麼大，有機會分享這些數據給大家知道~XD

回到效能問題，第一步可能要先引自Jazz大的名言：學控制的人都會知道『先能量測，才有辦法控制』，Jazz大建議安裝 Ganglia 跟 Munin來觀察，我們是裝OpenNms，總之要先確認Performace是出在哪裡：

• Host Dom0?
• CPU ?
• Memory ?
• Disk I/O ?
• Network I/O ?

 第二步，就是要確認你的虛擬化環境與設備：

• 你Server 的等級
• 你目前是使用哪一版的Xen?( 3.x、4.x)
• 你是使用PV  還是HVM?
• 你是使用怎樣的deivce當domU's disk?( file: ? tap:aio:? phy:?)

因為公司內部和專案開發都是使用Xen，所以之前倒是有為了效能問題去找了一堆資料，整理如下：

1. VM 盡量不要使用 SWAP (不過這還有爭議)
2. RAID10 array is recommended
3. 使用PV (半虛擬化)效能較好，或使用Xen PVHVM drivers for Linux HVM guests
4. 盡量直接使用 block device (without file system).
5. 如果還是要用File System 請參考這篇 Filesystem performance on Xen

 不過好像也沒啥特別的，大家應該都知道了，如果有其他作法，歡迎大家留言討論~:P

How to Install OpenvSwitch for CentOS 6.x + Xen4.x

前一陣子在CentOS6.x + Xen 4.x 上安裝 OpenvSwitch 遇到許多問題，雖說網路上有許多安裝範例，但是真的很多都沒辦法使用，都會遇到一些問題，索性就把安裝Openvswitch 的指令寫成Script 有興趣的朋友可以參考看看。


#!/bin/bash
# Licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License.
# A copy of the License is located at
#
# http://aws.amazon.com/apache2.0
#
# or in the "license" file accompanying this file. This file is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied. See the License for the specific language governing
# permissions and limitations under the License.
#
# Program:
# Install-openvSwitch.sh
#

if [ $UID != "0" ]; then
echo " You must be root to run the script or run it with sudo"
exit 1;
fi

if [ $# -lt 1 ]; then
echo " Usage: $0 "
echo " if src path in ~/ovs/ then =ovs "
echo " The script assumes the guest being clone is a Red Hat based system"
exit 1;
fi
INSTALLPATH=$1

pushd $INSTALLPATH

echo "Prepare need package......"
yum -y install tunctl

echo "Start download vswitch source......."
wget http://openvswitch.org/releases/openvswitch-1.3.0.tar.gz
tar zxvf openvswitch-1.3.0.tar.gz


pushd openvswitch-1.3.0/
./configure --with-linux=/lib/modules/`uname -r`/build 1>/dev/null 2>> $(errlog)
make 1>/dev/null 2>> $(errlog) && make install 1>/dev/null 2>> $(errlog)

#
#
#
rmmod bridge
insmod datapath/linux/openvswitch_mod.ko
virsh net-destroy default
virsh net-undefine default
service libvirtd restart


mkdir -p /usr/local/etc/openvswitch

ovsdb-tool create /usr/local/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema
ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \
--remote=db:Open_vSwitch,manager_options \
--private-key=db:SSL,private_key \
--certificate=db:SSL,certificate \
--bootstrap-ca-cert=db:SSL,ca_cert \
--pidfile --detach
ovs-vsctl --no-wait init
ovs-vswitchd --pidfile --detach



Install Xen 4.0.1 on CentOS 6

Reference document: RHEL6Xen4Tutorial
According to  reference document, I wrote a shell script to install Xen4.0.1 on CentOS 6

#!/bin/bash
# Program:
#       Install-xen4.0.1.sh
# History:
# 2011/10/20    howie YU   First release

if [ $UID != "0" ]; then
  echo " You must be root to run the script or run it with sudo"
  exit 1;
fi

if [ $# -lt 1 ]; then
  echo " Usage: ./install-xen.sh   "
  echo " if src path in ~/xen/ then =xen "
  echo " The script assumes the guest being clone is a Red Hat based system"
  exit 1;
fi
installPath=$1

chkconfig --list network
yum -y update
yum -y install screen vim wget tcpdump ntp ntpdate man smartmontools links lynx ethtool xorg-x11-xauth
yum -y groupinstall "Development tools" "Additional Development" "Debugging Tools" "System administration tools" "Compatibility libraries" "Console internet tools" "Desktop Platform Development"
yum -y install transfig wget texi2html libaio-devel dev86 glibc-devel e2fsprogs-devel gitk mkinitrd iasl xz-devel bzip2-devel pciutils-libs pciutils-devel SDL-devel libX11-devel gtk2-devel bridge-utils PyXML qemu-common qemu-img mercurial texinfo libuuid-devel
yum -y install glibc-devel.i686

echo "Remove old virtlib....."
yum -y groupremove "Virtualization"

echo "Prepare for qeum...."
wget http://download.fedora.redhat.com/pub/fedora/linux/releases/14/Everything/i386/os/Packages/openbios-common-1.0-2.fc12.noarch.rpm
wget ftp://ftp.pbone.net/mirror/download.fedora.redhat.com/pub/fedora/linux/releases/14/Everything/i386/os/Packages/openbios-ppc-1.0-2.fc12.noarch.rpm
wget ftp://ftp.isu.edu.tw/pub/Linux/Fedora/linux/releases/test/16-Alpha/Fedora/x86_64/os/Packages/seabios-bin-0.6.2-2.fc16.noarch.rpm
wget ftp://bo.mirror.garr.it/pub/1/fedora/linux/development/16/x86_64/os/Packages/seabios-0.6.2-2.fc16.x86_64.rpm
wget ftp://ftp.isu.edu.tw/pub/Linux/Fedora/linux/releases/15/Everything/i386/os/Packages/gpxe-roms-qemu-1.0.1-4.fc15.noarch.rpm


rpm -ivh openbios-common-1.0-2.fc12.noarch.rpm
rpm -ivh openbios-ppc-1.0-2.fc12.noarch.rpm
rpm -e seabios
rpm -ivh seabios-bin-0.6.2-2.fc16.noarch.rpm
rpm -ivh seabios-0.6.2-2.fc16.x86_64.rpm
rpm -e gpxe-roms-qemu
rpm -ivh gpxe-roms-qemu-1.0.1-4.fc15.noarch.rpm

wget http://dl.fedoraproject.org/pub/fedora/linux/updates/13/SRPMS/qemu-0.13.0-1.fc13.src.rpm
rpm -i qemu-0.13.0-1.fc13.src.rpm
rpmbuild -bb ~/rpmbuild/SPECS/qemu.spec

rpm -ivh  ~/rpmbuild/RPMS/x86_64/qemu-common-0.13.0-1.el6.x86_64.rpm
rpm -Uvh  ~/rpmbuild/RPMS/x86_64/qemu-img-0.13.0-1.el6.x86_64.rpm
rpm -Uvh  ~/rpmbuild/RPMS/x86_64/qemu-user-0.13.0-1.el6.x86_64.rpm
rpm -Uvh  ~/rpmbuild/RPMS/x86_64/qemu-system-*
rpm -ivh  ~/rpmbuild/RPMS/x86_64/qemu-kvm-*
rpm -ivh  ~/rpmbuild/RPMS/x86_64/qemu-debuginfo-0.13.0-1.el6.x86_64.rpm
rpm -ivh  ~/rpmbuild/RPMS/x86_64/qemu-0.13.0-1.el6.x86_64.rpm

echo "start to build xen"

wget ftp://ftp.isu.edu.tw/pub/Linux/Fedora/linux/releases/14/Fedora/source/SRPMS/xen-4.0.1-6.fc14.src.rpm
rpm -i xen-4.0.1-6.fc14.src.rpm
rpmbuild -bb ~/rpmbuild/SPECS/xen.spec

rpm -ivh ~/rpmbuild/RPMS/x86_64/xen-licenses-4.0.1-6.el6.x86_64.rpm
rpm -ivh ~/rpmbuild/RPMS/x86_64/xen-libs-4.0.1-6.el6.x86_64.rpm
rpm -ivh ~/rpmbuild/RPMS/x86_64/xen-hypervisor-4.0.1-6.el6.x86_64.rpm
rpm -ivh ~/rpmbuild/RPMS/x86_64/xen-runtime-4.0.1-6.el6.x86_64.rpm
rpm -ivh ~/rpmbuild/RPMS/x86_64/xen-doc-4.0.1-6.el6.x86_64.rpm
rpm -ivh ~/rpmbuild/RPMS/x86_64/xen-devel-4.0.1-6.el6.x86_64.rpm
rpm -ivh ~/rpmbuild/RPMS/x86_64/xen-debuginfo-4.0.1-6.el6.x86_64.rpm
rpm -ivh ~/rpmbuild/RPMS/x86_64/xen-4.0.1-6.el6.x86_64.rpm

echo "Prepare for libvirt install...."


yum -y install xhtml1-dtds libudev-devel libpciaccess-devel  yajl-devel libpcap-devel  libnl-devel avahi-devel  parted-devel device-mapper-devel numactl-devel  netcf-devel augeas radvd systemtap-sdt-devel gnutls-utils

wget ftp://ftp.redhat.com/pub/redhat/linux/enterprise/6Server/en/os/SRPMS/libvirt-0.8.7-18.el6_1.1.src.rpm
rpm -ivh libvirt-0.8.7-18.el6_1.1.src.rpm
wget http://pasik.reaktio.net/xen/patches/libvirt-spec-rhel6-enable-xen.patch
cp -a ~/rpmbuild/SPECS/libvirt.spec /root/rpmbuild/SPECS/libvirt.spec.orig
cd ~/rpmbuild/SPECS/
patch -p0 < ~/$installPath/libvirt-spec-rhel6-enable-xen.patch
cd ~/
rpmbuild -bb ~/rpmbuild/SPECS/libvirt.spec

echo "Remove old libvirt....."
yum -y erase libvirt-client

echo " reinstall libvirt......."

rpm -ivh  ~/rpmbuild/RPMS/x86_64/libvirt-client-0.8.7-18.el6.1.x86_64.rpm
rpm -ivh  ~/rpmbuild/RPMS/x86_64/libvirt-devel-0.8.7-18.el6.1.x86_64.rpm
rpm -ivh  ~/rpmbuild/RPMS/x86_64/libvirt-python-0.8.7-18.el6.1.x86_64.rpm
rpm -ivh  ~/rpmbuild/RPMS/x86_64/libvirt-debuginfo-0.8.7-18.el6.1.x86_64.rpm
rpm -ivh  ~/rpmbuild/RPMS/x86_64/libvirt-0.8.7-18.el6.1.x86_64.rpm

yum -y install virt-manager virt-top virt-viewer

#
# CentOS 6 and RHEL 6 has IGMP built into the bridge-utils. To prevent unnecessary error message, we will need to get Fedora version of bridge-utils source and rebuild it.
#
#
yum -y install  libsysfs-devel

wget ftp://ftp.icm.edu.pl/vol/rzm2/linux-fedora-secondary/development/15/source/SRPMS/bridge-utils-1.2-9.fc13.src.rpm
rpm -i bridge-utils-1.2-9.fc13.src.rpm
rpmbuild -bb ~/rpmbuild/SPECS/bridge-utils.spec

rpm -ivh --force ~/rpmbuild/RPMS/x86_64/bridge-utils-1.2-9.el6.x86_64.rpm


echo "prepare for kernel build "
yum -y install xmlto asciidoc newt-devel perl-ExtUtils-Embed


rpm -ivh kernel-2.6.32-131.12.1.el6.xendom0.src.rpm
rngd -r /dev/urandom
rpmbuild -bb --with baseonly --with firmware --without debuginfo --target=`uname -m`  ~/rpmbuild/SPECS/kernel.spec

rpm -Uvh ~/rpmbuild/RPMS/x86_64/kernel-firmware-2.6.32-131.12.1.el6.xendom0.x86_64.rpm
rpm -Uvh ~/rpmbuild/RPMS/x86_64/kernel-headers-2.6.32-131.12.1.el6.xendom0.x86_64.rpm
rpm -ivh ~/rpmbuild/RPMS/x86_64/kernel-devel-2.6.32-131.12.1.el6.xendom0.x86_64.rpm
rpm -ivh ~/rpmbuild/RPMS/x86_64/kernel-2.6.32-131.12.1.el6.xendom0.x86_64.rpm

chkconfig ksm off
chkconfig ksmtuned off

#
# title CentOS (2.6.18-128.1.10.el5)
# root (hd0,0)
# kernel /boot/vmlinuz-2.6.18-128.1.10.el5 ro root=LABEL=/ console=ttyS0
# initrd /boot/initrd-2.6.18-128.1.10.el5.img
#
#
#
echo "ttyS0" >> /etc/securetty

echo "Finish pleas check /etc/grub.conf"
echo " kernel /xen.gz dom0_mem=4096M console=ttyS0 loglvl=all guest_loglvl=all"
echo "module max_loop=256 earlyprintk=xen"