db2 purecale是ibm推出的基於共享磁盤的，可擴展的數據庫集群軟件，類似於oracle的rac。具體功能這裡就不過多介紹了，直接介紹在rehdat上安裝purescale的實戰經歷

1 安裝手冊

軟硬件版本

IB交換機：Mellanox

PC服務器：X3850 X5

存儲版本：DS5100

OS版本：RHEL6.1

DB2版本：db2 purescale10.1.0.2

建議使用purescale認證的軟硬件版本，否則可能會碰到各種莫名其妙的問題

1.1 規劃

主機名

IP地址

管理

IB地址

CF

member

Wtydb21

192.168.2.101

192.168.1.101

192.168.0.101

primary

YES

Wtydb22

192.168.2.102

192.168.1.102

192.168.0.102

second

YES

Wtydb23

192.168.2.103

192.168.1.103

192.168.0.103

YES

IB地址規劃

管理IP

浮動IP

IB

Switch-1

192.168.1.104

192.168.1.106

192.168.0.201

Switch-2

192.168.1.105

192.168.0.202

1.2 安裝前環境准備

1.2.1 Nfs+yum源配置

1.2.1.1 配置NFS

Wtydb21為NFS server，其它主機為client，為了在各服務器安裝軟件包方便，建議配置yum

1，手工安裝nfs軟件包

rpm –qa nfs

修改為nfs默認打開,所有服務器設置

[root@stydb1 ~]# chkconfig --list nfs

nfs 0:off 1:off 2:off 3:off 4:off 5:off 6:off

[root@stydb1 ~]# chkconfig nfs on

[root@stydb1 ~]# chkconfig --list nfs

nfs 0:off 1:off 2:on 3:on 4:on 5:on 6:off

service portmap start

service nfs start

service rpcbind start

2, 確保nfs server端防火牆處於關閉狀態

service iptables stop

3，nfs server端配置

A，Mount NFS磁盤mount nfs disks，本例共享/data目錄

mount /dev/mapper/vg_1-lv_data /data

B，vi /etc/fstab，添加如下一行，修改為自動掛載

/dev/mapper/vg_1-lv_data /data ext4 defaults 1 2

C，nfs共享/data目錄

vi /etc/exports

/data *(rw)

/data wtydb22(rw) wtydb23(rw)

/usr/sbin/exportfs -r

4，客戶端設置

A，查看nfs服務器端的設置

[root@wtydb22 ~]# showmount -e wtydb21

Export list for wtydb21:

/data wtydb23,wtydb22

B，vi /etc/fstab

wtydb21:/data /data nfs rw

rpm -qa|grep nfs

C，手工mount

mkdir /data

mount wtydb21:/data/ /data/

1.2.1.2 配置yum源

1，新建目錄mkidr/data/rhel6.1

cp 光盤內容到/data/rhel6.1目錄，也可以直接使用光盤或者iso文件的介質文件

2，配置yum（三台服務器上配置）

vi/etc/yum.repos.d/rhel6.repo

[Server]

name=rhel6server

baseurl=file:///data/rhel6.1/Server/

enable=1

gpcheck=1

gpgkey=file:///data/rhel6.1/RPM-GPG-KEY-redhat-release

3，測試

yum list

也可通過iso文件實現

mount -o loop /software/rhel-server-6.1-x86_64-dvd.iso /mnt/cdrom

vi /etc/yum.repos.d/rhel6.repo

[Server]

name=rhel6server

baseurl=file:///mnt/cdrom/Server/

enable=1

gpcheck=1

gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-redhat-release

1.2.2 ssh互通

1， 生成ssh key，在~/.ssh/目錄下會生成兩個文件

ssh-keygen -t dsa，一路默認回車

在.ssh目錄下

id_dsa (the private key) and id_dsa.pub (the public key) for DSA encryption

2，把三台主機的id_dsa.pub文件的內容復制到authorized_keys文件中，把authorized_keys放於三台主機的.ssh目錄下

1.2.3 GPFS安裝配置（可選）

正常情況下，GPFS不需要單獨安裝，隨著安裝db2 purescale軟件會自動完成gpfs安裝。.

1.2.3.1 手工安裝

[root@wtydb22 data]# cd /data/esefp2/server/db2/linuxamd64/gpfs

[root@wtydb22 gpfs]# ls

base db2ckgpfs db2gutil errMsg fp installGPFS uninstallGPFS

[root@wtydb22 gpfs]# ./installGPFS -i -g

"GPFS" is not installed.

DBI1070I Program installGPFS completed successfully.

# vi /tmp/gpfsprofile

wtydb21:quorum

wtydb22:quorum

wtydb23:quorum

[root@pure1, /]

mmcrcluster -n /tmp/gpfsprofile -p wtydb21-s wtydb22 -C gpfs_cluster -r /usr/bin/ssh -R /usr/bin/scp

# mmcrcluster -n /home/gpfs/gpfs.allnodes-p node1 -s node2 -C gpfs_cluster -r /usr/bin/ssh -R /usr/bin/scp

# cd /opt/ibm/db2/V10.1/bin

# ./db2cluster -cfs -add -license

The license for the shared file system

./db2cluster -cfs -start -all

./db2cluster -cfs -create -filesystem db2fs-disk /dev/dm-3 -mount /db2fs

1.2.3.2 安裝配置GPFS

1， 安裝gpfs軟件（三台主機上運行）

本次從db2 purescale安裝介質中直接安裝，其它安裝方法見參考文獻

rpm –qa gpfs驗證軟件是否已經正確安裝

2， cd /usr/lpp/mmfs/src && make Autoconfig && make World && make InstallImages

3， 修改PATH路徑（三台主機上運行）

在3 台主機上修改HOME目錄下的。bash_profile文件，在文件末尾增加

export PATH=$PATH:/usr/lpp/mmfs/bin

3， 新建目錄,用做GPFS文件系統（三台主機上運行）

mkdir /db2fs

4， 創建gpfs 群集配置文件

[root@wtydb21 tmp]# vi /tmp/gpfsprofile

wtydb21:quorum-manager

wtydb22:quorum-manager

wtydb23:quorum-manager

5，創建集群，注意指定ssh方式

[root@wtydb21 pam.d]#

[root@wtydb21 gpfs]# mmcrcluster -N /tmp/gpfsprofile -p wtydb21 -s wtydb22 -C gpfs_cluster -r /usr/bin/ssh -R /usr/bin/scp

Sat Apr 6 12:17:35 CST 2013: mmcrcluster: Processing node wtydb21

Sat Apr 6 12:17:35 CST 2013: mmcrcluster: Processing node wtydb22

Sat Apr 6 12:17:38 CST 2013: mmcrcluster: Processing node wtydb23

mmcrcluster: Command successfully completed

mmcrcluster: Warning: Not all nodes have proper GPFS license designations.

Use the mmchlicense command to designate licenses as needed.

mmcrcluster: Propagating the cluster configuration data to all

affected nodes. This is an asynchronous process.

mmcrcluster 命令其中參數含義

-C bgbcrun 設定集群名稱

-U bgbc 定義域名

-N /tmp/gpfs/nodefile 指定節點文件名

-p NSD1 指定主NSD 服務器為 NSD1

-s NSD1 指定備NSD 服務器為 NSD1

6，接受許可協議

[root@wtydb21 pam.d]# mmchlicense server --accept -N wtydb21,wtydb22,wtydb23

7，確認創建集群狀況

[root@wtydb21 ~]# mmlscluster

[root@wtydb21 gpfs]# mmlscluster

GPFS cluster information

========================

GPFS cluster name: gpfs_cluster.wtydb21

GPFS cluster id: 12146727015547904479

GPFS UID domain: gpfs_cluster.wtydb21

Remote shell command: /usr/bin/ssh

Remote file copy command: /usr/bin/scp

GPFS cluster configuration servers:

-----------------------------------

Primary server: wtydb21

Secondary server: wtydb22

Node Daemon node name IP address Admin node name Designation

-------------------------------------------------------------------

1 wtydb21 192.168.2.101 wtydb21 quorum-manager

2 wtydb22 192.168.2.102 wtydb22 quorum-manager

3 wtydb23 192.168.2.103 wtydb23 quorum-manager

8，生成nsd 盤，使用/dev/dm-3

[root@wtydb21 etc]# vi /tmp/nsdprofile

dm-3:::dataAndMetadata::

[root@wtydb21 gpfs]# mmcrnsd -F /tmp/nsdprofile

mmcrnsd: Processing disk dm-3

mmcrnsd: Propagating the cluster configuration data to all

affected nodes. This is an asynchronous process.

此時系統自動修改/tmp/gpfsnsdprofile 文件內容如下：

[root@wtydb21 ~]# cat /tmp/nsdprofile

# dm-4:::dataAndMetadata::

gpfs1nsd:::dataAndMetadata:-1::system

9,啟動集群

[root@wtydb21 /]# mmstartup -a

.

[root@wtydb21 src]# mmgetstate -a -L

Node number Node name Quorum Nodes up Total nodes GPFS state Remarks

------------------------------------------------------------------------------------

1 wtydb21 2 2 3 active quorum node

2 wtydb22 2 2 3 active quorum node

3 wtydb23 2 2 3 active

10,創建gpfs 文件系統

[root@wtydb21 src]# mmcrfs /db2fs gpfs_lv -F /tmp/nsdprofile -A yes -n 30 -v no

The following disks of gpfs_lv will be formatted on node wtydb21:

gpfs1nsd: size 100829184 KB

Formatting file system ...

Disks up to size 848 GB can be added to storage pool system.

Creating Inode File

Creating Allocation Maps

Creating Log Files

Clearing Inode Allocation Map

Clearing Block Allocation Map

Formatting Allocation Map for storage pool system

Completed creation of file system /dev/gpfs_lv.

mmcrfs: Propagating the cluster configuration data to all

affected nodes. This is an asynchronous process.

參數含義如下：

/datafs 文件系統 mount 點名

gpfs_lv 指定文件系統 lv 名

-F 指定 NSD 的文件名

-A 自動 mount 選項為 yes

-B 塊大小為64K

-n 掛載文件系統的節點估計數30 個

-v 校驗建立磁盤是否已有文件系統 為否

11，掛載文件系統

[root@wtydb21 /]# mount /db2fs

[root@wtydb21 /]# df

12，設置gpfs 開機自啟

[root@wtydb21 /]# mmchconfig autoload=yes

13，查詢GPFS 配置信息

[root@wtydb21 share]# mmlsconfig

[root@wtydb21 share]# mmgetstate -a

1.2.3.3 修改參數配置

db2 purescale要求必須符合以下參數配置

[root@wtydb21 src]# /opt/ibm/db2/V10.1/bin/db2cluster -cfs -verify -configuration

The shared file system cluster option 'adminMode' has not been set to the optimal value of 'allToAll'. See the DB2 Information Center for more details.

The shared file system cluster option 'maxFilesToCache' has been set too low. The value should be at least '10000'. See the DB2 Information Center for more details.

The shared file system cluster option 'usePersistentReserve' has not been set to the optimal value of 'yes'. See the DB2 Information Center for more details.

The shared file system cluster option 'verifyGpfsReady' has not been set to the optimal value of 'yes'. See the DB2 Information Center for more details.

The shared file system cluster option 'failureDetectionTime' could not be verified. Re-issue this command after the cluster manager peer domain has been created and is online.

A diagnostic log has been saved to '/tmp/ibm.db2.cluster.toBL0B'.修改過程如下

mmchconfig adminMode allToAll

mmchconfig maxFilesToCache=20000

mmchconfig pagepool=1024M

mmchconfig usePersistentReserve=yes

mmchconfig verifyGpfsReady=yes

mmshutdown -a

db2cluster -cfs -stop -all

mmchconfig failureDetectionTime=35

db2cluster -cfs -start -all

mmstartup -a

1.2.3.4 驗證GPFS

db2cluster -cfs -verify -configuration

db2cluster -cfs -verify -filesystem mygpfs1

實例的所有者db2sdin1對文件系統具有訪問權限

1.2.3.5 刪除GPFS集群（未驗證）

1.fuser –kcu /ods/gpfs

2. unmount /ods/gpfs #在所有節點上執行

3. mmdelfs gpfslv,

4. mmlsfs gpfslv #檢查結果

5. mmdelnsd –F /ods/gpfsnsdprofile

6. mmshutdown –a

7. mmdelnode -a

8. mmdelnode –f #最後清除集群

Optional. For DB2 managed GPFSinstallations, verify the remote shell and

remote file copy settings default todb2locssh and db2scp. For example:

/usr/lpp/mmfs/bin/mmlscluster

Remote shell command:/var/db2/db2ssh/db2locssh

Remote file copy command:/var/db2/db2ssh/db2scp

1，卸載所有gpfs文件系統，mmumount all -a

2，刪除文件系統，mmdelfs gpfs_lv

3，刪除nsd節點，mmdelnsd -F /tmp/nsdprofile

4，停止gpfs系統，mmshutdown-a

5，卸載gpfs軟件，先查看已經安裝的軟件rpm -qa|grep gpfs

rpm -e gpfs.gpl

rpm -e gpfs.msg.en_us

rpm -e gpfs.base

rpm -e gpfs.docs

6，刪除目錄 /var/mmfs和/usr/lpp/mmfs

7，刪除/var/adm/ras目錄下以mm開頭的文件

8，刪除/tmp/mmfs（如果存在）目錄和其內容

gpfs log file path /var/adm/ras

1.2.3.6 Explanation

The verifyGpfsReady=yesconfiguration attribute is set, but the /var/mmfs/etc/gpfsreadyscript could not be executed.

1.2.3.7 User response

Make sure /var/mmfs/etc/gpfsreadyexists and is executable, or disable the verifyGpfsReadyoption via mmchconfig verifyGpfsReady=no.

ERROR: DBI20093E The DB2 installer failed to set up thedb2sshid because the

GPFS file system is a user-managed filesystem.

參考文獻

1.2.4 網絡環境

1.2.4.1 1，修改vi /etc/hosts文件

[root@wtydb21 ~]# cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.0.101 wtydb21-ib0

192.168.3.101 wtydb21-ib3

192.168.0.102 wtydb22-ib0

192.168.3.102 wtydb22-ib3

192.168.0.103 wtydb23-ib0

192.168.3.103 wtydb23-ib3

192.168.2.101 wtydb21

192.168.2.102 wtydb22

192.168.2.103 wtydb23

備注：IP和主機名之間用空格，一定不要用tab鍵，否則報錯。lookback不要匹配主機名

1.2.4.2 2，修改主機名

第一步：

#hostnamewtydb21

第二步：

修改/etc/sysconfig/network中的hostname

第三步：

修改/etc/hosts文件

1.2.4.3 3，設置網卡綁定

eth0和eth4做網卡綁定，主備模式

使用ethtool確認網絡連接狀態

1，cd/etc/sysconfig/network-scripts/ 修改相關配置文件

新建 ifcfg-bond0

DEVICE=bond0

BOOTPROTO=static

IPADDR=192.168.2.101

NETMASK=255.255.255.0

GATEWAY=192.168.2.254

ONBOOT=YES

TYPE=Ethernet

USERCTL=no

BONDING_OPTS="miimon=100 mode=1"

修改ifcfg-eth0內容：

DEVICE=eth0

BOOTPROTO=none

ONBOOT=yes

USERCTL=no

MASTER=bond0

SLAVE=yes

修改ifcfg-eth4內容

DEVICE=eth4

BOOTPROTO=none

ONBOOT=yes

USERCTL=no

MASTER=bond0

SLAVE=yes

2，vi/etc/modprobe.conf /etc/modprobe.d/bonding.conf

aliasbond0 bonding

3，重啟網絡配置

servicenetwork

4， 關閉networkmanger(rhel6.3版本需要，之前rhel5配置時無此步驟)

chkconfig NetworkManager off;

/etc/init.d/NetworkManager stop;

chkconfig network on;

/etc/init.d/network restart

ifconfig bond1 up

1.2.4.4 4，關閉相關服務：防火牆和selinux

1，chkconfig iptablesoff

service iptables stop

2，selinux

vi /etc/selinux/config

#SELINUX=enforcing

SELINUX=disabled

將文件中的SELINUX="" 為 disabled ，然後重啟。

如果不想重啟系統，使用命令setenforce 0

1.2.5 IB環境准備

1.2.5.1 交換機配置

With the Mellanox SM HA software, thesystem manager can enter and modify

all IB subnet configurations for the subnetmanagers from a single location.

Assign a virtual IP address (VIP) to themanagement port of the switch to

manage the high availability domain. Thesystem manager must configure all

the switches in a Mellanox SM HAenvironment to join the same IB subnet, and

assign the subnet a name. After joining thesubnet, the subnet managers are

synchronized and you must select one as themaster subnet manager, and the

others become standby subnet managers.

1.2.5.2 安裝infiniband驅動軟件

infiniband軟件

Install the packages required and reboot:

rhel 6.3版本

[root@serv]# yum groupinstall "Infiniband Support"

[root@serv]# yum install infiniband-diags perftest qperf opensm

[root@serv]# chkconfig rdma on

[root@serv]# chkconfig opensm on

[root@serv]# shutdown -r now

1.2.5.3 配置IP地址

同普通以太網卡配置，例外db2數據庫使用RDMA協議，而不是TCP|IP協議

[root@wtydb22 network-scripts]# cat ifcfg-ib0

DEVICE="ib0"

BOOTPROTO="static"

IPADDR=192.168.0.102

#NETMASK=255.255.255.0

PREFIX=24

NM_CONTROLLED="yes"

ONBOOT="yes"

TYPE="InfiniBand"

1.2.5.4 參數配置

1，log_mtts_per_seg參數

如果 CF 或成員所在的主機有超過 64 GB 的內存，那麼 Mellanox HCA 驅動程序（mlx4_core）模塊的參數 log_mtts_per_seg 必須從 3（默認）增加到 7，以便有更大的內存注冊。

要增加大小，以根用戶身份發出以下命令：

在 SUSE 上：

echo "optionsmlx4_core log_mtts_per_seg=7" >>

/etc/modprobe.conf.local

在 RHEL 上：

echo "optionsmlx4_core log_mtts_per_seg=7" >> /etc/modprobe.conf

RHEL 6 上需要運行如下命令

echo "options mlx4_corelog_mtts_per_seg=7" >> /etc/modprobe.d/modprobe.conf

要讓該更改生效，您必須重啟服務器。要檢查您的更改在模塊上是否有效，輸入：

<host-name>/sys/module/mlx4_core/parameters# cat /sys/module/mlx4_core/parameters/log_mtts_per_seg

2，/etc/rdma/dat.conf配置文件

On RHEL 6.1, the DAT configuration file islocated in /etc/rdma/dat.conf

and it is updated by the group installationof the "InfiniBand Support"

package

系統默認配置好了

[root@wtydb21 Packages]# cat/etc/rdma/dat.conf

ofa-v2-mlx4_0-1 u2.0 nonthreadsafe defaultlibdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""

ofa-v2-mlx4_0-2 u2.0 nonthreadsafe defaultlibdaploscm.so.2 dapl.2.0 "mlx4_0 2" ""

ofa-v2-ib0 u2.0 nonthreadsafe defaultlibdaplofa.so.2 dapl.2.0 "ib0 0" ""

……

1.2.5.5 最佳實踐

網絡配置

2）檢查infiniband交換機的配置

a,管理軟件需要最低版本

image-PPC_M405EXEFM_1.1.2500.img

b,啟動子網管理SM，管理口配置VIP，互聯口配置IP地址

c，配置故障轉移

d,兩個交換機之間需要2根以上的互連線，建議CF和member連接至交換機的總port數除以2

1.2.6 依賴軟件安裝

使用yum install命令在三台主機同時安裝即可

yum install libstdc++

yum install pam-1.1.1

yum install pam_krb5-2.3.11

yum install pam-devel

yum install pam_pkcs11

所需軟件包，詳見官方安裝文檔說明。

libstdc++-4.4.5-6.el6.i686,

pam-1.1.1-8.el6.i686,

pam_krb5-2.3.11-6.el6.i686,

pam-devel-1.1.1-8.el6.i686,

pam_pkcs11-0.6.2-11.1.el6.i686,

pam_ldap-185-8.el6.i686

下列包rhel 6.1 均已經默認安裝

For InfiniBand network

type (both 32- bit and 64-bit libraries unless specified) :

libibcm

dapl (64-bit libraries only)

ibsim (64-bit libraries only)

ibutils (64-bit libraries only)

libibverbs

librdmacm

libcxgb3

libibmad

libibumad

libipathverbs (64-bit libraries only)

libmlx4

libmthca

libnes (64-bit libraries only)

libmlx4

rdma (no architecture)

ntp-4.2.4p8-2.el6.x86_64/ntpdate-4.2.4p8-2.el6.x86_64

libstdc++-4.4.5-6.el6.x86_64

libstdc++-4.4.5-6.el6.i686

glibc-2.12-1.25.el6.x86_64

glibc-2.12-1.25.el6.i686

gcc-c++-4.4.5-6.el6.x86_64

gcc-4.4.5-6.el6.x86_64

kernel-2.6.32-131.0.15.el6.x86_64

kernel-devel-2.6.32-131.0.15.el6.x86_64

kernel-headers-2.6.32-131.0.15.el6.x86_64

kernel-firmware-2.6.32-131.0.15.el6.noarch

ntp-4.2.4p8-2.el6.x86_64

ntpdate-4.2.4p8-2.el6.x86_64

sg3_utils-1.28-3.el6.x86_64

sg3_utils-libs-1.28-3.el6.x86_64

binutils-2.20.51.0.2-5.20.el6.x86_64

binutils-devel-2.20.51.0.2-5.20.el6.x86_64

openssh-5.3p1-52.el6.x86_64

cpp-4.4.5-6.el6.x86_64

ksh-20100621-16.el6.x86_64

安裝示例說明

[root@wtydb23 ~]# yum install compat-libstdc++-33-3.2.3-69.el6.x86_64.rpm

1.2.6.1 GPFS依賴軟件包

另外，GPFS軟件還需要I686等軟件包需要安裝

v For multiple communication adapter portson InfiniBand network and single or multiple communication adapter port at CFson 10GE network,

the minimum support level is RHEL 6.1.

i686 which is 32-bit packages might not getinstalled by default when installing x86_64 server. Make sure that all the32-bit dependencies are

explicitly installed. For example:

libstdc++-4.4.5-6.el6.i686,

pam-1.1.1-8.el6.i686,

pam_krb5-2.3.11-6.el6.i686,

pam-devel-1.1.1-8.el6.i686,

pam_pkcs11-0.6.2-11.1.el6.i686,

pam_ldap-185-8.el6.i686

Alternatively, run the yum command aftercreating a source from local DVD or after registering to RHN:

yum install *.i686

Ensure the following 32-bit RSCT packagesare installed:

– libibcm.i686

– libibverbs-rocee.i686

– librdmacm.i686

– libcxgb3.i686

– libibmad.i686

– libibumad.i686

– libmlx4-rocee.i686

– libmthca.i686·

DB2 pureScale Feature requireslibstdc++.so.6. Verify that the files exist with

the following commands:

ls /usr/lib/libstdc++.so.6*

ls /usr/lib64/libstdc++.so.6*

1.2.6.2 install SAM

[root@wtydb21 tsamp]# cat /tmp/prereqSAM.1.log

prereqSAM: >>> Prerequisite on wtydb21 check - log started : Fri Apr 12 14:22:16 CST 2013

prereqSAM: OPTIONS = ''

prereqSAM: OPT_SILENT = 0

prereqSAM: OPT_NOLICCHECK = 0

prereqSAM: Detected operating system Linux

prereqSAM: Detected architecture i386x

prereqSAM: Detected distribution RH

prereqSAM: Supported operating system versions RH Linux i386x - 5.0 6.0

prereqSAM: Detected operating system version

Red Hat Enterprise Linux Server release 6.1 (Santiago)

prereqSAM: rpm package and version installed 'ksh'

prereqSAM: Using default prerequisite checking on the following rpm package 'perl'

prereqSAM: rpm package and version installed 'perl'

prereqSAM: Using default prerequisite checking on the following rpm package 'libstdc++' 'i686'

prereqSAM: rpm package and version installed 'libstdc++' 'i686'

prereqSAM: Using default prerequisite checking on the following rpm package 'libstdc++' 'x86_64'

prereqSAM: rpm package and version installed 'libstdc++' 'x86_64'

prereqSAM: Using default prerequisite checking on the following rpm package 'compat-libstdc++-33' 'x86_64'

prereqSAM: Error: The following rpm package is not installed 'compat-libstdc++-33' 'x86_64'

prereqSAM: Using default prerequisite checking on the following rpm package 'pam' 'i686'

prereqSAM: rpm package and version installed 'pam' 'i686'

1 missing package: compat-libstdc++-33 (x86_64)

prereqSAM: Error: Prerequisite checking for the ITSAMP installation failed Linux i386x RH

Red Hat Enterprise Linux Server release 6.1 (Santiago)

prereqSAM: Most severe error code returned 21

prereqSAM: One or more prerequisite packages were not found installed

prereqSAM: <<< Prerequisite on wtydb21 check - log ended : Fri Apr 12 14:22:19 CST 2013

1.2.7 內核參數調整

db2 10.1 可以自動調整內核參數為最優值，但是個人設置一下還是沒啥壞處吧

able 1. Enforced minimum settings for Linuxinterprocess communication kernel parametersIPC kernel parameter Enforced minimum setting

kernel.shmmni (SHMMNI) 256 * <size of RAM in GB>

kernel.shmmax (SHMMAX) <size of RAM in bytes>

kernel.shmall (SHMALL) 2 * <size of RAM in the defaultsystem page size>

共享內存段的設置建議

Beginning with the first section on SharedMemory Limits, the SHMMAX limit is the maximum size of a shared memory segmenton a Linux system. The SHMALL limit is the maximum allocation of shared memorypages on a system.

It is recommended to set the SHMMAX value to be equal to the amount ofphysical memory on your system. However, the minimum required on x86 systems is268435456 (256 MB) and for 64-bit systems, it is 1073741824 (1 GB).

kernel.sem (SEMMNI) 256 * <size of RAM in GB>

kernel.sem (SEMMSL) 250

kernel.sem (SEMMNS) 256 000

kernel.sem (SEMOPM) 32

信號量相關的內核設置

kernel.msgmni (MSGMNI) 1 024 * <size of RAM in GB>

kernel.msgmax (MSGMAX) 65 536

kernel.msgmnb (MSGMNB) 65 536

消息相關的內核設置

設置的小例子

#Example for a computer with 128GB of RAM，100GB分配給數據庫使用

kernel.shmmni=4096

kernel.shmmax=107374182400

//單個共享內存段的最大大小，建議設置為最大物理內存大小。本例設置為100GB；大小為100*1024*1024*1024=107374182400

kernel.shmall=8388608

//系統中能夠分配的共享內存頁的大小，頁的大小是4KB，總大小設置為100GB，應該是100*1024*1024*1024/4/1024=26214400

#kernel.sem=<SEMMSL> <SEMMNS><SEMOPM> <SEMMNI>

kernel.sem=250 1024000 32 4096

kernel.msgmni=16384

kernel.msgmax=65536

kernel.msgmnb=65536

參數設置說明

具體設計的例子

vi /etc/sysctl.conf

2，網絡相關內核參數

注釋掉rp_filter=1的行，

# net.ipv4.conf.all.rp_filter = 1

# changes for DB2 pureScale

然後添加

net.ipv4.conf.all.rp_filter = 2

net.ipv4.conf.all.arp_ignore = 1

我的服務器設置內存128GB 100GB分配給數據庫使用

1.2.8 blacklist.conf

在所有的機器(pcf、scf、mb1、mb2)上，修改文件：

/etc/modprobe.d/blacklist.conf

添加如下內容：

# RSCT hatsd, add for pureScale

blacklist iTCO_wdt

blacklist iTCO_vendor_support

1.2.9 共享存儲准備

1.2.9.1 需求

一般至少需要四塊物理盤，

tirebreker disk 25M

實例共享文件系統 10G

數據文件系統

日志文件系統

1.2.9.2 安裝方法

1，安裝device-mapper-multipath rpm。

當在 DM-多路徑中添加新設備時，這些新設備會位於 /dev 目錄的三個不同位置：dev/mapper/mpathn、/dev/mpath/mpathn 和 /dev/dm-n。 /dev/mapper 中的設備是在引導過程中生成的。可使用這些設備訪問多路徑設備，例如在生成邏輯卷時。提供 /dev/mpath 中的設備是為了方便，這樣可在一個目錄中看到所有多路徑設備。這些設備是由 udev 設備管理器生成的，且在系統需要訪問它們時不一定能啟動。請不要使用這些設備生成邏輯卷或者文件系統。

所有 /dev/dm-n 格式的設備都只能是作為內部使用，且應該永遠不要使用。

（備注 db2 purescale不能使用/dev/mapper/mpath設備，只能使用dm-設備名稱）

yuminstall device-mapper-multipath*

2，打開multipathd

[root@wtydb21 ~]# chkconfig --list multipathd

multipathd 0:off 1:off 2:off 3:off 4:off 5:off 6:off

[root@wtydb21 ~]# chkconfig multipathd on

[root@wtydb21 ~]# chkconfig --list multipathd

multipathd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

3，修改配置文件，保持多路徑設備名稱保持一致（安裝db2 purescale 本步驟可省略）

一般可以修改/etc/multipath.conf文件，但為了保持集群（oraclerac），也可以修改/var/lib/multipath/bindings，並把文件復制到集群中的其它節點

4，查看配置情況

modprobe dm-multipath

service multipathd start

multipath -v2

multipath -v2 命令輸出多路徑設備的路徑來顯示哪些設備是多路徑設備。如果沒有輸出任何結果，請確定正確調試了所有 SAN 連接並使系統多路徑。

執行以下命令確定在引導時啟動了多路徑守護進程

mpathconf --enable --with_multipathd y

chkconfig multipathd on

chkconfig --list multipathd

$ service multipathd stop

$ service multipathd start

$ multipathd –F

$ multipathd –v2

參考文獻

https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/DM_Multipath/mpio_setup.html

1.2.10 用戶和組

可手工創建，也可在安裝db purescale軟件時配置。

兩個用戶分別屬於不同的組，集群中不同的主機上需要相同的UID和GID，相同的home目錄

You should use two different users with twodifferent groups. Each of the two users should have the same UID, GID, groupname, and the home directory on all the hosts.

Required user User name Groupname

Instance owner db2sdin1 db2iadm1

Fenced user db2sdfe1 db2fadm1

groupadd -g 999 db2iadm1

groupadd -g 998 db2fadm1

useradd -g db2iadm1 -u 1004-m db2sdin1

useradd -g db2fadm1 -u 1003 -m db2sdfe1

passwd db2sdin1

passwd db2fenc1

1.2.11 NTP設置

vi /etc/ntp.conf

server 192.168.0.81 # local clock

fudge 192.168.0.81 stratum 10

restrict 192.168.0.81 mask 255.255.255.255 nomodify notrap noquery

service ntp start

chkconfig ntp on

chkconfig --list ntpd

1.2.12 WatchDog設置

DB2使用自己的watch dog，因此需要關閉機器帶的intelwatch dog

In some installations, if Intel TCOWatchDog Timer Driver modules are loaded by default, they should beblacklisted, so that they do not start automatically or conflict with RSCT. Toblacklist the modules, edit the following files:

To verify if the modules are loaded

lsmod | grep -i iTCO_wdt; lsmod | grep -iiTCO_vendor_support

Edit the configuration files:

On RHEL 5.x and RHEL 6.1, edit file /etc/modprobe.d/blacklist.conf:

# RSCT hatsd

blacklist iTCO_wdt

blacklist iTCO_vendor_support

1.2.13 安裝前檢查

使用db2prereqcheck命令進行安裝前的檢查，一定確保檢查完成通過。其實即使了在安裝tivoli，gpfs等還需要一些依賴包，也可能出現安裝報錯

$ ./db2prereqcheck -i

檢查是否滿足具體安裝版本需求，使用-v命令

$ ./db2prereqcheck -v 10.1.0.0

To generate a report file containingvalidation information and output from the db2prereqcheck command (including asummary section listing all failing tests), use the -o <filename>parameter. Specify the full report filename path to generate the report inanother directory. The prerequisites for installing the DB2 product will onlybe displayed on the command screen without the -o <filename> parameter.

$ ./db2prereqcheck -i -o report.rpt

檢查IB網絡是否滿足要求

$ ./db2prereqcheck -i -p -t <networkconfiguration type>

db2prereqcheck –I –p –t MULT_IB_PORT_CLUSTER

1.2.14 接管GPFS（可選）

Purescale有個bug，不在認證中的陣列會出現不能使用/dev/dm-硬盤安裝的情況，可以先手工安裝gpfs，使用dm-盤，在使用db2cluster接管gpfs

解決不能直接使用/dev/dm- 進行安裝的問題

[root@wtydb21 db2sdin1]#/opt/ibm/db2/V10.1/bin/db2cluster -cfs -verify -configuration

/opt/ibm/db2/V10.1/instance/db2cluster_prepare-cfs_takeover

take over a user-managed GPFS cluster:

1. Log on as root on any machine in your cluster.

2. Run the db2cluster_prepare command with the following parameters:

db2cluster_prepare -cfs_takeover

3. Check the error return code using the echo $? command. If there are errors, resolve the errors and rerun the command as specified in Step 2.

4. To verify that you've properly added the record, run the following command:

[root@wtydb21 tmp]# db2greg -dump

S,GPFS,3.5.0.4,/usr/lpp/mmfs,-,-,0,0,-,1365173996,0

S,DB2,10.1.0.2,/opt/ibm/db2/V10.1,,,2,0,,1365174117,0

V,DB2GPRF,DB2SYSTEM,wtydb21,/opt/ibm/db2/V10.1,

V,GPFS_CLUSTER,NAME,gpfs_cluster.wtydb21,-,DB2_MANAGED

1.2.15 nable SCSI-3 PR

[root@wtydb21 src]#/usr/lpp/mmfs/bin/tsprinquiry dm-3

IBM :2810XIV :0000

vi /var/mmfs/etc/prcapdevices

IBM:2810XIV:0000

/usr/lpp/mmfs/bin/mmchconfigusePersistentReserve=yes

1.2.16 IB內核參數

echo "options mlx4_corelog_mtts_per_seg=7" >>/etc/modprobe.conf

1.3 安裝purescale Feature

僅安裝purescale Feature，不安裝db2 實例

1.3.1 db2setup圖形安裝

本文通過db2_install手工實現安裝

1.3.2 db2_install手工安裝

1.3.2.1 安裝軟件

需要在兩台主機分別進行安裝

[root@wtydb21 server]# ./db2_install -t /tmp/db2_install.trc -l db2_install.log

DBI1324W Support of the db2_install command is deprecated. For

more information, see the DB2 Information Center.

Default directory for installation of products - /opt/ibm/db2/V10.1

***********************************************************

Install into default directory (/opt/ibm/db2/V10.1) ? [yes/no]

yes

Specify one of the following keywords to install DB2 products.

AESE

ESE

CONSV

WSE

EXP

CLIENT

RTCL

Enter "help" to redisplay product names.

Enter "quit" to exit.

***********************************************************

ESE

***********************************************************

Do you want to install the DB2 pureScale Feature? [yes/no]

yes

DB2 installation is being initialized.

Total number of tasks to be performed: 50

Total estimated time for all tasks to be performed: 1910 second(s)

Task #1 start

Description: Checking license agreement acceptance

Estimated time 1 second(s)

Task #1 end

Task #2 start

Description: Base Client Support for installation with root privileges

Estimated time 3 second(s)

Task #2 end

Task #54 start

Description: Updating global profile registry

Estimated time 3 second(s)

Task #54 end

The execution completed successfully.

For more information see the DB2 installation log at

"/data/esefp2/server/db2_install.log"

.…….

1.3.2.2 新建實例

安裝前檢查，沒有輸出即正常

[root@wtydb21 ~]# preprpnode wtydb21 wtydb22wtydb23

開始安裝

/opt/ibm/db2/V10.1/instance/db2icrt -d -cf wtydb21 -cfnet wtydb21-ib0 -m wtydb22 -mnet wtydb22-ib0 -instance_shared_dev /dev/dm-15 -tbdev /dev/dm-2 -u db2sdfe1 db2sdin1

1.3.3 添加成員和CF

--添加第二個CF

/opt/ibm/db2/V10.1/instance/db2iupdt -d -add -cf wtydb22 -cfnet wtydb22-ib0 db2sdin1

--添加member

/opt/ibm/db2/V10.1/instance/db2iupdt -d -add -m wtydb22 -mnet wtydb22-ib0 db2sdin1

/opt/ibm/db2/V10.1/instance/db2iupdt -d -add -m wtydb23 -mnet wtydb23-ib0 db2sdin1

1.3.4 安裝補丁包

本次直接使用fp2最新補丁集安裝。

1.3.5 添加IB網卡(待實施)

更新前檢查db2nodes.cfg文件

[root@wtydb23 msdata]# cat /home/db2sdin1/sqllib/db2nodes.cfg

0 wtydb22 0 wtydb22-ib0 - MEMBER

1 wtydb23 0 wtydb23-ib0 - MEMBER

128 wtydb23 0 wtydb23-ib0 - CF

129 wtydb22 0 wtydb22-ib0 - CF

更新IB網絡配置

db2iupdt -update -cf wtydb23 -cfnet wtydb23-ib0,wtydb23-ib3 db2sdin1

db2iupdt -update -m wtydb23 -mnet wtydb23-ib0, wtydb23-ib3 db2sdin1

1.4 刪除軟件（可選）

安裝過程中如果出現問題，可以使用如下方法刪除相關軟件

1, 備份數據庫

2，使用db2stop 命令停止所有實例

3，使用db2idrop -g db2sdin1刪除所有主機上的數據庫實例.另外，db2idrop -g命令會保留最後主機上的gpfs數據庫，需要手工清除gpfs。

4，使用db2_deinstall -a命令刪除db2 purescale軟件

2 安裝中常見問題

1. /dev/mapper/mapth*設備不能使用

/dev/mapper/mapth*設備本質上是鏈接文件，GPFS不能使用，應該使用多路徑軟件映射後的/dev/mapth設備

2. 直接/dev/dm-*設備報錯

該問題是軟件的bug，主要是由於XIV存儲不在軟件支持列表導致，可以通過手工創建GPFS，然後db2cluster接管的方式或者改由

3 高可用測試

初始狀態，系統正常

[db2sdin1@wtydb22 ~]$ db2instance -list

ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME

-- ---- ----- --------- ------------ ----- ---------------- ------------ -------

0 MEMBER STARTED wtydb22 wtydb22 NO 0 0 wtydb22-ib0

1 MEMBER STARTED wtydb21 wtydb21 NO 0 0 wtydb21-ib0

2 MEMBER STARTED wtydb23 wtydb23 NO 0 0 wtydb23-ib0

128 CF PRIMARY wtydb21 wtydb21 NO - 0 wtydb21-ib0

129 CF PEER wtydb22 wtydb22 NO - 0 wtydb22-ib0

HOSTNAME STATE INSTANCE_STOPPED ALERT

-------- ----- ---------------- -----

wtydb23 ACTIVE NO NO

wtydb21 ACTIVE NO NO

wtydb22 ACTIVE NO NO

數據庫能正常連接

[db2sdin1@wtydb22 ~]$ db2 connect to msdb

Database Connection Information

Database server = DB2/LINUXX8664 10.1.2

SQL authorization ID = DB2SDIN1

Local database alias = MSDB

3.1 一台服務器問題

3.1.1 一台服務器公網斷

1，斷掉wtydb21主機的網卡

[root@wtydb21 ~]# ifdown bond0

2，此時，從其它主機觀察狀態，正常

[db2sdin1@wtydb22 ~]$ db2instance -list

ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME

-- ---- ----- --------- ------------ ----- ---------------- ------------ -------

0 MEMBER STARTED wtydb22 wtydb22 YES 0 0 wtydb22-ib0

1 MEMBER WAITING_FOR_FAILBACK wtydb21 wtydb23 YES 0 1 wtydb23-ib0

2 MEMBER STARTED wtydb23 wtydb23 YES 0 0 wtydb23-ib0

128 CF ERROR wtydb21 wtydb21 YES - 0 wtydb21-ib0

129 CF PEER wtydb22 wtydb22 NO - 0 wtydb22-ib0

HOSTNAME STATE INSTANCE_STOPPED ALERT

-------- ----- ---------------- -----

wtydb23 ACTIVE NO NO

wtydb21 ACTIVE YES YES

wtydb22 ACTIVE NO NO

There is currently an alert for a member, CF, or host in the data-sharing instance. For more information on the alert, its impact, and how to clear it, run the following command: 'db2cluster -cm -list -alert'.

[db2sdin1@wtydb22 ~]$ db2 connect to msdb

Database Connection Information

Database server = DB2/LINUXX8664 10.1.2

SQL authorization ID = DB2SDIN1

Local database alias = MSDB

3，網卡恢復後，系統能自動回復

一台服務器IB網絡斷

A,主機1 停止bond0網卡

[root@wtydb21 ~]# ifdown bond0

B, 主機2 仍能成功連接數據庫

[root@wtydb22 ~]# db2 connect to msdb

Database Connection Information

Database server = DB2/LINUXX8664 10.1.2

SQL authorization ID = ROOT

Local database alias = MSDB

C, 此時集群狀態，wtydb21處於waiting_for_fallback狀態

[db2sdin1@wtydb23 ~]$ db2instance -list

ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME

-- ---- ----- --------- ------------ ----- ---------------- ------------ -------

0 MEMBER STARTED wtydb22 wtydb22 NO 0 0 wtydb22-ib0

1 MEMBER STARTED wtydb23 wtydb23 NO 0 0 wtydb23-ib0

2 MEMBER WAITING_FOR_FAILBACK wtydb21 wtydb22 NO 0 1 wtydb22-ib0

128 CF PRIMARY wtydb23 wtydb23 NO - 0 wtydb23-ib0

129 CF PEER wtydb22 wtydb22 NO - 0 wtydb22-ib0

HOSTNAME STATE INSTANCE_STOPPED ALERT

-------- ----- ---------------- -----

wtydb21 ACTIVE NO YES

wtydb23 ACTIVE NO NO

wtydb22 ACTIVE NO NO

There is currently an alert for a member, CF, or host in the data-sharing instance. For more information on the alert, its impact, and how to clear it, run the following command: 'db2cluster -cm -list -alert'.

3.2 重啟一台服務器（wtydb23）

wtydb22主機上顯示，CF正常切換，數據庫仍能提供服務，測試通過。重新起來後，恢復正常

[db2sdin1@wtydb22 ~]$ db2instance -list

ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME

-- ---- ----- --------- ------------ ----- ---------------- ------------ -------

0 MEMBER STARTED wtydb22 wtydb22 NO 0 0 wtydb22-ib0

1 MEMBER WAITING_FOR_FAILBACK wtydb23 wtydb21 NO 0 1 wtydb21-ib0

2 MEMBER STARTED wtydb21 wtydb21 NO 0 0 wtydb21-ib0

128 CF ERROR wtydb23 wtydb23 YES - 0 wtydb23-ib0

129 CF PRIMARY wtydb22 wtydb22 NO - 0 wtydb22-ib0

HOSTNAME STATE INSTANCE_STOPPED ALERT

-------- ----- ---------------- -----

wtydb21 ACTIVE NO NO

wtydb23 INACTIVE NO YES

wtydb22 ACTIVE NO NO

There is currently an alert for a member, CF, or host in the data-sharing instance. For more information on the alert, its impact, and how to clear it, run the following command: 'db2cluster -cm -list -alert'

3.3 重啟IB交換機switch-1

wtydb22成為提供服務的主機

[db2sdin1@wtydb22 ~]$ db2instance -list

ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME

-- ---- ----- --------- ------------ ----- ---------------- ------------ -------

0 MEMBER STARTED wtydb22 wtydb22 NO 0 0 wtydb22-ib0

1 MEMBER WAITING_FOR_FAILBACK wtydb23 wtydb22 YES 0 2 wtydb22-ib0

2 MEMBER WAITING_FOR_FAILBACK wtydb21 wtydb22 YES 0 1 wtydb22-ib0

128 CF ERROR wtydb23 wtydb23 YES - 0 wtydb23-ib0

129 CF PRIMARY wtydb22 wtydb22 NO - 0 wtydb22-ib0

HOSTNAME STATE INSTANCE_STOPPED ALERT

-------- ----- ---------------- -----

wtydb21 ACTIVE NO YES

wtydb23 ACTIVE NO YES

wtydb22 ACTIVE NO NO

switch-1 起來以後，數據庫自動恢復

[db2sdin1@wtydb23 ~]$ db2instance -list

ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME

-- ---- ----- --------- ------------ ----- ---------------- ------------ -------

0 MEMBER STARTED wtydb22 wtydb22 NO 0 0 wtydb22-ib0

1 MEMBER STARTED wtydb23 wtydb23 NO 0 0 wtydb23-ib0

128 CF PEER wtydb23 wtydb23 NO - 0 wtydb23-ib0

129 CF PRIMARY wtydb22 wtydb22 NO - 0 wtydb22-ib0

HOSTNAME STATE INSTANCE_STOPPED ALERT

-------- ----- ---------------- -----

wtydb23 ACTIVE NO NO

wtydb22 ACTIVE NO NO

4 新增加一台服務器

前期環境准備，同前面

目標通過檢驗：/software/ese10.1.2/server/db2prereqcheck -p -tMULTI_IB_PORT_CLUSTER -v 10.1.0.2

/opt/ibm/db2/V10.1.01/instance/db2iupdt -d -add -m wtydb23 -mnet wtydb23-ib0 db2sdin1