捣腾Oracle RAC环境已经半月有余，从对RAC一无所知到环境搭建成功，期间遇到很多坑，填了很多坑；特写此篇，记录一下搭建的过程，以备忘！话不多说，开干！
1 2 3 4 5 主机名 Public网络 Priv网络 Storge网络 VIP rac1 192.168.100.11 172.16.10.11 172.16.20.11 92.168.100.21 rac2 192.168.100.12 172.16.10.12 172.16.20.12 192.168.100.22 rac-scan 192.168.100.23 freenas 172.16.20.13
软件环境：Oracle Linux6.9 Grid Oracle Database 11g ASMlib FreeNas
RAC搭建分为五个部分：
基础环境准备 共享存储搭建 网络搭建 Grid 安装 Database 安装 基础环境准备 基础环境如无特别指明，都是需要在两台机器上操作的。
虚拟机环境搭建 虚拟机资源配置 rac1:4核/8G/200G
网络环境配置 1 2 3 4 5 6 7 虚拟机名称 网卡1 网卡2 网卡3 rac1 VMnet1(NAT模式) VMnet2(仅主机模式) VMnet3(仅主机模式) rac2 VMnet1(NAT模式) VMnet2(仅主机模式) VMnet3(仅主机模式) freenas VMnet1(NAT模式) VMnet1:192.168.100.0/24 VMnet2:172.16.10.0/24 VMnet3:172.16.20.0/24
操作系统安装 安装图形界面环境
操作系统配置 修改主机名 rac1
1 2 vim /etc/sysconfig/network HOSTNAME=rac1
rac2
1 2 vim /etc/sysconfig/network HOSTNAME=rac2
配置hosts 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ########public####### 192.168.100.11 rac1 192.168.100.12 rac2 #######priv######### 172.16.10.11 rac1-priv 172.16.10.12 rac2-priv ######vip########### 192.168.100.21 rac1-vip 192.168.100.22 rac2-vip 192.168.100.23 rac-scan 172.16.20.13 freenas
配置时间同步 关闭ntpd服务，使用oracle自身的集群时间同步
1 2 3 4 5 6 7 8 cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime ntpdate cn.ntp.org.cn vim /etc/sysconfig/ntpd SYNC_HWCLOCK=yes OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid" /etc/init.d/ntpd stop chkconfig ntpd off mv /etc/ntp.conf /etc/ntp.conf.bak
基础软件安装 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 #默认是不安装32位软件包的，添加此选项 echo 'multilib_policy=all' >> /etc/yum.conf yum -y install binutils-* compat-libstdc++-* elfutils-libelf-* elfutils-libelf-devel-* gcc-* gcc-c++-* glibc-* glibc-common-* glibc-devel-* glibc-headers-* libaio-* libaio-devel-* libgcc-* libstdc++* libstdc++-devel* make-* sysstat-* unixODBC-* unixODBC-devel-* ksh-* mksh pdksh #监测一下软件是否安装全 rpm -q \ binutils \ compat-libstdc++-33 \ elfutils-libelf \ elfutils-libelf-devel \ expat \ gcc \ gcc-c++ \ glibc \ glibc-common \ glibc-devel \ glibc-headers \ libaio \ libaio-devel \ libgcc \ libstdc++ \ libstdc++-devel \ make \ pdksh \ sysstat \ unixODBC \ unixODBC-devel | grep "not installed"
修改PAM配置文件 1 2 vi /etc/pam.d/login session required /lib64/security/pam_limits.so
设置内核参数 1 2 3 4 5 6 7 8 9 10 11 12 13 14 vi /etc/sysctl.conf fs.aio-max-nr = 1048576 fs.file-max = 6815744 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 net.ipv4.ip_local_port_range = 9000 65500 net.core.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048586 重新加载生效 sysctl –p
创建用户grid 1 2 3 4 5 6 7 groupadd -g 1000 oinstall groupadd -g 1001 asmadmin groupadd -g 1002 asmdba groupadd -g 1003 asmoper useradd -u 1100 -g oinstall -G asmadmin,asmdba,asmoper -d /home/grid -s /bin/bash grid #设置密码grid用户密码为oracle echo oracle|passwd --stdin grid
配置grid用户环境变量 RAC1：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 su - grid #vi .bash_profile #export PS1="`/bin/hostname -s`->" export TMP=/tmp export TMPDIR=$TMP export ORACLE_SID=+ASM1 export ORACLE_BASE=/u01/app/grid export ORACLE_HOME=/u01/app/11.2.0/grid export TNS_ADMIN=$ORACLE_HOME/network/admin export PATH=/usr/sbin:$PATH export PATH=$PATH:$ORACLE_HOME/bin export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib export LANG=en_US.UTF-8 export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK #使配置文件生效 source .bash_profile
RAC2：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 su - grid #vi .bash_profile #export PS1="`/bin/hostname -s`->" export TMP=/tmp export TMPDIR=$TMP export ORACLE_SID=+ASM2 export ORACLE_BASE=/u01/app/grid export ORACLE_HOME=/u01/app/11.2.0/grid export TNS_ADMIN=$ORACLE_HOME/network/admin export PATH=/usr/sbin:$PATH export PATH=$PATH:$ORACLE_HOME/bin export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib export LANG=en_US.UTF-8 export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK #使配置文件生效 source .bash_profile
创建用户oracle 1 2 3 4 5 6 su - root groupadd -g 1200 dba groupadd -g 1201 oper useradd -u 1101 -g oinstall -G dba,oper,asmdba -d /home/oracle -s /bin/bash oracle #设置密码grid用户密码为oracle echo oracle|passwd --stdin oracle
配置oracle用户环境变量 RAC1：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 su - oracle #vi .bash_profile #export PS1="`/bin/hostname -s`->" export TMP=/tmp export TMPDIR=$TMP export ORACLE_HOSTNAME=rac1 export ORACLE_SID=orcl1 export ORACLE_UNQNAME=orcl export ORACLE_BASE=/u01/app/oracle export ORACLE_HOME=$ORACLE_BASE/11g export TNS_ADMIN=$ORACLE_HOME/network/admin export PATH=/usr/sbin:$PATH export PATH=$PATH:$ORACLE_HOME/bin export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib export LANG=en_US.UTF-8 export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK #使配置文件生效 source .bash_profile
RAC2：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 su - oracle #vi .bash_profile #export PS1="`/bin/hostname -s`->" export TMP=/tmp export TMPDIR=$TMP export ORACLE_HOSTNAME=rac2 export ORACLE_SID=orcl2 export ORACLE_UNQNAME=orcl export ORACLE_BASE=/u01/app/oracle export ORACLE_HOME=$ORACLE_BASE/11g export TNS_ADMIN=$ORACLE_HOME/network/admin export PATH=/usr/sbin:$PATH export PATH=$PATH:$ORACLE_HOME/bin export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib export LANG=en_US.UTF-8 export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK #使配置文件生效 source .bash_profile
创建目录，修改权限 1 2 3 4 5 6 7 8 9 10 su - root mkdir -p /u01/app/grid mkdir -p /u01/app/11.2.0/grid chown -R grid:oinstall /u01 chown -R grid:oinstall /u01/app/11.2.0/grid mkdir -p /u01/app/oracle chown -R oracle:oinstall /u01/app/oracle chmod -R 775 /u01 export LANG=en_US.UTF-8 export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK
配置profile 1 2 3 4 5 6 7 8 9 10 vim /etc/profile if [ \$USER = "oracle" ] || [ \$USER = "grid" ];then if [ \$SHELL = "/bin/ksh" ];then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi umask 022 fi
修改Oracle用户限制 1 2 3 4 5 6 7 8 9 10 11 12 13 14 vi /etc/security/limits.conf oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536 oracle soft stack 10240 oracle hard stack 32768 grid soft nproc 2047 grid hard nproc 16384 grid soft nofile 1024 grid hard nofile 65536 grid soft stack 10240 grid hard stack 32768
ssh 互信 1 2 3 4 su - grid ssh-keygen (一路回车) ssh-copy-id -i rac2 ssh-copy-id -i rac1
1 2 3 4 su - oracle ssh-keygen (一路回车) ssh-copy-id -i rac2 ssh-copy-id -i rac1
测试一下
1 2 3 4 ssh rac1 date Thu Nov 23 14:00:03 CST 2017 ssh rac2 date Thu Nov 23 14:00:24 CST 2017
使用FreeNas搭建iscsi存储 安装操作系统 使用FreeNAS-9.2.1.7-RELEASE-x64.iso镜像搭建NAS存储，一块系统磁盘，四块存储盘
配置网络 提示是否删除现有的配置，输入nhttp://172.16.20.13
配置iscsi 设置中文语言，上海时区
安装Grid Infrastructure 如果使用Centos6.x操作系统安装rac，建议使用UEK内核;uek内核是Oracle Linux、Oracle Database、中间件和硬件工程师团队合作的结果，是一个快速、现代、可靠的内核，专门针对Oracle软件和硬件进行了优化。随着硬件的快速发展，为了更好的兼容并支持快速发展的硬件，Linux系统版本及内核也在不断的更新，uek正是在这样的环境下应运而生的产物。https://docs.oracle.com/cd/E37670_01/E37355/html/ol_obtain_uek.html https://docs.oracle.com/cd/E52668_01/E39381/html/ol_import_gpg.html http://www.oracle.com/technetwork/server-storage/linux/asmlib/index-101839.html
ASM 分区创建 方法一：使用ASMlib创建ASM分区 1 2 yum -y install oracleasm-support oracle-rdbms-server-11gR2-preinstall rpm -ivh http://download.oracle.com/otn_software/asmlib/oracleasmlib-2.0.12-1.el6.x86_64.rpm
上传linux.x64_11gR2_grid.zip到/usr/local/src目录下
1 2 3 4 cd /usr/local/src/ unzip linux.x64_11gR2_grid.zip /usr/local/src/grid/rpm rpm -ivh cvuqdisk-1.0.7-1.rpm
初始化asm RAC1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [root@rac1 rpm]# /etc/init.d/oracleasm configure -i Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface [grid]: grid Default group to own the driver interface [oinstall]: oinstall Start Oracle ASM library driver on boot (y/n) [y]: y Scan for Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: done Initializing the Oracle ASMLib driver: [ OK ] Scanning the system for Oracle ASMLib disks: [ OK ]
1 /usr/sbin/oracleasm init
RAC2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [root@rac2 src]# /etc/init.d/oracleasm configure -i Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface [grid]: grid Default group to own the driver interface [oinstall]: oinstall Start Oracle ASM library driver on boot (y/n) [y]: y Scan for Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: done Initializing the Oracle ASMLib driver: [ OK ] Scanning the system for Oracle ASMLib disks: [ OK ]
配置iscsi客户端 1 yum -y install iscsi-initiator-utils
1 2 [root@rac1 rpm]# iscsiadm -m discovery -t sendtargets -p 172.16.20.13 172.16.20.13:3260,1 iqn.2017-11.cc.islocal.istgt:iscsi
1 2 3 [root@rac1 rpm]# /etc/init.d/iscsi restart Stopping iscsi: [ OK ] Starting iscsi: [ OK ]
iscsi命令拓展
1 2 3 iscsiadm -m node -p ip_addr iscsiadm -m node session -p ip_addr iscsiadm -m node discovery -P ip_addr
登录target连接
1 iscsiadm -m node -T iqn.2017-11.cc.islocal.istgt:iscsi -p ip_addr -l
登出target连接
1 iscsiadm -m node -T iqn.2017-11.cc.islocal.istgt:iscsi -p ip_addr -u
删除target条目记录
1 iscsiadm -m node -o delete -T iqn.2017-11.cc.islocal.istgt:iscsi -p ip_addr
磁盘分区 fdisk -l 可以发现已经识别到4个磁盘sdb,sdc,sdd,sde,对磁盘进行分区“n/p/1/回车/回车/w”
创建ASM分区 1 2 3 4 /etc/init.d/oracleasm createdisk ASMDATA /dev/sdb1 /etc/init.d/oracleasm createdisk BACKUP /dev/sdc1 /etc/init.d/oracleasm createdisk OCR /dev/sdd1 /etc/init.d/oracleasm createdisk VOTINGDISK /dev/sde1
1 2 3 4 5 6 7 [root@rac1 rpm]# /etc/init.d/oracleasm scandisks Scanning the system for Oracle ASMLib disks: [ OK ] [root@rac1 rpm]# /etc/init.d/oracleasm listdisks ASMDATA BACKUP OCR VOTINGDISK
在rac2上发现asm磁盘 1 yum -y install iscsi-initiator-utils
1 2 [root@rac2 rpm]# iscsiadm -m discovery -t sendtargets -p 172.16.20.13 172.16.20.13:3260,1 iqn.2017-11.cc.islocal.istgt:iscsi
1 2 3 [root@rac2 rpm]# /etc/init.d/iscsi restart Stopping iscsi: [ OK ] Starting iscsi: [ OK ]
1 2 3 4 5 6 7 8 9 [root@rac2 ~]# start_udev Starting udev: [ OK ] [root@rac2 src]# /etc/init.d/oracleasm scandisks Scanning the system for Oracle ASMLib disks: [ OK ] [root@rac2 src]# /etc/init.d/oracleasm listdisks ASMDATA BACKUP OCR VOTINGDISK
方法二：使用udev方式创建ASM分区 配置iscsi客户端 1 yum -y install iscsi-initiator-utils
1 2 [root@rac1 rpm]# iscsiadm -m discovery -t sendtargets -p 172.16.20.13 172.16.20.13:3260,1 iqn.2017-11.cc.islocal.istgt:iscsi
1 2 3 [root@rac1 rpm]# /etc/init.d/iscsi restart Stopping iscsi: [ OK ] Starting iscsi: [ OK ]
将ASM磁盘添加到udev，防止开始发现磁盘顺序不一致导致的异常问题
1 2 3 4 5 6 7 8 [root@rac1 ~]# /sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sdb1 3300000007a38757b [root@rac1 ~]# /sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sdc1 3300000008853f678 [root@rac1 ~]# /sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sdd1 3300000009b03058c [root@rac1 ~]# /sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sde1 3300000006968868f
1 2 3 4 5 [root@rac1 rules.d]# vim 99-iscsi-asm.rules KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="3300000007a38757b", NAME="asm-data", OWNER="grid", GROUP="oinstall", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="3300000008853f678", NAME="asm-backup", OWNER="grid", GROUP="oinstall", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="3300000009b03058c", NAME="asm-ocr", OWNER="grid", GROUP="oinstall", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="3300000006968868f", NAME="asm-vdisk", OWNER="grid", GROUP="oinstall", MODE="0660"
1 2 3 [root@rac1 rules.d]# scp 99-iscsi-asm.rules rac2:/etc/udev/rules.d/99-iscsi-asm.rules [root@rac1 rules.d]# start_udev Starting udev: [ OK ]
1 2 3 4 5 [root@rac1 rules.d]# ll /dev/asm-* brw-rw---- 1 grid asmadmin 8, 33 Nov 23 17:48 /dev/asm-backup brw-rw---- 1 grid asmadmin 8, 17 Nov 23 17:48 /dev/asm-data brw-rw---- 1 grid asmadmin 8, 49 Nov 23 17:48 /dev/asm-ocr brw-rw---- 1 grid asmadmin 8, 65 Nov 23 17:48 /dev/asm-vdisk
在rac2上发现asm磁盘
1 yum -y install iscsi-initiator-utils
1 2 [root@rac2 rpm]# iscsiadm -m discovery -t sendtargets -p 172.16.20.13 172.16.20.13:3260,1 iqn.2017-11.cc.islocal.istgt:iscsi
1 2 3 [root@rac2 rpm]# /etc/init.d/iscsi restart Stopping iscsi: [ OK ] Starting iscsi: [ OK ]
1 2 [root@rac2 ~]# start_udev Starting udev: [ OK ]
1 2 3 4 5 [root@rac2 ~]# ll /dev/asm-* brw-rw---- 1 grid asmadmin 8, 33 Nov 23 17:54 /dev/asm-backup brw-rw---- 1 grid asmadmin 8, 17 Nov 23 17:54 /dev/asm-data brw-rw---- 1 grid asmadmin 8, 49 Nov 23 17:54 /dev/asm-ocr brw-rw---- 1 grid asmadmin 8, 65 Nov 23 17:54 /dev/asm-vdisk
安装Grid 建议直接用grid用户登录图形桌面，而不是root登录su - grid切换过去。
1 2 rac1->cd /usr/local/src/grid rac1->./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -fixup -verbose
安装前监测一下环境，如果有failed，按照要求去修复，我这里有几个软件包太新，不是太影响，我们就开始安装吧。这里遇到几个几个坑，详细说一下吧；1.如果默认没有在候选磁盘(Candidate Disks)里,选择所有磁盘(All disks),看看是否可以显示出来；2.如果还是没有发现，点一下改变其他路径(Change Discovery Path),rhel系列系统默认路径是/dev/raw/*, 我这里是/dev/asm-*;3.如果磁盘显示出来是灰色，不能选择，可能是练习的时候，之前已经使用过磁盘做过asm了，可以执行“dd if=/dev/zero of=/dev/rhdisk2 bs=1024 count=100”，然后重新开始安装试试，如果还不行，请从存储方面开始排查；4.我这里配置的udev方式发现保证磁盘的唯一性，现在发现一个问题，start_udev后可以发现磁盘，/dev/asm-*都生成了；执行oraclescandisk后发现不了ASM磁盘，/dev/oracleasm/disks/目录下是空的，但是安装时候找到了，是可用状态，有点小疑惑。
1 2 3 4 CRS-4124: Oracle High Availability Services startup failed. CRS-4000: Command Start failed, or completed with errors. ohasd failed to start: Inappropriate ioctl for device ohasd failed to start at /opt/app/11.2.0/grid/crs/install/rootcrs.pl line 443.
此错误解决方法
1 2 3 删除之前的配置：/u01/app/11.2.0/grid/crs/install/roothas.pl -deconfig -force -verbose 窗口1：/u01/app/11.2.0/grid/root.sh 窗口2：/bin/dd if=/var/tmp/.oracle/npohasd of=/dev/null bs=1024 count=1 （刚开始执行root.sh时候，还没生成npohash文件，等日志出现“Adding daemon to inittab”的时候，在窗口2中使用root用户执行此命令，等root.sh执行成功后，ctrl+c结束此命令即可）
1 2 3 4 5 6 7 Configure Oracle Grid Infrastructure for a Cluster ... succeeded Updating inventory properties for clusterware Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 7999 MB Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oraInventory
这个错误没事，直接点击Next
验证安装状态 1 2 3 4 5 6 7 8 9 su - grid crs_stat -t crsctl check cluster -all olsnodes -s -n -t crsctl check crs srvctl status asm srvctl status scan srvctl status vip -n rac1 srvctl status scan_listener
创建ASM磁盘组 在grid 用户下，执行 asmca，启动 asm 磁盘组创建向导
安装oracle11gr2 database 软件 建议直接用oracle用户登录图形桌面，而不是root登录su - oracle切换过去。这个安装过程也很简单，基本上和单机版的一样；
1 2 rac1->cd /usr/local/src/database rac1->./runInstaller
(ps:这里的截图稍微有点问题，sid应该是orcl，不是orcl1)
1 2 3 4 5 Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. Finished product-specific root actions.
1 2 sqlplus system/oracle@orcl select inst_id,instance_number,instance_name from gv$instance;
参考资料：http://v.qq.com/vplus/91c845f5c21fd8a6a25e9827b38c171e http://www.dba-oracle.com/t_reset_an_asm_diskgroup.htm http://www.oracle.com/technetwork/cn/articles/hunter-rac11gr2-iscsi-083834-zhs.html http://blog.csdn.net/u010098331/article/details/50767353 https://docs.oracle.com/cd/E11882_01/doc.112/e28441/require.htm#CDMIG110 http://blog.sina.com.cn/s/blog_563c17780102ux53.html https://blogs.oracle.com/database4cn/11gr2-crsgrid-scansingle-client-access-name
