Change package IP address off line
Halt the package
# cmhaltpkg -v <package>
Update control files
/etc/cmcluster/package/package.conf
Propagate the changes manually or use cmapplyconf
(legacy or modular)
Start package
# cmrunpck <package>
Check that the IP was added.
# cmviewcl -v
Change package IP address on line
You can change the IP address while a package is up but there's some risky... but can be done.
The recommended is do it off line to avoid any problems.
Use the command cmmodnet to remove the IP addres and after to add.
Using netstat -in check that the ip address is correct.
Check the cluster status:
# cmviewcl -v
Update the package script, propagate the change to other nodes.
For more infos about cmmodnet check man pages.
Serviceguard - how to change package IP on line/ off line
Labels: MC ServiceGuard, Network
Serviceguard - Change node ip address
**In this situation we are changing just the ip, the subnet stil is the same**
Backup configuration files:
/etc/hosts
/etc/rc.config.d/netconf
/etc/cmcluster/
Move the package(s) to another node
# cmhaltpkg -v <package>
# cmrunpkg -v -n NODE <package>
Remove node from all packages
Run cmapplyconf (or send the configuration files for others nodes manualy)
Stop the node
# cmhaltnode -v <node>
Remove node from cluster configuration
Apply new cluster configuration
Change IP on node
/etc/rc.config.d/netconf
/etc/hosts
Restart the net deamon for apply the new network configuration
# /sbin/init.d/net stop
# /sbin/inti.d/net start
Add node to cluster
The bellow link is another post showing how to include a new node in the cluster running.
http://hp-ux-br.blogspot.com.br/2012/05/adding-new-node-in-running-cluster.html
Apply cluster configuration
Start the node again
# cmrunnode -v <node>
Add node to all relevant packages
Apply package configuration
Move package(s) back to node
# cmrunpkg -v -n NODE <package>
# cmhaltpkg -v <package>
Check cluster status
# cmviewcl -v
Labels: MC ServiceGuard, Network
Adding a new node in a running cluster
1- Add the new node in the file /etc/cmcluster/cmclnodelist.
2- Get the most up-to-date ASCII configuration file.
# cmgetconf -v -c clustername /etc/cmcluster/cluster.ascii
3- Query all nodes, including the new node, in the cluster.
# cmquerycl -v -c cluster_name -n node1 -n node2 cluster.ascii
4- Compare the ASCII files obtained from cmgetconf and cmquerycl.
5- Update the ASCII configuration file obtained from cmquerycl.
6- Check the new ASCII configuration file.
# cmcheckconf -v -C cluster.ascii
7- Compile and distribute the new binary cluster configuration file.
# cmapplyconf -v -C cluster.ascii
8- Start cluster services on the new node.
# cmrunnode -v newNode
9- Check the cluster status and the log file for validate.
# cmviewcl
tail -100 /var/adm/syslog/syslog.log
Labels: MC ServiceGuard
How to upgrade Serviceguard
1. First of all create a backup of /etc/cmcluster dir:
#cp -pR /etc/cmcluster /etc/cmcluster.bck
2. Also do a cmgetconf of the current configuration.
3. Halt ALL PKGs running on the first node and the node:
#cmhaltpkg pkg1
#cmhaltpkg pkg2
#cmhaltnode node
4. Get the service guard product name:
swlist |grep -i service
5. Uninstall Serviceguard:
#swremove -x enforce_dependencies=false
Select the Serviceguard product in the list.
6. Install the new Serviceguard version using swinstall.
7. After installation check the new version with swlist |grep -i service
8. start the node
#cmrunnode node
9. Now for validate...
Check cluster status:
#cmviewcl -v
Check cluster logs on /var/adm/syslog/syslog.log
Now with the node validate do the same for others node.
Finally start all PKGs.
Labels: MC ServiceGuard
MC Serviceguard - Replace Quorum Server
The first step is stop completely the cluster.. you can't do that without stop your cluster.
for each package:
# cmhaltpkg <package_name>
Stop the cluster:
# cmhaltcl
Now change the Quorun server:
# vi /etc/cmcluster/cmclconfig.ascii
...
QS_HOST quorum-server
Apply changes:
# cmapplyconf -v -C cmclconfig.ascii
Check the cluster configuration:
# cmcheckconf -v -C cmclconfig.ascii
Star the cluster again:
# cmruncl -v
Now check if the new quorum server is in use.
# cmviewcl -v
Labels: MC ServiceGuard
Making a new Serviceguard cluster
1- First of all install the ServiceGuard using swinstall
Or check with swlist if the Serviceguard is present on that machine:
# swlist Serviceguard
2- Add the node hostnames in /etc/cmcluster/cmclnodelist
3- Steps to check before to create the cluster:
Is this a two nodes cluster?
- If yes you need a quorum disk/host for your cluster.
Check the follow itens in all nodes:
- If the kernel parameter are consistents
- Comunication between the nodes must be ok.
- If all users and groups are using the same uids/gids.
- Comment out all VGs controlled by ServiceGuard in the /etc/fstab
- Copy all volume groups that will be countrolled by ServiceGuard to all nodes, use the follow procedure:
The Re-Import VG Procedure
4- Enable the NTP
5- Check active NICs have ipaddress associated with them (netconf) and standby NICs don't.
6- Edit /etc/lvmrc, the VGs controlled by ServiceGuard canno't be auto activated (this shouldn't include root VG).
7- Add all IPs used in the /etc/hosts
- Don't forget to include a hearbeat in a different network.
- If you have the heartbeat in your dns server include it in /etc/hosts too.
8- Create a basic cluster structure with one package.
Create cluster config file using cmquerycl
cd /etc/cmcluster/
# cmquerycl –v –C cmclconfig.ascii –n <node1> -n <node2>
Edit the ascii file, follow some points to change (example):
# vi cmclconfig.ascii
CLUSTER_NAME <myCluster1>
NODE_NAME <node1>
NETWORK_INTERFACE <lan0>
HEARTBEAT_IP <192.168.100.101>
NETWORK_INTERFACE <lan1>
HEARTBEAT_IP <164.39.8.92>
NETWORK_INTERFACE <lan2>
NODE_NAME <node2>
NETWORK_INTERFACE <lan0>
HEARTBEAT_IP <192.168.100.102>
NETWORK_INTERFACE <lan1>
HEARTBEAT_IP <164.39.8.93>
NETWORK_INTERFACE <lan2>
HEARTBEAT_INTERVAL 2000000
NODE_TIMEOUT 5000000
MAX_CONFIGURED_PACKAGES 10
Compile and distribute using cmapplyconf
# cmapplyconf –C /etc/cmcluster/cmclconfig.ascii
9- Start the cluster with cmruncl command
# cmruncl
10- Now you have a cluster but you don't have any package, follow few steps to create a basic package.
Create the package config file using cmmakepkg
# cd /etc/cmcluster
# mkdir pkg1
# cd pkg1
# cmmakepkg –p pkg1.conf
11- Edit as required for your app
12- Create package control script using cmmakepkg
- The control script is placed in the package directory and is given the same name as specified in the RUN_SCRIPT and HALT_SCRIPT parameters in the package configuration file.
- The control script must reside in a directory with the string cmcluster in the path.
# cmmakepkg –s pkg1.cntl
Customise the script as required
13- Validate the cluster and your packages using the follow procedure:
How to validate the cluster configurations
14- Copy the control script to other nodes, keep the same path and names.
15- Relocatable ipaddresses must be defined in DNS or /etc/hosts
16- Active the cluster autostart editing the follow configuration file as showed bellow
# vi /etc/rc.config.d/cmcluster
AUTOSTART_CMCLD=1
** Version 11.18+: you can create the package as a modular package or as a legacy package, for a modular package isn't necessary create the control files, just the configuration. This's not the only different you can see in released notes from Serviceguard 11.18 the all information about that. **
Labels: MC ServiceGuard
[MCSG] Where's the control file of my package?
During package start the corresponding package control script is called with the argument
start. The script is typicallly called /etc/cmcluster/package/package.cntl. The
command cmviewconf can be used to have a look at the script configuration:
# cmviewconf | egrep 'package (name|run|halt)'
package name: pkg1
package run script: /etc/cmcluster/pkg1/pkg1.cntl
package run timeout: (No Timeout)
package halt script: /etc/cmcluster/pkg1/pkg1.cntl
package halt timeout: (No Timeout)
Labels: MC ServiceGuard
The Re-Import VG Procedure
The following steps are needed to perform the re-import of an LVM cluster VG. We assume
that a cluster volume group /dev/vg01 with VG group special file minor number 0x010000
needs to be re-imported.
• Steps for the node that has the VG active:
1. Create a mapfile:
node1# vgexport –v –p –s –m /tmp/vg01.map vg01
Warning messages indicating that the VG is active should be ignored.
2. Copy mapfile to each of the other cluster nodes (using for example rcp, ftp, etc.)
node1# rcp /tmp/vg01.map node2:/tmp/vg01.map
...
• Steps to be performed on each of the other nodes:
3. Note VG minor number and permissions/ownership:
node2# ll /dev/vg01
total 12
drwxr-xr-x 2 root root 1024 Apr 16 12:04 ./
dr-xr-xr-x 19 bin bin 5120 Jun 20 06:44 ../
crw-r----- 1 root sys 64 0x010000 Apr 4 13:32 group
brw-r----- 1 root sys 64 0x010001 Apr 4 13:32 lvol1
crw-r----- 1 root sys 64 0x010001 Apr 4 13:32 rlvol1
4. Export the VG:
node2# vgexport vg01
5. Re-create VG directory:
node2# mkdir /dev/vg01
6. Re-create VG group special file, use minor number noted above.
node2# mknod /dev/vg01/group c 64 0x010000
7. Run vgimport using the copied mapfile:
node2# vgimport –v –s –m /tmp/vg01.map vg01
Messages indicating that no backup for this VG may exist should be ignored.
8. Change permissions and ownership of the VG directory and its device special files
according to the information noted above.
9. Test the activation of the VG in read-only mode and perform vgcfgbackup:
node2# vgchange –a r vg01
node2# vgcfgbackup vg01
node2# vgchange –a n vg01
Labels: MC ServiceGuard
MC ServiceGuard - Debug logging
Use the cmsetlog to change the log severity, changing it to the most verbose log level (in all log categories):
# cmsetlog 5
You can restrict for log categories:
# cmsetlog –M NET –M REM 6
In this case network and remote comunication will be restrict, check the full list of log categories on man pages.
Use the cmsetlog if you want change the log location, this can help you to analyze and don't forget with high log lever greater 3 your syslog will increase the size.
# cmsetlog -f /tmp/MCSG.log
The debug logging is automatically stopped and reset to default once the cluster halted. Toreset the debug logging to default modules, categories and loglevel on a running cluster,simply use the command:
# cmsetlog -r
If the ‘-f <file>’ option has been used with cmsetlog to redirect logging to another file, youshould re-direct it back to syslog.log with the command:
# cmsetlog -s
Debug logging for Advanced Tape Services (ATS) with stsetlog
stsetlog is an undocumented command that enables debug logging for the ATS feature of SG. The usage is:
# stsetlog <level>
Level can be in the range of 0 to 6, where 6 is the most verbose level that also logs themessages sent by ATS. To disable debug logging use “
# stsetlog 0
The default logfile for ATS debug logging is
/var/adm/cmcluster/sharedtape/cmtaped.log
This procedure you can use just with the cluster on line therefore you can't use if you have problems to start the cluster.
Labels: MC ServiceGuard
MC ServiceGuard - Reasons for TOC
Transfer of Control (TOC)
MCSG will TOC a system in order to release system resources and to ensure data integrity. There are several scenarios in which MCSG will TOC a system. They are:
Checking that:
# cmviewconf
Cluster information:cluster name: testcluster
version: 0
flags: 12 (single cluster lock)
heartbeat interval: 1.00 (seconds)
node timeout: 8.00 (seconds)
heartbeat connection timeout: 16.00 (seconds)
auto start timeout: 600.00 (seconds)
network polling interval: 2.00 (seconds)
first lock vg name: /dev/vglock
second lock vg name: (not configured)
Cluster Node information:Node ID 1:
Node name: node1
first lock pv name: /dev/dsk/c0t4d4
first lock disk interface type: c720
Network ID 1:
mac addr: 0x080009fd4375
hardware path: 8/16/6
network interface name: lan0
subnet: 15.140.8.0
subnet mask: 255.255.248.0
ip address: 15.140.10.236
flags: 1 (Heartbeat Network)
bridged net ID: 1
# lanscan
Hardware Station Crd Hdw Net-Interface NM MAC HP-DLPI DLPI
Path Address In# State NamePPA ID Type Support Mjr#
8/16/6 0x080009FD4375 0 UP lan0 snap0 1 ETHER Yes 119
8/8/2/0 0x00108318AFEE 2 UP lan2 snap2 2 ETHER Yes 119
8/8/1/0 0x00108318AFED 1 UP lan1 snap1 3 ETHER Yes 119
# cmscancl -n node -o /tmp/scan.log
Check the "link-level connectivity" in the log.
msg in the log:
Serviceguard: Unable to maintain contact with cmcld daemon.Performing TOC to ensure data integrity.
You can confirm TOC searching in the /etc/shutdownlog for something like that..
18:23 Thu Apr 24 2003. Reboot after panic: SafetyTimer expired, ...
Labels: MC ServiceGuard
How to validate Serviceguard configuration?
Use the follow command pointing to configuration file
cmcheckconf –P <configuration file.conf>
Labels: MC ServiceGuard
Check Serviceguard version
through swlist
swlist -l product ServiceGuard
through "what"
what /usr/lbin/cmcld
Labels: MC ServiceGuard
