//
you're reading...
Linux, Oracle ASM

Binding ASM Disk with Linux Multipath and Udev on SLES11 SP2

To use the Multipath and Udev, we have to understand what are they, why we want to use it.

What is ‘Device Mapper Multipath’ (multipath I/O)?

Device Mapper Multipath(DM-Multipath)can configure muliple I/O links between the server nodes to SAN as a single device. These I/O links might consist with various cables, switches and SCSI controllers  (physical links between SAN and Nodes). On the OS level, multipath will be able to integrate these physical links to a single new device. Multipath can be configured on both Windows and Linux.

What is ‘udev’?

udev is a utility software to manage all the physical devices in Linux OS, ‘udev’ can update the status of the hardware device of the computer, for example user insert a USB drive. The ‘udev’ file is under folder /dev. After the ‘udev’ started, It will only contain the actived hardware uder the /dev/ folder.

Multipath Technology with Oracle RAC system

When we are building an Oracle RAC Database System, the storage configuration is a very important component. Depend on the budget you have, you might have different SAN for your Oracle RAC database storage. The most common vendor like NETAPP, EMC, Dell, HP, IBM, NEC, HITACHI they all have their own mulitpath technology. For example

  • EMC PowerPath
  • NEC PathManager
  • Hitachi HiCommand Dynamic Link Manager

Also, DBAs or SAN administrator need to check the compatibility of the ASM with the SAN. In this case, we are using NETAPP SAN with Oracle ASM Disks.

list all the storage adapter card (In this case, I am using QLogic Fibre Channel adapater)

TESTDBnode01:~ # lspci | grep -i fibre
05:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
05:00.1 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02
TESTDBnode02:~ # lspci | grep -i fibre
05:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
05:00.1 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)

‘lispci’ will list the how many physical ports for your Fibre Channel Adapter on each server nodes.

Configure the file ‘/etc/multipath.conf’

defaults {
           udev_dir                /dev
           polling_interval        10
           path_selector           "round-robin 0"
           path_grouping_policy    failover
           getuid_callout          "/lib/udev/scsi_id -g -u -d /dev/%n"
           prio                    const
           prio_args               ""
           path_checker            directio
           rr_min_io               1000
           rr_weight               uniform
           failback                immediate
           #       no_path_retry           fail
           user_friendly_names     yes
           max_fds                 max
           queue_without_daemon    no
           flush_on_last_del       yes
}

blacklist {
          devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
          devnode "^hd[a-z]"
          #       devnode "sda"
          devnode "^dcssblk[0-9]*"
}

devices {
             device {
             vendor                  "NETAPP"
             product                 "LUN"
             getuid_callout          "/lib/udev/scsi_id -g -u -d /dev/%n"
             prio                   "alua"
             features                "3 queue_if_no_path pg_init_retries 50"
             hardware_handler        "1 alua"
             path_grouping_policy    group_by_prio
             failback                immediate
             rr_weight               uniform
             rr_min_io               128
             rr_min_io               1000
             rr_weight               uniform
             failback                immediate
             #       no_path_retry           fail
             user_friendly_names     yes
             max_fds                 max
             queue_without_daemon    no
             flush_on_last_del       yes
}

blacklist {
          devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
          devnode "^hd[a-z]"
           #       devnode "sda"
          devnode "^dcssblk[0-9]*"
}

devices {
             device {
             vendor                  "NETAPP"
             product                 "LUN"
             getuid_callout          "/lib/udev/scsi_id -g -u -d /dev/%n"
             prio                   "alua"
             features                "3 queue_if_no_path pg_init_retries 50"
             hardware_handler        "1 alua"
             path_grouping_policy    group_by_prio
             failback                immediate
             rr_weight               uniform
             rr_min_io               128
             path_checker            tur
             }
}

multipaths {
            multipath {
                       wwid    3600a06504176347c472b443145474a43
                       alias   testdb_data01
            }
            multipath {
                       wwid    3600a06504176347c472b443145474a44
                       alias   testdb_fra01
            }
            multipath {
                       wwid    3600a06504176347c472b443145474a49
                       alias   ocr_vote01
            }
            multipath {
                       wwid    3600a06504176347c472b443145474a4a
                       alias   ocr_vote02
            }
            multipath {
                       wwid    3600a06504176347c472b443145474a4b
                       alias   ocr_vote03
            }
}

Configure the ‘udev’

Edit the file ‘/etc/udev/rule.d/12-dm-permissions.rules’ as:

# DM_LV_NAME - logical volume name (not set if LVM device not present)
# DM_VG_NAME - volume group name (not set if LVM device not present)
# DM_LV_LAYER - logical volume layer (not set if LVM device not present)

# "add" event is processed on coldplug only!
ACTION!="add|change", GOTO="dm_end"
ENV{DM_UDEV_RULES_VSN}!="?*", GOTO="dm_end"

#ocr_vote001
ENV{DM_UUID}=="mpath-3600a06504176347c472b443145474a49", OWNER:="grid", GROUP:="asmadmin", MODE:="660"

#ocr_vote002
ENV{DM_UUID}=="mpath-3600a06504176347c472b443145474a4a", OWNER:="grid", GROUP:="asmadmin", MODE:="660"

#ocr_vote003
ENV{DM_UUID}=="mpath-3600a06504176347c472b443145474a4b", OWNER:="grid", GROUP:="asmadmin", MODE:="660"

#testdb_data01
ENV{DM_UUID}=="mpath-3600a06504176347c472b443145474a43", OWNER:="grid", GROUP:="asmadmin", MODE:="660"

#testdb_fra01
ENV{DM_UUID}=="mpath-3600a06504176347c472b443145474a44", OWNER:="grid", GROUP:="asmadmin", MODE:="660"

Edit the file ‘/etc/udev/rule.d/99-oracle-asmdevices.rules’ as:

KERNEL=="*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="mpath-3600a06504176347c472b443145474a49", NAME+="/dev/mapper/ocr_vote01", OWNER="grid", GROUP="asmadmin", MODE="0660" , OPTIONS="last_rule"
KERNEL=="*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="mpath-3600a06504176347c472b443145474a4a", NAME+="/dev/mapper/ocr_vote02", OWNER="grid", GROUP="asmadmin", MODE="0660" , OPTIONS="last_rule"
KERNEL=="*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="mpath-3600a06504176347c472b443145474a4b", NAME+="/dev/mapper/ocr_vote03", OWNER="grid", GROUP="asmadmin", MODE="0660" , OPTIONS="last_rule"
KERNEL=="*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="mpath-3600a06504176347c472b443145474a43", NAME+="/dev/mapper/testdb_data01", OWNER="grid", GROUP="asmadmin", MODE="0660" , OPTIONS="last_rule"
KERNEL=="*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="mpath-3600a06504176347c472b443145474a44", NAME+="/dev/mapper/testdb_fra01", OWNER="grid", GROUP="asmadmin", MODE="0660" , OPTIONS="last_rule"

Repeat the setup above on every single nodes for your Oracle RAC

Start the ‘multipath’

TESTDBnode01:~ # multipath -F
TESTDBnode01:~ # multipath -v2

TESTDBnode02:~ # multipath -F
TESTDBnode02:~ # multipath -v2

Udev rules are extremely flexible. We can identify devices by using:

TESTDBnode01:~ # udevadm info --query=all --path=/dev/mapper/testdb_data01
Advertisements

About daviewning

I am an Oracle DBA

Discussion

No comments yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: