Site Search
Latest Topics
SAN/Storage DMX
[DMX]In the event of FA Director Failure what is the procedure for the replacement?
In the event of FA Director Failure, follow below steps:
1. To validate the status of FA director, run symevent to get the event log:
c:\>symevent -sid 1234 list
Detection time Dir Src Category Severity Error Num
------------------------ ------ ---- ------------ ------------ ----------
Sat May 31 17:39:52 2008 FA-8A Symm Director Fatal 0x0040
A Symmetrix Director is not responding
Sat May 31 17:39:53 2008 FA-8B Symm Director Fatal 0x0040
A Symmetrix Director is not responding
Sat May 31 17:39:55 2008 FA-8C Symm Director Fatal 0x0040
A Symmetrix Director is not responding
Sat May 31 17:39:56 2008 FA-8D Symm Director Fatal 0x0040
A Symmetrix Director is not responding
2. Next run symmaskdb to backup vcm database and extract to text file, this step is to identify servers that has been affected by the FA port/director failure. This information will be used to create change if necessory in your environment.
C:\>symmaskdb -sid 1234 backup -f c:\June01_2008_vcm_bkup.bin
C:\>symmaskdb -f c:\c:\June01_2008_vcm_bkup.bin list database >c:\c:\June01_2008_vcm_bkup.txt
Here is part of the output:
Director Identification : FA-8A
Director Port : 0
User-generated
Identifier Type Node Name Port Name Devices
---------------- ----- --------------------------------- ---------
2100001b320f2712 Fibre DBP01 hba0 021F
027F
210000e08b86d195 Fibre SQLDB01 hba0 051A
0522
…,...
06A7:06A9
210000e08b8936e3 Fibre SQLDB02 hba0 051A
0522
,…,...
06A7:06A9
210000e08b89e2e5 Fibre SQLDB03 hba0 051A
0522
…,...
069F
Director Identification : FA-8C
Director Port : 0
User-generated
Identifier Type Node Name Port Name Devices
---------------- ----- --------------------------------- ---------
10000000c94647e3 Fibre RPMDBPRD1 hba0 0297
029B
…,…
04F9
Director Identification : FA-8C
Director Port : 1
User-generated
Identifier Type Node Name Port Name Devices
---------------- ----- --------------------------------- ---------
10000000c9407518 Fibre SERV06 hba0 03DF
04BB:04E4
04FB:0509
10000000c9449fa7 Fibre 10000000c9449fa7 10000000c9449fa7 03D3
03D7
03DB
0443
0445
0447
07E3:07E8
Next to validate the paths for every server:
C:\>symmaskdb -sid 1234 list capacity -host DBP01
Symmetrix ID : 000000001234
Host Name : DBP01
Identifiers Found : 2100001b320f2712
2100001b320f4212
Device Cap(MB) Attr Dir:P
------ ------- ---- ----
021F 400890 (M) 7A:0, 8A:0, 9A:0,10A:0
027F 100222 (M) 7A:0, 8A:0, 9A:0,10A:0
-----------------------------
MB Total: 501112
GB Total: 489.4
C:\>symmaskdb -sid 1234 list capacity -host MLISCDPSQLDB01
Symmetrix ID : 000000001234
Host Name : SQLDB01
Identifiers Found : 210000e08b86d195
210000e08b86a796
Device Cap(MB) Attr Dir:P
------ ------- ---- ----
051A 69720 (M) 7A:0, 8A:0, 9A:0,10A:0
…,...
06A9 8715 7A:0, 8A:0, 9A:0,10A:0
-----------------------------
MB Total:2117745
GB Total: 2068.1
C:\>symmaskdb -sid 1234 list capacity -host MLISCDPSQLDB02
Symmetrix ID : 000000001234
Host Name : SQLDB02
Identifiers Found : 210000e08b8936e3
210000e08b893be4
Device Cap(MB) Attr Dir:P
------ ------- ---- ----
051A 69720 (M) 7A:0, 8A:0, 9A:0,10A:0
…,..,
06A9 8715 7A:0, 8A:0, 9A:0,10A:0
-----------------------------
MB Total:2117745
GB Total: 2068.1
C:\>symmaskdb -sid 1234 list capacity -host MLISCDPSQLDB03
Symmetrix ID : 000000001234
Host Name : SQLDB03
Identifiers Found : 210000e08b89e2e5
210000e08b89a7e1
Device Cap(MB) Attr Dir:P
------ ------- ---- ----
051A 69720 (M) 7A:0, 8A:0, 9A:0,10A:0
…,…,
06A9 8715 7A:0, 8A:0, 9A:0,10A:0
-----------------------------
MB Total:2117745
GB Total: 2068.1
C:\>symmaskdb -sid 1234 list capacity -host RPMDBPRD1
Symmetrix ID : 000000001234
Host Name : RPMDBPRD1
Identifiers Found : 10000000c94647e3
10000000c9464671
Device Cap(MB) Attr Dir:P
------ ------- ---- ----
0297 34860 (M) 8C:0, 9C:0
…,...
04F9 17430 (M) 8C:0, 9C:0
-----------------------------
MB Total:1429260
GB Total: 1395.8
C:\>symmaskdb -sid 1234 list capacity -host SERV06
Symmetrix ID : 000000001234
Host Name : SERV06
Identifiers Found : 10000000c9407518
10000000c94074be
Device Cap(MB) Attr Dir:P
------ ------- ---- ----
03DF 17430 (M) 7C:0, 8C:1, 9C:1,10C:0
…,…
0509 8715 7C:0, 8C:1, 9C:1,10C:0
-----------------------------
MB Total: 514185
GB Total: 502.1
3. Once the ECR being approved, EMC Engineer should be contacted for dispatch.
4. Before the Director replacement, contact Netops to place following servers in maintenance mode
5. Once the replace is done, contact Host team (Unix or intel) to validate if the paths' being restored automatically.. If not possible reboot is needed.
Below are the output of all three servers which provided by UNIX team. Here we can see all paths are restored.
RPMDBPRD1
Symmetrix logical device count=78
CLARiiON logical device count=0
Hitachi logical device count=0
Invista logical device count=0
HP xp logical device count=0
Ess logical device count=0
HP HSx logical device count=0
==============================================================================
----- Host Bus Adapters --------- ------ I/O Paths ----- ------ Stats ------
### HW Path Summary Total Dead IO/Sec Q-IOs Errors
==============================================================================
0 fscsi0 optimal 78 0 - 1 0
1 fscsi1 optimal 78 0 - 1 0
===============================
SERV06
Symmetrix logical device count=58
CLARiiON logical device count=0
Hitachi logical device count=0
Invista logical device count=0
HP xp logical device count=0
Ess logical device count=0
HP HSx logical device count=0
==============================================================================
----- Host Bus Adapters --------- ------ I/O Paths ----- ------ Stats ------
### HW Path Summary Total Dead IO/Sec Q-IOs Errors
==============================================================================
0 fscsi0 optimal 116 0 - 0 0
1 fscsi1 optimal 116 0 - 0 0
===============================
SERV05
Symmetrix logical device count=12
CLARiiON logical device count=66
Hitachi logical device count=0
Invista logical device count=0
HP xp logical device count=0
Ess logical device count=0
HP HSx logical device count=0
==============================================================================
----- Host Bus Adapters --------- ------ I/O Paths ----- ------ Stats ------
### HW Path Summary Total Dead IO/Sec Q-IOs Errors
==============================================================================
0 fscsi0 optimal 156 0 - 0 0
1 fscsi1 optimal 156 0 - 0 0
===============================