AIX 서버 장애 로그 내용 분석 부탁드립니다.

구십평   
   조회 7835   추천 0    

 주말에 aix 서버에서 장애가 발생하여 로그를 긁어왔는데, 분석좀 부탁드리겠습니다.

(현재는 정상 운영중입니다.)

db01:/>errpt
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
DCB47997 0524125420 T H hdisk3 DISK OPERATION ERROR
3074FEB7 0524125320 T H fscsi1 ADAPTER ERROR
825849BF 0524125320 T H fcs1 ADAPTER ERROR
C43F90ED 0524075520 P H hdisk15 SUBSYSTEM COMPONENT FAILURE
C43F90ED 0524075520 P H hdisk17 SUBSYSTEM COMPONENT FAILURE
db01:/>errpt -a
---------------------------------------------------------------------------
LABEL: SC_DISK_ERR4
IDENTIFIER: DCB47997

Date/Time: Sun May 24 12:54:05 2020
Sequence Number: 1038
Machine Id: 00C8CA254C00
Node Id: db01
Class: H
Type: TEMP
Resource Name: hdisk3
Resource Class: disk
Resource Type: mpioapdisk
Location: U789C.001.DQD5N52-P1-C3-T1-W20130080E517E362-L1000000000000
VPD:
Manufacturer................IBM
Machine Type and Model......1814 FAStT
ROS Level and ID............31303630
Serial Number...............
Device Specific.(Z0)........3*50*4***052*032
Device Specific.(Z1)........

Description
DISK OPERATION ERROR

Probable Causes
MEDIA
DASD DEVICE

User Causes
MEDIA DEFECTIVE

Recommended Actions
FOR REMOVABLE MEDIA, CHANGE MEDIA AND RETRY
PERFORM PROBLEM DETERMINATION PROCEDURES

Failure Causes
MEDIA
DISK DRIVE

Recommended Actions
FOR REMOVABLE MEDIA, CHANGE MEDIA AND RETRY
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
PATH ID
1
SENSE DATA
0A00 2A00 0C71 08D8 0000 0804 0000 0000 0000 0000 0000 0000 0102 0000 7000 0600
0000 009E 0000 0000 2904 0000 0000 0000 0100 0000 0000 0000 0000 0000 0000 0000
0008 3600 002A 000C 7108 D800 0008 0400 0000 0000 0000 0000 5351 3934 3630 3035
3838 2020 2020 2020 0760 1300 0001 FF00 0900 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 00FF 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 075F 3035 3233 3230 2F32 3030 3731 3900 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 1049
32D1 0082 0080
---------------------------------------------------------------------------
LABEL: FSCSI_ERR4
IDENTIFIER: 3074FEB7

Date/Time: Sun May 24 12:53:21 2020
Sequence Number: 1037
Machine Id: 00C8CA254C00
Node Id: db01
Class: H
Type: TEMP
Resource Name: fscsi1
Resource Class: driver
Resource Type: efscsi
Location: U789C.001.DQD5N52-P1-C3-T1

Description
ADAPTER ERROR

Probable Causes
ADAPTER HARDWARE OR CABLE
ADAPTER MICROCODE
FIBRE CHANNEL SWITCH OR FC-AL HUB

Failure Causes
ADAPTER
CABLES AND CONNECTIONS
DEVICE

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
CHECK CABLES AND THEIR CONNECTIONS
VERIFY DEVICE CONFIGURATION

Detail Data
SENSE DATA
0000 0000 0000 002C 0000 0000 0200 0000 0000 0000 0000 0000 0000 0000 0000 2000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0700 0000 0000
0001 0400 2400 0000 2420 0700 0000 0000 0001 0001 0000 0000 0000 0000 2038 0200
069C 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 2000 0000 0030 0700 0000 0000 1801 0000 0040 0700 0000 0000 0000 0000
0000 2038 0000 0000 0000 0000 0000 0000 0000 0000 F100 0110 1448 9000 F100 0110
1448 8000 F100 0110 1448 3024 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0FFA 2000
---------------------------------------------------------------------------
LABEL: FCS_ERR2
IDENTIFIER: 825849BF

Date/Time: Sun May 24 12:53:20 2020
Sequence Number: 1036
Machine Id: 00C8CA254C00
Node Id: db01
Class: H
Type: TEMP
Resource Name: fcs1
Resource Class: adapter
Resource Type: df1000fe
Location: U789C.001.DQD5N52-P1-C3-T1
VPD:
Part Number.................10N7249
Serial Number...............1C9490875D
Manufacturer................001C
EC Level....................D76626
Customer Card ID Number.....5773
FRU Number.................. 10N7249
Device Specific.(ZM)........3
Network Address.............10000000C995E94F
ROS Level and ID............02E8277F
Device Specific.(Z0)........2057706D
Device Specific.(Z1)........00000000
Device Specific.(Z2)........00000000
Device Specific.(Z3)........03000909
Device Specific.(Z4)........FFE01212
Device Specific.(Z5)........02E8277F
Device Specific.(Z6)........06E32715
Device Specific.(Z7)........07E3277F
Device Specific.(Z8)........20000000C995E94F
Device Specific.(Z9)........ZS2.71X15
Device Specific.(ZA)........Z1D2.70A5
Device Specific.(ZB)........Z2D2.71X15
Device Specific.(ZC)........00000000

Description
ADAPTER ERROR

Probable Causes
ADAPTER HARDWARE
SYSTEM I/O BUS

Failure Causes
ADAPTER

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
SENSE DATA
0000 0000 0000 0027 0502 001A 0000 0000 0001 0700 0947 5F76 0000 23CD 0000 012C
0000 0001 0000 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0000 0000
0000 0000 0610 0000 0000 0010 0000 0000 0000 0000 0000 2710 0000 07D0 0000 076C
0000 0064 0000 000F 5000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0002 0902 8A00 0200 0019 0000 0000 0000 0000 0000 0403 4EF0 0000 0001 0000 0000
0000 0000 0000 0000 0000 0001 636D 4643 F100 0110 1448 B0E8 F100 0110 1448 B15C
F100 0110 1448 306C 0000 0000 0007 50E8 0000 0000 0007 515C 0000 0000 0007 206C
0000 0000
---------------------------------------------------------------------------
LABEL: SC_DISK_PCM_ERR1
IDENTIFIER: C43F90ED

Date/Time: Sun May 24 07:55:11 2020
Sequence Number: 1035
Machine Id: 00C8CA254C00
Node Id: db01
Class: H
Type: PERM
Resource Name: hdisk15
Resource Class: disk
Resource Type: mpioapdisk
Location: U789C.001.DQD5N52-P1-C3-T1-W20130080E517E362-LD000000000000
VPD:
Manufacturer................IBM
Machine Type and Model......1814 FAStT
ROS Level and ID............31303630
Serial Number...............
Device Specific.(Z0)........5*30**4*05**2032
Device Specific.(Z1)........

Description
SUBSYSTEM COMPONENT FAILURE

Probable Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT

Failure Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
PATH ID
1
SENSE DATA
0600 0308 0000 FF04 0000 0000 0000 0000 0000 0000 0000 0000 0102 0000 7000 0B00
0000 009E 0000 0000 4400 0000 0000 0000 0100 0000 0000 0000 0000 0000 0000 0000
0002 0100 0003 0800 00FF 0400 0000 0000 0000 0000 0000 0000 5351 3934 3630 3035
3838 2020 2020 2020 0760 1300 000D 0000 0B00 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 72D3 95D2 3035 3233 3230 2F31 3530 3835 3500 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 102D
E0F7 0085 0080
---------------------------------------------------------------------------
LABEL: SC_DISK_PCM_ERR1
IDENTIFIER: C43F90ED

Date/Time: Sun May 24 07:55:11 2020
Sequence Number: 1034
Machine Id: 00C8CA254C00
Node Id: db01
Class: H
Type: PERM
Resource Name: hdisk17
Resource Class: disk
Resource Type: mpioapdisk
Location: U789C.001.DQD5N52-P1-C3-T1-W20130080E517E362-LF000000000000
VPD:
Manufacturer................IBM
Machine Type and Model......1814 FAStT
ROS Level and ID............31303630
Serial Number...............
Device Specific.(Z0)........2**34***0505*032
Device Specific.(Z1)........

Description
SUBSYSTEM COMPONENT FAILURE

Probable Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT

Failure Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
PATH ID
1
SENSE DATA
0600 0308 0000 FF04 0000 0000 0000 0000 0000 0000 0000 0000 0102 0000 7000 0B00
0000 009E 0000 0000 4400 0000 0000 0000 0100 0000 0000 0000 0000 0000 0000 0000
0002 0100 0003 0800 00FF 0400 0000 0000 0000 0000 0000 0000 5351 3934 3630 3035
3838 2020 2020 2020 0760 1300 000F 0000 0B00 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 72D3 95D1 3035 3233 3230 2F31 3530 3835 3500 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 102D
E0F7 0085 0080
짧은글 일수록 신중하게.
FOXBI 2020-05
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
DCB47997 0524125420 T H hdisk3 DISK OPERATION ERROR
3074FEB7 0524125320 T H fscsi1 ADAPTER ERROR
825849BF 0524125320 T H fcs1 ADAPTER ERROR

이거는 temp성이니 일단 두시고

C43F90ED 0524075520 P H hdisk15 SUBSYSTEM COMPONENT FAILURE
C43F90ED 0524075520 P H hdisk17 SUBSYSTEM COMPONENT FAILURE

Perm 이니 좀 보셔야 할것 같습니다.

C43F90ED  항목보시면

Probable Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT

Failure Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

원인과  fault 가 두가지도 예측됩니다.

array쪽 DASD 미디어나, 파워또는 AMD팬모듈입니다.

근데 위치나 리소스 이런게 sysplanar 같은걸 안찍은거 보니  array쪽일 확률이 있어보이긴하네요..


일전에도 문의하신 적있는거 같은데 중복 발생인거 같습니다..

http://www.2cpu.co.kr/QnA/748947

그때 강승훈님께서

https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014283446

이런 검색 결과 알려주셨네요..거의 유사한 결과인거 같습니다...

Check your Storage Manager Major Event log (MEL), it is probably a fan or battery problem.


저도 아는건 여기까지라.. P장비는 엔지니어 불러서 일처리해서 ^^;;
멜랑꼴링 2020-05
lsdev -Cc adpater
명령으로 출력되는 결과로
fcs1번의 path확인과
lsdev -Cc disk
명령으로 출력되는 결과의
스토리지 디스크 path를 확인해서 보여주세요~~
중간에 05-01
이런식으로 나올겁니다
로그인 하시면 댓글을 남길 수 있습니다


QnA
쓰기
제목Page 4510/5680
2014-05   4946225   1 정은준1
2015-12   1482899   1  백메가
2014-09   8006   1 가빠로구나
2014-09   5384   1 설영수
2020-06   2863   1 준호
2023-06   1653   1 화란
2016-01   4241   1 홍방장00
2021-11   1583   1 공백기
2021-11   1414   1 나기냥
2023-06   1271   1  삐돌이슬픔이
2016-12   5767   1 건전지
2021-11   2081   1 KurTurCho
2023-07   2517   1 I79700k
2010-03   17507   1 박종용
2016-12   5265   1 공돌이에요
2016-01   3480   1 재마
2023-07   2654   1 눈팅돌이
2010-03   8197   1 이선규
2020-07   2511   1 미나리나물
2023-07   1540   1 빨강토끼
2014-10   4120   1  방o효o문
2014-10   6263   1  김황중
목록
쓰기