멀쩡히 잘 돌아가던 서버가 갑자기 재시동되었는데,
secure 로그나 messages 로그나 Webmin 로그를 살펴봤지만, 힌트가 될 만한 기록은 없고,
boot.log 를 보는데, 어떻게 이해를 해야할 지 조언을 구합니다.
Dell R320 / CentOS 6.8 / Webmin 1.801 / Virtualmin GPL 5.03
하드는 SSD 250GB + 1TB 하드 2개 non-raid로 구성되어 있습니다.
%Gata_id[293]: HDIO_GET_IDENTITY failed for '/dev/sda'
ata_id[295]: HDIO_GET_IDENTITY failed for '/dev/sdb'
ata_id[297]: HDIO_GET_IDENTITY failed for '/dev/sdc'
Welcome to [0;36mCentOS[0;39m
Starting udev: %Gata_id[701]: HDIO_GET_IDENTITY failed for '/dev/sda'
ata_id[706]: HDIO_GET_IDENTITY failed for '/dev/sdb'
ata_id[716]: HDIO_GET_IDENTITY failed for '/dev/sdc'
[60G[[0;32m OK [0;39m]
Setting hostname huso.kr: [60G[[0;32m OK [0;39m]
Checking filesystems
/dev/sda6: clean, 15159/1281120 files, 280730/5120000 blocks
/dev/sda1: recovering journal
/dev/sda1: clean, 60/128016 files, 149322/512000 blocks
/dev/sda8: recovering journal
/dev/sdc1: recovering journal
/dev/sdb1: recovering journal
/dev/sda8: Clearing orphaned inode 20 (uid=27, gid=27, mode=0100600, size=0)
/dev/sda8: Clearing orphaned inode 19 (uid=27, gid=27, mode=0100600, size=0)
/dev/sda8: Clearing orphaned inode 18 (uid=27, gid=27, mode=0100600, size=0)
/dev/sda8: Clearing orphaned inode 16 (uid=27, gid=27, mode=0100600, size=0)
/dev/sda8: Clearing orphaned inode 13 (uid=27, gid=27, mode=0100600, size=0)
/dev/sda8: clean, 195/5808128 files, 411728/23217408 blocks
/dev/sda5: recovering journal
/dev/sdc1: clean, 75775/61054976 files, 4488702/244190208 blocks
/dev/sdb1: clean, 42524/61054976 files, 7724547/244190208 blocks
/dev/sda5: Clearing orphaned inode 916018 (uid=0, gid=0, mode=0100755, size=1383392)
/dev/sda5: clean, 160618/1602496 files, 808978/6400000 blocks
/dev/sda2: recovering journal
/dev/sda2: clean, 44/3203072 files, 247106/12800000 blocks
/dev/sda3: recovering journal
/dev/sda3: Clearing orphaned inode 786507 (uid=0, gid=0, mode=0140777, size=0)
/dev/sda3: Clearing orphaned inode 786506 (uid=0, gid=0, mode=0100600, size=0)
/dev/sda3: Clearing orphaned inode 786505 (uid=0, gid=0, mode=0100600, size=0)
/dev/sda3: clean, 7102/3203072 files, 437005/12800000 blocks
[60G[[0;32m OK [0;39m]
Remounting root filesystem in read-write mode: [60G[[0;32m OK [0;39m]
Mounting local filesystems: [60G[[0;32m OK [0;39m]
Enabling local filesystem quotas: [60G[[0;32m OK [0;39m]
Enabling /etc/fstab swaps: [60G[[0;32m OK [0;39m]
Entering non-interactive startup
ipset: Loaded with no configuration
ip6tables: No config file.[60G[[0;33mWARNING[0;39m]
iptables: Applying firewall rules: [60G[[0;32m OK [0;39m]
Bringing up loopback interface: [60G[[0;32m OK [0;39m]
Bringing up interface em1: Determining if ip address 112.216.156.205 is already in use for device em1...
[60G[[0;32m OK [0;39m]
Bringing up interface em2: Determining if ip address 192.168.10.205 is already in use for device em2...
[60G[[0;32m OK [0;39m]
Starting auditd: [60G[[0;32m OK [0;39m]
Starting portreserve: [60G[[0;32m OK [0;39m]
Starting system logger: [60G[[0;32m OK [0;39m]
Starting system message bus: [60G[[0;32m OK [0;39m]
Setting network parameters... [60G[[0;32m OK [0;39m]
Starting NetworkManager daemon: [60G[[0;32m OK [0;39m]
Starting named: [60G[[0;32m OK [0;39m]
Mounting filesystems: [60G[[0;32m OK [0;39m]
Starting HAL daemon: [60G[[0;32m OK [0;39m]
Retrigger failed udev events[60G[[0;32m OK [0;39m]
Starting kdump:[60G[[0;32m OK [0;39m]
Starting sshd: [60G[[0;32m OK [0;39m]
Starting mysqld: [60G[[0;32m OK [0;39m]
Starting Dovecot Imap: [60G[[0;32m OK [0;39m]
Starting saslauthd: [60G[[0;32m OK [0;39m]
Starting postfix: [60G[[0;32m OK [0;39m]
Starting proftpd: [60G[[0;32m OK [0;39m]
Starting httpd: [Thu Jun 30 16:18:22 2016] [warn] module ssl_module is already loaded, skipping
httpd: Could not reliably determine the server's fully qualified domain name, using huso.kr for ServerName
[Thu Jun 30 16:18:22 2016] [warn] _default_ VirtualHost overlap on port 443, the first has precedence
[60G[[0;32m OK [0;39m]
Starting crond: [60G[[0;32m OK [0;39m]
Starting fail2ban: [60G[[0;32m OK [0;39m]
델의 AS가 가능한 상태면 AS를 요청해서 HW 로그를 뽑아보셔야 할 것 같습니다.
델의 A/S는 지났고, HW 로그라는 것이 있는지 몰랐습니다.
더 살펴보겠습니다.
감사합니다.
http://www.2cpu.co.kr/PDS/10028
다른 방법도 있지만, 이 방법이 가장 좋습니다.
갑작스런 장애는 보드 또는 쿨링팬 등의 장애일 가능성이 높기 때문에
hw 로그를 반드시 보셔야 해요.
Dell 상담원의 지원을 받아서 Hardware Log를 채집하였고, Dell 에 보냈습니다.
최근 부분은 아래와 같이 되어 있습니다.
Severity Date and Time Description Raw SEL Data
Mon May 30 19:51:32 2016 The power supplies are redundant. 0xB20002C4994C5720000408740B00FFFFh
Mon May 30 19:51:27 2016 The input power for power supply 1 has been restored. 0xB10002BF994C572000040862EF03FFFFh
Mon May 30 19:51:27 2016 Power supply 1 is operating normally. 0xB00002BF994C572000040862EF01FFFFh
Mon May 30 19:50:44 2016 Fan failure detected on power supply 1. 0xAF000294994C5720000408626F8107FFh
Mon May 30 19:50:37 2016 Power supply redundancy is lost. 0xAE00028D994C5720000408740B01FFFFh
Mon May 30 19:50:32 2016 The power input for power supply 1 is lost. 0xAD000288994C5720000408626F03FFFFh
Mon May 30 19:49:17 2016 The power supplies are redundant. 0xAC00023D994C5720000408740B00FFFFh
Mon May 30 19:49:07 2016 The input power for power supply 1 has been restored. 0xAB000233994C572000040862EF03FFFFh
Mon May 30 19:49:07 2016 Power supply 1 is operating normally. 0xAA000233994C572000040862EF01FFFFh
Mon May 30 19:48:53 2016 Fan failure detected on power supply 1. 0xA9000225994C5720000408626F8107FFh
Mon May 30 19:48:46 2016 Power supply redundancy is lost. 0xA800021E994C5720000408740B01FFFFh
Mon May 30 19:48:41 2016 The power input for power supply 1 is lost. 0xA7000219994C5720000408626F03FFFFh
Wed May 25 22:25:59 2016 The power supplies are redundant. 0xA600027726465720000408740B00FFFFh
Wed May 25 22:25:49 2016 The input power for power supply 1 has been restored. 0xA500026D2646572000040862EF03FFFFh
Wed May 25 22:25:49 2016 Power supply 1 is operating normally. 0xA400026D2646572000040862EF01FFFFh
Wed May 11 20:04:00 2016 Fan failure detected on power supply 1. 0xA300023090335720000408626F8107FFh
Wed May 11 20:03:58 2016 Power supply redundancy is lost. 0xA200022E90335720000408740B01FFFFh
Wed May 11 20:03:48 2016 The power input for power supply 1 is lost. 0xA100022490335720000408626F03FFFFh
Wed May 11 17:03:17 2016 The power supplies are redundant. 0xA00002D565335720000408740B00FFFFh
Wed May 11 17:03:07 2016 The input power for power supply 1 has been restored. 0x9F0002CB6533572000040862EF03FFFFh
Wed May 11 17:03:07 2016 Power supply 1 is operating normally. 0x9E0002CB6533572000040862EF01FFFFh
Wed May 11 12:21:50 2016 Fan failure detected on power supply 1. 0x9D0002DE23335720000408626F8107FFh
Wed May 11 12:21:45 2016 Power supply redundancy is lost. 0x9C0002D923335720000408740B01FFFFh
Wed May 11 12:21:40 2016 The power input for power supply 1 is lost. 0x9B0002D423335720000408626F03FFFFh
Wed May 11 08:41:35 2016 The power supplies are redundant. 0x9A00023FF0325720000408740B00FFFFh
Wed May 11 08:41:30 2016 The input power for power supply 1 has been restored. 0x9900023AF032572000040862EF03FFFFh
Wed May 11 08:41:30 2016 Power supply 1 is operating normally. 0x9800023AF032572000040862EF01FFFFh
Tue May 10 12:39:01 2016 Fan failure detected on power supply 1. 0x97000265D6315720000408626F8107FFh
Tue May 10 12:38:58 2016 Power supply redundancy is lost. 0x96000262D6315720000408740B01FFFFh
Tue May 10 12:38:48 2016 The power input for power supply 1 is lost. 0x95000258D6315720000408626F03FFFFh
Tue May 10 12:34:17 2016 The power supplies are redundant. 0x94000249D5315720000408740B00FFFFh