1. 목표
- 사무실에서 사용 할 저렴한 가격에 안정적인 스토리지 획득
- RAID5, 1TB 6개로 5TB용량 확보
- 안정적인 디스크컨트롤러 확보
2. 소프트웨어 선택
- ZFS를 사용가능한 OS
FreeBSD, FreeNAS, Solaris, OpenSolaris, Nexentar
- Solaris, OpenSolaris, Nexentar
ZFS의 원조라는 장점이 있지만, Solaris는 상용으로 돌아섰으며, Opensolaris는 지원이 끊긴지 오래됐고, 넥센타는 솔라리스계열이라 하드웨어 지원이 빈약함, 신형 칩셋들(디스크컨트롤러)을 지원 못함.
- FreeNAS
FreeBSD기반, APM(Apache, PHP, MySQL) 설치할것이라면 FReeBSD가 낫다는 판단.
- FreeBSD
현재 8.2릴리즈와 9.0RC1이 나와있는데, 안정성면에서 8.2릴리즈 선택, 설치 후 8.2-STABLE 로 업그레이드하면 ZFS 28로 업그레이드됨
3. 하드웨어 선택
- CPU
소프트웨어레이드의 특성상 어느정도의 CPU연산능력 필요, Deduplication을 사용안한다면 듀얼코어로 충분함
- RAM
ZFS의 메모리 활용 특성상 4G 이상 권장
- Mainboard
대역폭이 보장된 많은 내장 SATA포트, 안정성
- 하드웨어정보
메인보드 : Supermicro X7SB3
CPU : Intel Dualcore CPU E2200 @ 2.20GHz
RAM : 2G * 3
HDD : 1TB * 6
OS-HDD : 64G SSD(SLC) (SAMSUNG MCCOE64G8MPP-0VA PS105006 )
- HDD 상세정보
da0: <ATA ST31000340NS SN05> Fixed Direct Access SCSI-5 device
da1: <ATA ST31000340NS SN05> Fixed Direct Access SCSI-5 device
da2: <ATA Hitachi HDT72101 A31B> Fixed Direct Access SCSI-5 device
da3: <ATA Hitachi HDT72101 A31B> Fixed Direct Access SCSI-5 device
da4: <ATA Hitachi HDT72101 A31B> Fixed Direct Access SCSI-5 device
da5: <ATA ST31000340NS SN05> Fixed Direct Access SCSI-5 device
4. OS 설치 및 업데이트
- 이미지 다운받아서 CD에 구워 설치
http://ftp.kaist.ac.kr/FreeBSD/ISO-IMAGES-amd64/8.2/FreeBSD-8.2-RELEASE-amd64-disc1.iso
- 설치 매뉴얼
http://www.kr.freebsd.org/doc/KoreanFreeBSDHandbook/
http://www.freebsd.org/doc/handbook/install.html
- 시스템 업데이트
업데이트 후 ZFS 버전 업그레이드
- 업데이트 전 ZFS 버전
[ys ~ # ] zpool upgrade -v
This system is currently running ZFS pool version 15.
The following versions are supported:
VER DESCRIPTION
--- --------------------------------------------------------
1 Initial ZFS version
2 Ditto blocks (replicated metadata)
3 Hot spares and double parity RAID-Z
4 zpool history
5 Compression using the gzip algorithm
6 bootfs pool property
7 Separate intent log devices
8 Delegated administration
9 refquota and refreservation properties
10 Cache devices
11 Improved scrub performance
12 Snapshot properties
13 snapused property
14 passthrough-x aclinherit
15 user/group space accounting
For more information on a particular version, including supported releases, see:
http://www.opensolaris.org/os/community/zfs/version/N
Where 'N' is the version number.
[ys ~ # ] zfs upgrade -v
The following filesystem versions are supported:
VER DESCRIPTION
--- --------------------------------------------------------
1 Initial ZFS filesystem version
2 Enhanced directory entries
3 Case insensitive and File system unique identifer (FUID)
4 userquota, groupquota properties
For more information on a particular version, including supported releases, see:
http://www.opensolaris.org/os/community/zfs/version/zpl/N
Where 'N' is the version number.
- 업데이트 후 ZFS 버전
This system is currently running ZFS pool version 28.
The following versions are supported:
VER DESCRIPTION
--- --------------------------------------------------------
1 Initial ZFS version
2 Ditto blocks (replicated metadata)
3 Hot spares and double parity RAID-Z
4 zpool history
5 Compression using the gzip algorithm
6 bootfs pool property
7 Separate intent log devices
8 Delegated administration
9 refquota and refreservation properties
10 Cache devices
11 Improved scrub performance
12 Snapshot properties
13 snapused property
14 passthrough-x aclinherit
15 user/group space accounting
16 stmf property support
17 Triple-parity RAID-Z
18 Snapshot user holds
19 Log device removal
20 Compression using zle (zero-length encoding)
21 Deduplication
22 Received properties
23 Slim ZIL
24 System attributes
25 Improved scrub stats
26 Improved snapshot deletion performance
27 Improved snapshot creation performance
28 Multiple vdev replacements
For more information on a particular version, including supported releases,
see the ZFS Administration Guide.
[ys ~ # ] zfs upgrade -v
The following filesystem versions are supported:
VER DESCRIPTION
--- --------------------------------------------------------
1 Initial ZFS filesystem version
2 Enhanced directory entries
3 Case insensitive and File system unique identifier (FUID)
4 userquota, groupquota properties
5 System attributes
For more information on a particular version, including supported releases,
see the ZFS Administration Guide.
5. ZFS 설치
- 볼륨 만들기 및 마운트
[ys ~ # ] zpool create ZFS raidz da0 da1 da2 da3 da4 da5
- ZFS 시작프로그램 등록
[ys ~ # ] echo 'zfs_enable="YES"' >> /etc/rc.conf
6. ZFS 설치 확인 및 어플리케이션 설치
- ZFS 상태 확인
[ys ~ # ] zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
ZFS 5.44T 1.14T 4.30T 20% 1.00x ONLINE -
[ys ~ # ] df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ad0s1a 12G 543M 10G 5% /
devfs 1.0k 1.0k 0B 100% /dev
/dev/ad0s1e 28G 5G 21G 19% /usr
/dev/ad0s1d 9.7G 173M 8.8G 2% /var
ZFS 4.5T 969G 3.5T 21% /ZFS
[ys ~ # ] zpool status
pool: ZFS
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
ZFS ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
da3 ONLINE 0 0 0
da4 ONLINE 0 0 0
da5 ONLINE 0 0 0
errors: No known data errors
[ys ~ # ] zpool get all ZFS
NAME PROPERTY VALUE SOURCE
ZFS size 5.44T -
ZFS capacity 20% -
ZFS altroot - default
ZFS health ONLINE -
ZFS guid 3165830405713802798 default
ZFS version 28 default
ZFS bootfs - default
ZFS delegation on default
ZFS autoreplace off default
ZFS cachefile - default
ZFS failmode wait default
ZFS listsnapshots off default
ZFS autoexpand off default
ZFS dedupditto 0 default
ZFS dedupratio 1.00x -
ZFS free 4.30T -
ZFS allocated 1.14T -
ZFS readonly off -
[ys ~ # ] zfs get all ZFS
NAME PROPERTY VALUE SOURCE
ZFS type filesystem -
ZFS creation 월 10 31 17:59 2011 -
ZFS used 969G -
ZFS available 3.50T -
ZFS referenced 969G -
ZFS compressratio 1.00x -
ZFS mounted yes -
ZFS quota none default
ZFS reservation none default
ZFS recordsize 128K default
ZFS mountpoint /ZFS default
ZFS sharenfs off default
ZFS checksum on default
ZFS compression off default
ZFS atime on default
ZFS devices on default
ZFS exec on default
ZFS setuid on default
ZFS readonly off default
ZFS jailed off default
ZFS snapdir hidden default
ZFS aclmode discard default
ZFS aclinherit restricted default
ZFS canmount on default
ZFS xattr off temporary
ZFS copies 1 default
ZFS version 5 -
ZFS utf8only off -
ZFS normalization none -
ZFS casesensitivity sensitive -
ZFS vscan off default
ZFS nbmand off default
ZFS sharesmb off default
ZFS refquota none default
ZFS refreservation none default
ZFS primarycache all default
ZFS secondarycache all default
ZFS usedbysnapshots 0 -
ZFS usedbydataset 969G -
ZFS usedbychildren 20.9M -
ZFS usedbyrefreservation 0 -
ZFS logbias latency default
ZFS dedup off default
ZFS mlslabel -
ZFS sync standard default
ZFS refcompressratio 1.00x
- 어플리케이션 설치
[ys ~ # ] pkg_add -r samba
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8-stable/Latest/samba.tbz... Done.
7. 성능시험
- 스크립트
dd if=/dev/zero of=/ZFS/test01 bs=1024k count=1000
dd if=/dev/zero of=/ZFS/test01 bs=1024k count=2000
dd if=/dev/zero of=/ZFS/test01 bs=1024k count=4000
dd if=/dev/zero of=/ZFS/test01 bs=1024k count=8000
time bonnie -s 1024 -d /ZFS
time bonnie -s 2048 -d /ZFS
time bonnie -s 4096 -d /ZFS
time bonnie -s 8192 -d /ZFS
- dd 시험 결과
1회 : 1048576000 bytes transferred in 3.311334 secs (316662698 bytes/sec)
2회 : 2097152000 bytes transferred in 7.727088 secs (271402630 bytes/sec)
3회 : 4194304000 bytes transferred in 16.621016 secs (252349435 bytes/sec)
4회 : 8388608000 bytes transferred in 31.444447 secs (266775497 bytes/sec)
- bonnie 시험 결과
-------Sequential Output-------- ---Sequential Input-- --Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
1회 : 1024 167482 68.3 170106 51.2 192174 70.4 230958 99.5 986913 94.1 104835.5 170.4
2회 : 2048 146949 59.2 197874 64.1 132736 50.9 194864 87.9 381413 54.1 458.3 1.6
3회 : 4096 161796 67.0 196955 64.7 136108 53.3 173639 81.0 363994 52.1 216.9 1.2
4회 : 8192 162792 67.2 187785 61.6 133866 53.0 173529 81.6 382054 54.0 164.5 1.0
- 시험중 시스템 상태
[ys ~ # ] vmstat 1
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr ad0 da0 in sy cs us sy id
2 0 0 738M 289M 0 0 0 0 82694 0 0 46 241 41518 6367 0 54 46
1 0 0 738M 282M 66 0 0 0 149547 0 0 647 3077 9978 25296 1 95 4
1 0 0 738M 221M 0 0 0 0 75278 0 0 41 250 36666 6804 0 59 41
1 0 0 738M 286M 12 0 0 0 185231 0 0 508 3068 28971 26960 0 80 19
2 0 0 738M 268M 0 0 0 0 121960 0 0 365 1838 19188 15595 0 81 19
1 0 0 738M 285M 3 0 0 0 120820 0 0 259 1504 40534 16231 2 61 37
2 0 0 738M 279M 6 0 0 0 147139 0 0 626 2907 11167 24565 1 93 6
1 0 0 738M 218M 36 0 0 0 81377 0 0 60 1144 42255 11208 3 61 36
1 0 0 738M 282M 0 0 0 0 174121 0 0 521 3017 24227 26570 0 83 17
2 0 0 738M 250M 9 0 0 0 100255 0 0 270 1351 24553 11705 1 76 24
2 0 0 738M 282M 1 0 0 0 142577 0 0 336 2198 35206 20491 1 70 29
1 0 0 738M 265M 1 0 0 0 133733 0 0 533 2764 13620 21705 1 91 8
1 0 0 738M 281M 0 0 0 0 99956 0 0 132 904 41003 11736 2 57 42
2 0 0 738M 276M 0 0 0 0 146644 0 0 634 2989 9659 22534 0 91 9
1 0 0 738M 280M 108 0 0 18 83950 0 22 81 1274 44865 11499 3 54 43
1 0 0 738M 277M 2 0 0 0 151835 0 3 635 2845 12526 21737 0 98 2
1 0 0 738M 213M 0 0 0 0 76730 0 0 65 440 33713 8346 0 64 36
1 0 0 738M 276M 2 0 0 0 173750 0 0 550 2959 24705 24278 0 76 24
2 0 0 738M 237M 0 0 0 0 91778 0 0 167 772 29615 8592 0 75 25
2 0 0 738M 276M 4 0 0 0 162449 0 7 421 2351 31842 21850 1 67 32
1 0 0 738M 272M 0 0 0 0 146013 0 0 634 2822 9553 22894 0 93 7
[ys ~ # ] iostat ad0 da0 da1 da2 da3 da4 da5 1
tty ad0 da0 da1 da2 da3 da4 da5 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id
0 238 2.00 1 0.00 38.92 950 36.11 38.04 972 36.11 29.19 1267 36.11 29.40 1258 36.10 28.65 1291 36.11 41.68 887 36.11 0 0 83 3 14
0 524 0.00 0 0.00 113.29 289 32.02 113.24 271 30.02 113.10 292 32.29 113.06 283 31.29 113.31 303 33.56 113.24 300 33.21 0 0 78 2 20
0 238 0.00 0 0.00 103.26 403 40.61 104.71 417 42.61 102.81 402 40.33 103.55 409 41.33 101.59 394 39.06 101.97 396 39.41 0 0 72 3 25
0 233 0.00 0 0.00 113.19 343 37.89 113.07 325 35.90 113.08 378 41.76 112.98 364 40.21 113.27 395 43.68 113.08 347 38.29 0 0 77 3 20
0 237 0.00 0 0.00 95.48 354 33.05 98.26 365 35.07 93.60 319 29.12 95.94 328 30.68 91.95 303 27.17 97.59 342 32.64 0 0 70 1 30
0 390 0.00 0 0.00 112.43 371 40.73 111.96 354 38.70 112.23 387 42.40 112.53 383 42.08 112.08 344 37.65 112.86 363 40.00 0 0 76 3 21
0 236 0.00 0 0.00 99.88 329 32.12 100.96 346 34.14 99.81 312 30.44 101.83 309 30.76 100.60 358 35.20 99.70 337 32.84 0 0 68 3 29
0 232 0.00 0 0.00 113.61 420 46.55 113.57 391 43.32 113.73 420 46.59 113.78 468 52.04 113.37 393 43.47 113.45 405 44.82 0 0 82 1 16
0 237 0.00 0 0.00 96.66 255 24.06 99.40 281 27.25 95.36 258 24.01 91.80 207 18.56 97.24 286 27.13 98.92 267 25.77 0 0 63 0 36
0 232 0.00 0 0.00 113.54 544 60.37 113.51 550 61.02 113.53 601 66.68 113.40 630 69.81 113.52 583 64.67 113.53 589 65.35 1 0 87 2 10
0 238 0.00 0 0.00 71.07 137 9.50 70.79 128 8.84 40.29 81 3.18 0.87 50 0.04 51.14 104 5.19 51.27 90 4.50 2 0 53 2 43
0 142 0.00 0 0.00 113.18 625 69.10 113.27 603 66.68 113.00 626 69.10 113.00 626 69.10 113.99 616 68.62 113.18 625 69.10 1 0 91 3 6
0 195 0.00 0 0.00 64.25 105 6.62 74.44 118 8.56 64.74 108 6.82 67.71 105 6.92 63.15 114 7.06 65.06 101 6.39 3 0 62 3 32
0 231 0.00 0 0.00 106.44 598 62.14 106.96 596 62.24 106.37 596 61.89 107.04 591 61.76 106.00 600 62.09 106.54 600 62.41 0 0 80 3 17
0 237 0.00 0 0.00 111.38 156 17.00 110.97 142 15.42 112.08 177 19.39 111.90 172 18.82 112.10 174 19.07 111.89 164 17.95 0 0 70 0 29
0 235 0.00 0 0.00 104.47 517 52.76 105.73 526 54.31 105.39 489 50.38 106.14 491 50.94 104.18 498 50.70 105.83 501 51.81 0 0 74 3 23
8. 결론
- ZFS는 쓸만하다.
- 허접한 강좌라도 쓰기는 어렵다.
* 장비 외관
전에 저게 느슨해지더니 스파크가 튀더군요 하드 날려먹었던...뉴뉴
궁금한 점이 있습니다.
OS디스크를 SSD로 구성하신 이유가 있으신지요? -> 저는 OS는 USB 메모리 스틱정도로만 구성하려 했습니다.
On-board의 6개 Sata port를 모두 Data용으로 할당하고 싶어서요.
그나마 디스크IO가 좀 있을만한것들은 모두 ZFS파티션으로 옮겼습니다.
당장 죽어도 그리 큰 문제는 없는서버라, SLC이니 한 이삼년 잘 버텨주겠지하고 기대하고있습니다.
그리고 USB도 생각을 해보긴 했는데 아무래도 안정성괜찮은 USB를 찾기 어려워 포기했습니다.
솔라리스와 넥센타, 프리나스도 많은 유혹이 있더군요.
어제 솔라리스11 릴리즈됐다는 소식을 듣고는 또 갈아엎을까하는 마음이 꿈틀하더군요. ^^
아직까지 별 문제 없네요, 운영도중 뻑나거나 문제발생하면 바로 보고하도록하겠습니다. ^^