StudyOverseas 大佬有话说 :
为ZFS zpool添加固态硬盘缓存 ZIL写缓存|ARC写缓存
原文：https://www.liujason.com/article/800.html
新建loop虚拟盘
本地磁盘就是NVME的，如果可以重新分区那是最好，不过我这里嫌麻烦就直接用loop虚拟盘了，性能会差一点点但是不会太多（毕竟本地已经格式化为Ext4了）。
创建镜像文件
大家都喜欢用dd，不过fallocate我觉得更舒服。这里新建两个镜像文件，分别是读缓存和写缓存的：
root@HPC-HEL1-S101 /home # fallocate -l 300G zfs-log-cache.img
root@HPC-HEL1-S101 /home # fallocate -l 300G zfs-read-cache.img
要注意的是，读缓存损坏后数据不会丢失，写缓存损坏的话会导致数据丢失，慎用！
挂载为loop
这个没啥好说的，用losetup即可：
root@HPC-HEL1-S101 /home # losetup -fP zfs-log-cache.img
root@HPC-HEL1-S101 /home # losetup -fP zfs-read-cache.img
检查挂载情况
root@HPC-HEL1-S101 /home # losetup -a
/dev/loop1: :37691140 (/home/zfs-log-cache.img)
/dev/loop2: :37691139 (/home/zfs-read-cache.img)
/dev/loop0: :52272397 (/var/lib/vz/images/100/vm-100-disk-0.raw)
root@HPC-HEL1-S101 /home # lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 64G0 loop
loop1 7:1 0 300G0 loop
loop2 7:2 0 300G0 loop
添加到zpool
查看zpool
首先检查一下zpool的情况：
root@HPC-HEL1-S101 /home # zpool status
pool: Hel1-S101
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
Hel1-S101 ONLINE 0 0 0
raidz1-0ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
sdg ONLINE 0 0 0
sdh ONLINE 0 0 0
sdi ONLINE 0 0 0
sdj ONLINE 0 0 0
errors: No known data errors
添加缓存
分别添加读缓存和写缓存：
root@HPC-HEL1-S101 /home # zpool add Hel1-S101 log /dev/loop1
root@HPC-HEL1-S101 /home # zpool add Hel1-S101 cache /dev/loop2
检查缓存情况
root@HPC-HEL1-S101 /home # zpool status
pool: Hel1-S101
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
Hel1-S101 ONLINE 0 0 0
raidz1-0ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
sdg ONLINE 0 0 0
sdh ONLINE 0 0 0
sdi ONLINE 0 0 0
sdj ONLINE 0 0 0
logs
loop1 ONLINE 0 0 0
cache
loop2 ONLINE 0 0 0
errors: No known data errors
缓存效果
用zpool iostat -v可以明显看到SSD的读写缓存已经生效了，后续热文件将直接从cache中读取，降低机械盘的负载。
root@HPC-HEL1-S101 ~ # zpool iostat -v
capacity operations bandwidth
pool alloc free readwrite readwrite
—————————————-
Hel1-S101 132G90.8T 0 893.30K1.91M
raidz1 132G90.8T 0 813.30K 991K
sda – – 0 8 33399.5K
sdb – – 0 7 31598.6K
sdc – – 0 8 34999.6K
sdd – – 0 7 33498.7K
sde – – 0 8 36299.7K
sdf – – 0 8 32698.7K
sdg – – 0 8 35499.6K
sdh – – 0 8 32898.6K
sdi – – 0 8 34999.6K
sdj – – 0 8 32298.5K
logs – – – – – –
loop1 11.8M 298G 0 10 81.21M
cache – – – – – –
loop2 224M 300G 0 0 4981.3K
—————————————-
fio测试-16k小文件随机
测试代码：fio -filename=/Hel1-S101/fiotest.bin -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=10G -numjobs=30 -runtime=10 -group_reporting -name=zfs-cache-test
Jobs: 30 (f=30):
zfs-cache-test: (groupid=0, jobs=30): err= 0: pid=21034: Thu Jun4 18:49:56 2020
write: IOPS=11.1k, BW=173MiB/s (182MB/s)(1734MiB/10002msec); 0 zone resets
clat (usec): min=5, max=88218, avg=2700.66, stdev=2176.33
lat (usec): min=6, max=88218, avg=2700.98, stdev=2176.36
clat percentiles (usec):
|1.00th=[ 10],5.00th=[ 45], 10.00th=[ 52], 20.00th=,
| 30.00th=[ 1696], 40.00th=[ 2311], 50.00th=[ 2737], 60.00th=[ 3130],
| 70.00th=[ 3556], 80.00th=[ 3982], 90.00th=[ 4883], 95.00th=[ 6063],
| 99.00th=[ 7373], 99.50th=[ 8291], 99.90th=, 99.95th=,
| 99.99th=
bw (KiB/s): min= 3552, max=40271, per=3.33%, avg=5913.96, stdev=4550.90, samples=592
iops : min=222, max= 2516, avg=369.60, stdev=284.39, samples=592
lat (usec) : 10=1.10%, 20=0.79%, 50=7.09%, 100=5.98%, 250=2.23%
lat (usec) : 500=0.92%, 750=1.26%, 1000=3.18%
lat (msec) : 2=11.54%, 4=46.17%, 10=19.38%, 20=0.29%, 50=0.07%
lat (msec) : 100=0.01%
cpu : usr=0.16%, sys=2.21%, ctx=205649, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,110988,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=173MiB/s (182MB/s), 173MiB/s-173MiB/s (182MB/s-182MB/s), io=1734MiB (1818MB), run=10002-10002msec
fio测试-大文件1G连续
测试代码：fio -filename=/Hel1-S101/fiotest.bin -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=1G -size=20G -numjobs=30 -runtime=10 -group_reporting -name=zfs-cache-test
zfs-cache-test: (groupid=0, jobs=30): err= 0: pid=6117: Thu Jun4 18:52:20 2020
write: IOPS=0, BW=785MiB/s (823MB/s)(30.0GiB/39131msec); 0 zone resets
clat (msec): min=14642, max=38803, avg=24851.81, stdev=8958.50
lat (msec): min=14770, max=39124, avg=25138.77, stdev=8987.50
clat percentiles (msec):
|1.00th=,5.00th=, 10.00th=, 20.00th=,
| 30.00th=, 40.00th=, 50.00th=, 60.00th=,
| 70.00th=, 80.00th=, 90.00th=, 95.00th=,
| 99.00th=, 99.50th=, 99.90th=, 99.95th=,
| 99.99th=
cpu : usr=0.54%, sys=1.83%, ctx=264186, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,30,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=785MiB/s (823MB/s), 785MiB/s-785MiB/s (823MB/s-823MB/s), io=30.0GiB (32.2GB), run=39131-39131msec
h20 大佬有话说 :
拿SSD的生命换HDD的速度？