iozone用了很久，查看源码，发现iozone其实还附带两个工具fileop和pit_server，fileop测试了POSIX常用的函数，pit_server用来测试TCP或UDP服务
今天用了一下fileop测试ceph的读写瓶颈，一下就找到问题了。首先看一下帮助
[root@node2 iozone]# ./fileop -h
--------------------------------------
| Fileop |
| $Revision: 1.61 $ |
| |
| by |
| |
| Don Capps |
--------------------------------------
fileop [-f X ]|[-l # -u #] [-s Y] [-e] [-b] [-w] [-d <dir>] [-t] [-v] [-h]
-f # Force factor. X^ files will be created and removed.
-l # Lower limit on the value of the Force factor.
-u # Upper limit on the value of the Force factor.
-s # Optional. Sets filesize for the create/write. May use suffix 'K' or 'M'.
-e Excel importable format.
-b Output best case results.
-i # Increment force factor by this increment.
-w Output worst case results.
-d <dir> Specify starting directory.
-U <dir> Mount point to remount between tests.
-t Verbose output option.
-v Version information.
-h Help text.
The structure of the file tree is:
X number of Level directories, with X number of
level directories, with X number of files in each
of the level directories.
Example: fileop
dir_1 dir_2
/ \ / \
sdir_1 sdir_2 sdir_1 sdir_2
/ \ / \ / \ / \
file_1 file_2 file_1 file_2 file_1 file_2 file_1 file_2
Each file will be created, and then Y bytes is written to the file.例如：1.我们测试一个文件读写最好最坏和平均的结果。(单位：次/s)
[root@node2 iozone]# ./fileop -f -d ./ -e -b -w
. mkdir chdir rmdir create open read write close stat access chmod readdir link unlink delete Total_files
A
B
W
2.我们用verbose打印详细日志查看结果
[root@node2 iozone]# ./fileop -t -f -d ./ Fileop: Working in ., File size is , Output is in Ops/sec. (A=Avg, B=Best, W=Worst)
mkdir: Dirs = Total Time = 0.006633043 seconds
Avg mkdir(s)/sec = 452.28 ( 0.002211014 seconds/op)
Best mkdir(s)/sec = 529.38 ( 0.001888990 seconds/op)
Worst mkdir(s)/sec = 411.17 ( 0.002432108 seconds/op) chdir: Dirs = Total Time = 0.000052691 seconds
Avg chdir(s)/sec = 56936.25 ( 0.000017564 seconds/op)
Best chdir(s)/sec = 113359.57 ( 0.000008821 seconds/op)
Worst chdir(s)/sec = 28532.68 ( 0.000035048 seconds/op) rmdir: Dirs = Total Time = 0.011839867 seconds
Avg rmdir(s)/sec = 253.38 ( 0.003946622 seconds/op)
Best rmdir(s)/sec = 909.24 ( 0.001099825 seconds/op)
Worst rmdir(s)/sec = 165.32 ( 0.006048918 seconds/op) create: Files = Total Time = 0.002378941 seconds
Avg create(s)/sec = 420.36 ( 0.002378941 seconds/op)
Best create(s)/sec = 420.36 ( 0.002378941 seconds/op)
Worst create(s)/sec = 420.36 ( 0.002378941 seconds/op) write: Files = Total Time = 0.000038147 seconds
Avg write(s)/sec = 26214.40 ( 0.000038147 seconds/op)
Best write(s)/sec = 26214.40 ( 0.000038147 seconds/op)
Worst write(s)/sec = 26214.40 ( 0.000038147 seconds/op) close: Files = Total Time = 0.000033855 seconds
Avg close(s)/sec = 29537.35 ( 0.000033855 seconds/op)
Best close(s)/sec = 29537.35 ( 0.000033855 seconds/op)
Worst close(s)/sec = 29537.35 ( 0.000033855 seconds/op) stat: Files = Total Time = 0.000011206 seconds
Avg stat(s)/sec = 89240.51 ( 0.000011206 seconds/op)
Best stat(s)/sec = 89240.51 ( 0.000011206 seconds/op)
Worst stat(s)/sec = 89240.51 ( 0.000011206 seconds/op) open: Files = Total Time = 0.000014067 seconds
Avg open(s)/sec = 71089.90 ( 0.000014067 seconds/op)
Best open(s)/sec = 71089.90 ( 0.000014067 seconds/op)
Worst open(s)/sec = 71089.90 ( 0.000014067 seconds/op) read: Files = Total Time = 0.000015020 seconds
Avg read(s)/sec = 66576.25 ( 0.000015020 seconds/op)
Best read(s)/sec = 66576.25 ( 0.000015020 seconds/op)
Worst read(s)/sec = 66576.25 ( 0.000015020 seconds/op) access: Files = Total Time = 0.000008821 seconds
Avg access(s)/sec = 113359.57 ( 0.000008821 seconds/op)
Best access(s)/sec = 113359.57 ( 0.000008821 seconds/op)
Worst access(s)/sec = 113359.57 ( 0.000008821 seconds/op) chmod: Files = Total Time = 0.000021935 seconds
Avg chmod(s)/sec = 45590.26 ( 0.000021935 seconds/op)
Best chmod(s)/sec = 45590.26 ( 0.000021935 seconds/op)
Worst chmod(s)/sec = 45590.26 ( 0.000021935 seconds/op) readdir: Files = Total Time = 0.001602888 seconds
Avg readdir(s)/sec = 623.87 ( 0.001602888 seconds/op)
Best readdir(s)/sec = 623.87 ( 0.001602888 seconds/op)
Worst readdir(s)/sec = 623.87 ( 0.001602888 seconds/op) link: Files = Total Time = 0.000864983 seconds
Avg link(s)/sec = 1156.09 ( 0.000864983 seconds/op)
Best link(s)/sec = 1156.09 ( 0.000864983 seconds/op)
Worst link(s)/sec = 1156.09 ( 0.000864983 seconds/op) unlink: Files = Total Time = 0.000801086 seconds
Avg unlink(s)/sec = 1248.30 ( 0.000801086 seconds/op)
Best unlink(s)/sec = 1248.30 ( 0.000801086 seconds/op)
Worst unlink(s)/sec = 1248.30 ( 0.000801086 seconds/op) delete: Files = Total Time = 0.000800133 seconds
Avg delete(s)/sec = 1249.79 ( 0.000800133 seconds/op)
Best delete(s)/sec = 1249.79 ( 0.000800133 seconds/op)
Worst delete(s)/sec = 1249.79 ( 0.000800133 seconds/op)
从结果中，我们可以得出结论，ceph在FS挂载的情况下，其实性能瓶颈主要是在readdir:rmdir:mkdir:三个函数上面。