本文使用的是 Fluid 1.0 版本，高版本的配置文件路径发生了变化，需要根据实际情况调整。
1. 制作镜像
1.1 fluid_config_init.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
| #!/usr/bin/env python
import json
import os
rawStr = ""
with open("/etc/fluid/config.json", "r") as f:
rawStr = f.readlines()
rawStr = rawStr[0]
script = """
#!/bin/sh
set -ex
MNT_TO=$targetPath
trap "umount ${MNT_TO}" SIGTERM
mkdir -p ${MNT_TO} || true
s3fs $bucket:$bucketPath $MNT_TO -o url=$url -o allow_other -f -d -o f2
echo "mounted and exit code: $?"
sleep inf
"""
obj = json.loads(rawStr)
with open("/root/.passwd-s3fs", "w") as f:
f.write("%s:%s\n" % (obj["mounts"][0]["options"]["s3-access-key"], obj["mounts"][0]["options"]["s3-access-secret"]))
# set /root/.passwd-s3fs to be read only by owner
os.chmod("/root/.passwd-s3fs", 0o600)
bucketPath = obj["mounts"][0]["mountPoint"].lstrip("s3://").rstrip("/")
bucket = bucketPath.split("/")[0]
bucketPath = bucketPath[len(bucket):]
with open("/mount-s3.sh", "w") as f:
f.write('targetPath="%s"\n' % obj["targetPath"])
f.write('bucket="%s"\n' % bucket)
f.write('bucketPath="%s"\n' % bucketPath)
f.write('url="%s"\n' % obj["mounts"][0]["options"]["url"])
f.write(script)
|
1.2 entrypoint.sh
1
2
3
4
5
6
| #!/usr/bin/env bash
set +x
python /fluid_config_init.py
chmod u+x /mount-s3.sh
bash /mount-s3.sh
|
1.3 Dockerfile
1
2
3
4
5
6
| FROM hubimage/runtime-ubuntu:20.04
RUN apt-get update && apt-get install -y python s3fs
COPY ./fluid_config_init.py /
COPY ./entrypoint.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/entrypoint.sh
ENTRYPOINT []
|
1.4 构建镜像
1
2
| docker build . --network=host -f Dockerfile -t shaowenchen/demo-fluid-s3fs:latest
docker push shaowenchen/demo-fluid-s3fs:latest
|
1. 挂载 S3 存储
1
2
3
4
| export ENDPOINT=obs.ap-southeast-3.myhuaweicloud.com
export BUCKET=
export AK=
export SK=
|
1
2
3
4
5
6
7
8
9
10
11
12
13
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
name: mys3fs
spec:
mounts:
- mountPoint: s3://${BUCKET}/test1
options:
s3-access-key: ${AK}
s3-access-secret: ${SK}
url: https://${ENDPOINT}
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: ThinRuntimeProfile
metadata:
name: s3fs-profile
spec:
fileSystemType: s3fs
fuse:
image: shaowenchen/demo-fluid-s3fs
imageTag: latest
imagePullPolicy: Always
command:
- "/usr/local/bin/entrypoint.sh"
EOF
|
1
2
3
4
5
6
7
8
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: ThinRuntime
metadata:
name: mys3fs-37
spec:
profileName: s3fs-profile
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: mys3fs
spec:
containers:
- name: mys3fs
image: shaowenchen/demo-ubuntu
volumeMounts:
- mountPath: /data
name: mys3fs
volumes:
- name: mys3fs
persistentVolumeClaim:
claimName: mys3fs
EOF
|
2. 性能测试
进入 Pod
1
| kubectl exec -it mys3fs -- bash
|
执行
1
| curl -sSL https://d.juicefs.com/install | sh -
|
安装 JuiceFS 客户端
1
2
3
4
5
6
7
8
9
10
11
12
13
| juicefs bench --block-size 4 --big-file-size 1024 /data
Benchmark finished!
BlockSize: 4.0 MiB, BigFileSize: 1.0 GiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 1
+------------------+-----------------+----------------+
| ITEM | VALUE | COST |
+------------------+-----------------+----------------+
| Write big file | 194.80 MiB/s | 5.26 s/file |
| Read big file | 100.50 MiB/s | 10.19 s/file |
| Write small file | 3.6 files/s | 277.70 ms/file |
| Read small file | 51.1 files/s | 19.59 ms/file |
| Stat file | 26301.5 files/s | 0.04 ms/file |
+------------------+-----------------+----------------+
|
3. 清理资源
1
2
3
| kubectl delete pod mys3fs
kubectl delete thinruntime mys3fs
kubectl delete dataset mys3fs
|
