在 K8S 中使用 IP SAN 设备

术语

缩写	定义
iSCSI	Internet Small Computer System Interface
IQN	iSCSI Qualified Name
NFS	Network File System

IP SAN 设备配置简介

IP SAN 配置略
本文主要探究 iSCSI、NFS两种协议的使用方式

k8s 对存储的支持情况

Volume Plugin	ReadWriteOnce	ReadOnlyMany	ReadWriteMany
HostPath	✓	-	-
iSCSI	✓	✓	-
NFS	✓	✓	✓

NFS 相对支持比较好点，支持多读多写，推荐使用

CentOS 7 使用 iSCSI 连接设备

首先安装 iSCSI 控制器

yum install iscsi-initiator-utils -y

搜索设备

搜索下设备，找到 IQN

[root@master1 ~]# iscsiadm -m discoverydb -t sendtargets -p 192.168.88.100 -I default --discover
...
192.168.88.100:3260,1 iqn.2471-05.storos.t:27
10.15.117.190:3260,1 iqn.2471-05.storos.t:27
...

挂载设备

通过fdisk -l 看下挂载前情况

[root@master1 ~]# fdisk -l

Disk /dev/sda: 480.1 GB, 480103981056 bytes, 937703088 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
...

看到只有一块硬盘 /dev/sda

然后执行挂载指令，并查看挂载后情况

[root@master1 ~]# iscsiadm -m node -T iqn.2471-05.storos.t:27 -p 192.168.88.100 -l
[root@master1 ~]# fdisk -l

Disk /dev/sda: 480.1 GB, 480103981056 bytes, 937703088 sectors
Units = sectors of 1 * 512 = 512 bytes
...

Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 2621440 sectors
Units = sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 524288 bytes

可以看到 /dev/sdb 已经被挂载到本地
随后使用就和其他磁盘一样，可以格式化、挂载

创建分区

[root@master1 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xc26993b8.

Command (m for help): p

Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 2621440 sectors
Units = sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 524288 bytes
Disk label type: dos
Disk identifier: 0xc26993b8

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 
First sector (256-2621439, default 256): 
Using default value 256
Last sector, +sectors or +size{K,M,G} (256-2621439, default 2621439): 
Using default value 2621439
Partition 1 of type Linux and of size 10 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

格式化分区

1	mkfs.ext4 /dev/sdb1

会话查看与卸载

[root@master1 ~]# iscsiadm -m session
tcp: [1] 192.168.88.100:3260,1 iqn.2471-05.storos.t:27 (non-flash)
[root@master1 ~]# 
[root@master1 ~]# iscsiadm -m node -T iqn.2471-05.storos.t:27 -p 192.168.88.100 -u
Logging out of session [sid: 1, target: iqn.2471-05.storos.t:27, portal: 192.168.88.100,3260]
Logout of [sid: 1, target: iqn.2471-05.storos.t:27, portal: 192.168.88.100,3260] successful.

CHAP authentication

在IP SAN 里面创建用户test_chap ，密码为test_chap

[root@master1 ~]# iscsiadm -m node -T iqn.2471-05.storos.t:27 -p 192.168.88.100 -o update --name=node.session.auth.username --value=test_chap
[root@master1 ~]# iscsiadm -m node -T iqn.2471-05.storos.t:27 -p 192.168.88.100 -o update --name=node.session.auth.password --value=test_chap
[root@master1 ~]# iscsiadm -m node -T iqn.2471-05.storos.t:27 -p 192.168.88.100 -l
Logging in to [iface: default, target: iqn.2471-05.storos.t:27, portal: 192.168.88.100,3260] (multiple)
Login to [iface: default, target: iqn.2471-05.storos.t:27, portal: 192.168.88.100,3260] successful.

这里只描述创建会话的认证过程，搜索过程没做认证

K8S 使用 iSCSI 作为pv后端

创建pv

apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-mysql
  labels:
    app: test-mysql
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  iscsi:
    targetPortal: 192.168.88.100:3260
    iqn: iqn.2471-05.storos.t:27
    fsType: ext4
    lun: 0

其中lun 为客户端卷号（在san管理/iSCSI管理界面查询，依据设备不同而不同）

磁盘上如果数据格式不对，无法挂载，最好新建磁盘，让 K8S 自行去做初始化

Apr 11 13:46:58 master1 kubelet[16033]: E0411 13:46:58.912465   16033 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/iscsi/192.168.88.100:3260:iqn.2471-05.storos.t:27:0\"" failed. No retries permitted until 2018-04-11 13:47:06.91242994 +0800 CST (durationBeforeRetry 8s). Error: MountVolume.WaitForAttach failed for volume "test-mysql" (UniqueName: "kubernetes.io/iscsi/192.168.88.100:3260:iqn.2471-05.storos.t:27:0") pod "test-mysql-dd6df7d44-d4jkj" (UID: "6b529d7b-3d4b-11e8-8ab1-0cc47ae55938") : failed to mount the volume as "ext4", it already contains unknown data, probably partitions. Mount error: mount failed: exit status 32

挂载过程

kubelet 挂载过程

[root@master1 ~]# journalctl -u kubelet -f | grep iscsi
Apr 11 14:19:35 master1 kubelet[16033]: I0411 14:19:35.194301   16033 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "test-mysql" (UniqueName: "kubernetes.io/iscsi/192.168.88.100:3260:iqn.2471-05.storos.t:27:0") pod "test-mysql-dd6df7d44-lqj4j" (UID: "38528d0a-3d50-11e8-8ab1-0cc47ae55938")
Apr 11 14:19:35 master1 kubelet[16033]: I0411 14:19:35.198210   16033 operation_generator.go:684] Controller attach succeeded for volume "test-mysql" (UniqueName: "kubernetes.io/iscsi/192.168.88.100:3260:iqn.2471-05.storos.t:27:0") pod "test-mysql-dd6df7d44-lqj4j" (UID: "38528d0a-3d50-11e8-8ab1-0cc47ae55938") device path: ""
Apr 11 14:19:35 master1 kubelet[16033]: I0411 14:19:35.294607   16033 reconciler.go:257] operationExecutor.MountVolume started for volume "test-mysql" (UniqueName: "kubernetes.io/iscsi/192.168.88.100:3260:iqn.2471-05.storos.t:27:0") pod "test-mysql-dd6df7d44-lqj4j" (UID: "38528d0a-3d50-11e8-8ab1-0cc47ae55938")
Apr 11 14:19:35 master1 kubelet[16033]: I0411 14:19:35.294653   16033 operation_generator.go:416] MountVolume.WaitForAttach entering for volume "test-mysql" (UniqueName: "kubernetes.io/iscs/192.168.88.100:3260:iqn.2471-05.storos.t:27:0") pod "test-mysql-dd6df7d44-lqj4j" (UID: "38528d0a-3d50-11e8-8ab1-0cc47ae55938") DevicePath ""
Apr 11 14:19:35 master1 kubelet[16033]: E0411 14:19:35.314813   16033 iscsi_util.go:233] iscsi: failed to rescan session with error: iscsiadm: No session found.
Apr 11 14:19:36 master1 kubelet[16033]: Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/192.168.88.100:3260-iqn.2471-05.storos.t:27-lun-0 --scope -- mount -t ext4 -o defaults /dev/disk/by-path/ip-192.168.88.100:3260-iscsi-iqn.2471-05.storos.t:27-lun-0 /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/192.168.88.100:3260-iqn.2471-05.storos.t:27-lun-0
Apr 11 14:19:36 master1 kubelet[16033]: I0411 14:19:36.546450   16033 mount_linux.go:404] Disk "/dev/disk/by-path/ip-192.168.88.100:3260-iscsi-iqn.2471-05.storos.t:27-lun-0" appears to be unformatted, attempting to format as type: "ext4" with options: [-F /dev/disk/by-path/ip-192.168.88.100:3260-iscsi-iqn.2471-05.storos.t:27-lun-0]
Apr 11 14:19:38 master1 kubelet[16033]: I0411 14:19:38.003506   16033 mount_linux.go:408] Disk successfully formatted (mkfs): ext4 - /dev/disk/by-path/ip-192.168.88.100:3260-iscsi-iqn.2471-05.storos.t:27-lun-0 /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/192.168.88.100:3260-iqn.2471-05.storos.t:27-lun-0
Apr 11 14:19:38 master1 kubelet[16033]: I0411 14:19:38.022322   16033 operation_generator.go:425] MountVolume.WaitForAttach succeeded for volume "test-mysql" (UniqueName: "kubernetes.io/iscsi/192.168.88.100:3260:iqn.2471-05.storos.t:27:0") pod "test-mysql-dd6df7d44-lqj4j" (UID: "38528d0a-3d50-11e8-8ab1-0cc47ae55938")
Apr 11 14:19:38 master1 kubelet[16033]: I0411 14:19:38.022420   16033 operation_generator.go:446] MountVolume.MountDevice succeeded for volume "test-mysql" (UniqueName: "kubernetes.io/iscsi/192.168.88.100:3260:iqn.2471-05.storos.t:27:0") pod "test-mysql-dd6df7d44-lqj4j" (UID: "38528d0a-3d50-11e8-8ab1-0cc47ae55938") device mount path "/var/lib/kubelet/plugins/kubernetes.io/iscs/iface-default/192.168.88.100:3260-iqn.2471-05.storos.t:27-lun-0"
Apr 11 14:19:38 master1 kubelet[16033]: I0411 14:19:38.061696   16033 operation_generator.go:484] MountVolume.SetUp succeeded for volume "test-mysql" (UniqueName: "kubernetes.io/iscsi/192.168.88.100:3260:iqn.2471-05.storos.t:27:0") pod "test-mysql-dd6df7d44-lqj4j" (UID: "38528d0a-3d50-11e8-8ab1-0cc47ae55938")
Apr 11 14:20:21 master1 kubelet[16033]: I0411 14:20:21.347063   16033 reconciler.go:407] Reconciler sync states: could not find pod information in desired or actual states or pending operation, update it in both states: &{volumeName:kubernetes.io/iscsi/test-mysql:test-mysql:0 podName:38528d0a-3d50-11e8-8ab1-0cc47ae55938 volumeSpec:0xc422c2f760 outerVolumeSpecName:test-mysql pod:0xc42389c380 pluginIsAttachable:true volumeGidValue: devicePath: reportedInUse:false mounter:0xc420bdfc80}
Apr 11 14:20:21 master1 kubelet[16033]: I0411 14:20:21.351063   16033 reconciler.go:547] Volume: kubernetes.io/iscsi/test-mysql:test-mysql:0 is mounted

主机挂载点查看

1 2	[root@master1 ~]# df -h \| grep sdb /dev/sdb 9.8G 246M 9.0G 3% /var/lib/kubelet/pods/38528d0a-3d50-11e8-8ab1-0cc47ae55938/volumes/kubernetes.io~iscsi/test-mysql

[root@master1 ~]# mount |grep kub
/dev/sdb on /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/192.168.88.100:3260-iqn.2471-05.storos.t:27-lun-0 type ext4 (rw,relatime,stripe=128,data=ordered)
/dev/sdb on /var/lib/kubelet/pods/38528d0a-3d50-11e8-8ab1-0cc47ae55938/volumes/kubernetes.io~iscsi/test-mysql type ext4 (rw,relatime,stripe=128,data=ordered)
...

CHAP authentication

修改PV 的配置文件

---
apiVersion: v1
kind: Secret
metadata:
  name: chap-secret
type: "kubernetes.io/iscsi-chap"  
data:
  node.session.auth.username: aGlrY2xvdWRfY2hhcA==
  node.session.auth.password: aGlrY2xvdWRfY2hhcA==
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-mysql
  labels:
    app: test-mysql
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  iscsi:
    targetPortal: 192.168.88.100:3260
    iqn: iqn.2471-05.storos.t:27
    fsType: ext4
    lun: 0
    chapAuthDiscovery: false
    chapAuthSession: true
    secretRef: 
       name: chap-secret

secret中其他配置项参见 Open-iSCSI

CentOS 7 使用 NFS 连接设备

首先安装nfs工具

yum -y install nfs-utils

查看NAS的挂载点

[root@master1 ~]# showmount -e 192.168.88.100
Export list for 192.168.88.100:
/nas/test_nfs/test_nfs 0.0.0.0/0.0.0.0
/nas/nfs_falcon_db/nfs1        0.0.0.0/0.0.0.0
/nas/nfs_falcon_db/db          0.0.0.0/0.0.0.0
/nas/nfs_falcon_rtsp/rtsp      0.0.0.0/0.0.0.0
/mnt/backup/indexbackup        0.0.0.0/0.0.0.0

挂载

1 2	mkdir -p /root/test_nfs mount -t nfs 192.168.88.100:/nas/test_nfs/test_nfs /root/test_nfs

K8S 使用 NFS 作为pv后端

创建pv

apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-mysql
  labels:
    app: test-mysql
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.88.100
    path: "/nas/test_nfs/test_nfs"

这里遇到一个问题，MySQL 无法启动：

[root@master1 ~]# kubectl logs -f  mysql-test
Initializing database
2018-04-13T03:46:31.653727Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2018-04-13T03:46:31.695064Z 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 37
2018-04-13T03:46:31.695129Z 0 [ERROR] InnoDB: Operating system error number 37 in a file operation.
2018-04-13T03:46:31.695179Z 0 [ERROR] InnoDB: Error number 37 means 'No locks available'
2018-04-13T03:46:31.695194Z 0 [ERROR] InnoDB: Cannot open datafile './ibdata1'
2018-04-13T03:46:31.695216Z 0 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2018-04-13T03:46:31.695228Z 0 [ERROR] InnoDB: InnoDB Database creation was aborted with error Cannot open a file. You may need to delete the ibdata1 file before trying to start up again.
2018-04-13T03:46:32.295901Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2018-04-13T03:46:32.295925Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2018-04-13T03:46:32.295936Z 0 [ERROR] Failed to initialize builtin plugins.
2018-04-13T03:46:32.295941Z 0 [ERROR] Aborting

分析了下就是.lock 文件不能创建，然后就退出了
对应的BUG记录mysql bug
意思就是这个是和使用的NFSv3 协议有关系，据说 NFSv4 已经解决，不幸的是我们设备是NFSv3协议的，随后改了下mountOptions，就可以正常使用了:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-mysql
  labels:
    app: test-mysql
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
   mountOptions:
  - nolock
  - hard
  - rw
  - proto=tcp
  - vers=3
  - mountvers=3
  - nfsvers=3
  - rsize=524288
  - wsize=524288
  - namlen=255
  nfs:
    server: 192.168.88.100
    path: "/nas/test_nfs/test_nfs"

l1b0k

在 K8S 中使用 IP SAN 设备

TOC

术语

IP SAN 设备配置简介

k8s 对存储的支持情况

CentOS 7 使用 iSCSI 连接设备

搜索设备

挂载设备

创建分区

格式化分区

会话查看与卸载

CHAP authentication

K8S 使用 iSCSI 作为pv后端

创建pv

挂载过程

kubelet 挂载过程

主机挂载点查看

CHAP authentication

CentOS 7 使用 NFS 连接设备

查看NAS的挂载点

挂载

K8S 使用 NFS 作为pv后端

创建pv