kubernetes 接 ceph 當 storage backend

kubernetes 版本：1.13.0
不同版本的設定可能會不一樣，我在201903測試過可以用
未來可能會有些許不同
不過原則應該都一樣




原則：
kubernetes 支援用 rbd 當 storage backend。但是 kube-controller-manager container 未必會包含 rbd 指令。如果 kube-controller-manager container 未包含 rbd 指令，你建立 storage 時就會看到 『failed to create rbd image: executable file not found in $PATH, command output:』的錯誤。這時候，你就要提供新的 container ，讓 kubernetes 可以透過你提供的 container （之後都叫 provider ）來執行 rbd 相關操作。該 provider 放在 kube-system 下。

腳本簡易說明：

一開始建立名為 rbd-provisioner 的 serviceAccount ，之後在建立 secret 用來放 ceph keyring。最後建立 storage class ( 裡面放 ceph 相關資訊 )

完整腳本如下：


https://gist.github.com/kjelly/97ebc4133b1293eff9135eada974d670

---
	# kubectl -n kube-system create -f create-kube-ceph-pbrc.yaml
	kind: ClusterRole
	apiVersion: rbac.authorization.k8s.io/v1
	metadata:
	name: rbd-provisioner
	rules:
	- apiGroups: [""]
	resources: ["persistentvolumes"]
	verbs: ["get", "list", "watch", "create", "delete"]
	- apiGroups: [""]
	resources: ["persistentvolumeclaims"]
	verbs: ["get", "list", "watch", "update"]
	- apiGroups: ["storage.k8s.io"]
	resources: ["storageclasses"]
	verbs: ["get", "list", "watch"]
	- apiGroups: [""]
	resources: ["events"]
	verbs: ["create", "update", "patch"]
	- apiGroups: [""]
	resources: ["services"]
	resourceNames: ["kube-dns","coredns"]
	verbs: ["list", "get"]
	- apiGroups: [""]
	resources: ["endpoints"]
	verbs: ["get", "list", "watch", "create", "update", "patch"]
	---
	kind: ClusterRoleBinding
	apiVersion: rbac.authorization.k8s.io/v1
	metadata:
	name: rbd-provisioner
	subjects:
	- kind: ServiceAccount
	name: rbd-provisioner
	namespace: kube-system
	roleRef:
	kind: ClusterRole
	name: rbd-provisioner
	apiGroup: rbac.authorization.k8s.io
	---
	apiVersion: rbac.authorization.k8s.io/v1beta1
	kind: Role
	metadata:
	name: rbd-provisioner
	rules:
	- apiGroups: [""]
	resources: ["secrets"]
	verbs: ["get"]
	---
	apiVersion: rbac.authorization.k8s.io/v1
	kind: RoleBinding
	metadata:
	name: rbd-provisioner
	roleRef:
	apiGroup: rbac.authorization.k8s.io
	kind: Role
	name: rbd-provisioner
	subjects:
	- kind: ServiceAccount
	name: rbd-provisioner
	namespace: kube-system
	---
	apiVersion: v1
	kind: ServiceAccount
	metadata:
	name: rbd-provisioner
	---
	apiVersion: extensions/v1beta1
	kind: Deployment
	metadata:
	name: rbd-provisioner
	spec:
	replicas: 1
	strategy:
	type: Recreate
	template:
	metadata:
	labels:
	app: rbd-provisioner
	spec:
	containers:
	- name: rbd-provisioner
	image: "quay.io/external_storage/rbd-provisioner:latest"
	env:
	- name: PROVISIONER_NAME
	value: ceph.com/rbd
	serviceAccount: rbd-provisioner
	---
	apiVersion: v1
	kind: Secret
	metadata:
	name: ceph-secret-admin
	namespace: kube-system
	data:
	# change the value
	# sudo ceph auth get-key client.admin | base64
	key: QVFDYlFubGNVZjVzRkJBQUtKanN1R1FuWUpvNlRweWZQT0E0d3c9PQ==
	type: kubernetes.io/rbd
	---
	apiVersion: v1
	kind: Secret
	metadata:
	name: ceph-secret-user
	namespace: kube-system
	data:
	# change the value
	# sudo ceph auth get-key client.admin | base64
	key: QVFDYlFubGNVZjVzRkJBQUtKanN1R1FuWUpvNlRweWZQT0E0d3c9PQ==
	type: kubernetes.io/rbd
	---
	kind: StorageClass
	apiVersion: storage.k8s.io/v1
	metadata:
	name: ceph-dynamic
	annotations:
	storageclass.beta.kubernetes.io/is-default-class: "true"
	provisioner: ceph.com/rbd
	parameters:
	monitors: 10.20.0.5:6789 # change it if needed.
	adminId: admin # change it if needed.
	adminSecretName: ceph-secret-admin # change it if needed.
	adminSecretNamespace: kube-system # change it if needed.
	pool: kube # change it if needed.
	userId: admin # change it if needed.
	userSecretName: ceph-secret-user # change it if needed.
	userSecretNamespace: kube-system # change it if needed.
	imageFormat: "2"
	imageFeatures: "layering"