19 Statefulset 部署有状态应用

发表于 | 分类于


使用 Statefulset 部署应用

1
2
3
4
5
6
7
8
# 区别:
1. 无状态应用 无需持久化数据，实例挂掉没有影响,重新拉起,用户无感知
2. 有状态应用 每个实例都需要有自己独立的持久化存储,实例挂掉，新的实例必须有就实例保持一致的状态和标识（相同的名称和网络标识）

# 有状态应用提供:
1. 稳定的网络标识:Statefulset创建的每个Pod都由一个从0开始的顺序索引，在pod的名称和主机名上,还有Pod的存储上
2. 提供稳定的专属存储：Statefulset的每个Pod都需要关联到不同的持久卷声明(pvc),也就是自己独自专属的持久卷
3. 需要再Pod中添加 卷声明模板(volumeClaimTemplates)
1
2
3
4
5
# 持久卷的创建和删除
1. 当一个声明(pvc)被删除,与之绑定的持久卷就会被回收或者删除,数据就会被丢失
2. 缩容 Statefulset 只会删除一个Pod，而声明(pvc)会遗留下来
3. 当你需要释放持久卷时,需要手动删除对应的持久卷声明
4. 缩容时不删除pvc，扩容时可以重新挂载上

创建应用和容器镜像

1
2
# 该应用接收到POST请求时,会把Body数据写入 /var/data/kubia.txt中
# 收到Get请求时，返回主机名和存储数据(文件中的内容)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
[root@k8s-master2 kubia-pet-image]# vim app.js 

const http = require('http');
const os = require('os');
const fs = require('fs');

const dataFile = "/var/data/kubia.txt";

function fileExists(file) {
  try {
    fs.statSync(file);
    return true;
  } catch (e) {
    return false;
  }
}

var handler = function(request, response) {
  if (request.method == 'POST') {
    var file = fs.createWriteStream(dataFile);
    file.on('open', function (fd) {
      request.pipe(file);
      console.log("New data has been received and stored.");
      response.writeHead(200);
      response.end("Data stored on pod " + os.hostname() + "\n");
    });
  } else {
    var data = fileExists(dataFile) ? fs.readFileSync(dataFile, 'utf8') : "No data posted yet";
    response.writeHead(200);
    response.write("You've hit " + os.hostname() + "\n");
    response.end("Data stored on this pod: " + data + "\n");
  }
};

var www = http.createServer(handler);
www.listen(8080);
1
2
3
4
5
6
7
8
[root@k8s-master2 kubia-pet-image]# vim Dockerfile 

FROM node:7
ADD app.js /app.js
ENTRYPOINT ["node", "app.js"]

[root@k8s-master2 kubia-pet-image]# docker build -t 172.31.228.68/game/kubia-pet .
[root@k8s-master2 kubia-pet-image]# docker push 172.31.228.68/game/kubia-pet

部署有状态应用

1
2
3
4
5
6
# 需要创建的对象
1. 存储数据的持久卷(如果不支持动态供给,还需要手动创建)
2. Statefulset 的 Service
3. Statefulset 控制器

# 对于每一个Pod，Statefulset 都会创建一个绑定到一个持久卷上的持久卷声明

手动创建 PV 持久化存储卷

1
2
3
4
5
6
7
8
# nfs共享存储里创建好目录
[root@k8s-master2 nfs]# mkdir -p pv-{a..c}

[root@k8s-master2 nfs]# ls -l
total 12
drwxr-xr-x 2 root root 4096 Mar 24 10:58 pv-a
drwxr-xr-x 2 root root 4096 Mar 24 10:58 pv-b
drwxr-xr-x 2 root root 4096 Mar 24 10:58 pv-c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
[root@k8s-master1 statefulset]# vim pv-nfs.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-a
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/nfs/pv-a
    server: 172.31.228.68

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-b
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/nfs/pv-b
    server: 172.31.228.68

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-c
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/nfs/pv-c
    server: 172.31.228.68


# 创建 pv-a pv-b pv-c 三个持久卷 
# 回收策略 Recycle 当卷的声明被释放后,空间会被回收再利用

[root@k8s-master1 statefulset]# kubectl create -f pv-nfs.yaml 
persistentvolume/pv-a created
persistentvolume/pv-b created
persistentvolume/pv-c created

[root@k8s-master1 statefulset]# kubectl get pv
NAME   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
pv-a   2Gi        RWX            Recycle          Available                                   6s
pv-b   2Gi        RWX            Recycle          Available                                   6s
pv-c   2Gi        RWX            Recycle          Available                                   6s

创建 Headless Service

1
2
3
# 常规 Service：一组POD的访问策略，提供负载均衡服务发现
# Headless Service: 不需要Cluster-IP，他会直接绑定到POD IP
# 没有IP 使用DNS保证网络唯一标识符 coredns
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[root@k8s-master1 statefulset]# vim kubia-service-headless.yaml

apiVersion: v1
kind: Service
metadata:
  name: kubia
spec:
  clusterIP: None
  selector:
    app: kubia
  ports:
  - name: http
    port: 80

[root@k8s-master1 statefulset]# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP   14d
kubia        ClusterIP   None         <none>        80/TCP    3s

创建 Statefulset

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
[root@k8s-master1 statefulset]# vim kubia-statefulset.yaml 

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kubia
spec:
  serviceName: kubia
  replicas: 2
  selector:
    matchLabels:
      app: kubia                        # Pod 标签 app: kubia
  template:
    metadata:
      labels:
        app: kubia
    spec:
      containers:
      - name: kubia
        image: 172.31.228.68/game/kubia-pet
        ports:
        - name: http
          containerPort: 8080
        volumeMounts:
        - name: data
          mountPath: /var/data           # 容器挂载目录
  volumeClaimTemplates:                  # 持久卷声明模板
  - metadata:
      name: data
    spec:
     # storageClassName: "managed-nfs-storage" 自动供给，本次是手动
      resources:
        requests:
          storage: 1Gi
      accessModes:
      - ReadWriteMany
1
2
3
# volumeClaimTemplates  持久卷声明模板
1. 定义1个 data 卷声明，根据这个模板为每个Pod都创建一个持久卷声明(pvc)
2. statefulset 创建Pod时,会自动将PVC添加到Pod描述中

查看 生成的有状态 Pod

1
2
3
4
5
6
7
8
[root@k8s-master1 statefulset]# kubectl create -f kubia-statefulset.yaml 
[root@k8s-master1 statefulset]# kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
kubia-0                                   1/1     Running   0          5s
kubia-1                                   1/1     Running   0          3s

1. 第二个Pod会在第一个Pod运行处于就绪状态后创建
2. 依次创建启动每个Pod，以免竞争
1
2
3
4
5
6
7
8
9
10
11
[root@k8s-master1 statefulset]# kubectl get pod kubia-0 -o yaml
...
    volumeMounts:
    - mountPath: /var/data           # 挂载点
      name: data
...
  volumes:                           # 创建的持久卷声明
  - name: data
    persistentVolumeClaim:
      claimName: data-kubia-0        # 名称 =  volumeClaimTemplates的name + pod name
...
1
2
3
4
5
6
7
8
9
10
11
# 查看 PVC
[root@k8s-master1 statefulset]# kubectl get pvc
NAME           STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-kubia-0   Bound    pv-c     2Gi        RWX                           48s
data-kubia-1   Bound    pv-a     2Gi        RWX                           47s

[root@k8s-master1 statefulset]# kubectl get pv
NAME   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                  STORAGECLASS   REASON   AGE
pv-a   2Gi        RWX            Recycle          Bound       default/data-kubia-1                           2m17s
pv-b   2Gi        RWX            Recycle          Available                                                  2m17s
pv-c   2Gi        RWX            Recycle          Bound       default/data-kubia-0                           2m17s

使用 Pod 测试数据

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# 通过API服务器与Pod通信
# 创建一个新代理
[root@k8s-master1 statefulset]# kubectl proxy
Starting to serve on 127.0.0.1:8001

[root@k8s-master1 ~]# curl localhost:8001/api/v1/namespaces/default/pods/kubia-0/proxy/
You've hit kubia-0
Data stored on this pod: No data posted yet

# 请求被正确收到 

# 发送一个POST请求
[root@k8s-master1 ~]# curl -X POST -d "Hey there! to kubia-0" localhost:8001/api/v1/namespaces/default/pods/kubia-0/proxy/
Data stored on pod kubia-0

# 再来Get请求看看能否查看数据
[root@k8s-master1 ~]# curl localhost:8001/api/v1/namespaces/default/pods/kubia-0/proxy/
You've hit kubia-0
Data stored on this pod: Hey there! to kubia-0

# 看看对应的存储里面的数据
# data-kubia-0   Bound    pv-c 
[root@k8s-master2 pv-c]# cat kubia.txt
Hey there! to kubia-0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# 测试下其他集群节点数据 kubia-1
# 没有数据正确
[root@k8s-master1 statefulset]# curl localhost:8001/api/v1/namespaces/default/pods/kubia-1/proxy/
You've hit kubia-1
Data stored on this pod: No data posted yet

# 插入数据到 kubia-1
[root@k8s-master1 statefulset]# curl -X POST -d "Hey there! to kubia-1" localhost:8001/api/v1/namespaces/default/pods/kubia-1/proxy/
Data stored on pod kubia-1

# 每个节点的存储都保存自己的数据
[root@k8s-master1 statefulset]# curl localhost:8001/api/v1/namespaces/default/pods/kubia-1/proxy/
You've hit kubia-1
Data stored on this pod: Hey there! to kubia-1

[root@k8s-master1 statefulset]# curl localhost:8001/api/v1/namespaces/default/pods/kubia-0/proxy/
You've hit kubia-0
Data stored on this pod: Hey there! to kubia-0

[root@k8s-master2 pv-a]# cat kubia.txt 
Hey there! to kubia-1

删除一个有状态Pod

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# 删除一个有状态Pod 查看重新调度的Pod 是否关联了之前的数据
[root@k8s-master1 statefulset]# kubectl delete pod kubia-0
pod "kubia-0" deleted

# 当Pod成功终止后,Statefulset会立即拉起一个新的具有相同名称的Pod，有可能在其他node上,Pod地址肯定是变化了
[root@k8s-master1 ~]# kubectl get pod -o wide
NAME                                      READY   STATUS        RESTARTS   AGE     IP             NODE          NOMINATED NODE   READINESS GATES
kubia-0                                   0/1     Terminating   0          3m      10.244.1.104   k8s-node2     <none>           <none>
kubia-1                                   1/1     Running       0          10m     10.244.2.83    k8s-master1   <none>           <none>
nfs-client-provisioner-56f4b98d47-v4nf6   1/1     Running       9          4d12h   10.244.0.78    k8s-node1     <none>           <none>

[root@k8s-master1 ~]# kubectl get pod -o wide
NAME                                      READY   STATUS              RESTARTS   AGE     IP            NODE          NOMINATED NODE   READINESS GATES
kubia-0                                   0/1     ContainerCreating   0          1s      <none>        k8s-node2     <none>           <none>
kubia-1                                   1/1     Running             0          10m     10.244.2.83   k8s-master1   <none>           <none>
nfs-client-provisioner-56f4b98d47-v4nf6   1/1     Running             9          4d12h   10.244.0.78   k8s-node1     <none>           <none>

[root@k8s-master1 ~]# kubectl get pod -o wide
NAME                                      READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE   READINESS GATES
kubia-0                                   1/1     Running   0          1s      10.244.1.105   k8s-node2     <none>           <none>
kubia-1                                   1/1     Running   0          10m     10.244.2.83    k8s-master1   <none>           <none>
nfs-client-provisioner-56f4b98d47-v4nf6   1/1     Running   9          4d12h   10.244.0.78    k8s-node1     <none>           <none>

# 新的Pod保留名称 主机名 存储
# pod的名称是被保留的,通过访问Pod来确认

[root@k8s-master1 statefulset]# curl localhost:8001/api/v1/namespaces/default/pods/kubia-0/proxy/
You've hit kubia-0
Data stored on this pod: Hey there! to kubia-0

# 说明重新拉起的Pod的主机名和持久化数据 与 之前的保持一致,即保证了服务状态,基本完全一致的新Pod

在Statefulset 中发现其他伙伴节点

SRV 记录

1
2
3
4
5
6
7
8
9
10
11
# 运行一个DNS查询工具 dig 命令 

[root@k8s-master1 ~]# kubectl run -it srvlookup --image=tutum/dnsutils --rm --restart=Never -- dig SRV kubia.default.svc.cluster.local
...
;; ANSWER SECTION:
kubia.default.svc.cluster.local. 5 IN	SRV	0 50 80 kubia-0.kubia.default.svc.cluster.local.
kubia.default.svc.cluster.local. 5 IN	SRV	0 50 80 kubia-1.kubia.default.svc.cluster.local.

;; ADDITIONAL SECTION:
kubia-0.kubia.default.svc.cluster.local. 5 IN A	10.244.1.106
kubia-1.kubia.default.svc.cluster.local. 5 IN A	10.244.2.86

通过DNS查找所有Pod节点 并获取数据

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
[root@k8s-master1 pub]# vim kubia-service-public.yaml

apiVersion: v1
kind: Service
metadata:
  name: kubia-public
spec:
  selector:
    app: kubia
  ports:
  - port: 80
    targetPort: 8080

# 通过API服务器访问集群内部服务
# 让请求随机分配到一个statefulset的pod上
[root@k8s-master1 pub]# kubectl create -f kubia-service-public.yaml 
service/kubia-public created

[root@k8s-master1 pub]# kubectl get svc
NAME           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes     ClusterIP   10.0.0.1     <none>        443/TCP   14d
kubia-public   ClusterIP   10.0.0.241   <none>        80/TCP    8s

[root@k8s-master1 pub]# curl localhost:8001/api/v1/namespaces/default/services/kubia-public/proxy/
You've hit kubia-1
Data stored on this pod: Hey there! to kubia-1

[root@k8s-master1 pub]# curl localhost:8001/api/v1/namespaces/default/services/kubia-public/proxy/
You've hit kubia-0
Data stored on this pod: Hey there! to kubia-0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
[root@k8s-master2 kubia-pet-peers-image]# cat app.js 
const http = require('http');
const os = require('os');
const fs = require('fs');
const dns = require('dns');

const dataFile = "/var/data/kubia.txt";
const serviceName = "kubia.default.svc.cluster.local";
const port = 8080;


function fileExists(file) {
  try {
    fs.statSync(file);
    return true;
  } catch (e) {
    return false;
  }
}

function httpGet(reqOptions, callback) {
  return http.get(reqOptions, function(response) {
    var body = '';
    response.on('data', function(d) { body += d; });
    response.on('end', function() { callback(body); });
  }).on('error', function(e) {
    callback("Error: " + e.message);
  });
}

var handler = function(request, response) {
  if (request.method == 'POST') {
    var file = fs.createWriteStream(dataFile);
    file.on('open', function (fd) {
      request.pipe(file);
      response.writeHead(200);
      response.end("Data stored on pod " + os.hostname() + "\n");
    });
  } else {
    response.writeHead(200);
    if (request.url == '/data') {
      var data = fileExists(dataFile) ? fs.readFileSync(dataFile, 'utf8') : "No data posted yet";
      response.end(data);
    } else {
      response.write("You've hit " + os.hostname() + "\n");
      response.write("Data stored in the cluster:\n");
      dns.resolveSrv(serviceName, function (err, addresses) {
        if (err) {
          response.end("Could not look up DNS SRV records: " + err);
          return;
        }
        var numResponses = 0;
        if (addresses.length == 0) {
          response.end("No peers discovered.");
        } else {
          addresses.forEach(function (item) {
            var requestOptions = {
              host: item.name,
              port: port,
              path: '/data'
            };
            httpGet(requestOptions, function (returnedData) {
              numResponses++;
              response.write("- " + item.name + ": " + returnedData + "\n");
              if (numResponses == addresses.length) {
                response.end();
              }
            });
          });
        }
      });
    }
  }
};

var www = http.createServer(handler);
www.listen(port);

更新 Statefulset

1
2
3
4
5
[root@k8s-master1 ~]# kubectl edit statefulset kubia
replicas: 3                                         # 更新副本个数
...
- image: 172.31.228.68/game/kubia-pet-peers         # 更新镜像
...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[root@k8s-master1 ~]# kubectl get pods -o wide
NAME                                      READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE   READINESS GATES
kubia-0                                   1/1     Running   0          5s      10.244.1.109   k8s-node2     <none>           <none>
kubia-1                                   1/1     Running   0          45s     10.244.2.87    k8s-master1   <none>           <none>
kubia-2                                   1/1     Running   0          80s     10.244.0.81    k8s-node1     <none>           <none>

# 新增的kubia-2  之前的连个Pod也被滚动更新

[root@k8s-master1 ~]# kubectl get pvc
NAME           STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-kubia-0   Bound    pv-c     2Gi        RWX                           3h13m
data-kubia-1   Bound    pv-a     2Gi        RWX                           3h13m
data-kubia-2   Bound    pv-b     2Gi        RWX                           101s

[root@k8s-master1 ~]# kubectl get pv
NAME   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   REASON   AGE
pv-a   2Gi        RWX            Recycle          Bound    default/data-kubia-1                           3h14m
pv-b   2Gi        RWX            Recycle          Bound    default/data-kubia-2                           3h14m
pv-c   2Gi        RWX            Recycle          Bound    default/data-kubia-0                           3h14m

测试集群数据

1
2
3
4
5
6
7
8
9
10
[root@k8s-master1 ~]# kubectl proxy
Starting to serve on 127.0.0.1:8001

# 写入到不同的pod中

[root@k8s-master1 pub]# curl -X POST -d "123" localhost:8001/api/v1/namespaces/default/services/kubia-public/proxy/
Data stored on pod kubia-2

[root@k8s-master1 pub]# curl -X POST -d "567" localhost:8001/api/v1/namespaces/default/services/kubia-public/proxy/
Data stored on pod kubia-0

通过 DNS 获取 POD IP

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
[root@k8s-master1 statefulset]# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP   14d
kubia        ClusterIP   None         <none>        80/TCP    3s

[root@k8s-master1 demo]# vim busybox.yaml 

apiVersion: v1
kind: Pod
metadata:
  name: dns-test
spec:
  containers:
  - name: busybox
    image: busybox:1.28.4
    args:
    - /bin/sh
    - -c
    - sleep 36000
  restartPolicy: Never

[root@k8s-master1 statefulset]# kubectl create -f busybox.yaml 

[root@k8s-master1 statefulset]# kubectl exec -it dns-test sh
/ #  nslookup kubia
Server:    10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name:      kubia
Address 1: 10.244.2.89 kubia-1.kubia.default.svc.cluster.local
Address 2: 10.244.1.111 kubia-0.kubia.default.svc.cluster.local
Address 3: 10.244.0.82 kubia-2.kubia.default.svc.cluster.local

/ #  nslookup kubia-1.kubia.default.svc.cluster.local
Server:    10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name:      kubia-1.kubia.default.svc.cluster.local
Address 1: 10.244.2.89 kubia-1.kubia.default.svc.cluster.local

StatefulSet 总结

1
2
3
4
5
6
7
8
9
10
11
StatefulSet 与 Deployment区别: 有身份的,有网络标识,主机名,固定的存储
身份三要素:
域名
主机名
存储(PVC)

statefulset 的 POD 名字 == 主机名
ClusterIP A记录: <service-name>.<namespace-name>.svc.cluster.local 
ClusterIP = Node A记录格式: <StatefulsetName-index>.<namespace-name>.<service-name>.svc.cluster.local
nginx-statefulset-0.nginx.default.svc.cluster.local
这个就是 statefulset 的 POD 的固定的访问地址 ，通过域名访问到后面的 POD ，通过DNS维持身份