09 k8s Ingress 与外界连通

发表于 | 分类于


Ingress Controller 部署

  1. Ingress 全局负载均衡，7层负载均衡(域名url), nodeport只支持4层(端口)
  2. 用户 -> Ingress Controller(Node) -> Pod
  3. 他会使用宿主机网络的 80和443端口,要确保node这俩个端口不要被占用
1
2
3
1. 通过service相关联
2. 通过Ingress Controller实现Pod的负载均衡
3. 支持TCP/UDP 4层和HTTP 7层

部署

1
2
部署文档：
https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/index.md
1
2
3
注意事项：
1. 镜像地址修改成国内的：lizhenliang/nginx-ingress-controller:0.20.0 
2. 使用宿主机网络：hostNetwork: true
1
2
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
# 可以先下载到本地
1
[root@k8s-master demo2]# kubectl create -f mandatory.yaml

[root@k8s-master1 ~]# kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
nginx-ingress-controller-8nbng 1/1 Running 12 6d8h
nginx-ingress-controller-dksf2 1/1 Running 10 6d8h
nginx-ingress-controller-rp5t2 1/1 Running 0 2m32s

1
2
[root@k8s-node1 ~]# netstat -antp|grep 80
[root@k8s-node1 ~]# netstat -antp|grep 443

Ingress 配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[root@k8s-master1 demo]# vim app01-ingress.yaml 

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: app01-ingress
  # 注解
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  # 域名
  - host: foo.bar.com
    http:
      paths:
      - path: /
        backend:
          # 转发到哪个 service
          serviceName: app01
          # 对应 clusterip 端口
          servicePort: 80
1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@k8s-master demo2]# kubectl apply -f ingress01.yaml 

[root@k8s-master1 demo]# kubectl get pods,svc,ing
NAME                         READY   STATUS    RESTARTS   AGE
pod/app01-64d7b49995-4fdn5   1/1     Running   0          5m17s
pod/app01-64d7b49995-6rnsp   1/1     Running   0          5m35s
pod/app01-64d7b49995-d4qsx   1/1     Running   0          5m17s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
service/app01        NodePort    10.0.0.225   <none>        80:31000/TCP   4m56s
service/kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP        6d8h

NAME                               HOSTS         ADDRESS   PORTS   AGE
ingress.extensions/app01-ingress   foo.bar.com             80      56s
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 测试增加域名解析
# 不添加hosts的话 需要将域名接下到 部署控制器的 node节点上
# 实际工作环境 就是将 域名 解析到 NodeIP上
[root@k8s-master1 demo]# cat /etc/hosts
39.106.100.108 foo.bar.com 
39.106.168.181 foo.bar.com 
60.205.217.112 foo.bar.com

[root@k8s-master1 demo]# curl -I foo.bar.com
HTTP/1.1 200 OK
Server: nginx/1.15.5
Date: Mon, 23 Dec 2019 09:28:42 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Tue, 13 Aug 2019 10:05:00 GMT
ETag: "5d528b4c-264"
Accept-Ranges: bytes

深入的去看看 ingress-nginx

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@k8s-master1 demo]# kubectl get pods -n ingress-nginx -o wide
NAME                             READY   STATUS    RESTARTS   AGE    IP              NODE          NOMINATED NODE   READINESS GATES
nginx-ingress-controller-8nbng   1/1     Running   12         6d8h   172.17.70.254   k8s-node2     <none>           <none>
nginx-ingress-controller-dksf2   1/1     Running   10         6d8h   172.17.70.251   k8s-master1   <none>           <none>
nginx-ingress-controller-rp5t2   1/1     Running   0          28m    172.17.70.253   k8s-node1     <none>           <none>

[root@k8s-master1 demo]# kubectl exec -it nginx-ingress-controller-8nbng bash -n ingress-nginx
www-data@k8s-node2:/etc/nginx$ 

# 运行着 nginx 
www-data@k8s-node2:/etc/nginx$ ps -ef|grep nginx

www-data@k8s-node2:/etc/nginx$ nginx -v
nginx version: nginx/1.15.5

# ginx-ingress-controller 进程 实时监听apiserver里所有svc，如果有更新就刷新 nginx里面的负载

Ingress HTTPS

  1. 域名 -> service 端口 -> pod
  2. https 需要定义 tls数字证书
  3. https 需要启动 crt和key
  4. 权威结构购买数字域名证书
  5. 自签证书 不受浏览器信任
  6. 证书保存到 secret里面 然后保存到 secretName下

自签证书 并 引用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
# 1. 生成证书
[root@k8s-master1 cert]# vim certs.sh 

# 生成 CA 证书
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json <<EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

# 域名证书 
cat > sslexample.foo.com-csr.json <<EOF
{
  "CN": "sslexample.foo.com",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing"
    }
  ]
}
EOF

# 生成 sslexample.foo.com 域名证书 crt和key 
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes sslexample.foo.com-csr.json | cfssljson -bare sslexample.foo.com

[root@k8s-master1 cert]# sh certs.sh 
[root@k8s-master1 cert]# ls sslexample.foo.com*.pem
sslexample.foo.com-key.pem  sslexample.foo.com.pem
1
2
3
4
5
6
7
8
9
# 保存到 secret 
[root@k8s-master1 cert]# kubectl create secret tls sslexample-foo-com --cert=sslexample.foo.com.pem --key=sslexample.foo.com-key.pem
secret/sslexample-foo-com created

[root@k8s-master1 cert]# kubectl get secret
NAME                                 TYPE                                  DATA   AGE
default-token-l2x75                  kubernetes.io/service-account-token   3      7d2h
nfs-client-provisioner-token-75bdm   kubernetes.io/service-account-token   3      6d2h
sslexample-foo-com                   kubernetes.io/tls                     2      16s
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# 创建 ingress

[root@k8s-master1 demo]# vim app01-ingress-https.yaml 

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: tls-example-ingress
spec:
  tls:
  - hosts:
    # 域名
    - sslexample.foo.com
    # secret 名称 保存的证书配置
    secretName: sslexample-foo-com
  rules:
    # 域名
    - host: sslexample.foo.com
      http:
        paths:
        - path: /
          backend:
            # service name
            serviceName: app01
            # service port 
            servicePort: 80
1
2
3
4
5
6
7
8
9
10
# 启动 ingress-https
[root@k8s-master1 demo]# kubectl delete ing app01-ingress
ingress.extensions "app01-ingress" deleted

[root@k8s-master1 demo]# kubectl apply -f app01-ingress-https.yaml 
ingress.networking.k8s.io/tls-example-ingress created

[root@k8s-master1 demo]# kubectl get ing
NAME                  HOSTS                ADDRESS   PORTS     AGE
tls-example-ingress   sslexample.foo.com             80, 443   4s
1
2
3
4
5
6
7
8
9
10
# 测试
[root@k8s-master1 demo]# cat /etc/hosts

39.106.100.108 foo.bar.com sslexample.foo.com
39.106.168.181 foo.bar.com sslexample.foo.com
60.205.217.112 foo.bar.com sslexample.foo.com

# 更换个页面测试查看 
[root@k8s-master1 demo]# kubectl exec -it app01-64d7b49995-d4qsx bash
root@app01-64d7b49995-d4qsx:/# echo 789 > /usr/share/nginx/html/index.html

使用默认的 Ingress 证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# k8s 会默认自签证书 
# 我们就不用自签证书 创建secert 

[root@k8s-master1 demo]# cp app01-ingress-https.yaml app01-ingress-default-https.yaml 

[root@k8s-master1 demo]# vim app01-ingress-default-https.yaml 

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: tls-example-ingress
spec:
  tls:
  - hosts:
    # 域名
    - sslexample.foo.com
    # secret  没有这个secret 
    secretName: sslexample-foo-com-100
  rules:
    # 域名
    - host: sslexample.foo.com
      http:
        paths:
        - path: /
          backend:
            # service name
            serviceName: app01
            # service port 
            servicePort: 80
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# 启动
[root@k8s-master1 demo]# kubectl delete -f app01-ingress-https.yaml 
ingress.networking.k8s.io "tls-example-ingress" deleted

[root@k8s-master1 demo]# kubectl apply -f app01-ingress-default-https.yaml 
ingress.networking.k8s.io/tls-example-ingress created

# 并没有创建新的 secret 
[root@k8s-master1 demo]# kubectl get secret
NAME                                 TYPE                                  DATA   AGE
default-token-l2x75                  kubernetes.io/service-account-token   3      7d2h
nfs-client-provisioner-token-75bdm   kubernetes.io/service-account-token   3      6d3h
sslexample-foo-com                   kubernetes.io/tls                     2      20m

[root@k8s-master1 demo]# kubectl get ing
NAME                  HOSTS                  ADDRESS   PORTS     AGE
tls-example-ingress   mysslexample.foo.com             80, 443   34s

[root@k8s-master1 demo]# cat /etc/hosts
39.106.100.108 foo.bar.com mysslexample.foo.com
39.106.168.181 foo.bar.com mysslexample.foo.com
60.205.217.112 foo.bar.com mysslexample.foo.com
1
# 测试 k8s自己颁发的证书

总结

  1. Ingress 支持 四层、七层负载均衡转发
  2. 支持自定义service访问策略
  3. 只支持基于域名的网站访问
  4. 支持TLS
  5. 用户 -> 域名 -> 负载均衡 -> 固定NODE的Ingress Controller(node) -> Pod\
  6. 部署多个 nginx-ingress-controller
  7. DaemonSet 方式部署到 多个指定标签的node节点上