ingress最佳方式对外暴露应用

1972字约7分钟

2024-09-29

Ingress是什么

NodePort存在的不足：

一个端口只能一个服务使用，端口需提前规划
只支持4层负载均衡

Ingress：Ingress公开了从集群外部到集群内服务的HTTP和HTTPS路由的规则集合，而具体实现流量路由则是由Ingress Controller负责。

Ingress：K8s中的一个抽象资源，给管理员提供一个暴露应用的入口定义方法
Ingress Controller：根据Ingress生成具体的路由规则，并对Pod负载均衡器

Ingress Controller

Ingress Controller有很多实现，我们这里采用官方维护的Nginx控制器。

项目地址：https://github.com/kubernetes/ingress-nginx

部署：kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx0.30.0/deploy/static/mandatory.yaml

注意事项：

镜像地址修改成国内的：lizhenliang/nginx-ingress-controller:0.30.0
将Ingress Controller暴露，一般使用宿主机网络（hostNetwork: true）或者使用NodePort

其他控制器：https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/

Ingress

部署deployment与serivce

vim web1-deploy-svc.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web1
  namespace: default
spec:
  replicas: 3 # Pod副本预期数量
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web # Pod副本的标签
    spec:
      containers:
      - name: web1
        image: nginx:1.15
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: web
  name: web1
spec:
  type: ClusterIP # 服务类型
  ports:
  - port: 80 # Service端口
    protocol: TCP # 协议
    targetPort: 80 # 容器端口
   # nodePort: 30009    #nodeport暴露的端口
  selector:
    app: web # 指定关联Pod的标签

访问

kubectl apply -f web1-deploy-svc.yaml 
kubectl get svc
curl 10.99.227.165

部署ingress（基于域名方式访问）

vim ingress-web1.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web1
spec:
  rules:
  - host: web1.ctnrs.com        #域名
    http:
      paths:
      - path: /                  #类似nginx配置文件的location /访问路径
        pathType: Prefix
        backend:
          service:
            name: web1            #service的名字
            port:
              number: 80          #ClusterIP的端口

浏览器访问web1.ctnrs.com

kubectl apply -f ingress-web1.yaml 
kubectl get ingress

测试：本地电脑绑定hosts记录对应ingress里面配置的域名

例： <Ingress Controller Pod所在Node IP> foo.bar.com

kubectl get pod -n ingress-nginx -o wide

Ingress：基于URI路由多个服务

vim web2-deploy-svc.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web2
  namespace: default
spec:
  replicas: 1 # Pod副本预期数量
  selector:
    matchLabels:
      app: web2
  template:
    metadata:
      labels:
        app: web2 # Pod副本的标签
    spec:
      containers:
      - name: nginx
        image: nginx:1.15
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: web2
  name: web2
spec:
  type: ClusterIP # 服务类型
  ports:
  - port: 80 # Service端口
    protocol: TCP # 协议
    targetPort: 80 # 容器端口
   # nodePort: 30009    #nodeport暴露的端口
  selector:
    app: web2 # 指定关联Pod的标签

vim web22-deploy-svc.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web22
  namespace: default
spec:
  replicas: 1 # Pod副本预期数量
  selector:
    matchLabels:
      app: web22
  template:
    metadata:
      labels:
        app: web22 # Pod副本的标签
    spec:
      containers:
      - name: web22
        image: lizhenliang/java-demo
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: web22
  name: web22
spec:
  type: ClusterIP # 服务类型
  ports:
  - port: 8080 # Service端口
    protocol: TCP # 协议
    targetPort: 8080 # 容器端口
   # nodePort: 30009    #nodeport暴露的端口
  selector:
    app: web22 # 指定关联Pod的标签

vim ingress-web2.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web2
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/server-snippet: |
      rewrite ^/css/(.*)$ /bar/css/$1 redirect;
      rewrite ^/images/(.*)$ /bar/images/$1 redirect;
      rewrite ^/js/(.*)$ /bar/js/$1 redirect;

spec:
  rules:
  - host: web2.ctnrs.com        #域名
    http:
      paths:
      - path: /foo                  #类似nginx配置文件的location /访问路径
        pathType: Prefix
        backend:
          service:
            name: web2            #service的名字
            port:
              number: 80          #ClusterIP的端口
      - path: /bar(/|$)(.*)                  #类似nginx配置文件的location /访问路径
        pathType: Prefix
        backend:
          service:
            name: web22            #service的名字
            port:
              number: 8080          #ClusterIP的端口

kubectl apply -f web2-deploy-svc.yaml 
kubectl apply -f web22-deploy-svc.yaml 
kubectl apply -f ingress-web2.yaml 
kubectl get pod 
kubectl get svc
kubectl get ingress

访问

http://web2.ctnrs.com/bar
http://web2.ctnrs.com/foo

Ingress ：基于名称的虚拟主机

vim web3-deploy-svc.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web3
  namespace: default
spec:
  replicas: 3 # Pod副本预期数量
  selector:
    matchLabels:
      app: web-tomcat
  template:
    metadata:
      labels:
        app: web-tomcat # Pod副本的标签
    spec:
      containers:
      - name: java-demo
        image: lizhenliang/java-demo
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: web-tomcat
  name: web3
spec:
  type: ClusterIP # 服务类型
  ports:
  - port: 88 # Service端口
    protocol: TCP # 协议
    targetPort: 8080 # 容器端口
   # nodePort: 30009    #nodeport暴露的端口
  selector:
    app: web-tomcat # 指定关联Pod的标签

vim ingress-web3.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web3
spec:
  rules:
  - host: web3-1.ctnrs.com        #域名
    http:
      paths:
      - path: /                  #类似nginx配置文件的location /访问路径
        pathType: Prefix
        backend:
          service:
            name: web1            #service的名字
            port:
              number: 80          #ClusterIP的端口
  - host: web3-2.ctnrs.com        #域名
    http:
      paths:
      - path: /                  #类似nginx配置文件的location /访问路径
        pathType: Prefix
        backend:
          service:
            name: web3            #service的名字
            port:
              number: 88          #ClusterIP的端口

kubectl apply -f web3-deploy-svc.yaml 
kubectl apply -f ingress-web3.yaml 
kubectl get ingress

访问

web3-1.ctnrs.com
web3-2.ctnrs.com

Ingress：HTTPS

配置HTTPS步骤：

1、准备域名证书文件（来自：openssl/cfssl工具自签或者权威机构颁发）

vim cfssl.sh

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl*
mv cfssl_linux-amd64 /usr/bin/cfssl
mv cfssljson_linux-amd64 /usr/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

vim certs.sh

cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json <<EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

cat > web4.ctnrs.com-csr.json <<EOF
{
  "CN": "web4.ctnrs.com",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes web4.ctnrs.com-csr.json | cfssljson -bare web4.ctnrs.com

生成证书

bash cfssl.sh
bash certs.sh

2、将证书文件保存到Secret

kubectl create secret tls web4-ctnrs-com --cert=ssl/web4.ctnrs.com.pem --key=ssl/web4.ctnrs.com-key.pem
kubectl get secrets

3、Ingress规则配置tls

vim ingress-web4-https.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web4
spec:
  tls:
  - hosts:
    - web4.ctnrs.com                 #自签证书对应的域名
    secretName: web4-ctnrs-com       #Secret的名字
  rules:
  - host: web4.ctnrs.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web1
            port:
              number: 80

kubectl apply -f ingress-web4-https.yaml 
kubectl get ingress

配置本地hosts文件解析

192.168.1.12      web1.ctnrs.com  web4.ctnrs.com  
192.168.1.13      web1.ctnrs.com  web4.ctnrs.com

访问

Ingress：个性化配置

示例1：设置代理超时参数

vim ingress-annotations.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: annotations
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
spec:
  rules:
  - host: annotations.ctnrs.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web1
            port:
              number: 80

kubectl apply -f ingress-annotations.yaml 
kubectl get ingress
kubectl get pod -n ingress-nginx 
kubectl exec -it nginx-ingress-controller-qj4vg bash -n ingress-nginx 
$vi /etc/nginx/nginx.conf

查看是否有代理的配置

示例2：设置客户端上传文件大小

vim ingress-annotations.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: annotations
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
spec:
  rules:
  - host: annotations.ctnrs.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web1
            port:
              number: 80

kubectl apply -f ingress-annotations.yaml 
kubectl get pod -n ingress-nginx 
kubectl exec -it nginx-ingress-controller-qj4vg bash -n ingress-nginx 
$vi /etc/nginx/nginx.conf

查看annotations.ctnrs.com域名下有没有相关的配置

示例3：重定向

nginx.ingress.kubernetes.io/rewrite-target: https://www.baidu.com

示例4：自定义规则

    nginx.ingress.kubernetes.io/server-snippet: |
      if ($http_user_agent ~* '(Android|iPhone)') {
        rewrite ^/(.*) http://m.baidu.com break;
      }

vim web5-deploy-svc.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web5
  namespace: default
spec:
  replicas: 1 # Pod副本预期数量
  selector:
    matchLabels:
      app: web5
  template:
    metadata:
      labels:
        app: web5 # Pod副本的标签
    spec:
      containers:
      - name: web5-java-demo
        image: lizhenliang/java-demo
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: web5
  name: web5
spec:
  type: ClusterIP # 服务类型
  ports:
  - port: 8888 # Service端口
    protocol: TCP # 协议
    targetPort: 8080 # 容器端口
   # nodePort: 30009    #nodeport暴露的端口
  selector:
    app: web5 # 指定关联Pod的标签

vim ingress-web5.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web5
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
      if ($http_user_agent ~* '(Android|iPhone)') {
        rewrite ^/(.*) http://m.baidu.com break;
      }
spec:
  rules:
  - host: web5.ctnrs.com        #域名
    http:
      paths:
      - path: /                  #类似nginx配置文件的location /访问路径
        pathType: Prefix
        backend:
          service:
            name: web5        #service的名字
            port:
              number: 8888          #ClusterIP的端口

验证访问

kubectl apply -f web5-deploy-svc.yaml 
kubectl apply -f ingress-web5.yaml 
kubectl get ingress

电脑端访问：http://web5.ctnrs.com/

使用电脑火狐浏览器模仿手机端访问跳转到百度页面

注：上面这些配置都是针对Nginx Server块生效

更多使用方法：https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md

Ingress Controller

Ingress Contronler怎么工作的？

Ingress Contronler通过与 Kubernetes API 交互，动态的去感知集群中 Ingress 规则变化，然后读取它，按照自定义的规则，规则就是写明了哪个域名对应哪个service，生成一段 Nginx 配置，应用到管理的 Nginx服务，然后热加载生效。以此来达到Nginx负载均衡器配置及动态更新的问题。

流程包流程：客户端 ->Ingress Controller（nginx） -> 分布在各节点Pod

Ingress Controller高可用方案

一般Ingress Controller会以DaemonSet+nodeSelector部署到几台特定 Node，然后将这几台挂载到公网负载均衡器对外提供服务。