简述

需求：OpenStack endpoint 统一格式为 https + ip

环境信息

  • 系统：CentOS Linux release 7.7.1908 (Core)

  • Kubernetes版本：v1.19.0

  • OpenStack版本：Train

  • 部署方式：StarlingX Armada部署

网络架构

Kubernetes上部署的OpenStack，通过以下3中方式提供服务：

  • nodePort：OpenStack Horizon服务使用该方式。将端口映射至宿主机的端口，可通过宿主机ip + 端口方式访问。
  • ingress：OpenStack admin和public类型的endpoint使用该方式。
  • service：OpenStack internal类型的endpoint使用该方式。

以上3种方式的网络，升级为HTTPS方案分别如下：

  • nodePort：通过HAProxy进行HTTPS的证书验证和卸载，转发至后端nodePort端口

  • ingress：ingress运行在7层，支持配置HTTPS，在ingress增加tls模块即可

  • service：service运行在4层，不支持配置HTTPS，需要上层增加ingress

这里根据需求，所有服务改为nodePort方式，通过HAProxy进行HTTPS证书验证和卸载

证书卸载

ingress方式配置HTTPS

ingress对应的yaml，增加tls模块内容，即可开启HTTPS。

Kubernetes官方提供的ingress TLS的配置参考如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tls-example-ingress
spec:
  tls:
  - hosts:
      - https-example.foo.com
    secretName: testsecret-tls
  rules:
  - host: https-example.foo.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: service1
            port:
              number: 80

步骤

TLS证书

1. 制作证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
#!/bin/bash
certificates_dir=/tmp/certificates
service_name=inspur

# 创建目录
mkdir -p ${certificates_dir}/private

# 配置证书参数
# 认证多域名，增加alt_names即可
cat > ${certificates_dir}/${service_name}-openssl.cnf << EOF
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req

[req_distinguished_name]
countryName = CN
stateOrProvinceName = Xian
localityName = Xian
organizationalUnitName = Inspur
commonName = ${service_name}

[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = *.openstack.svc.cluster.local
DNS.2 = *.openstack
IP.1 = 111.111.111.88
EOF

# Creating Key
openssl genrsa -out ${certificates_dir}/private/${service_name}.key
chmod 0600 ${certificates_dir}/private/${service_name}.key

# Creating Server Certificate
openssl req -new -nodes -sha256 -x509 \
    -subj "/C=CN/ST=Xian/L=Xian/O=Inspur/CN=${service_name}" \
    -config ${certificates_dir}/${service_name}-openssl.cnf \
    -days 3650 \
    -extensions v3_req \
    -key ${certificates_dir}/private/${service_name}.key \
    -out ${certificates_dir}/private/${service_name}.crt

# Creating CA Certificate File
cp ${certificates_dir}/private/${service_name}.crt ${certificates_dir}/${service_name}-ca.crt

# Creating Server PEM File
cat ${certificates_dir}/private/${service_name}.crt ${certificates_dir}/private/${service_name}.key \
    | tee ${certificates_dir}/${service_name}.pem

openrc.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Clear any old environment that may conflict.
for key in $( set | awk '{FS="="}  /^OS_/ {print $1}' ); do unset $key ; done
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_AUTH_URL=https://111.111.111.80:35357/v3
export OS_CACERT=/etc/kolla/certificates/haproxy-ca.crt    # CA Certificate File
export OS_INTERFACE=internal
export OS_ENDPOINT_TYPE=internalURL
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne
export OS_AUTH_PLUGIN=password

Kubernetes管理证书

1. 创建Secret资源

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# 使用Secret方式上传证书至Kubernetes，挂载给pod即可使用该证书文件

# Creating Secret方式
kubectl create secret tls -n openstack tls-secret \
    --cert=/tmp/certificates/private/inspur.crt \
    --key=/tmp/certificates/private/inspur.key

# Secret yaml方式
cat > tls-secret.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: tls-secret
  namespace: openstack
data:
  tls.crt: < base64 encoded cert >
  tls.key: < base64 encoded key >
type: kubernetes.io/tls
EOF

kubectl create -f tls-secret.yaml

2. Pod使用证书

更新 Deployment 中，volumes 和 volumeMounts，设置secret和挂载目录。

Kubernetes官方提供的Pod的配置参考如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apiVersion: v1
kind: Pod
metadata:
  name: secret-test-pod
spec:
  containers:
    - name: test-container
      image: nginx
      volumeMounts:
        # name must match the volume name below
        - name: secret-volume
          mountPath: /etc/secret-volume
  # The secret data is exposed to Containers in the Pod through a Volume.
  volumes:
    - name: secret-volume
      secret:
        secretName: test-secret

HAProxy基础配置

haproxy.cfg 中，增加如下内容：

1
2
3
4
global
    ssl-default-bind-ciphers DEFAULT:!MEDIUM:!3DES
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11
    tune.ssl.default-dh-param 4096

Keystone配置证书

  1. 使用http协议，先创建 https endpoint
1
2
3
openstack endpoint create --region RegionOne identity public https://111.111.111.88:32500/v3
openstack endpoint create --region RegionOne identity internal https://111.111.111.88:32500/v3
openstack endpoint create --region RegionOne identity admin https://111.111.111.88:32500/v3

  1. 删除http endpoint。

    openrc.sh中 OS_INTERFACE 字段和 OS_ENDPOINT_TYPE 替换为admin类型可以避免出错。

  2. 更新haproxy配置，增加keystone https认证，重启haproxy。此时keystone的https端口正常响应

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#---------------------------------------------------------------------
# Keystone
#---------------------------------------------------------------------
frontend keystone_front
    mode http
    http-request del-header X-Forwarded-Proto
    option httplog
    option forwardfor
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    bind 111.111.111.88:32500 ssl crt /tmp/certificates/inspur.pem
    default_backend keystone_back

backend keystone_back
    mode http
    server node01 111.111.111.89:30500 check inter 2000 rise 2 fall 5
  1. 验证
1
openstack endpoint list

Glance配置证书

  1. 更新haproxy配置，增加glance https认证，重启haproxy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#---------------------------------------------------------------------
# Glance
#---------------------------------------------------------------------
frontend glance_api_front
    mode http
    http-request del-header X-Forwarded-Proto
    timeout client 6h
    option httplog
    option forwardfor
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    bind 111.111.111.88:32092 ssl crt /tmp/certificates/inspur.pem
    default_backend glance_api_back

backend glance_api_back
    mode http
    timeout server 6h
    server node01 111.111.111.89:30092 check inter 2000 rise 2 fall 5
  1. 在Kubernetes界面中，更新glance服务配置文件：Secret >> glance-etc >> glance-api.conf
1
2
3
4
5
6
7
[keystone_authtoken]
auth_uri = https://111.111.111.88:32500/v3
auth_url = https://111.111.111.88:32500/v3

# 配置pod中证书挂载地址
cafile = /etc/tls/tls.crt
insecure = true
  1. 更新Secret后，需要重建pod，以应用新的配置。参考命令：
1
2
# 删除pod后会自动重建
kubectl delete pod -n openstack < glance-api pod name >
  1. 更新Glance endpoint，替换为https地址
  2. 验证
1
openstack image list

OpenStack其他服务

按照上述Glance流程，参考文档【OpenStack部署：HTTPS】修改其余服务设置即可

参考文档