简述 需求: 需要为openstack所有服务更新为https方式访问
方案: 使用HAProxy进行负载均衡和https证书卸载,通过HAProxy frontend进行https证书验证,验证通过后,转发至backend的后端http server。openstack各服务之间,通过https进行访问,需要配置各个服务的CA证书,同时更新keystone的endpoint
环境
系统:CentOS Linux release 7.7.1908 (Core)
OpenStack版本:Train
kolla-ansible版本:9.2.0
Docker版本:docker-ce-18.09.6-3.el7.x86_64
OpenStack部署方式:kolla-ansible部署(OpenStack服务运行在Docker上)
OpenStack部署规模:control节点1台,compute节点3台
步骤 TLS证书 生成证书
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 # 使用kolla生成证书 kolla-ansible certificates # openssl-kolla.cnf [req] distinguished_name = req_distinguished_name req_extensions = v3_req [req_distinguished_name] countryName = US stateOrProvinceName = NC localityName = RTP organizationalUnitName = kolla commonName = 111.111.111.80 [v3_req] subjectAltName = @alt_names [alt_names] IP.1 = 111.111.111.80
分发证书
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 # haproxy cp /etc/kolla/certificates/haproxy.pem /etc/kolla/haproxy/ # glance cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/glance-api/ # cinder cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/cinder-api/ cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/cinder-backup/ cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/cinder-scheduler/ cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/cinder-volume/ scp /etc/kolla/certificates/haproxy-ca.crt node02:/etc/kolla/cinder-backup/ scp /etc/kolla/certificates/haproxy-ca.crt node03:/etc/kolla/cinder-backup/ scp /etc/kolla/certificates/haproxy-ca.crt node02:/etc/kolla/cinder-volume/ scp /etc/kolla/certificates/haproxy-ca.crt node03:/etc/kolla/cinder-volume/ # neutron cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/neutron-server/ cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/neutron-openvswitch-agent/ cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/neutron-metadata-agent/ cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/neutron-l3-agent/ cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/neutron-dhcp-agent/ scp /etc/kolla/certificates/haproxy-ca.crt node02:/etc/kolla/neutron-openvswitch-agent/ scp /etc/kolla/certificates/haproxy-ca.crt node03:/etc/kolla/neutron-openvswitch-agent/ # placement cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/placement-api/ # nova cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/nova-api/ cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/nova-api-bootstrap/ cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/nova-cell-bootstrap/ cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/nova-compute/ cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/nova-conductor/ cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/nova-novncproxy/ cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/nova-scheduler/ scp /etc/kolla/certificates/haproxy-ca.crt node02:/etc/kolla/nova-compute/ scp /etc/kolla/certificates/haproxy-ca.crt node03:/etc/kolla/nova-compute/ # horizon cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/horizon/
admin-openrc.sh
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # Clear any old environment that may conflict. for key in $( set | awk '{FS="="} /^OS_/ {print $1}' ); do unset $key ; done export OS_PROJECT_DOMAIN_NAME=Default export OS_USER_DOMAIN_NAME=Default export OS_PROJECT_NAME=admin export OS_TENANT_NAME=admin export OS_USERNAME=admin export OS_PASSWORD=admin export OS_AUTH_URL=https://111.111.111.80:35357/v3 export OS_CACERT=/etc/kolla/certificates/haproxy-ca.crt export OS_INTERFACE=internal export OS_ENDPOINT_TYPE=internalURL export OS_IDENTITY_API_VERSION=3 export OS_REGION_NAME=RegionOne export OS_AUTH_PLUGIN=password
Keystone配置证书
使用http协议,先创建 https endpoint
1 2 3 openstack endpoint create --region RegionOne identity public https://111.111.111.80:5000 openstack endpoint create --region RegionOne identity internal https://111.111.111.80:5000 openstack endpoint create --region RegionOne identity admin https://111.111.111.80:35357
删除http endpoint
1 2 3 4 5 6 7 8 9 # 如果环境变量OS_INTERFACE使用的是internal,则删除掉internal http的endpoint之后,删除其他endpoint会报错,属于正常现象。 # 原因:https尚未在haproxy上配置,请求https端口会报错 # 解决:需要执行第3步,更新haproxy。配置https证书验证后,可正常执行其余操作 root@node01: kolla # openstack endpoint delete 9200b3fcad1e42e8a8d8277e8c7ba71c Failed to contact the endpoint at https://111.111.111.80:5000 for discovery. Fallback to using that endpoint as the base url. Failed to contact the endpoint at https://111.111.111.80:5000 for discovery. Fallback to using that endpoint as the base url. Failed to contact the endpoint at https://111.111.111.80:5000 for discovery. Fallback to using that endpoint as the base url. Failed to delete endpoint with ID '9200b3fcad1e42e8a8d8277e8c7ba71c': SSL exception connecting to https://111.111.111.80:5000/endpoints?: HTTPSConnectionPool(host='111.111.111.80', port=5000): Max retries exceeded with url: /endpoints (Caused by SSLError(SSLError(1, u'[SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:618)'),)) 1 of 1 endpoints failed to delete.
更新haproxy中 keystone.cfg
配置,重启haproxy。此时keystone的https端口正常响应
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 frontend keystone_internal_front mode http http-request del-header X-Forwarded-Proto option httplog option forwardfor http-request set-header X-Forwarded-Proto https if { ssl_fc } bind 111.111.111.80:5000 ssl crt /var/lib/kolla/config_files/haproxy.pem default_backend keystone_internal_back backend keystone_internal_back mode http server node01 111.111.111.81:5000 check inter 2000 rise 2 fall 5 frontend keystone_admin_front mode http option forwardfor http-request del-header X-Forwarded-Proto option httplog http-request set-header X-Forwarded-Proto https if { ssl_fc } bind 111.111.111.80:35357 ssl crt /var/lib/kolla/config_files/haproxy.pem default_backend keystone_admin_back backend keystone_admin_back mode http server node01 111.111.111.81:35357 check inter 2000 rise 2 fall 5
继续删除其余未删除的keystone http endpoint
更新其他组件endpoint协议,切换为https
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 # !/bin/bash openstack endpoint list | sed '1,3d' | sed '$d' > /tmp/endpoint while read -r line do echo $line id=`echo $line | awk '{print $2}'` region=`echo $line | awk '{print $4}'` service_name=`echo $line | awk '{print $6}'` service_type=`echo $line | awk '{print $8}'` interface=`echo $line | awk '{print $12}'` url=`echo $line | awk '{print $14}'` if [ $(echo $url | awk -F : '{print $1}') == 'http' ]; then openstack endpoint delete $id openstack endpoint create --region $region \ $service_type $interface \ $(echo $url | sed 's/http/https/g') fi done < /tmp/endpoint
验证
Glance配置证书
更新haproxy中 glance-api.cfg
配置,重启haproxy
1 2 3 4 5 6 7 8 9 10 11 12 13 14 frontend glance_api_front mode http http-request del-header X-Forwarded-Proto timeout client 6h option httplog option forwardfor http-request set-header X-Forwarded-Proto https if { ssl_fc } bind 111.111.111.80:9292 ssl crt /var/lib/kolla/config_files/haproxy.pem default_backend glance_api_back backend glance_api_back mode http timeout server 6h server node01 111.111.111.81:9292 check inter 2000 rise 2 fall 5
更新glance服务配置文件:glance-api.conf
,重启glance服务
1 2 3 4 5 [keystone_authtoken] www_authenticate_uri = https://111.111.111.80:5000 auth_url = https://111.111.111.80:35357 cafile = /var/lib/kolla/config_files/haproxy-ca.crt insecure = true
验证
Cinder配置证书
更新haproxy中 cinder-api.cfg
配置,重启haproxy
1 2 3 4 5 6 7 8 9 10 11 12 frontend cinder_api_front mode http http-request del-header X-Forwarded-Proto option httplog option forwardfor http-request set-header X-Forwarded-Proto https if { ssl_fc } bind 111.111.111.80:8776 ssl crt /var/lib/kolla/config_files/haproxy.pem default_backend cinder_api_back backend cinder_api_back mode http server node01 111.111.111.81:8776 check inter 2000 rise 2 fall 5
更新服务配置文件:cinder.conf
。同步所有节点配置文件,重启cinder相关服务
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 [DEFAULT] glance_api_servers = https://111.111.111.80:9292 glance_api_insecure = true glance_ca_certificates_file = /var/lib/kolla/config_files/haproxy-ca.crt [nova] auth_url = https://111.111.111.80:35357 cafile = /var/lib/kolla/config_files/haproxy-ca.crt insecure = true [keystone_authtoken] www_authenticate_uri = https://111.111.111.80:5000 auth_url = https://111.111.111.80:35357 cafile = /var/lib/kolla/config_files/haproxy-ca.crt insecure = true
验证
1 2 3 4 5 6 7 8 9 10 11 12 root@node01: kolla # openstack volume service list +------------------+--------------+------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+--------------+------+---------+-------+----------------------------+ | cinder-scheduler | node01 | nova | enabled | up | 2021-01-18T06:13:08.000000 | | cinder-volume | node01@lvm-1 | nova | enabled | up | 2021-01-18T06:13:09.000000 | | cinder-volume | node03@lvm-1 | nova | enabled | up | 2021-01-18T06:13:11.000000 | | cinder-volume | node02@lvm-1 | nova | enabled | up | 2021-01-18T06:13:11.000000 | | cinder-backup | node01 | nova | enabled | up | 2021-01-18T06:13:17.000000 | | cinder-backup | node02 | nova | enabled | up | 2021-01-18T06:13:10.000000 | | cinder-backup | node03 | nova | enabled | up | 2021-01-18T06:13:10.000000 | +------------------+--------------+------+---------+-------+----------------------------+
Neutron配置证书
更新haproxy中 neutron-server.cfg
配置,重启haproxy
1 2 3 4 5 6 7 8 9 10 11 12 13 frontend neutron_server_front mode http http-request del-header X-Forwarded-Proto option http-tunnel option httplog option forwardfor http-request set-header X-Forwarded-Proto https if { ssl_fc } bind 111.111.111.80:9696 ssl crt /var/lib/kolla/config_files/haproxy.pem default_backend neutron_server_back backend neutron_server_back mode http server node01 111.111.111.81:9696 check inter 2000 rise 2 fall 5
更新neutron服务配置文件:neutron.conf
。同步所有节点配置文件,重启neutron相关服务
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 [nova] auth_url = https://111.111.111.80:35357 cafile = /var/lib/kolla/config_files/haproxy-ca.crt insecure = true [keystone_authtoken] www_authenticate_uri = https://111.111.111.80:5000 auth_url = https://111.111.111.80:35357 cafile = /var/lib/kolla/config_files/haproxy-ca.crt insecure = true [placement] auth_url = https://111.111.111.80:35357 cafile = /var/lib/kolla/config_files/haproxy-ca.crt insecure = true
验证
1 2 3 4 5 6 7 8 9 10 11 root@node01: kolla # openstack network agent list +--------------------------------------+--------------------+--------+-------------------+-------+-------+---------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+--------------------+--------+-------------------+-------+-------+---------------------------+ | 07a53564-2518-4550-b222-e1d740e51772 | Open vSwitch agent | node01 | None | :-) | UP | neutron-openvswitch-agent | | 07c00fbe-9da4-4747-9a60-5af739733eb3 | Open vSwitch agent | node03 | None | :-) | UP | neutron-openvswitch-agent | | 0fd1203e-1fdc-4261-bf87-0a76b5883f5b | Metadata agent | node01 | None | :-) | UP | neutron-metadata-agent | | 333408d8-3a49-4ae1-8251-e52f2a09da78 | Open vSwitch agent | node02 | None | :-) | UP | neutron-openvswitch-agent | | 48c2a3d0-5818-4df2-8054-41befae51b44 | L3 agent | node01 | nova | :-) | UP | neutron-l3-agent | | 4df37b6c-2a77-4f1f-8dad-a15e758955e9 | DHCP agent | node01 | nova | :-) | UP | neutron-dhcp-agent | +--------------------------------------+--------------------+--------+-------------------+-------+-------+---------------------------+
Placement配置证书
更新haproxy中 placement-api.cfg
配置,重启haproxy
1 2 3 4 5 6 7 8 9 10 11 12 frontend placement_api_front mode http http-request del-header X-Forwarded-Proto option httplog option forwardfor http-request set-header X-Forwarded-Proto https if { ssl_fc } bind 111.111.111.80:8780 ssl crt /var/lib/kolla/config_files/haproxy.pem default_backend placement_api_back backend placement_api_back mode http server node01 111.111.111.81:8780 check inter 2000 rise 2 fall 5
更新placement服务配置文件:placement.conf
。同步所有节点配置文件,重启placement相关服务
1 2 3 4 5 [keystone_authtoken] www_authenticate_uri = https://111.111.111.80:5000 auth_url = https://111.111.111.80:35357 cafile = /var/lib/kolla/config_files/haproxy-ca.crt insecure = true
验证
1 2 3 4 5 6 7 # 配置nova后,查看nova_compute日志,是否有如下报错: 2021-01-18 02:28:39.196 7 WARNING keystoneauth.discover [req-a08df9fe-e10d-4836-b2cd-3fc922461ea3 - - - - -] Failed to contact the endpoint at https://111.111.111.80:8780 for discovery. Fallback to using that endpoint as the base url.: SSLError: SSL exception connecting to https://111.111.111.80:8780: HTTPSConnectionPool(host='111.111.111.80', port=8780): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_record', 'wrong version number')],)",),)) # 查看placement日志,有如下信息,200表示请求正常响应: 111.111.111.82 - - [18/Jan/2021:02:30:59 -0500] "GET / HTTP/1.1" 200 136 2191 "-" "nova-compute keystoneauth1/3.17.3 python-requests/2.22.0 CPython/2.7.5" 111.111.111.80 - - [18/Jan/2021:02:31:06 -0500] "GET / HTTP/1.1" 200 136 2366 "-" "nova-compute keystoneauth1/3.17.3 python-requests/2.22.0 CPython/2.7.5" 111.111.111.83 - - [18/Jan/2021:02:32:01 -0500] "GET / HTTP/1.1" 200 136 4198 "-" "nova-compute keystoneauth1/3.17.3 python-requests/2.22.0 CPython/2.7.5"
Nova配置证书
更新haproxy中 nova-api.cfg
, nova-novncproxy.cfg
配置,重启haproxy
nova-api.cfg
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 frontend nova_api_front mode http http-request del-header X-Forwarded-Proto option httplog option forwardfor http-request set-header X-Forwarded-Proto https if { ssl_fc } bind 111.111.111.80:8774 ssl crt /var/lib/kolla/config_files/haproxy.pem default_backend nova_api_back backend nova_api_back mode http server node01 111.111.111.81:8774 check inter 2000 rise 2 fall 5 frontend nova_metadata_front mode http http-request del-header X-Forwarded-Proto option httplog option forwardfor http-request set-header X-Forwarded-Proto https if { ssl_fc } bind 111.111.111.80:8775 ssl crt /var/lib/kolla/config_files/haproxy.pem default_backend nova_metadata_back backend nova_metadata_back mode http server node01 111.111.111.81:8775 check inter 2000 rise 2 fall 5
nova-novncproxy.cfg
1 2 3 4 5 6 7 8 9 10 11 12 13 frontend nova_novncproxy_front mode http http-request del-header X-Forwarded-Proto option httplog option forwardfor http-request set-header X-Forwarded-Proto https if { ssl_fc } bind 111.111.111.80:6080 ssl crt /var/lib/kolla/config_files/haproxy.pem default_backend nova_novncproxy_back backend nova_novncproxy_back mode http timeout tunnel 1h server node01 111.111.111.81:6080 check inter 2000 rise 2 fall 5
更新nova服务配置文件:nova.conf
。同步所有节点配置文件,重启nova相关服务
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 [glance] api_servers = https://111.111.111.80:9292 cafile = /var/lib/kolla/config_files/haproxy-ca.crt insecure = true [cinder] auth_url = https://111.111.111.80:35357 cafile = /var/lib/kolla/config_files/haproxy-ca.crt insecure = true [neutron] auth_url = https://111.111.111.80:35357 cafile = /var/lib/kolla/config_files/haproxy-ca.crt insecure = true [keystone_authtoken] www_authenticate_uri = https://111.111.111.80:5000 auth_url = https://111.111.111.80:35357 cafile = /var/lib/kolla/config_files/haproxy-ca.crt insecure = true [placement] auth_url = https://111.111.111.80:35357 cafile = /var/lib/kolla/config_files/haproxy-ca.crt insecure = true
验证
1 2 3 4 5 6 7 8 9 10 11 root@node01: kolla # openstack compute service list +----+----------------+--------+----------+---------+-------+----------------------------+ | ID | Binary | Host | Zone | Status | State | Updated At | +----+----------------+--------+----------+---------+-------+----------------------------+ | 10 | nova-scheduler | node01 | internal | enabled | up | 2021-01-18T17:37:46.000000 | | 1 | nova-conductor | node01 | internal | enabled | up | 2021-01-18T17:37:44.000000 | | 6 | nova-compute | node01 | nova | enabled | up | 2021-01-18T17:37:52.000000 | | 7 | nova-compute | node02 | nova | enabled | up | 2021-01-18T17:37:37.000000 | | 8 | nova-compute | node03 | nova | enabled | up | 2021-01-18T17:37:51.000000 | +----+----------------+--------+----------+---------+-------+----------------------------+ root@node01: kolla # openstack server create --image cirros --flavor m1.tiny --key-name mykey --network demo-net --boot-from-volume 1 demo
常见问题
问题1: HostMappingNotFound: Host ‘node01’ is not mapped to any cell
解决: nova_api pod中执行如下命令,重新发现node01 nova-manage cell_v2 discover_hosts –verbos
Horizon配置证书
更新haproxy中 nova-api.cfg
, nova-novncproxy.cfg
配置,重启haproxy
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 frontend horizon_redirect_front mode http bind 111.111.111.80:80 redirect scheme https code 301 if !{ ssl_fc } frontend horizon_front mode http http-request del-header X-Forwarded-Proto option httplog option forwardfor http-request set-header X-Forwarded-Proto https if { ssl_fc } bind 111.111.111.80:443 ssl crt /var/lib/kolla/config_files/haproxy.pem default_backend horizon_back backend horizon_back mode http balance source server node01 111.111.111.81:80 check inter 2000 rise 2 fall 5
更改horizon服务的配置文件:local_settings
, 重启horizon、memcached服务
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 # Set SSL proxy settings: # Pass this header from the proxy after terminating the SSL, # and don't forget to strip it from the client' s request. # For more information see: # https://docs.djangoproject.com/en/dev/ref/settings/ SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https') # If Horizon is being served through SSL, then uncomment the following two # settings to better secure the cookies from security exploits CSRF_COOKIE_SECURE = True SESSION_COOKIE_SECURE = True OPENSTACK_KEYSTONE_URL = "https://111.111.111.80:5000/v3" # Disable SSL certificate checks in the OpenStack clients (useful for self-signed certificates). # 自签名证书,需要设置为True,不验证SSL OPENSTACK_SSL_NO_VERIFY = True # When unset or set to None the default CA certificate on the system is used for SSL verification. # # When set with the path to a custom CA certificate file, this overrides use of the default system CA certificate. This custom certificate is used to verify all connections to openstack services when making API calls. # 自签名证书,不需要设置此项 OPENSTACK_SSL_CACERT = '/var/lib/kolla/config_files/haproxy-ca.crt'
验证
1 2 # 访问vip:80端口,自动跳转至https 443端口 # horizon可正常登录,界面功能正常
常见问题
问题1 :horizon登录后,界面显示验证错误,后端haproxy出现如下报错: {“Payload”:”Jan 19 11:01:56 haproxy[33]: 111.111.111.80:60552 [19/Jan/2021:11:01:56.143] keystone_internal_front/1: SSL handshake failure “,”log_level”:”info”,”Hostname”:”node01”,”programname”:”haproxy”}
期望结果:{“Payload”:”Jan 19 03:06:58 haproxy[32]: 111.111.111.80:53822 [19/Jan/2021:03:06:58.236] keystone_internal_front~ keystone_internal_back/node01 3/0/0/10/13 300 538 – —- 64/1/0/1/0 0/0 "GET / HTTP/1.1"“,”log_level”:”info”,”Hostname”:”node01”,”programname”:”haproxy”}
原因: horizon请求后端接口时,使用的证书验证存在异常,未通过keystone_internal_front的SSL验证,无法转到keystone后端server服务
解决: 自签名证书,SSL验证存在异常。需要Disable SSL certificate checks in the OpenStack clients。设置 local_settings
配置中的 OPENSTACK_SSL_NO_VERIFY = True
问题2: 界面显示:无法与keystone服务终端建立连接。
解决: 检查 local_settings
配置中的 OPENSTACK_KEYSTONE_URL
是否切换为https
参考脚本 单节点更新配置文件 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # !/bin/bash module='cinder' module_config='cinder-api/cinder.conf' config_root_dir='/etc/kolla' cd $config_root_dir dir_lists=`ls | grep $module` for dir in $dir_lists do if [ $dir != $(echo $module_config | awk -F '/' '{print $1}') ]; then \cp $module_config $dir fi done
参考文档