简述

需求:需要为openstack所有服务更新为https方式访问

方案:使用HAProxy进行负载均衡和https证书卸载,通过HAProxy frontend进行https证书验证,验证通过后,转发至backend的后端http server。openstack各服务之间,通过https进行访问,需要配置各个服务的CA证书,同时更新keystone的endpoint

证书卸载

环境

  • 系统:CentOS Linux release 7.7.1908 (Core)
  • OpenStack版本:Train
  • kolla-ansible版本:9.2.0
  • Docker版本:docker-ce-18.09.6-3.el7.x86_64
  • OpenStack部署方式:kolla-ansible部署(OpenStack服务运行在Docker上)
  • OpenStack部署规模:control节点1台,compute节点3台

步骤

TLS证书

生成证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# 使用kolla生成证书
kolla-ansible certificates

# openssl-kolla.cnf
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req

[req_distinguished_name]
countryName = US
stateOrProvinceName = NC
localityName = RTP
organizationalUnitName = kolla
commonName = 111.111.111.80

[v3_req]
subjectAltName = @alt_names

[alt_names]
IP.1 = 111.111.111.80

分发证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# haproxy
cp /etc/kolla/certificates/haproxy.pem /etc/kolla/haproxy/

# glance
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/glance-api/

# cinder
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/cinder-api/
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/cinder-backup/
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/cinder-scheduler/
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/cinder-volume/
scp /etc/kolla/certificates/haproxy-ca.crt node02:/etc/kolla/cinder-backup/
scp /etc/kolla/certificates/haproxy-ca.crt node03:/etc/kolla/cinder-backup/
scp /etc/kolla/certificates/haproxy-ca.crt node02:/etc/kolla/cinder-volume/
scp /etc/kolla/certificates/haproxy-ca.crt node03:/etc/kolla/cinder-volume/

# neutron
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/neutron-server/
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/neutron-openvswitch-agent/
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/neutron-metadata-agent/
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/neutron-l3-agent/
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/neutron-dhcp-agent/
scp /etc/kolla/certificates/haproxy-ca.crt node02:/etc/kolla/neutron-openvswitch-agent/
scp /etc/kolla/certificates/haproxy-ca.crt node03:/etc/kolla/neutron-openvswitch-agent/

# placement
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/placement-api/

# nova
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/nova-api/
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/nova-api-bootstrap/
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/nova-cell-bootstrap/
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/nova-compute/
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/nova-conductor/
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/nova-novncproxy/
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/nova-scheduler/
scp /etc/kolla/certificates/haproxy-ca.crt node02:/etc/kolla/nova-compute/
scp /etc/kolla/certificates/haproxy-ca.crt node03:/etc/kolla/nova-compute/

# horizon
cp /etc/kolla/certificates/haproxy-ca.crt /etc/kolla/horizon/

admin-openrc.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Clear any old environment that may conflict.
for key in $( set | awk '{FS="="} /^OS_/ {print $1}' ); do unset $key ; done
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_AUTH_URL=https://111.111.111.80:35357/v3
export OS_CACERT=/etc/kolla/certificates/haproxy-ca.crt
export OS_INTERFACE=internal
export OS_ENDPOINT_TYPE=internalURL
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne
export OS_AUTH_PLUGIN=password

Keystone配置证书

  1. 使用http协议,先创建 https endpoint
1
2
3
openstack endpoint create --region RegionOne identity public https://111.111.111.80:5000
openstack endpoint create --region RegionOne identity internal https://111.111.111.80:5000
openstack endpoint create --region RegionOne identity admin https://111.111.111.80:35357
  1. 删除http endpoint
1
2
3
4
5
6
7
8
9
# 如果环境变量OS_INTERFACE使用的是internal,则删除掉internal http的endpoint之后,删除其他endpoint会报错,属于正常现象。
# 原因:https尚未在haproxy上配置,请求https端口会报错
# 解决:需要执行第3步,更新haproxy。配置https证书验证后,可正常执行其余操作
root@node01: kolla # openstack endpoint delete 9200b3fcad1e42e8a8d8277e8c7ba71c
Failed to contact the endpoint at https://111.111.111.80:5000 for discovery. Fallback to using that endpoint as the base url.
Failed to contact the endpoint at https://111.111.111.80:5000 for discovery. Fallback to using that endpoint as the base url.
Failed to contact the endpoint at https://111.111.111.80:5000 for discovery. Fallback to using that endpoint as the base url.
Failed to delete endpoint with ID '9200b3fcad1e42e8a8d8277e8c7ba71c': SSL exception connecting to https://111.111.111.80:5000/endpoints?: HTTPSConnectionPool(host='111.111.111.80', port=5000): Max retries exceeded with url: /endpoints (Caused by SSLError(SSLError(1, u'[SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:618)'),))
1 of 1 endpoints failed to delete.
  1. 更新haproxy中 keystone.cfg 配置,重启haproxy。此时keystone的https端口正常响应
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
frontend keystone_internal_front
mode http
http-request del-header X-Forwarded-Proto
option httplog
option forwardfor
http-request set-header X-Forwarded-Proto https if { ssl_fc }
bind 111.111.111.80:5000 ssl crt /var/lib/kolla/config_files/haproxy.pem
default_backend keystone_internal_back

backend keystone_internal_back
mode http
server node01 111.111.111.81:5000 check inter 2000 rise 2 fall 5

frontend keystone_admin_front
mode http
option forwardfor
http-request del-header X-Forwarded-Proto
option httplog
http-request set-header X-Forwarded-Proto https if { ssl_fc }
bind 111.111.111.80:35357 ssl crt /var/lib/kolla/config_files/haproxy.pem
default_backend keystone_admin_back

backend keystone_admin_back
mode http
server node01 111.111.111.81:35357 check inter 2000 rise 2 fall 5
  1. 继续删除其余未删除的keystone http endpoint
  2. 更新其他组件endpoint协议,切换为https
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#!/bin/bash
openstack endpoint list | sed '1,3d' | sed '$d' > /tmp/endpoint

while read -r line
do
echo $line
id=`echo $line | awk '{print $2}'`
region=`echo $line | awk '{print $4}'`
service_name=`echo $line | awk '{print $6}'`
service_type=`echo $line | awk '{print $8}'`
interface=`echo $line | awk '{print $12}'`
url=`echo $line | awk '{print $14}'`

if [ $(echo $url | awk -F : '{print $1}') == 'http' ]; then
openstack endpoint delete $id
openstack endpoint create --region $region \
$service_type $interface \
$(echo $url | sed 's/http/https/g')
fi
done < /tmp/endpoint
  1. 验证
1
openstack endpoint list

Glance配置证书

  1. 更新haproxy中 glance-api.cfg 配置,重启haproxy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
frontend glance_api_front
mode http
http-request del-header X-Forwarded-Proto
timeout client 6h
option httplog
option forwardfor
http-request set-header X-Forwarded-Proto https if { ssl_fc }
bind 111.111.111.80:9292 ssl crt /var/lib/kolla/config_files/haproxy.pem
default_backend glance_api_back

backend glance_api_back
mode http
timeout server 6h
server node01 111.111.111.81:9292 check inter 2000 rise 2 fall 5
  1. 更新glance服务配置文件:glance-api.conf,重启glance服务
1
2
3
4
5
[keystone_authtoken]
www_authenticate_uri = https://111.111.111.80:5000
auth_url = https://111.111.111.80:35357
cafile = /var/lib/kolla/config_files/haproxy-ca.crt
insecure = true
  1. 验证
1
openstack image list

Cinder配置证书

  1. 更新haproxy中 cinder-api.cfg 配置,重启haproxy
1
2
3
4
5
6
7
8
9
10
11
12
frontend cinder_api_front
mode http
http-request del-header X-Forwarded-Proto
option httplog
option forwardfor
http-request set-header X-Forwarded-Proto https if { ssl_fc }
bind 111.111.111.80:8776 ssl crt /var/lib/kolla/config_files/haproxy.pem
default_backend cinder_api_back

backend cinder_api_back
mode http
server node01 111.111.111.81:8776 check inter 2000 rise 2 fall 5
  1. 更新服务配置文件:cinder.conf 。同步所有节点配置文件,重启cinder相关服务
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[DEFAULT]
glance_api_servers = https://111.111.111.80:9292
glance_api_insecure = true
glance_ca_certificates_file = /var/lib/kolla/config_files/haproxy-ca.crt

[nova]
auth_url = https://111.111.111.80:35357
cafile = /var/lib/kolla/config_files/haproxy-ca.crt
insecure = true

[keystone_authtoken]
www_authenticate_uri = https://111.111.111.80:5000
auth_url = https://111.111.111.80:35357
cafile = /var/lib/kolla/config_files/haproxy-ca.crt
insecure = true
  1. 验证
1
2
3
4
5
6
7
8
9
10
11
12
root@node01: kolla # openstack volume service list
+------------------+--------------+------+---------+-------+----------------------------+
| Binary | Host | Zone | Status | State | Updated At |
+------------------+--------------+------+---------+-------+----------------------------+
| cinder-scheduler | node01 | nova | enabled | up | 2021-01-18T06:13:08.000000 |
| cinder-volume | node01@lvm-1 | nova | enabled | up | 2021-01-18T06:13:09.000000 |
| cinder-volume | node03@lvm-1 | nova | enabled | up | 2021-01-18T06:13:11.000000 |
| cinder-volume | node02@lvm-1 | nova | enabled | up | 2021-01-18T06:13:11.000000 |
| cinder-backup | node01 | nova | enabled | up | 2021-01-18T06:13:17.000000 |
| cinder-backup | node02 | nova | enabled | up | 2021-01-18T06:13:10.000000 |
| cinder-backup | node03 | nova | enabled | up | 2021-01-18T06:13:10.000000 |
+------------------+--------------+------+---------+-------+----------------------------+

Neutron配置证书

  1. 更新haproxy中 neutron-server.cfg 配置,重启haproxy
1
2
3
4
5
6
7
8
9
10
11
12
13
frontend neutron_server_front
mode http
http-request del-header X-Forwarded-Proto
option http-tunnel
option httplog
option forwardfor
http-request set-header X-Forwarded-Proto https if { ssl_fc }
bind 111.111.111.80:9696 ssl crt /var/lib/kolla/config_files/haproxy.pem
default_backend neutron_server_back

backend neutron_server_back
mode http
server node01 111.111.111.81:9696 check inter 2000 rise 2 fall 5
  1. 更新neutron服务配置文件:neutron.conf 。同步所有节点配置文件,重启neutron相关服务
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[nova]
auth_url = https://111.111.111.80:35357
cafile = /var/lib/kolla/config_files/haproxy-ca.crt
insecure = true

[keystone_authtoken]
www_authenticate_uri = https://111.111.111.80:5000
auth_url = https://111.111.111.80:35357
cafile = /var/lib/kolla/config_files/haproxy-ca.crt
insecure = true

[placement]
auth_url = https://111.111.111.80:35357
cafile = /var/lib/kolla/config_files/haproxy-ca.crt
insecure = true
  1. 验证
1
2
3
4
5
6
7
8
9
10
11
root@node01: kolla # openstack network agent list
+--------------------------------------+--------------------+--------+-------------------+-------+-------+---------------------------+
| ID | Agent Type | Host | Availability Zone | Alive | State | Binary |
+--------------------------------------+--------------------+--------+-------------------+-------+-------+---------------------------+
| 07a53564-2518-4550-b222-e1d740e51772 | Open vSwitch agent | node01 | None | :-) | UP | neutron-openvswitch-agent |
| 07c00fbe-9da4-4747-9a60-5af739733eb3 | Open vSwitch agent | node03 | None | :-) | UP | neutron-openvswitch-agent |
| 0fd1203e-1fdc-4261-bf87-0a76b5883f5b | Metadata agent | node01 | None | :-) | UP | neutron-metadata-agent |
| 333408d8-3a49-4ae1-8251-e52f2a09da78 | Open vSwitch agent | node02 | None | :-) | UP | neutron-openvswitch-agent |
| 48c2a3d0-5818-4df2-8054-41befae51b44 | L3 agent | node01 | nova | :-) | UP | neutron-l3-agent |
| 4df37b6c-2a77-4f1f-8dad-a15e758955e9 | DHCP agent | node01 | nova | :-) | UP | neutron-dhcp-agent |
+--------------------------------------+--------------------+--------+-------------------+-------+-------+---------------------------+

Placement配置证书

  1. 更新haproxy中 placement-api.cfg 配置,重启haproxy
1
2
3
4
5
6
7
8
9
10
11
12
frontend placement_api_front
mode http
http-request del-header X-Forwarded-Proto
option httplog
option forwardfor
http-request set-header X-Forwarded-Proto https if { ssl_fc }
bind 111.111.111.80:8780 ssl crt /var/lib/kolla/config_files/haproxy.pem
default_backend placement_api_back

backend placement_api_back
mode http
server node01 111.111.111.81:8780 check inter 2000 rise 2 fall 5
  1. 更新placement服务配置文件:placement.conf 。同步所有节点配置文件,重启placement相关服务
1
2
3
4
5
[keystone_authtoken]
www_authenticate_uri = https://111.111.111.80:5000
auth_url = https://111.111.111.80:35357
cafile = /var/lib/kolla/config_files/haproxy-ca.crt
insecure = true
  1. 验证
1
2
3
4
5
6
7
# 配置nova后,查看nova_compute日志,是否有如下报错:
2021-01-18 02:28:39.196 7 WARNING keystoneauth.discover [req-a08df9fe-e10d-4836-b2cd-3fc922461ea3 - - - - -] Failed to contact the endpoint at https://111.111.111.80:8780 for discovery. Fallback to using that endpoint as the base url.: SSLError: SSL exception connecting to https://111.111.111.80:8780: HTTPSConnectionPool(host='111.111.111.80', port=8780): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_record', 'wrong version number')],)",),))

# 查看placement日志,有如下信息,200表示请求正常响应:
111.111.111.82 - - [18/Jan/2021:02:30:59 -0500] "GET / HTTP/1.1" 200 136 2191 "-" "nova-compute keystoneauth1/3.17.3 python-requests/2.22.0 CPython/2.7.5"
111.111.111.80 - - [18/Jan/2021:02:31:06 -0500] "GET / HTTP/1.1" 200 136 2366 "-" "nova-compute keystoneauth1/3.17.3 python-requests/2.22.0 CPython/2.7.5"
111.111.111.83 - - [18/Jan/2021:02:32:01 -0500] "GET / HTTP/1.1" 200 136 4198 "-" "nova-compute keystoneauth1/3.17.3 python-requests/2.22.0 CPython/2.7.5"

Nova配置证书

  1. 更新haproxy中 nova-api.cfg, nova-novncproxy.cfg 配置,重启haproxy

nova-api.cfg

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
frontend nova_api_front
mode http
http-request del-header X-Forwarded-Proto
option httplog
option forwardfor
http-request set-header X-Forwarded-Proto https if { ssl_fc }
bind 111.111.111.80:8774 ssl crt /var/lib/kolla/config_files/haproxy.pem
default_backend nova_api_back

backend nova_api_back
mode http
server node01 111.111.111.81:8774 check inter 2000 rise 2 fall 5

frontend nova_metadata_front
mode http
http-request del-header X-Forwarded-Proto
option httplog
option forwardfor
http-request set-header X-Forwarded-Proto https if { ssl_fc }
bind 111.111.111.80:8775 ssl crt /var/lib/kolla/config_files/haproxy.pem
default_backend nova_metadata_back

backend nova_metadata_back
mode http
server node01 111.111.111.81:8775 check inter 2000 rise 2 fall 5

nova-novncproxy.cfg

1
2
3
4
5
6
7
8
9
10
11
12
13
frontend nova_novncproxy_front
mode http
http-request del-header X-Forwarded-Proto
option httplog
option forwardfor
http-request set-header X-Forwarded-Proto https if { ssl_fc }
bind 111.111.111.80:6080 ssl crt /var/lib/kolla/config_files/haproxy.pem
default_backend nova_novncproxy_back

backend nova_novncproxy_back
mode http
timeout tunnel 1h
server node01 111.111.111.81:6080 check inter 2000 rise 2 fall 5
  1. 更新nova服务配置文件:nova.conf 。同步所有节点配置文件,重启nova相关服务
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[glance]
api_servers = https://111.111.111.80:9292
cafile = /var/lib/kolla/config_files/haproxy-ca.crt
insecure = true

[cinder]
auth_url = https://111.111.111.80:35357
cafile = /var/lib/kolla/config_files/haproxy-ca.crt
insecure = true

[neutron]
auth_url = https://111.111.111.80:35357
cafile = /var/lib/kolla/config_files/haproxy-ca.crt
insecure = true

[keystone_authtoken]
www_authenticate_uri = https://111.111.111.80:5000
auth_url = https://111.111.111.80:35357
cafile = /var/lib/kolla/config_files/haproxy-ca.crt
insecure = true

[placement]
auth_url = https://111.111.111.80:35357
cafile = /var/lib/kolla/config_files/haproxy-ca.crt
insecure = true
  1. 验证
1
2
3
4
5
6
7
8
9
10
11
root@node01: kolla # openstack compute service list
+----+----------------+--------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+----------------+--------+----------+---------+-------+----------------------------+
| 10 | nova-scheduler | node01 | internal | enabled | up | 2021-01-18T17:37:46.000000 |
| 1 | nova-conductor | node01 | internal | enabled | up | 2021-01-18T17:37:44.000000 |
| 6 | nova-compute | node01 | nova | enabled | up | 2021-01-18T17:37:52.000000 |
| 7 | nova-compute | node02 | nova | enabled | up | 2021-01-18T17:37:37.000000 |
| 8 | nova-compute | node03 | nova | enabled | up | 2021-01-18T17:37:51.000000 |
+----+----------------+--------+----------+---------+-------+----------------------------+
root@node01: kolla # openstack server create --image cirros --flavor m1.tiny --key-name mykey --network demo-net --boot-from-volume 1 demo
  1. 常见问题

问题1:HostMappingNotFound: Host ‘node01’ is not mapped to any cell

解决:nova_api pod中执行如下命令,重新发现node01
nova-manage cell_v2 discover_hosts –verbos

Horizon配置证书

  1. 更新haproxy中 nova-api.cfg, nova-novncproxy.cfg 配置,重启haproxy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
frontend horizon_redirect_front
mode http
bind 111.111.111.80:80
redirect scheme https code 301 if !{ ssl_fc }

frontend horizon_front
mode http
http-request del-header X-Forwarded-Proto
option httplog
option forwardfor
http-request set-header X-Forwarded-Proto https if { ssl_fc }
bind 111.111.111.80:443 ssl crt /var/lib/kolla/config_files/haproxy.pem
default_backend horizon_back

backend horizon_back
mode http
balance source
server node01 111.111.111.81:80 check inter 2000 rise 2 fall 5
  1. 更改horizon服务的配置文件:local_settings, 重启horizon、memcached服务
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Set SSL proxy settings:
# Pass this header from the proxy after terminating the SSL,
# and don't forget to strip it from the client's request.
# For more information see:
# https://docs.djangoproject.com/en/dev/ref/settings/#secure-proxy-ssl-header
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

# If Horizon is being served through SSL, then uncomment the following two
# settings to better secure the cookies from security exploits
CSRF_COOKIE_SECURE = True
SESSION_COOKIE_SECURE = True

OPENSTACK_KEYSTONE_URL = "https://111.111.111.80:5000/v3"

# Disable SSL certificate checks in the OpenStack clients (useful for self-signed certificates).
# 自签名证书,需要设置为True,不验证SSL
OPENSTACK_SSL_NO_VERIFY = True

# When unset or set to None the default CA certificate on the system is used for SSL verification.
#
# When set with the path to a custom CA certificate file, this overrides use of the default system CA certificate. This custom certificate is used to verify all connections to openstack services when making API calls.
# 自签名证书,不需要设置此项
OPENSTACK_SSL_CACERT = '/var/lib/kolla/config_files/haproxy-ca.crt'
  1. 验证
1
2
# 访问vip:80端口,自动跳转至https 443端口
# horizon可正常登录,界面功能正常
  1. 常见问题

问题1:horizon登录后,界面显示验证错误,后端haproxy出现如下报错:
{“Payload”:”Jan 19 11:01:56 haproxy[33]: 111.111.111.80:60552 [19/Jan/2021:11:01:56.143] keystone_internal_front/1: SSL handshake failure“,”log_level”:”info”,”Hostname”:”node01”,”programname”:”haproxy”}

期望结果:{“Payload”:”Jan 19 03:06:58 haproxy[32]: 111.111.111.80:53822 [19/Jan/2021:03:06:58.236] keystone_internal_front~ keystone_internal_back/node01 3/0/0/10/13 300 538 – —- 64/1/0/1/0 0/0 "GET / HTTP/1.1"“,”log_level”:”info”,”Hostname”:”node01”,”programname”:”haproxy”}

原因:horizon请求后端接口时,使用的证书验证存在异常,未通过keystone_internal_front的SSL验证,无法转到keystone后端server服务

解决:自签名证书,SSL验证存在异常。需要Disable SSL certificate checks in the OpenStack clients。设置 local_settings 配置中的 OPENSTACK_SSL_NO_VERIFY = True

问题2:界面显示:无法与keystone服务终端建立连接。

解决:检查 local_settings 配置中的 OPENSTACK_KEYSTONE_URL 是否切换为https

参考脚本

单节点更新配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/bin/bash
module='cinder'
module_config='cinder-api/cinder.conf'
config_root_dir='/etc/kolla'

cd $config_root_dir
dir_lists=`ls | grep $module`

for dir in $dir_lists
do
if [ $dir != $(echo $module_config | awk -F '/' '{print $1}') ]; then
\cp $module_config $dir
fi
done

参考文档