Glance/Cinder-store-proposal

Glance Cinder Driver

There is currently a Cinder store in Glance, however it does not support uploads and downloads. This is a proposal to allow Cinder to act as a data store for Glance images that supports upload and download by using Cinder volumes as containers for image data. There would be a 1-to-1 mapping of Glance images to Cinder volumes.

What are the benefits of this approach?

This will allow Glance to store images on the large and diverse array of SAN and NAS storage systems supported by Cinder. It also allows for storing images as block devices to enable the features mentioned in the “Subsequent Features” section of this wiki page.

Glance and Cinder using common backend storage will allow for storage optimizations when creating volumes from images or storing images as volumes. Some examples are deduplication and fast replication (create-volume-from-image).

How do you configure Glance on Cinder?

These new configuration options will need to be added to the current Cinder store options:

  • cinder_store_tenant - string - The Cinder tenant in which to store images
  • cinder_store_user - string - The Cinder user used to access Cinder
  • cinder_store_key - string - The Cinder key used to access Cinder
  • cinder_volume_type - string (optional)- The uuid of the volume type to assign to volumes created by Glance

How would an image upload work?

Glance-cinder-upload

  1. The user begins streaming the image data to Glance as usual.
  2. Make a request to the Cinder API to create a new volume with the configured volume type with the volume size being the smallest possible volume size that is greater than the image size.
  3. Attach the newly created volume to the Glance node using the Brick library.
  4. Stream the incoming bits into the volume.
  5. Detach the volume from the Glance node using the Brick library, set the image location to “cinder://‘, and return image details to the user.

How would an image download work?

Glance-cinder-download

  1. The user sends a request to download the image
  2. Retrieve the backend volume id from the image location and attach the volume to the Glance node using Brick library
  3. Stream the bits from the volume to the user
  4. Detach the volume from the glance node using Brick library

How would a Nova boot work?

Nova would download the image via Glance without needing knowledge that Cinder is storing the images.

Single tenant vs Multi tenant? How do we prevent the volume being deleted/changed out from under Glance?

Single-tenant

The Cinder store will only allow for ‘single-tenant’ use. Glance will use the credentials and tenant provided in glance-api.conf to create and attach to volumes. Credentials will not be stored in the image location. Volumes cannot be deleted without the Glance service knowing since only the glance user will have access to it’s volumes in Cinder.

Multi-tenant

Multi-tenant, I.E. storing volumes in the users tenant, should not be supported because:

  • There is no way to support image sharing
  • Reauthentication due to the user token expiring would present problems
  • There is no way to prevent users from deleting/modifying volumes without Glance having knowledge.

How do we allow for Image Sharing?

This is trivial because the service user will have access to all image volumes in cinder.

Example Configuration

以 kolla/centos-source-glance-api:train 镜像在Kubernetes中部署为例，Cinder使用iSCSI后端存储，介绍Glance Cinder Driver配置思路。

服务配置

以Cinder使用iSCSI后端存储为例，Glance用户需要通过iSCSI方式，把Cinder卷挂载至本地，挂载中Glance需要有访问和使用iSCSI的权限，涉及提权操作，需要配置如下相关配置文件：

  • /etc/glance/glance-api.conf（配置cinder连接信息）
  • /etc/glance/rootwrap.conf（提权相关）
  • /etc/glance/rootwrap.d/glance_cinder_store.filters（提权相关）
  • /etc/sudoers.d/glance-sudoers（提权相关）

/etc/glance/glance-api.conf

1
2
3
4
5
6
7
8
9
10
[glance_store]
default_store = cinder
stores = cinder
cinder_http_retries = 3
cinder_catalog_info = volumev2::publicURL
rootwrap_config = /etc/glance/rootwrap.conf
[cinder]
cinder_http_retries = 3
cinder_catalog_info = volumev2::publicURL
rootwrap_config = /etc/glance/rootwrap.conf

/etc/glance/rootwrap.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# Configuration for nova-rootwrap
# This file should be owned by (and only-writeable by) the root user

[DEFAULT]
# List of directories to load filter definitions from (separated by ',').
# These directories MUST all be only writeable by root !
filters_path=/etc/glance/rootwrap.d

# List of directories to search executables in, in case filters do not
# explicitely specify a full path (separated by ',')
# If not specified, defaults to system PATH environment variable.
# These directories MUST all be only writeable by root !
exec_dirs=/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/bin,/usr/local/sbin,/var/lib/openstack/bin,/var/lib/kolla/venv/bin

# Enable logging to syslog
# Default value is False
use_syslog=False

# Which syslog facility to use.
# Valid values include auth, authpriv, syslog, local0, local1...
# Default value is 'syslog'
syslog_log_facility=syslog

# Which messages to log.
# INFO means log all usage
# ERROR means only log unsuccessful attempts
syslog_log_level=ERROR

/etc/glance/rootwrap.d/glance_cinder_store.filters

1
2
3
4
5
6
7
8
9
10
11
12
# glance-rootwrap command filters for glance cinder store
# This file should be owned by (and only-writable by) the root user

[Filters]
# cinder store driver
disk_chown: RegExpFilter, chown, root, chown, \d+, /dev/(?!.*/\.\.).*

# os-brick library commands
# os_brick.privileged.run_as_root oslo.privsep context
# This line ties the superuser privs with the config files, context name,
# and (implicitly) the actual python code invoked.
privsep-rootwrap: RegExpFilter, privsep-helper, root, privsep-helper, --config-file, /etc/(?!\.\.).*, --privsep_context, os_brick.privileged.default, --privsep_sock_path, /tmp/.*

/etc/sudoers.d/glance-sudoers

1
2
3
4
5
Defaults:glance !requiretty

Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/var/lib/openstack/bin:/var/lib/kolla/venv/bin"

glance ALL = (root) NOPASSWD: /var/lib/kolla/venv/bin/glance-rootwrap /etc/glance/rootwrap.conf *, /var/lib/openstack/bin/glance-rootwrap /etc/glance/rootwrap.conf *

POD配置

Glance容器需要与主机的iSCSI服务有交互，同时有提权的需求。参考相关配置项如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
spec:
  ...
  template:
    ...
    spec:
    ...
      volumes:
        ...
        - name: hostdir-iscsi
          hostPath:
            path: /etc/iscsi
            type: ''
        - name: dev
          hostPath:
            path: /dev
            type: ''
        ...
      hostNetwork: true
      containers:
      ...
          volumeMounts:
            - name: hostdir-iscsi
              mountPath: /etc/iscsi/
            - name: dev
              mountPath: /dev
              mountPropagation: HostToContainer
            ...
          securityContext:
            privileged: true
            readOnlyRootFilesystem: false
            allowPrivilegeEscalation: true
          ...

参考文档