OpenStack研发：Glance Image Cache

The Glance Image Cache

The Glance API server may be configured to have an optional local image cache. A local image cache stores a copy of image files, essentially enabling multiple API servers to serve the same image file, resulting in an increase in scalability due to an increased number of endpoints serving an image file.

This local image cache is transparent to the end user – in other words, the end user doesn’t know that the Glance API is streaming an image file from its local cache or from the actual backend storage system.

Configuration options for the Image Cache

Config File：glance-api.conf

[DEFAULT]
# This is the base directory where Glance stores the cache data (Required to be set, as does not have a default).
image_cache_dir = /var/lib/glance/image-cache

# Path to the sqlite file database that will be used for cache management.
image_cache_driver = sqlite

# This is a relative path from the image_cache_dir directory (Default:cache.db)
image_cache_sqlite_db = cache.db

# The size when the glance-cache-pruner will remove the oldest images, to reduce the bytes until under this value. (Default:10 GB)
image_cache_max_size = 1073741824

[paste_deploy]
# Enabling the Image Cache Management Middlewar. There are three types you can chose: 
#   - cachemanagement
#   - keystone+cachemanagement
#   - trusted-auth+cachemanagement.
flavor = keystone+cachemanagement

Managing the Glance Image Cache

While image files are automatically placed in the image cache on successful requests to GET /v2/images/{image_id}/file, eg: openstack image save --file <file name> <image id>.

the image cache is not automatically managed. Here, we describe the basics of how to manage the local image cache on Glance API servers and how to automate this cache management.

Glance Image Cache API

Description	Method	URL
Get cached image	GET	/v2/cached_images
Delete cached image	DELETE	/v2/cached_images/{image_id}
Delete all cached image	DELETE	/v2/cached_images
Get queued image	GET	/v2/queued_images
Put image to queue	PUT	/v2/queued_images/{image_id}
Delete queued image	DELETE	/v2/queued_images/{image_id}
Delete all queued image	DELETE	/v2/queued_images

Src File：glance/api/middleware/cache_manage.py

# Copyright 2011 OpenStack Foundation
# All Rights Reserved.
#
#    Licensed under the Apache License, Version 2.0 (the "License"); you may
#    not use this file except in compliance with the License. You may obtain
#    a copy of the License at
#
#         http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#    License for the specific language governing permissions and limitations
#    under the License.

"""
Image Cache Management API
"""

from oslo_log import log as logging
import routes

from glance.api.v2 import cached_images
from glance.common import wsgi
from glance.i18n import _LI

LOG = logging.getLogger(__name__)


class CacheManageFilter(wsgi.Middleware):
    def __init__(self, app):
        mapper = routes.Mapper()
        resource = cached_images.create_resource()

        mapper.connect("/v2/cached_images",
                       controller=resource,
                       action="get_cached_images",
                       conditions=dict(method=["GET"]))

        mapper.connect("/v2/cached_images/{image_id}",
                       controller=resource,
                       action="delete_cached_image",
                       conditions=dict(method=["DELETE"]))

        mapper.connect("/v2/cached_images",
                       controller=resource,
                       action="delete_cached_images",
                       conditions=dict(method=["DELETE"]))

        mapper.connect("/v2/queued_images/{image_id}",
                       controller=resource,
                       action="queue_image",
                       conditions=dict(method=["PUT"]))

        mapper.connect("/v2/queued_images",
                       controller=resource,
                       action="get_queued_images",
                       conditions=dict(method=["GET"]))

        mapper.connect("/v2/queued_images/{image_id}",
                       controller=resource,
                       action="delete_queued_image",
                       conditions=dict(method=["DELETE"]))

        mapper.connect("/v2/queued_images",
                       controller=resource,
                       action="delete_queued_images",
                       conditions=dict(method=["DELETE"]))

        self._mapper = mapper
        self._resource = resource

        LOG.info(_LI("Initialized image cache management middleware"))
        super(CacheManageFilter, self).__init__(app)

    def process_request(self, request):
        # Map request to our resource object if we can handle it
        match = self._mapper.match(request.path_info, request.environ)
        if match:
            request.environ['wsgiorg.routing_args'] = (None, match)
            return self._resource(request)
        # Pass off downstream if we don't match the request path
        else:
            return None

数据库

sqlite数据库

sqlite> .tables
cached_images
sqlite> .schema cached_images
CREATE TABLE cached_images (
                        image_id TEXT PRIMARY KEY,
                        last_accessed REAL DEFAULT 0.0,
                        last_modified REAL DEFAULT 0.0,
                        size INTEGER DEFAULT 0,
                        hits INTEGER DEFAULT 0,
                        checksum TEXT
                    );

• image_id：缓存镜像id
• last_accessed：镜像最近一次被访问的时间
• last_modified：镜像最近一次更改的时间
• size：镜像大小
• hits：镜像缓存命中次数，命中一次 +1
• checksum：校验值

Src File：glance/image_cache/drivers/sqlite.py

SQLite Driver创建表的流程如下：

def initialize_db(self):
    db = CONF.image_cache_sqlite_db
    self.db_path = os.path.join(self.base_dir, db)
    lockutils.set_defaults(self.base_dir)

    @lockutils.synchronized('image_cache_db_init', external=True)
    def create_db():
        try:
            conn = sqlite3.connect(self.db_path, check_same_thread=False,
                                   factory=SqliteConnection)
            conn.executescript("""
                CREATE TABLE IF NOT EXISTS cached_images (
                    image_id TEXT PRIMARY KEY,
                    last_accessed REAL DEFAULT 0.0,
                    last_modified REAL DEFAULT 0.0,
                    size INTEGER DEFAULT 0,
                    hits INTEGER DEFAULT 0,
                    checksum TEXT
                );
            """)
            conn.close()
        except sqlite3.DatabaseError as e:
            msg = _("Failed to initialize the image cache database. "
                    "Got error: %s") % e
            LOG.error(msg)
            raise exception.BadDriverConfiguration(driver_name='sqlite',
                                                   reason=msg)

    create_db()

源码解析

Glance image cache流程

Src File：glance/image_cache/__init__.py

入口函数：主要传入CHUNKSIZE值

def cache_image_file(self, image_id, image_file):
        """
        Cache an image file.

        :param image_id: Image ID
        :param image_file: Image file to cache

        :returns: True if image file was cached, False otherwise
        """
        CHUNKSIZE = 64 * units.Mi

        return self.cache_image_iter(image_id,
                                     utils.chunkiter(image_file, CHUNKSIZE))

cache_image_iter() > get_caching_iter() > cache_tee_iter，打开缓存文件写入缓存内容，文件名为image_id

def cache_image_iter(self, image_id, image_iter, image_checksum=None):
    """
    Cache an image with supplied iterator.

    :param image_id: Image ID
    :param image_file: Iterator retrieving image chunks
    :param image_checksum: Checksum of image

    :returns: True if image file was cached, False otherwise
    """
    if not self.driver.is_cacheable(image_id):
        return False

    for chunk in self.get_caching_iter(image_id, image_checksum,
                                       image_iter):
        pass
    return True

def get_caching_iter(self, image_id, image_checksum, image_iter):
    """
    Returns an iterator that caches the contents of an image
    while the image contents are read through the supplied
    iterator.

    :param image_id: Image ID
    :param image_checksum: checksum expected to be generated while
                           iterating over image data
    :param image_iter: Iterator that will read image contents
    """
    if not self.driver.is_cacheable(image_id):
        return image_iter

    LOG.debug("Tee'ing image '%s' into cache", image_id)

    return self.cache_tee_iter(image_id, image_iter, image_checksum)

def cache_tee_iter(self, image_id, image_iter, image_checksum):
    try:
        current_checksum = hashlib.md5()

        with self.driver.open_for_write(image_id) as cache_file:
            for chunk in image_iter:
                try:
                    cache_file.write(chunk)
                finally:
                    current_checksum.update(chunk)
                    yield chunk
            cache_file.flush()

            if (image_checksum and
                    image_checksum != current_checksum.hexdigest()):
                msg = _("Checksum verification failed. Aborted "
                        "caching of image '%s'.") % image_id
                raise exception.GlanceException(msg)

    except exception.GlanceException as e:
        with excutils.save_and_reraise_exception():
            # image_iter has given us bad, (size_checked_iter has found a
            # bad length), or corrupt data (checksum is wrong).
            LOG.exception(encodeutils.exception_to_unicode(e))
    except Exception as e:
        LOG.exception(_LE("Exception encountered while tee'ing "
                          "image '%(image_id)s' into cache: %(error)s. "
                          "Continuing with response.") %
                      {'image_id': image_id,
                       'error': encodeutils.exception_to_unicode(e)})

        # If no checksum provided continue responding even if
        # caching failed.
        for chunk in image_iter:
            yield chunk

Glance image cache清理流程

源生为一次性可执行命令，官方建议使用cron运行glance-cache-pruner

Src File：glance/image_cache/__init__.py

当前缓存文件大小与配置中的image_cache_max_size值进行比较，大于image_cache_max_size则开启清理操作，根据缓存文件的last_accessed字段，依次删除活跃度最低的镜像，直至缓存文件大小小于image_cache_max_size值

def prune(self):
    """
    Removes all cached image files above the cache's maximum
    size. Returns a tuple containing the total number of cached
    files removed and the total size of all pruned image files.
    """
    max_size = CONF.image_cache_max_size
    current_size = self.driver.get_cache_size()
    if max_size > current_size:
        LOG.debug("Image cache has free space, skipping prune...")
        return (0, 0)

    overage = current_size - max_size
    LOG.debug("Image cache currently %(overage)d bytes over max "
              "size. Starting prune to max size of %(max_size)d ",
              {'overage': overage, 'max_size': max_size})

    total_bytes_pruned = 0
    total_files_pruned = 0
    entry = self.driver.get_least_recently_accessed()
    while entry and current_size > max_size:
        image_id, size = entry
        LOG.debug("Pruning '%(image_id)s' to free %(size)d bytes",
                  {'image_id': image_id, 'size': size})
        self.driver.delete_cached_image(image_id)
        total_bytes_pruned = total_bytes_pruned + size
        total_files_pruned = total_files_pruned + 1
        current_size = current_size - size
        entry = self.driver.get_least_recently_accessed()

    LOG.debug("Pruning finished pruning. "
              "Pruned %(total_files_pruned)d and "
              "%(total_bytes_pruned)d.",
              {'total_files_pruned': total_files_pruned,
               'total_bytes_pruned': total_bytes_pruned})
    return total_files_pruned, total_bytes_pruned

参考文档

• OpenStack Documentation：The Glance Image Cache

• OpenStack Documentation：Glance Administration Guide

• OpenStack Documentation：Basic Configuration