含有多个设备的GPU

常见型号：Quadro RTX 6000/8000

简述

系统内lspci查看显卡信息如下，可以看到同一个PCI插槽上（b1）包含了4个设备：VGA compatible、Audio device、USB、Serial bus，同时4个设备对应的驱动各不相同。

lspci信息

原始驱动

目前cyborg暂不支持此类加速设备的管理，cyborg采集的attach handle info中，只包含了function 0，即生成的xml中仅会透传GPU中的1个设备进入虚机，由于设备不完整，会导致libvirt会无法拉起虚机。

cyborg采集到的信息

使用此类GPU，需要通过nova compute的PCI透传功能实现，总体思路如下：

  1. 切换GPU卡各类设备的驱动为vfio_pci。
  2. 透传GPU卡的所有设备至虚机。

若这两点有一点不满足，libvirt在拉起虚机时均会报如下报错：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
2022-08-20 16:39:06.047 3711 INFO nova.compute.claims [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Claim successful on node node01
2022-08-20 16:39:06.259 3711 WARNING nova.virt.osinfo [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] Cannot find OS information - Reason: (No configuration information found for operating system Windows): OsInfoNotFound: No configuration information found for operating system Windows
2022-08-20 16:39:06.282 3711 INFO nova.virt.libvirt.driver [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Ignoring supplied device name: /dev/vda. Libvirt can't honour user-supplied dev names
2022-08-20 16:39:06.283 3711 WARNING nova.virt.osinfo [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] Cannot find OS information - Reason: (No configuration information found for operating system Windows): OsInfoNotFound: No configuration information found for operating system Windows
2022-08-20 16:39:06.402 3711 INFO nova.virt.block_device [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Booting with volume-backed-image eb3aee11-f487-45e3-ac36-984904ad010a at /dev/vda
2022-08-20 16:39:12.973 3711 WARNING nova.virt.osinfo [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] Cannot find OS information - Reason: (No configuration information found for operating system Windows): OsInfoNotFound: No configuration information found for operating system Windows
2022-08-20 16:39:12.975 3711 INFO nova.virt.libvirt.driver [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Creating image
2022-08-20 16:39:13.008 3711 WARNING nova.virt.osinfo [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] Cannot find OS information - Reason: (No configuration information found for operating system Windows): OsInfoNotFound: No configuration information found for operating system Windows
2022-08-20 16:39:13.013 3711 WARNING nova.virt.libvirt.driver [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] USB tablet requested for guests by host configuration. In order to accept this request VNC should be enabled or SPICE and SPICE agent disabled on host.
2022-08-20 16:39:15.811 3711 ERROR nova.virt.libvirt.guest [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] Error launching a defined domain with XML: <domain type='kvm'>
  <name>instance-00000007</name>
  <uuid>285e807d-426d-4b4a-b500-e89f92b9172a</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="20.6.1"/>
      <nova:name>win10-1</nova:name>
      <nova:creationTime>2022-08-20 16:39:12</nova:creationTime>
      <nova:flavor name="win10_8C16G">
        <nova:memory>16384</nova:memory>
        <nova:disk>100</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>8</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="20f2649ce88142f19048b0aa090b78a1">admin</nova:user>
        <nova:project uuid="a1670d65a7e644e2a78fa31a001f284b">admin</nova:project>
      </nova:owner>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <shares>8192</shares>
    <vcpupin vcpu='0' cpuset='3-15,35-47'/>
    <vcpupin vcpu='1' cpuset='3-15,35-47'/>
    <vcpupin vcpu='2' cpuset='3-15,35-47'/>
    <vcpupin vcpu='3' cpuset='3-15,35-47'/>
    <vcpupin vcpu='4' cpuset='3-15,35-47'/>
    <vcpupin vcpu='5' cpuset='3-15,35-47'/>
    <vcpupin vcpu='6' cpuset='3-15,35-47'/>
    <vcpupin vcpu='7' cpuset='3-15,35-47'/>
    <emulatorpin cpuset='3-15,35-47'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>20.6.1</entry>
      <entry name='serial'>285e807d-426d-4b4a-b500-e89f92b9172a</entry>
      <entry name='uuid'>285e807d-426d-4b4a-b500-e89f92b9172a</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <topology sockets='2' cores='1' threads='4'/>
    <numa>
      <cell id='0' cpus='0-7' memory='16777216' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
      </auth>
      <source protocol='rbd' name='cinder-volumes/a728e5a5-ca72-4d70-8d2f-4b193d88a595'>
        <host name='10.99.22.45' port='6789'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <serial>a728e5a5-ca72-4d70-8d2f-4b193d88a595</serial>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='usb' index='1' model='ehci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='usb' index='2' model='nec-xhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='usb' index='3' model='ehci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='usb' index='4' model='nec-xhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:58:02:a6'/>
      <source bridge='br-int'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='2fdb3a80-a1ca-4927-b11e-2c40024dad69'/>
      </virtualport>
      <target dev='tap2fdb3a80-a1'/>
      <model type='virtio'/>
      <mtu size='1500'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <log file='/var/lib/nova/instances/285e807d-426d-4b4a-b500-e89f92b9172a/console.log' append='off'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <log file='/var/lib/nova/instances/285e807d-426d-4b4a-b500-e89f92b9172a/console.log' append='off'/>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/org.qemu.guest_agent.0.instance-00000007.sock'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <sound model='ac97'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='32768' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xb1' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='3' port='1'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='3' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='4' port='1'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='4' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='3' port='3'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='3' port='4'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='4' port='3'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='4' port='4'/>
    </redirdev>
    <memballoon model='virtio'>
      <stats period='10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </memballoon>
  </devices>
</domain>
: libvirtError: internal error: process exited while connecting to monitor: 2022-08-20T16:39:15.220119Z qemu-kvm: -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=32,max_outputs=1,bus=pci.0,addr=0x2: vhost_region_add_section:Section rounded to 0 prior to previous c0000
2022-08-20 16:39:15.813 3711 ERROR nova.virt.libvirt.driver [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Failed to start libvirt guest: libvirtError: internal error: process exited while connecting to monitor: 2022-08-20T16:39:15.220119Z qemu-kvm: -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=32,max_outputs=1,bus=pci.0,addr=0x2: vhost_region_add_section:Section rounded to 0 prior to previous c0000
2022-08-20 16:39:15.864 3711 INFO nova.virt.libvirt.driver [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Deleting instance files /var/lib/nova/instances/285e807d-426d-4b4a-b500-e89f92b9172a_del
2022-08-20 16:39:15.865 3711 INFO nova.virt.libvirt.driver [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Deletion of /var/lib/nova/instances/285e807d-426d-4b4a-b500-e89f92b9172a_del complete
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Instance failed to spawn: libvirtError: internal error: process exited while connecting to monitor: 2022-08-20T16:39:15.220119Z qemu-kvm: -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=32,max_outputs=1,bus=pci.0,addr=0x2: vhost_region_add_section:Section rounded to 0 prior to previous c0000
2022-08-20T16:39:15.220245Z qemu-kvm: -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=32,max_outputs=1,bus=pci.0,addr=0x2: vhost_region_add_section:Section rounded to 0 prior to previous c0000
2022-08-20T16:39:15.224010Z qemu-kvm: -device vfio-pci,host=0000:b1:00.0,id=hostdev0,bus=pci.0,addr=0xb: vfio error: 0000:b1:00.0: group 183 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.
unload codec helper module
unload codec helper module done
unload x264 encoder module done
unload aac encode compress module
unload aac encode compress done
unload aac decode compress module
unload aac decode compress done
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Traceback (most recent call last):
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2808, in _build_resources
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     yield resources
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2567, in _build_and_run_instance
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     accel_info=accel_info)
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3524, in spawn
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     power_on=power_on)
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6431, in _create_domain_and_network
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     destroy_disks_on_failure)
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     self.force_reraise()
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     six.reraise(self.type_, self.value, self.tb)
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6397, in _create_domain_and_network
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     post_xml_callback=post_xml_callback)
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6328, in _create_domain
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     guest.launch(pause=pause)
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 145, in launch
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     self._encoded_xml, errors='ignore')
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     self.force_reraise()
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     six.reraise(self.type_, self.value, self.tb)
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 140, in launch
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     return self._domain.createWithFlags(flags)
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 190, in doit
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 148, in proxy_call
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     rv = execute(f, *args, **kwargs)
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 129, in execute
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     six.reraise(c, e, tb)
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     rv = meth(*args, **kwargs)
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1110, in createWithFlags
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] libvirtError: internal error: process exited while connecting to monitor: 2022-08-20T16:39:15.220119Z qemu-kvm: -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=32,max_outputs=1,bus=pci.0,addr=0x2: vhost_region_add_section:Section rounded to 0 prior to previous c0000
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] 2022-08-20T16:39:15.220245Z qemu-kvm: -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=32,max_outputs=1,bus=pci.0,addr=0x2: vhost_region_add_section:Section rounded to 0 prior to previous c0000
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] 2022-08-20T16:39:15.224010Z qemu-kvm: -device vfio-pci,host=0000:b1:00.0,id=hostdev0,bus=pci.0,addr=0xb: vfio error: 0000:b1:00.0: group 183 is not viable
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Please ensure all devices within the iommu_group are bound to their vfio bus driver.
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] unload codec helper module
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] unload codec helper module done
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] unload x264 encoder module done
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] unload aac encode compress module
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] unload aac encode compress done
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] unload aac decode compress module
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] unload aac decode compress done
2022-08-20 16:39:15.950 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]
2022-08-20 16:39:15.956 3711 INFO nova.compute.manager [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Terminating instance
2022-08-20 16:39:15.961 3711 INFO nova.virt.libvirt.driver [-] [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Instance destroyed successfully.
2022-08-20 16:39:15.998 3711 INFO nova.virt.libvirt.driver [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Deletion of /var/lib/nova/instances/285e807d-426d-4b4a-b500-e89f92b9172a_del complete
2022-08-20 16:39:16.104 3711 INFO nova.compute.manager [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Took 0.15 seconds to destroy the instance on the hypervisor.
2022-08-20 16:39:16.412 3711 INFO nova.compute.manager [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Took 0.31 seconds to detach 1 volumes for instance.
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Failed to build and run instance: libvirtError: internal error: process exited while connecting to monitor: 2022-08-20T16:39:15.220119Z qemu-kvm: -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=32,max_outputs=1,bus=pci.0,addr=0x2: vhost_region_add_section:Section rounded to 0 prior to previous c0000
2022-08-20T16:39:15.220245Z qemu-kvm: -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=32,max_outputs=1,bus=pci.0,addr=0x2: vhost_region_add_section:Section rounded to 0 prior to previous c0000
2022-08-20T16:39:15.224010Z qemu-kvm: -device vfio-pci,host=0000:b1:00.0,id=hostdev0,bus=pci.0,addr=0xb: vfio error: 0000:b1:00.0: group 183 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.
unload codec helper module
unload codec helper module done
unload x264 encoder module done
unload aac encode compress module
unload aac encode compress done
unload aac decode compress module
unload aac decode compress done
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Traceback (most recent call last):
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2567, in _build_and_run_instance
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     accel_info=accel_info)
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3524, in spawn
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     power_on=power_on)
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6431, in _create_domain_and_network
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     destroy_disks_on_failure)
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     self.force_reraise()
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     six.reraise(self.type_, self.value, self.tb)
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6397, in _create_domain_and_network
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     post_xml_callback=post_xml_callback)
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6328, in _create_domain
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     guest.launch(pause=pause)
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 145, in launch
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     self._encoded_xml, errors='ignore')
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     self.force_reraise()
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     six.reraise(self.type_, self.value, self.tb)
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 140, in launch
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     return self._domain.createWithFlags(flags)
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 190, in doit
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 148, in proxy_call
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     rv = execute(f, *args, **kwargs)
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 129, in execute
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     six.reraise(c, e, tb)
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     rv = meth(*args, **kwargs)
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1110, in createWithFlags
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] libvirtError: internal error: process exited while connecting to monitor: 2022-08-20T16:39:15.220119Z qemu-kvm: -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=32,max_outputs=1,bus=pci.0,addr=0x2: vhost_region_add_section:Section rounded to 0 prior to previous c0000
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] 2022-08-20T16:39:15.220245Z qemu-kvm: -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=32,max_outputs=1,bus=pci.0,addr=0x2: vhost_region_add_section:Section rounded to 0 prior to previous c0000
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] 2022-08-20T16:39:15.224010Z qemu-kvm: -device vfio-pci,host=0000:b1:00.0,id=hostdev0,bus=pci.0,addr=0xb: vfio error: 0000:b1:00.0: group 183 is not viable
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Please ensure all devices within the iommu_group are bound to their vfio bus driver.
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] unload codec helper module
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] unload codec helper module done
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] unload x264 encoder module done
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] unload aac encode compress module
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] unload aac encode compress done
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] unload aac decode compress module
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] unload aac decode compress done
2022-08-20 16:39:16.770 3711 ERROR nova.compute.manager [instance: 285e807d-426d-4b4a-b500-e89f92b9172a]
2022-08-20 16:39:18.018 3711 INFO nova.compute.manager [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] [instance: 285e807d-426d-4b4a-b500-e89f92b9172a] Took 1.20 seconds to deallocate network for instance.
2022-08-20 16:39:18.330 3711 INFO nova.scheduler.client.report [req-5656f7a4-5fcf-4f5a-b72b-2f425087cfb4 20f2649ce88142f19048b0aa090b78a1 a1670d65a7e644e2a78fa31a001f284b - default default] Deleted allocation for instance 285e807d-426d-4b4a-b500-e89f92b9172a
2022-08-20 17:00:09.337 3711 INFO nova.compute.manager [req-c902ff9e-9ef9-484a-87ed-30e51a8c7b03 - - - - -] Running instance usage audit for host node01 from 2022-08-20 16:00:00 to 2022-08-20 17:00:00. 3 instances.

操作

a. 更换设备的驱动为vfio_pci

  1. 配置加载vfio-pci模块，编辑/etc/modules-load.d/openstack-gpu.conf，添加如下内容：

    1
    2
    3
    4
    5
    6
    vfio_pci
    pci_stub
    vfio
    vfio_iommu_type1
    kvm
    kvm_intel

  2. 配置使用vfio驱动的设备（这里的设备就是lspci查到的GPU设备信息），编辑/etc/modprobe.d/vfio.conf，添加如下配置：

    1
    options vfio-pci ids=10de:1e30,10de:10f7,10de:1ad6,10de:1ad7

  3. 重启系统，通过dmesg查看驱动加载情况。

dmesg信息

  1. 更新驱动后，GPU卡的各类设备驱动信息如下，Audio device与Serial bus的驱动变为vfio-pci。

更换vfio驱动后

b. 更改nova配置信息

  1. 修改nova-api.conf，增加如下PCI信息：

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    ## 配置直通PCI信息
    # 需直通的PCI Function均需配置
    #
    # 参数配置原则：
    # - name：名称随便定义，通常一块卡上的多个设备名称类似
    # - product_id：PCI Function对应的产品id，lspci -nn命令可查看product_id，显示格式：[<vendor_id>: <product_id>]
    # - vendor_id：PCI Function对应的厂商id，lspci -nn命令可查看vendor_id，显示格式：[<vendor_id>: <product_id>]
    # - device_type：设备类型，PCI设备固定为：type-PCI
    [pci]
    alias = {"name":"a1","product_id":"1e30","vendor_id":"10de","device_type":"type-PCI"}
    alias = {"name":"a2","product_id":"10f7","vendor_id":"10de","device_type":"type-PCI"}
    alias = {"name":"a3","product_id":"1ad6","vendor_id":"10de","device_type":"type-PCI"}
    alias = {"name":"a4","product_id":"1ad7","vendor_id":"10de","device_type":"type-PCI"}

  2. 修改nova-scheduler.conf文件，添加PciPassthroughFilter，同时添加available_filters = nova.scheduler.filters.all_filters，如下：

    1
    2
    3
    4
    ## 启用PciPassthroughFilter过滤器
    [filter_scheduler]
    enabled_filters = RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,AggregateCoreFilter,AggregateDiskFilter,DifferentHostFilter,SameHostFilter,PciPassthroughFilter
    available_filters = nova.scheduler.filters.all_filters

  3. 配置计算节点，编辑nova-compute.conf文件，如下：

    1
    2
    3
    4
    5
    6
    7
    8
    ## 配置直通白名单
    # 所有直通的PCI Function均需要配置
    [pci]
    alias = {"name":"a1","product_id":"1e30","vendor_id":"10de","device_type":"type-PCI"}
    alias = {"name":"a2","product_id":"10f7","vendor_id":"10de","device_type":"type-PCI"}
    alias = {"name":"a3","product_id":"1ad6","vendor_id":"10de","device_type":"type-PCI"}
    alias = {"name":"a4","product_id":"1ad7","vendor_id":"10de","device_type":"type-PCI"}
    passthrough_whitelist = [{"vendor_id": "10de", "product_id": "1e30"}, {"vendor_id": "10de", "product_id": "10f7"}, {"vendor_id": "10de", "product_id": "1ad6"}, {"vendor_id": "10de", "product_id": "1ad7"}]

c. 创建GPU Flavor

1
2
3
4
openstack flavor create --ram 2048 --disk 20 --vcpus 2 m1.large

# alias需要指定同一个PCI卡所包含的所有function设备
openstack flavor set m1.large --property pci_passthrough:alias='a1:1,a2:1,a3:1,a4:1'

使用该flavor创建虚机即可。

备注

正确instacne xml内容如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
()[root@node01 /]# virsh dumpxml instance-00000008
<domain type='kvm' id='5'>
  <name>instance-00000008</name>
  <uuid>1f4bb25c-d665-4f03-a0f4-6857af066781</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="20.6.1"/>
      <nova:name>win10-1</nova:name>
      <nova:creationTime>2022-08-20 17:14:33</nova:creationTime>
      <nova:flavor name="win10_4C8G_GPU">
        <nova:memory>8192</nova:memory>
        <nova:disk>100</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="20f2649ce88142f19048b0aa090b78a1">admin</nova:user>
        <nova:project uuid="a1670d65a7e644e2a78fa31a001f284b">admin</nova:project>
      </nova:owner>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <shares>4096</shares>
    <vcpupin vcpu='0' cpuset='17-31,49-63'/>
    <vcpupin vcpu='1' cpuset='17-31,49-63'/>
    <vcpupin vcpu='2' cpuset='17-31,49-63'/>
    <vcpupin vcpu='3' cpuset='17-31,49-63'/>
    <emulatorpin cpuset='17-31,49-63'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='1'/>
    <memnode cellid='0' mode='strict' nodeset='1'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>20.6.1</entry>
      <entry name='serial'>1f4bb25c-d665-4f03-a0f4-6857af066781</entry>
      <entry name='uuid'>1f4bb25c-d665-4f03-a0f4-6857af066781</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Cascadelake-Server</model>
    <vendor>Intel</vendor>
    <topology sockets='2' cores='1' threads='2'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='avx512ifma'/>
    <feature policy='require' name='sha-ni'/>
    <feature policy='require' name='avx512vbmi'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='pku'/>
    <feature policy='require' name='avx512vbmi2'/>
    <feature policy='require' name='gfni'/>
    <feature policy='require' name='vaes'/>
    <feature policy='require' name='vpclmulqdq'/>
    <feature policy='require' name='avx512bitalg'/>
    <feature policy='require' name='avx512-vpopcntdq'/>
    <feature policy='require' name='la57'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='xsaves'/>
    <feature policy='require' name='ibpb'/>
    <feature policy='disable' name='mpx'/>
    <feature policy='disable' name='arat'/>
    <numa>
      <cell id='0' cpus='0-3' memory='8388608' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
      </auth>
      <source protocol='rbd' name='cinder-volumes/9dc4fcd1-a258-4f5c-8f3f-9029f699b064'>
        <host name='10.99.22.45' port='6789'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <serial>9dc4fcd1-a258-4f5c-8f3f-9029f699b064</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </disk>
    <controller type='usb' index='1' model='ehci'>
      <alias name='usb1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='usb' index='2' model='nec-xhci'>
      <alias name='usb2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='usb' index='3' model='ehci'>
      <alias name='usb3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='usb' index='4' model='nec-xhci'>
      <alias name='usb4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:f1:a6:a9'/>
      <source bridge='br-int'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='f91ebe33-a80c-4038-a2f2-e543dbe509d9'/>
      </virtualport>
      <target dev='tapf91ebe33-a8'/>
      <model type='virtio'/>
      <mtu size='1500'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/5'/>
      <log file='/var/lib/nova/instances/1f4bb25c-d665-4f03-a0f4-6857af066781/console.log' append='off'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/5'>
      <source path='/dev/pts/5'/>
      <log file='/var/lib/nova/instances/1f4bb25c-d665-4f03-a0f4-6857af066781/console.log' append='off'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/org.qemu.guest_agent.0.instance-00000008.sock'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='spice' port='5903' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <sound model='ac97'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='32768' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0xb1' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0xb1' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0xb1' slot='0x00' function='0x2'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0xb1' slot='0x00' function='0x3'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
      <address type='usb' bus='3' port='1'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
      <address type='usb' bus='3' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir2'/>
      <address type='usb' bus='4' port='1'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir3'/>
      <address type='usb' bus='4' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir4'/>
      <address type='usb' bus='3' port='3'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir5'/>
      <address type='usb' bus='3' port='4'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir6'/>
      <address type='usb' bus='4' port='3'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir7'/>
      <address type='usb' bus='4' port='4'/>
    </redirdev>
    <memballoon model='virtio'>
      <stats period='10'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+0</label>
    <imagelabel>+0:+0</imagelabel>
  </seclabel>
</domain>

参考文档