OpenStack迁移虚拟机流程分析

1、 在OpenStack中，虚拟机的迁移类似分为三种，分别是冷迁移、热迁移和故障迁移。冷迁移实现原理：使原来所需的资源在标节点上重新创建个虚拟机。云主机冷迁移流程图：更详细的过程图：发起云主机冷迁移后，先调到的是nova/api/openstack/compute/migrate_server.py的_migrate函数：wsgi.response(202)extensions.expected_errors(400, 403, 404, 409)wsgi.action()def_migrate(self, req, id, body):Permit admins to migrate a se

2、rver to a new host.context = req.environnova.contextcontext.can(ms_policies.POLICY_ROOT % )host = bodymigratehostinstance = common.get_instance(pute_api, context, id)try:pute_api.resize(req.environ nova.context, instance, host=host).这的核代码是调到了resize函数，openstack本还有个resize功能，它的作是对云主机的配置进升级，但只能往上升，冷迁移的流

3、程跟resize作流程样，只不过是flavor没有发改变。实现代码是在nova/compute/api.py：check_instance_lockcheck_instance_cellcheck_instance_state(vm_state=vm_states.ACTIVE, vm_states.STOPPED)defresize(self, context, instance, flavor_id=None, clean_shutdown=True,host=None,*extra_instance_updates):# 检查flavor是否是新flavor，如果是resize则进配额预

4、留分配、修改虚拟机的状态、# 提交迁移记录和成的宿主机需满的条件对象pute_task_api.resize_instance(context, instance,extra_instance_updates, scheduler_hint=scheduler_hint,flavor=new_instance_type,reservations=quotas.reservations or,clean_shutdown=clean_shutdown,request_spec=request_spec,host=host)这的核是调resize_instance函数进处理，该函数实现在nova/

5、conductor/api.py，该函数再调了nova/conductor/rpcapi.py件中的migrate_server函数：defmigrate_server(self, context, instance, scheduler_hint, live, rebuild,flavor, block_migration, disk_over_commit,reservations=None, clean_shutdown=True, request_spec=None, host=None):# 根据版本号构建kw参数returncctxt.call(context, migrate_s

6、erver, *kw)通过远程调conductor进程的migrate_server函数，此时进nova/conductor/manage.py件中的migrate_server函数：defmigrate_server(self, context, instance, scheduler_hint, live, rebuild,flavor, block_migration, disk_over_commit, reservations=None,clean_shutdown=True, request_spec=None, host=None):# 些条件判断，判断是进冷迁移还是热迁移if

7、notlive andnotrebuildandflavor:# 冷迁移这个逻辑instance_uuid = instance.uuidwith compute_utils.EventReporter(context, cold_migrate,instance_uuid):self._cold_migrate(context, instance, flavor,scheduler_hint filter_properties,reservations, clean_shutdown, request_spec,host=host)else:raiseNotImplementedError(

8、)核代码是调了_cold_migrate函数：wrap_instance_event(prefix=conductor)def_cold_migrate(self, context, instance, flavor, filter_properties,reservations, clean_shutdown, request_spec, host=None):image = utils.get_image_from_system_metadata(instance.system_metadata)task = self._build_cold_migrate_task(context, i

9、nstance, flavor,request_spec,reservations, clean_shutdown, host=host)task.execute()构建迁移任务，然后执任务：def_build_cold_migrate_task(self, context, instance, flavor,request_spec, reservations,clean_shutdown, host=None):# nova/conductor/tasks/migrate.pyreturnmigrate.MigrationTask(context, instance, flavor,req

10、uest_spec,reservations, clean_shutdown,pute_rpcapi,self.scheduler_client,host=host)这是返回了MigrationTask类实例，该类继承的TaskBase基类的execute函数会调_execute函数，所以我们直接看该MigrationTask类的execute函数实现即可：def_execute(self):# 选择个宿主机pute_rpcapi.prep_resize(self.context, self.instance, legacy_spec ,self.flavor, host, self.rese

11、rvations,request_spec=legacy_spec, filter_properties=legacy_props,node=node, clean_shutdown=self.clean_shutdown)这调到了nova/compute/rpcapi.py中的prep_resize函数：defprep_resize(self, ctxt, instance, image, instance_type, host,reservations=None, request_spec=None,filter_properties=None, node=None,clean_shutd

12、own=True):image_p = jsonutils.to_primitive(image)msg_args = instance: instance,instance_type: instance_type,: image_p,reservations: reservations,request_spec: request_spec,filter_properties: filter_properties,node: node,clean_shutdown: clean_shutdownversion = 4.1client = self.router.by_host(ctxt, ho

13、st)if notclient.can_send_version(version):version = 4.0msg_argsinstance_type = objects_base.obj_to_primitive(instance_type)cctxt = client.prepare(server=host, version=version)# 远程调到宿主机上让其准备将要迁移过去的虚拟机的资源cctxt.cast(ctxt, , *msg_args)这远程调到了nova/compute/manager.py中的prep_resize函数，该函数的核代码是调了_prep_resize：d

14、ef_prep_resize(self, context, image, instance, instance_type,quotas, request_spec, filter_properties, node,clean_shutdown=True):.rt = self._get_resource_tracker()# 这进了资源的检查和预留分配并更新数据库宿主机更新后的资源with rt.resize_claim(context, instance, instance_type, node,image_meta=image, limits=limits) as claim:LOG.in

15、fo(_LI(Migrating), instance=instance)pute_rpcapi.resize_instance(context, instance, claim.migration, image,instance_type, quotas.reservations,clean_shutdown)分配好资源后调pute_rpcapi.resize_instance函数：defresize_instance(self, ctxt, instance, migration, image, instance_type,reservations=None, clean_shutdown

16、=True):msg_args = instance: instance, migration: migration,: image, reservations: reservations,instance_type: instance_type,clean_shutdown: clean_shutdown,version = 4.1client = self.router.by_instance(ctxt, instance)if notclient.can_send_version(version):msg_args instance_type = objects_base.obj_to_

17、primitive(instance_type)version = 4.0cctxt = client.prepare(server=_compute_host(None, instance),version=version)# 到源主机的nova/compute/manager.py中执resize_instance函数cctxt.cast(ctxt, resize_instance, *msg_args)进源主机的resize_instance函数中：defresize_instance(self, context, instance, image,reservations, migrat

18、ion, instance_type,clean_shutdown):# 获取要迁移的云主机卡信息# 修改数据库云主机状态# 迁移事件通知# 关机并进磁盘迁移disk_info = self.driver.migrate_disk_and_power_off(context, instance, migration.dest_host,instance_type, network_info,block_device_info,timeout, retry_interval)# 开始为虚拟机迁移络work_api.migrate_instance_start(context,instance,m

19、igration_p)pute_rpcapi.finish_resize(context, instance,migration, image, disk_info,migration.dest_compute, reservations=quotas.reservations)此时远程调的主机的finish_resize函数：deffinish_resize(self, context, disk_info, image, instance,reservations, migration):# 提交配额.self._finish_resize(context, instance, migra

20、tion,disk_info, image_meta)def_finish_resize(self, context, instance, migration, disk_info,image_meta):# 初始化络work_api.setup_networks_on_host(context, instance,migration dest_compute)migration_p = obj_base.obj_to_primitive(migration)work_api.migrate_instance_finish(context,instance,migration_p)# 获取当前

21、云主机的络信息network_info = work_api.get_instance_nw_info(context, instance)# 更新数据库状态instance.task_state = task_states.RESIZE_FINISHinstance.save(expected_task_state=task_states.RESIZE_MIGRATED)# nova/virt/libvirt/driver.pyself.driver.finish_migration(context, migration, instance,disk_info,network_info,im

22、age_meta, resize_instance,block_device_info, power_on)最后还要再出发次confirmresize函数完成整个冷迁移过程，该函数是确认在源主机上删除云主机的数据和络数据等，函数件在nova/api/openstack/compute/servers.py：wsgi.action(confirmResize)def_action_confirm_resize(self, req, id, body):pute_api.confirm_resize(context, instance)defconfirm_resize(self, context

23、, instance, migration=None):Confirms a migration/resize and deletes the old instance.# 修改迁移状态和更新配额.pute_rpcapi.confirm_resize(context,instance,migration,migration.source_compute,quotas.reservations or)wsgi.action(confirmResize)def_action_confirm_resize(self, req, id, body):pute_api.confirm_resize(co

24、ntext, instance)defconfirm_resize(self, context, instance, migration=None):Confirms a migration/resize and deletes the old instance.# 修改迁移状态和更新配额.pute_rpcapi.confirm_resize(context,instance,migration,migration.source_compute,quotas.reservations or)通过rpc调到源宿主机上进confirm_resize，这函数中较核的部分是调了_confirm_res

25、ize函数：def_confirm_resize(self, context, instance, quotas,migration=None):Destroys the source instance.self._notify_about_instance_usage(context, instance,resize.confirm.start)# NOTE(tr3buchet): tear down networks on source host# 断掉络work_api.setup_networks_on_host(context, instance,migration.source_c

26、ompute, teardown=True)network_info = work_api.get_instance_nw_info(context,instance)# TODO(mriedem): GetBDMs here and pass them to the driver.# 删除虚拟机self.driver.confirm_migration(context, migration, instance,network_info)# 更新迁移状态migration.status = confirmedwith migration.obj_as_admin():migration.sav

27、e()# 更新资源rt = self._get_resource_tracker()rt.drop_move_claim(context, instance, migration.source_node,old_instance_type, prefix= )instance.drop_migration_context()冷迁移过程总结：（1）nova-api收到冷迁移请求，验证权限、配额等并获取虚拟机信息，通过消息队列向nova-conductor发起冷迁移请求（2）nova-conductor通过消息队列请求nova-scheduler选择可迁移的宿主机（3）获取到的宿主机后，nova-

28、conductor通过消息队列请求的宿主机的nova-compute服务做资源准备作（4）的宿主机进资源准备作，如执claim机制检测和预分配资源，完成后通过消息队列请求源宿主机进虚拟机迁移准备作（5）源宿主机进关机、卸载络设备、磁盘等资源，完成后通过消息队列请求的宿主机让它初始化好虚拟机所需的资源，如络虚拟设备的创建和磁盘挂载等并更新虚拟机状态为等待被确认。（6）最后需要发起confirm_resize命令删除源主机上的该虚拟机的资源并更新虚拟机状态为正常，如备份的虚拟机的录热迁移实现原理：热迁移与冷迁移作流程类似，但热迁移由于是在运中迁移，进了较多的兼容性判断，如两个宿主机之间的cpu兼容

29、性等。热迁移其实并没有业务中断，只是在迁移的最后时刻，虚拟机会有短暂挂起，快速完成最后次内存复制。影响热迁移的关键因素有两个：（1）虚拟机内存脏页的速度，迭代复制是以页为单位的；（2）络带宽，如果脏页的速度远于迭代复制内存页的速度，在段时间内迁移是不成功的。libvirtd数据迁移逻辑：（1）标记所有的脏内存；（2）传输所有的脏内存，然后开始重新计算新产的脏内存，如此迭代，知道某个条件退出；（3）暂停虚拟机，传输剩余数据；第（2）步的某个条件可以是：（1）50%或者更少的内存需要迁移；（2）迭代次数不超过多少次；热迁移流程图：发起热迁移请求后，nova中的函数是nova-api的migrate

30、_server.py件中的_migrate_live函数：wsgi.action(os-migrateLive)validation.schema(migrate_server.migrate_live, 2.0, 2.24)validation.schema(migrate_server.migrate_live_v2_25, 2.25, 2.29)validation.schema(migrate_server.migrate_live_v2_30, 2.30)def_migrate_live(self, req, id, body):pute_api.live_migrate(conte

31、xt, instance, block_migration,disk_over_commit, host, force, async)调到nova/compute/api.py中：deflive_migrate(self, context, instance, block_migration,disk_over_commit, host_name, force=None, async=False):# 修改虚拟机状态# 成request_spec.# nova/conductor/pute_task_api.live_migrate_instance(context, instance,hos

32、t_name, block_migration=block_migration,disk_over_commit=disk_over_commit,request_spec=request_spec, async=async)live_migrate_instance调到了同级下的rpcapi.py中的live_migrate_instance函数：deflive_migrate_instance(self, context, instance, scheduler_hint,block_migration, disk_over_commit, request_spec):kw = insta

33、nce: instance, scheduler_hint: scheduler_hint,block_migration: block_migration,disk_over_commit: disk_over_commit,request_spec: request_spec,version = 1.15cctxt = self.client.prepare(version=version)# 交到conductor进程中处理cctxt.cast(context, live_migrate_instance, *kw)wrap_instance_event(prefix=conductor

34、)deflive_migrate_instance(self, context, instance, scheduler_hint,block_migration, disk_over_commit, request_spec):self._live_migrate(context, instance, scheduler_hint,block_migration, disk_over_commit, request_spec)def_live_migrate(self, context, instance, scheduler_hint,block_migration, disk_over_

35、commit, request_spec):destination = scheduler_hint.get( host)# 初始化个热迁移任务跟踪对象migration = objects.Migration(context=context.elevated()migration.dest_compute = destinationmigration.status = acceptedmigration.instance_uuid = instance.uuidmigration.source_compute = instance.hostmigration.migration_type =

36、 live-migrationif instance.obj_attr_is_set(flavor):migration.old_instance_type_id = instance.flavor.idmigration.new_instance_type_id = instance.flavor.idelse:migration.old_instance_type_id = instance.instance_type_idmigration.new_instance_type_id = instance.instance_type_idmigration.create()# 创建热迁移任

37、务task = self._build_live_migrate_task(context, instance, destination,block_migration, disk_over_commit,migration, request_spec)task.execute()def_build_live_migrate_task(self, context, instance, destination,block_migration, disk_over_commit, migration,request_spec=None):# nova/conductor/tasks/live_mi

38、grate.pyreturnlive_migrate.LiveMigrationTask(context, instance,destination, block_migration,disk_over_commit, migration,pute_rpcapi,self.servicegroup_api,self.scheduler_client,request_spec)我们可以直接看nova/conductor/tasks/live_migrate.py件中的类LiveMigrationTask中的_execute函数实现：def_execute(self):# 检查云主机是否activ

39、e状态self._check_instance_is_active()# 检查源主机宿主机是否可状态self._check_host_is_up(self.source)# 如果没有指定的主机，则调scheduler去选择合适的的主机if notself.destination:self.destination = self._find_destination()self.migration.dest_compute = self.destinationself.migration.save()else:# 如果指定了还需要对它做些条件检查，判断是否的主机满热迁移self._check_req

40、uested_destination()# TODO(johngarbutt) need to move complexity out of compute manager# TODO(johngarbutt) disk_over_commit?pute_rpcapi.live_migration(self.context,host=self.source,instance=self.instance,dest=self.destination,block_migration=self.block_migration,migration=self.migration,migrate_data=

41、self.migrate_data)def_check_requested_destination(self):# 确保源主机和宿主机不是同物理主机self._check_destination_is_not_source()# 检查的主机是否可self._check_host_is_up(self.destination)# 检查的主机是否有够的内存self._check_destination_has_enough_memory()# 检查两个源主机和宿主机之间是否兼容self._check_compatible_with_source_hypervisor(self.destinatio

42、n)# 检查下是否可以对的主机执热迁移操作self._call_livem_checks_on_host(self.destination)其中_call_livem_checks_on_host函数会远程调到的主机上去执check_can_live_migrate_destination函数来检验的主机是否满热迁移，同时的主机也会远程调check_can_live_migrate_source函数检查源主机是否持热迁移。def_do_check_can_live_migrate_destination(self, ctxt, instance, block_migration, disk_o

43、ver_commit):src_compute_info = obj_base.obj_to_primitive(self._get_compute_info(ctxt, instance.host)dst_compute_info = obj_base.obj_to_primitive(self._get_compute_info(ctxt, CONF.host)# nova/virt/libvirt/driver.pydest_check_data = self.driver.check_can_live_migrate_destination(ctxt,instance, src_com

44、pute_info, dst_compute_info,block_migration, disk_over_commit)LOG.debug(destination check data is %s, dest_check_data)try:# 远程调回源主机上检查源主机是否可热迁移migrate_data = pute_rpcapi.check_can_live_migrate_source(ctxt, instance,dest_check_data)finally:# 删除些检查时产的临时件self.driver.cleanup_live_migration_destination_c

45、heck(ctxt,dest_check_data)returnmigrate_data接着便是调到了nova/compute/rpcapi.py中的live_migration函数，该函数远程调了nova-compute服务的live_migrate法，交给nova-compute服务来进处理：deflive_migration(self, context, dest, instance, block_migration,migration, migrate_data):self._do_live_migration(*args, *kwargs)def_do_live_migration(

46、self, context, dest, instance, block_migration,migration, migrate_data):# nova/compute/rpcapi.pymigrate_data = pute_rpcapi.pre_live_migration(context, instance,block_migration, disk, dest, migrate_data)self.driver.live_migration(context, instance, dest,self._post_live_migration,self._rollback_live_m

47、igration,block_migration, migrate_data)_do_live_migration函数中的核代码是pre_live_migration和live_migration的调。先看pre_live_migration函数，其是远程调到的主机上执pre_live_migration函数：defpre_live_migration(self, context, instance, block_migration, disk,migrate_data):block_device_info = self._get_instance_block_device_info(cont

48、ext, instance, refresh_conn_info=True)network_info = work_api.get_instance_nw_info(context, instance)self._notify_about_instance_usage(context, instance, live_migration.pre.start,network_info=network_info)# 做连接上磁盘和挂载上络的作等migrate_data = self.driver.pre_live_migration(context,instance,block_device_inf

49、o,network_info,disk,migrate_data)LOG.debug(driver pre_live_migration data is %s, migrate_data)# NOTE(tr3buchet): setup networks on destination host# 初始化好络work_api.setup_networks_on_host(context, instance,self.host)# Creating filters to hypervisors and firewalls.# An example is that nova-instance-ins

50、tance-xxx,# which is written to libvirt.xml(Check virsh nwfilter-list)# This nwfilter is necessary on the destination host.# In addition, this method is creating filtering rule# onto destination host.# 在热迁移进前在的主机上创建好那些络过滤规则self.driver.ensure_filtering_rules_for_instance(instance,network_info)在的主机上执完

51、pre_live_migration函数后，源主机上调live_migration开始执热迁移操作，调到nova/virt/libvirt/driver.py中的live_migration函数，再调到_live_migration函数：def_live_migration(self, context, instance, dest, post_method,recover_method, block_migration,migrate_data):opthread = utils.spawn(self._live_migration_operation,context, instance,

52、dest,block_migration,migrate_data, guest,device_names)self._live_migration_monitor(context, instance, guest, dest,post_method, recover_method,block_migration, migrate_data,finish_event, disk_paths)这主要有两个核调，个是_live_migration_operation进迁移操作，个是调_live_migration_monitor函数以监控迁移进度。def_live_migration_operat

53、ion(self, context, instance, dest,block_migration, migrate_data, guest,device_names):# 调libvirt的api来实现迁移 # nova/virt/libvirt/guest.pyguest.migrate(self._live_migration_uri(dest),migrate_uri=migrate_uri,flags=migration_flags,params=params,domain_xml=new_xml_str,bandwidth=CONF.libvirt.live_migration_b

54、andwidth)defmigrate(self, destination, migrate_uri=None, params=None, flags=0,domain_xml=None, bandwidth=0):# 调了libvirt的python接virDomainMigrateToURI来实现从当前云主机迁移domain对象到给定的标主机if domain_xml is None:self._domain.migrateToURI(destination, flags=flags, bandwidth=bandwidth)else:if params:if migrate_uri:#

55、In migrateToURI3 this paramenter is searched in# the params dictparams = migrate_uriself._domain.migrateToURI3(destination, params=params, flags=flags)else:self._domain.migrateToURI2(destination, miguri=migrate_uri, dxml=domain_xml,flags=flags, bandwidth=bandwidth)_live_migration_monitor的主要实现则是调了lib

56、virt的job_info函数获取进度情况。热迁移过程总结：（1）nova-api收到冷迁移请求，验证权限、配额等并获取虚拟机信息，通过消息队列向nova-conductor发起热迁移请求（2）nova-conductor检查虚拟机是否是开机状态并检查源宿主机计算服务是否正常，然后通过消息队列请求nova-scheduler服务选择的宿主机（3）获取到的宿主机后，对的宿主机进多项是否符合热迁移条件的检查，如两宿主机的CPU是否兼容、的宿主机是否可以做热迁移（这会有个互相检查对是否可以进热迁移）等（4）nova-conductor通过消息队列服务让的宿主机执热迁移作（5）的宿主机进络初始化、络过

57、滤规则和磁盘准备作等并通过消息队列让源宿主机执热迁移操作（6）源宿主机调libvirt的热迁移API进热迁移操作故障迁移实现原理：根据在数据库中保存的配置，重新成个样的云主机，前提是需要能访问到故障云主机的磁盘数据，所以使共享存储可以实现云主机故障迁移。函数代码件：nova/api/openstack/compute/evacuate.py发起故障迁移请求后进的函数是：def_evacuate(self, req, id, body):Permit admins to evacuate a server from a failed hostto a new one.pute_api.evacu

58、ate(context, instance, host,on_shared_storage, password, force)执nova/compute/api.py中的evacuate函数：defevacuate(self, context, instance, host, on_shared_storage,admin_password=None, force=None):# 检查源宿主机是否可，可则停继续执# 修改云主机状态和成request_spec.# nova/conductor/pute_task_api.rebuild_instance(context,instance=ins

59、tance,new_pass=admin_password,injected_files=None,image_ref=None,orig_image_ref=None,orig_sys_metadata=None,bdms=None,recreate=True,on_shared_storage=on_shared_storage,host=host,request_spec=request_spec,)rpcnova/compute/manager.py rebuild_instance的法进处理：defrebuild_instance(self, context, instance, o

60、rig_image_ref, image_ref,injected_files, new_pass, orig_sys_metadata,bdms,recreate, on_shared_storage=None,preserve_ephemeral=False, migration=None,scheduled_node=None, limits=None):# 获取物理主机资源# 获取镜像元数据self._do_rebuild_instance_with_claim(claim_ctxt, context, instance, orig_image_ref,image_ref, injec