The last thing that I want to mention very briefly is VM migration. Why do we need to do virtual machine migration? Well, one important reason is rebalancing of resources. You may have spun up more VMs on one physical machine, and in the data center, you might suddenly see that, or some physical machines are not getting utilized as much, and therefore, I want to rebalance the resources by moving some of the VM from one physical machine to another. Another reason could be that I'm doing incremental upgrades of the physical resources, and in which case, I may want to take some VMs that are running on some physical machines, move it over so that I can upgrade it, or deal with faults that happen and correcting the faults and so on. But the challenge in doing all of this is because of the fact that we have SLAs with individual users of the data center. You have to make sure that you minimize the downtime and avoid disruption of active services. Active services are things in which there is a human in the loop. If you take airline reservation or Gmail and things like that, those are active services as opposed to background activity. This background activities are machine learning algorithm that are running in the background, then it's not disrupting any active service. But when you're doing migration of active services like Gmail and supporting portals that are interactive, you have to make sure that you avoid the disruption of services. Terms of options for migration, there is a push phase, a stop and copy phase, and then a pull phase. I'll just elaborate on these different phases of a migration operation. The generic steps in migration, there are lots of details associated with VM migration and how to do it efficiently and so on. I'll have resources that you can look at if you're interested in knowing more about that. But the generic steps in doing VM migration is that there's a pre-migration step, where you have to reserve some resources. If I have some physical resource here and some physical resource here, and there is a VM that I want to move from here to here, you want to first do reservation of the resources. This is resource 1 and resource 2, and I want to make sure that there's capacity here for me to move this guy over to this physical resource. Once I have negotiated that and I've reserved it, then in order to reduce the downtime, this is important, you want to reduce the downtime. I cannot say, "Okay, I'm going to stop this VM on R1 and then start it on R2." If I'm going to start to do a cold start, it's going to take a lot of time to do that. What you do is, to reduce the downtime, you do iterative pre-copy, meaning you start migrating states into this guy corresponding to this VM. When most of the immutable states have been migrated, then you may want to make sure that evolving states have to be checkpointed and migrated, and that's the time when you have to do stop and copy. In that case, what you're going to do is, you're going to suspend this guy on this VM, and then you may have to do a whole bunch of things in terms of network redirection, and file redirection, and all of these kinds of things that are associated with the environment in which this virtual machine is operating, and that might require contacting DNS servers to set up the routing table and things like this. All of these things they'll be done, and once all of that is done, routing redirections have been done, and you've synced up all the VM state, then you can copy this entirely over to this guy and now you're ready. This is now existing in some sense in both places in exactly the same state except that it is not running in either one of them. At this point, you can release the resources on the source machine and then you can activate it on the target machine, and then the migration is complete. This is the way you reduce the amount of downtime by doing iterative pre-copy to make sure that most of the working set is moved over, and only the things that are changing, you're allowed to change, but at some point is suspended and most of it has been moved over, and then you've reset the redirection and so on. Then you can release the resources and activate it. Those are generic steps in migration. Like I said, there are lots of details associated with VM migration and how to do it efficiently. There are ways in which you all have been using ZooKeeper and things like that, where there is an active copy and a background copy ready for fault tolerance. This is more on the context of balancing resources, upgrading, but the technologies that ZooKeeper has is similar to the technologies that I'm talking about in a generic sense.
