Large-scale machine learning (ML) clusters deploy hundreds of models packaged as large container images, making scheduling and image pulling dominant contributors to deployment latency. Current systems adopt a hybrid-granularity strategy: they schedule using image-level metadata, reuse on-node layer caches, and lazily fetch content at the file or chunk level. This fragmentation leads to suboptimal cross-image reuse. We show that an intermediate unit, software granularity, can be uniformly applied at scheduling, caching, and pulling to improve deployment performance. We present Containerlet, a software-scoped layer that reorganizes standard images into one private layer plus multiple reusable per-software units that reside in compatible registries and are pulled by unmodified runtimes. Containerlet promotes cross-image reuse while bounding the number of units that the scheduler must consider. We design a file backtracking method to convert standard images into Containerlet images, which includes one private layer and multiple Containerlets. We also extend Kubernetes and container runtime to support software-granular scheduling, caching, and lazy pulling. End-to-end deployment experiments on diverse ML images and production cluster traces show consistent gains, reducing deployment overhead by 18% compared with state-of-the-art systems.

Due to the unprecedented development of low-latency interconnect technology, building large-scale disaggregated architecture is drawing more and more attention from both industry and academia. Resource disaggregation is a new way to organize the hardware resources of datacenters, and has the potential to overcome the limitations, e.g., low resource utilization and low reliability, of conventional datacenters. However, the emerging disaggregated architecture brings severe performance and latency problems to the existing cloud systems. In this paper, we take memory disaggregation as an example to demonstrate the unique challenges that the disaggregated datacenter poses to the existing cloud software stacks, e.g., programming interface, language runtime, and operating system, and further discuss the possible ways to reinvent the cloud systems.