Design and Implementation of a Distributed Versioning File System for Cloud Rendering

Abstract

Rendering is widely used for visual effects in animations, video games, and movies. As the computational load in rendering workloads fluctuates greatly over time, it is attractive to use cloud infrastructures for cost-effective rendering. However, we analyze that cloud rendering has several technical challenges involved in the handling of rendering input data. In this article, we analyze the workload characteristics of popular rendering projects, and find out the following three observations. First, total size of rendering input files reaches tens to hundreds of gigabytes, and uploading these large data to cloud increases the startup latency of rendering significantly. Second, the consistency requirement of file systems in cloud rendering is complicated compared to that of traditional distributed file systems. Third, file accesses in rendering are highly skewed such that the top 20% files account for 60-80% of total accesses, whereas 40-70% are never used or used only once. Based on these observations, we design and implement a new file system for cloud rendering, which has the function of version control, on-demand fetch, and distributed cooperative caching for rendering data. This allows for minimizing data transmission overhead caused by the large input data of rendering and satisfying the rendering data consistency. Measurement studies under synthetic and real workloads show that the proposed file system performs better than the conventional uploading scheme and NFS by 55.4% and 29.5% on average, respectively.

Keywords

• Cloud
• rendering
• file system
• cooperative caching
• versioning file system