Evolution of the CloudVeneto.it private cloud to support research and innovation

Abstract

CloudVeneto.it was initially funded and deployed by INFN in 2014 for serving the computational and storage demands of INFN research projects mainly related to HEP and Nuclear Physics. It is an OpenStack-based scientific cloud with resources spread across two different sites connected with a high speed optical link: INFN Padova Unit and the INFN Legnaro National Laboratories. The infrastructure has grown throughout the years with additional funds from ten University of Padova departments, and nowadays supports a broader range of scientific and engineering disciplines. Its hardware resources provide around 2500 computational cores and 360 TB of storage to about 250 users working for more than 70 projects. In the last months we enhanced the cloud platform in two ways: 1) by integrating a number of heterogeneous GPU cards to address the special needs of user communities whose computations involve machine learning training; 2) by enabling the users to simply deploy on-demand Kubernetes clusters for Big Data Analytics applications taking advantage of the operator framework. In particular, the Kubernetes operators for Apache Kafka and Spark platforms were integrated to address real-time data ingestion and streaming processing on the cloud. This article describes the technical details of these two solutions and their integration with the cloud infrastructure.