scVAG: Unified single-cell clustering via variational-autoencoder integration with Graph Attention Autoencoder

Abstract

Single-cell RNA sequencing (scRNA-seq) enables high-resolution transcriptional profiling of cell heterogeneity. However, analyzing this noisy, high-dimensional matrix remains challenging. We present scVAG, an integrated deep learning framework combining Variational-Autoencoder (VAE) and Graph Attention Autoencoder (GATE) for enhanced single-cell clustering. Building upon scGAC, our approach replaces its restrictive linear principal component analysis (PCA) with nonlinear dimensionality reduction better suited for scRNA-seq data. Specifically, we integrate VAE and GATE to enable more flexible latent space encoding. Extensive experiments on 20 datasets demonstrate scVAG's superior performance over previous state-of-the-art methods including scGAC, SCEA, SC3, Seurat, scGNN, scASGC, DESC, NIC, scLDS2, DRJCC, sLMIC, and jSRC. On average, scVAG improves clustering accuracy by 5 percent in ARI and 4 percent in NMI parameters. Visualizations highlight scVAG's capacity to recover interpretable biological structures. Our VAE-GATE pipeline extracts intricate expression patterns into compact representations that precisely delineate cell subpopulations consistent with ground truth labels. Overall, scVAG establishes a robust architecture for elucidating cell taxonomies from noisy transcriptomic inputs.

Keywords

• Single-cell RNA sequencing
• Variational-autoencoder
• Graph attention autoencoder
• Clustering
• Dimensionality reduction