Abstract

Generative adversarial networks (GANs) have emerged as a powerful unsupervised method to model the statistical patterns of real-world data sets, such as natural images. These networks are trained to map random inputs in their latent space to new samples representative of the learned data. However, the structure of the latent space is hard to intuit due to its high dimensionality and the non-linearity of the generator, limiting the usefulness of the models.

Understanding the latent space requires a way to identify input codes for existing real-world images (inversion), and a way to identify directions with known image transformations (interpretability). Here, we use a geometric framework to address both issues simultaneously. We develop an architecture-agnostic method to compute the Riemannian metric of the image manifold created by GANs. The eigen-decomposition of the metric isolates axes that account for different levels of image variability.

An empirical analysis of several pretrained GANs shows that image variation around each position is concentrated along surprisingly few major axes (the space is highly anisotropic) and the directions that create this large variation are similar at different positions in the space (the space is homogeneous). We show that many of the top eigenvectors correspond to interpretable transforms in the image space, with a substantial part of eigenspace corresponding to minor transforms which could be compressed out.

This geometric understanding unifies key previous results related to GAN interpretability. We show that the use of this metric allows for more efficient optimization in the latent space (e.g. GAN inversion) and facilitates unsupervised discovery of interpretable axes. Our results illustrate that defining the geometry of the GAN image manifold can serve as a general framework for understanding GANs.

Introduction

Deep generative models, particularly Generative Adversarial Networks (GANs), have revolutionized the field of unsupervised learning by enabling the generation of highly realistic and diverse images. Despite their remarkable success in producing photorealistic samples, the underlying structure of their latent spaces remains poorly understood. The high-dimensional, non-linear nature of these spaces presents significant challenges for interpretation and practical application.

This paper introduces a geometric perspective to analyze and understand the latent spaces of GANs. By treating the generator as a smooth mapping from latent space to image space, we can apply tools from Riemannian geometry to characterize the structure of the resulting image manifold. This approach provides a unified framework for addressing two fundamental challenges in GAN research: latent space inversion (finding codes for real images) and interpretability (identifying meaningful directions in latent space).

Our work demonstrates that the Riemannian metric of the GAN manifold reveals crucial properties about its geometry, including anisotropy and homogeneity, which have direct implications for both theoretical understanding and practical applications of generative models.

Background

Generative adversarial networks learn patterns that characterize complex datasets and subsequently generate new samples representative of that set. In recent years, there has been tremendous success in training GANs to generate high-resolution and photorealistic images. Well-trained GANs show smooth transitions between image outputs when interpolating in their latent input space, which makes them useful in applications such as high-level image editing (changing attributes of faces), object segmentation, and image generation for art and neuroscience.

However, there is no systematic approach for understanding the latent space of any given GAN or its relationship to the manifold of natural images. Because a generator provides a smooth map onto image space, one relevant conceptual model for GAN latent space is a Riemannian manifold. To define the structure of this manifold, we have to ask questions such as: are images homogeneously distributed on a sphere? What is the structure of its tangent space — do all directions induce the same amount of variance in image transformation?

Here we develop a method to compute the metric of this manifold and investigate its geometry directly, and then use this knowledge to navigate the space and improve several applications. To define a Riemannian geometry, we need to have a smooth map and a notion of distance on it, defined by the metric tensor. For image applications, the relevant notion of distance is in image space rather than code space. Thus, we can pull back the distance function from the image space onto the latent space. Differentiating this distance function on latent space, we will get a differential geometric structure (Riemannian metric) on the image manifold.

Methodology: Computing the Riemannian Metric

Our approach to understanding GAN latent spaces centers on computing the Riemannian metric tensor H of the image manifold. This metric captures how small changes in the latent space translate to changes in the image space, providing a mathematical foundation for analyzing the geometry of the manifold.

The Riemannian metric is defined as H = J^T J, where J is the Jacobian matrix of the generator function with respect to the latent code. This metric tensor encodes the local stretching and compression of the mapping from latent space to image space. By analyzing the eigenvalues and eigenvectors of this metric, we can identify directions in the latent space that correspond to significant image transformations.

Our method is architecture-agnostic and can be applied to any differentiable generator network. We compute the metric efficiently using automatic differentiation, allowing us to analyze large-scale generative models without prohibitive computational costs. The key steps in our methodology include:

• Sampling points from the latent space
• Computing the Jacobian of the generator at each point
• Calculating the Riemannian metric tensor H = J^T J
• Performing eigen-decomposition of H to identify principal directions
• Analyzing the distribution of eigenvalues across different regions of the latent space

This geometric framework provides a principled way to understand the structure of GAN latent spaces and enables various applications, from interpretable axis discovery to efficient optimization.

Empirical Analysis of GAN Manifolds

We conducted extensive empirical analysis of several pretrained GANs, including StyleGAN and BigGAN, to investigate the geometric properties of their image manifolds. Our findings reveal several surprising and consistent patterns across different architectures and datasets.

Our analysis shows that GAN manifolds exhibit strong anisotropy, meaning that image variation is not uniformly distributed across all directions in the latent space. Instead, a small number of directions account for the majority of image variability, while most directions produce negligible changes in the generated images.

Furthermore, we observed that the space is remarkably homogeneous—the directions that create large image variations are similar at different positions in the latent space. This property suggests that the semantic structure of the latent space is consistent across different regions, facilitating navigation and interpretation.

We found that many of the top eigenvectors correspond to interpretable transformations in image space, such as changes in pose, lighting, color, or semantic attributes. This connection between geometric structure and semantic meaning provides a foundation for unsupervised discovery of interpretable axes in GAN latent spaces.

Network Architecture and Training Effects

The geometric properties of GAN manifolds are influenced by both network architecture and training procedures. We investigated how different architectural choices affect the Riemannian metric and the resulting manifold structure.

Style-based architectures, such as StyleGAN, exhibit different geometric properties compared to traditional generator networks. The style-based approach introduces additional structure to the latent space through the use of intermediate latent codes and adaptive instance normalization, which affects the distribution of curvature and the anisotropy of the resulting manifold.

Training procedures, including the choice of loss functions, regularization techniques, and dataset characteristics, also significantly impact the geometry of the learned manifold. We observed that well-trained GANs tend to have more structured and interpretable latent spaces, with clearer separation between major and minor variation directions.

Progressive growing techniques, commonly used in high-resolution image generation, introduce additional geometric structure to the latent space. The multi-scale nature of these models creates a hierarchical organization of image features, which is reflected in the eigenvalue spectrum of the Riemannian metric.

Understanding these architectural and training effects is crucial for designing better generative models and for developing effective methods for latent space manipulation and interpretation.

Applications and Practical Implications

The geometric understanding of GAN manifolds enables several practical applications and provides insights for improving existing methods. We demonstrate how our framework can be applied to enhance GAN inversion, interpretable axis discovery, and latent space navigation.

GAN Inversion: Traditional GAN inversion methods often struggle with finding accurate latent codes for real images. By incorporating the Riemannian metric into the optimization process, we can guide the search toward directions that produce meaningful image changes, resulting in more efficient and accurate inversion. The metric provides a natural distance measure that accounts for the perceptual importance of different directions in the latent space.

Interpretable Axis Discovery: Our method enables unsupervised discovery of interpretable directions in the latent space. By analyzing the eigenvectors of the Riemannian metric, we can identify directions that correspond to semantically meaningful transformations without requiring labeled data or manual inspection. This approach unifies and generalizes previous methods for finding interpretable directions, such as SeFa and GANSpace.

Latent Space Navigation: The geometric framework provides principled methods for navigating the latent space. By following geodesics (shortest paths on the manifold) rather than straight lines in the latent space, we can generate more natural and semantically consistent interpolations between images. This approach minimizes unnecessary image distortions during traversal of the latent space.

Model Compression and Efficiency: The highly anisotropic nature of GAN manifolds suggests opportunities for compression. Since most directions in the latent space produce negligible image changes, we can potentially reduce the effective dimensionality of the latent space without significant loss of image quality or diversity.

Key Insights

Our geometric analysis of GAN manifolds reveals several fundamental insights about the structure and properties of deep generative models:

• Concentrated Variability: Image variation in GANs is highly concentrated along a small number of directions, indicating that the effective dimensionality of the image manifold is much lower than the nominal dimensionality of the latent space.
• Structural Consistency: The major variation directions are consistent across different regions of the latent space, suggesting a homogeneous structure that facilitates navigation and interpretation.
• Semantic-Geometric Correspondence: There is a strong correspondence between the geometric structure (eigenvectors of the metric) and semantic meaning, enabling unsupervised discovery of interpretable axes.
• Architecture Independence: The observed geometric properties are consistent across different GAN architectures, suggesting that they reflect fundamental aspects of how deep networks learn to generate images.
• Unifying Framework: The Riemannian geometric perspective provides a unifying framework that connects and explains various previous findings in GAN interpretability and manipulation.

These insights have implications for both theoretical understanding and practical applications of generative models, suggesting directions for future research and development.

Conclusion

We have presented a geometric framework for analyzing and understanding the latent spaces of deep generative models. By computing the Riemannian metric of the image manifold, we can characterize its local and global properties, including anisotropy, homogeneity, and curvature. Our empirical analysis reveals that GAN manifolds exhibit consistent geometric patterns across different architectures and datasets.

The geometric perspective provides a unified understanding of various phenomena in GAN latent spaces and enables practical applications such as improved GAN inversion, unsupervised discovery of interpretable axes, and more natural latent space navigation. The correspondence between geometric structure and semantic meaning suggests that the Riemannian metric captures fundamental aspects of how generative models organize visual information.

Our work demonstrates that defining the geometry of the GAN image manifold can serve as a general framework for understanding and improving generative models. Future research could explore how these geometric principles extend to other types of generative models, such as variational autoencoders and diffusion models, and how they can inform the design of more interpretable and controllable generative systems.

The geometric approach to understanding deep generative models represents a promising direction for bridging the gap between the impressive empirical performance of these models and our theoretical understanding of how they work. By bringing tools from differential geometry to bear on deep learning, we can develop more principled methods for analyzing, interpreting, and manipulating complex neural networks.