The 3D morphology of beaks across the avian tree of life

Bird beaks have a stunning diversity of forms that are linked to their feeding ecology [15,16]. At the same time, significant constraints on their form exist due to the requirements of flight (e.g., they must not be heavy) and also due to the existence of strong developmental constraints [17]; [18]. The shape of a bird’s beak also has consequences for its ability to deal with climate variations, and for characteristics of its song [19–21], and so having a deep understanding of variations in shape that can capture its full complexity could have very important downstream uses.

We explore the benefits of deep generative models for analyzing 3D morphology using the full 3D shape of the surface of bird beaks, collected from over 2,000 bird species across the avian tree of life.

Exploring the latent space

A DeepSDF model was successfully trained on the 3D bird beak mesh dataset, using a single GPU (achieving a mean absolute error of 0.1 over 5000 epochs, note the loss is difficult to interpret given the shape is mostly determined by low error near the surface only). A visualization of the latent space in two dimensions reveals that the model produces a latent space that smoothly transitions between different types of beak shapes (S1 Movie), suggesting that the estimated vectors capture meaningful and continuous variation in beak shape. A comparison of reconstructed beak meshes with original meshes showed good correspondence in overall shape (Fig 1). An examination of all 2,021 reconstructions of the original data suggested that the model captured the overall shape of beaks excellently but struggled to capture finer details such as the nares or elaborate ornaments on some species (see supporting document on figshare with all beak reconstructions: [22]).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Some example reconstructions from the DeepSDF model, chosen randomly.

Top row is the original 3D meshes, bottom row is the reconstruction of the same bird bill from the fitted DeepSDF model. The model has captured effectively the overall shape of each bill, even when the mesh has clear defects (the model having filled this in from what it has learned about the full distribution of bill shapes). On the other hand, small details are not captured well such as the nares, but this was not desired in this study.

https://doi.org/10.1371/journal.pcbi.1012887.g001

Morphological coherence

To see whether the estimated latent space has morphological’coherence’, we discovered vectors through the latent space associated with a bird beak’s’elongation’ and’broadness’.

Figs 2 and 3 show that the discovered vectors of elongation and broadness have strong and consistent effects. Regardless of which starting beak the elongation vector was applied to, it led to an increase (or a decrease for the negative direction) in how elongated the beak was. The same was true for broadness -- applying the vector led to broader beaks in the positive direction. Moreover, the vectors did not appear to be ’entangled’ with other features. Entanglement is the phenomenon, observed in some deep generative models, where some vectors can ’entangle’ multiple features together as we move along them. Often features are correlated in the training sample, and so they are in a sense ’unentangled’ by the model, which appears to have happened at least for these simple types of features.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The latent morphological space contains an ’elongation vector’.

The beaks highlighted in the red box are generated from random latent vectors. The beaks on the right and left are after moving the latent vector along the elongation vector in the positive or negative direction respectively. The elongation vector was found by regressing the latent space against independent measurements of the length, width and depth of the beaks in the observed dataset (see Methods for details).

https://doi.org/10.1371/journal.pcbi.1012887.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The latent morphological space contains an ’Broadness vector’.

The beaks highlighted in the red box are generated from random latent vectors. The beaks on the right and left are after moving the latent vector along the broadness vector in the positive or negative direction respectively. The elongation vector was found by regressing the latent space against independent measurements of the length and width of the beaks in the observed dataset (see Methods for details). Note that the highlighted beaks are the same as those highlighted in Fig 2, but viewed from a different angle to make their degree of broadness easier to see.

https://doi.org/10.1371/journal.pcbi.1012887.g003

Ecological meaning

To see if the latent vector estimated by the DeepSDF model held ecologically important information, we analysed their association with the trophic niche of bird species’ they belonged to.

In Fig 4 we show a plot of the beak morphology latent space, reduced to two dimensions using the UMAP algorithm. Points are coloured by trophic niche which shows, broadly, how trophic niches are distributed in the latent space.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. UMAP dimension reduction of the 64 latent beak morphology variables, plotted by the trophic niche of the bird species each beak belongs to.

We can see good separation for some niches (nectarivores and granivores, aquatic predators). Both invertivores and omnivores are spread out in their morphology, spanning across most of the space covered by other trophic niches. Panel a shows all bird species, coloured by their trophic niche. Panel b shows density contours for individual trophic niches in the UMAP space. The two most distinct clusters have been labeled with their taxonomic group.

https://doi.org/10.1371/journal.pcbi.1012887.g004

Trophic niche prediction

To see whether the latent variables could predict trophic niche we fit classification Random Forest models with trophic niche as the response variable and the 64 latent morphological variables as the predictors.

Training a Random Forest model to predict the trophic niche of a bird based on its estimated latent vector produced a model with an overall 60% accuracy on a test dataset held out from training representing 20% of the original bird beaks. Balanced accuracy, designed to account for imbalanced classes, and defined as the average of specificity and sensitivity, was high (0.78, maximum is 1.0). The discrepancy between the measures is largely explained by the fact that the model did poorly at predicting the trophic niche of omnivores, which has a fairly large number of members (see Discussion). Looking at the confusion matrices of the Random Forests model provides more insight (Fig 5). Most trophic niches had most of their members correctly classified, except omnivores, which had more of their members incorrectly classified, typically as invertivores. Most trophic niches that had incorrect classifications had most of them misclassified as omnivores or invertivores. These misclassifications make sense because invertivores and omnivores appear to have the most widely spread distribution in the morphological latent space (Fig 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Confusion matrices of a random forest models trained to predict a bird’s trophic niche based on the 64 dimensional latent space estimated by DeepSDF, and 64 PCA variables using landmarks from Cooney et al (2017) [

23]. Predictions are evaluated on a held-out set of test data that were not used for training the model. On the x axis is the true trophic niches and on the y axis are the predicted trophic niches. Colours and square sizes are proportional to the percentage of observed samples that were correctly classified in each trophic class. The trophic niche that the model predicts the worst is omnivory, which makes sense given we expect this trophic niche to have the most generalized morphology.

https://doi.org/10.1371/journal.pcbi.1012887.g005

On the other hand, a Random Forest model trained with 64 PCA axes calculated from the aligned landmark data from Cooney et al. [23] did very slightly worse at classifying trophic niche (60% accuracy, 0.77 balanced accuracy).

We also trained a generalized linear model (with regularization) using the glmnet R package using both our latent variables and PCA scores as predictors. For comparison with the Random Forest models, we used elastic net regularization with the same cross-validation procedure to tune the mixing parameter (alpha) between L1 and L2 regularization, as well as the overall regularization strength (lambda). The multinomial regression model achieved higher accuracy with the DeepSDF latent variables (0.79 balanced accuracy) compared to the PCA variables (0.757 balanced accuracy). This suggests that while the overall predictive power of DeepSDF latent variables is similar to PCA, they may be more amenable to linear classification methods, possibly due to the tendency of deep generative models to encode interpretable concepts linearly in their latent space

Phylogenetic structure and signal

Phylogenetic analysis using the geomorph package [24,25] revealed distinct patterns between PCA and DeepSDF representations. While both showed similar overall phylogenetic signal (multivariate K = 0.23 and 0.16 for PCA and DeepSDF respectively), signal distribution differed markedly. PCA concentrated signal in early axes, and this persisted to some degree even after phylogenetic PCA analysis. In contrast, DeepSDF distributed signal uniformly across all 64 dimensions (Fig 6A).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Phylogenetic Signal Distribution:

a) Distribution of phylogenetic signal (Blomberg’s K) across individual axes for three different representations: standard PCA on aligned landmarks, phylogenetic PCA on landmarks, and DeepSDF latent variables. Lines show smoothed trends with 95% confidence intervals. DeepSDF shows remarkably uniform distribution of signal across all axes, while both PCA approaches show concentration of signal in early axes despite phylogenetic correction. b) Comparison of phylogenetic signal in PACA components between DeepSDF (coral) and landmark-based PCA (teal). Higher bars indicate stronger phylogenetic signal in that component. DeepSDF components show stronger concentration of phylogenetic signal when explicitly aligned using PACA, suggesting better preservation of phylogenetic structure despite more uniform distribution in the original space.

https://doi.org/10.1371/journal.pcbi.1012887.g006

PACA alignment demonstrated DeepSDF variables could concentrate phylogenetic signal more effectively. The first PACA axis from DeepSDF showed stronger signal (Blomberg’s K = 0.81) versus PCA (K = 0.57) (Fig 6B). This indicates DeepSDF preserves phylogenetic structure while maintaining uniform distribution across unaligned dimensions.

To see whether phylogenetic signal in latent variables largely explained their ability to predict trophic niche, we repeated the classification analysis using PACA with the first axis removed (hence removing the largest part of the phylogenetic signal). This revealed phylogenetic signal alone did not explain predictive ability for either DeepSDF or PCA. In fact, classification balanced accuracy actually was slightly higher using PACA minus the first axis, and it also increased slightly the advantage of DeepSDF over PCA (balanced accuracy = 0.81 for DeepSDF and 0.78 for PCA). This suggests DeepSDF variables capture additional non-phylogenetic variation relevant to trophic ecology.

DeepSDF produces a morphologically coherent latent space

The latent space estimated by the DeepSDF model is coherent in that it produces smooth transitions between similar beak morphologies as one moves through it (S1 Movie). Vectors associated with distinct aspects of general beak shape can be found in the latent space (Figs 2 and 3), suggesting shape is captured meaningfully by the model. There also is some separation of different trophic niches in a reduced dimensional visualization, though it varies substantially between different trophic niches (Fig 4). We explored this more formally by training a Random Forest model to predict trophic niche from morphological latent vectors.

Trophic niche is well predicted by morphological latent variables

The results of the Random Forest classification model to predict trophic niche from a bird beaks latent code provided some interesting results. The model correctly predicted the trophic niche of most birds, in both the training set and a held-out test set. Where the model produced incorrect classifications, on the other hand, provides some interesting insight into how bird beak morphology is similar or dissimilar between trophic niches.

For example, the results of the model confirm that nectarivores have highly distinctive beak morphology, since the model almost never incorrectly classified beaks in the nectarivore trophic niche, in either the training or test set. This was also evident in the UMAP dimension reduction (Fig 4), where nectarivores were the most distinct cluster. Our conditional VAE model also produced highly distinctive bird beak samples when asked to generate them while conditioning on the nectarivore trophic niche (Fig 7).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Probablistic generative samples from a conditional variational autoencoder trained on the latent space of the bird beak DeepSDF model, and conditioned on the trophic niche of the bird species each beak belonged to.

Each panel represents a random sample from the estimated conditional probability distribution given a particular trophic niche. The trophic niches with the most distinctive generated distributions tend to correspond to the trophic niches that have the lowest false negative rate for the Random Forest classifier (Fig 5). For example, Granivores, Nectarivores, and Vertivores stand out with distinctive generative distributions and had low false negatives in the classifier. On the other hand, generative distribution with a lot of beak shape diversity correspond to trophic niches that had low classification accuracy (e.g., Omnivores, Invertivores, Frugivores). Images were generated using the R package impac (https://github.com/rdinnager/impac).

https://doi.org/10.1371/journal.pcbi.1012887.g007

Another interesting trophic niche class was omnivores. Omnivores were frequently misclassified in the test set as invertivores, suggesting that the beak morphology of omnivores is difficult to distinguish from that of invertivores. This is intriguing given that a previous macroevolutionary analysis has shown that omnivory appears to be an evolutionary ’sink’ [26]. Burin et al. (2016) showed that omnivores frequently evolved from most other trophic niches, except for invertivores. Once they had evolved, omnivores had a low diversification rate and had almost no transitions into other trophic niches. However, the one exception was invertivores -- omnivores occasionally transitioned to invertivores. The same study also showed that most omnivores had some invertebrates in their diet. Taken together, this suggests that omnivores are in an evolutionary transitional state from other trophic niches to invertivory, but they rarely complete this transition in the macroevolutionary record. Given that omnivores are frequently misclassified as invertivores in the Random Forest model, this could suggest that the transition to invertivory in beak morphology is, in fact, almost complete in most lineages of omnivores. This suggests a speculative explanation for omnivory appearing as an evolutionary ’dead-end’. It could be that the final transition to invertivory from omnivory might be prevented by heavy competition with incumbent invertivores during the final evolutionary approach, leading to frequent competition-driven extinction. Though beyond the scope of this paper, there may be ways to test this speculative theory using a combination of macroevolutionary transition models and the latent beak morphology data provided by this study, perhaps by taking advantage of a conditional or joint VAE trained on the DeepSDF vectors, similar to the one we used for generation in this study. It is also interesting to note that our conditional VAE model generated highly diverse beak shapes when conditioning on the omnivore trophic niche, supporting the idea that omnivores have been derived from a diversity of different ancestral trophic niches (Fig 7).

On the other hand, the DeepSDF latent variables did not perform considerably better than simpler measures of morphology provided by Principle Components Analysis. This means we can say only that the DeepSDF latent variables contain as much predictive power for trophic niches as a simpler linear method like PCA. Furthermore, our results are quite consistent with those of [27], who used PCA scores from only 4 measurements of beaks to predict trophic niches of birds. Like us, they found that omnivores and frugivores were most difficult to predict. However, they discovered that when they also used measurements of leg morphology as an additional set of predictors, that the model was able to distinguish frugivores and a number of smaller trophic classes better than before. This suggests that trophic niche prediction is not an especially difficult task, with the exception of predicting the omnivore class, meaning that it perhaps is not surprising that it was unable to distinguish between DeepSDF and the simpler PCA based method. However, it seems unlikely that any better representation of beak morphology, no matter how complex, will improve trophic class prediction since omnivore are probably not a good natural class, that is, it is transitional between other classes with more extreme diets. Therefore we expect it to be highly heterogeneous in the morphology of its members and it may not ever be possible to predict it well. What is needed is more fine-grained data on bird diets that includes relative percentages of different diet items. Predicting this sort of data would provide a much more informative test of different numerical representations for beak morphology.

An intriguing result was that when we used a simpler linear model for the trophic classification task, the DeepSDF variables performed better than the PCA scores. This could be a result of a general phenomenon that has been observed in deep generative models, that their estimated latent spaces tend to separate human-interpretable ’concepts’ linearly. This is sometimes referred to as the ’linear representation hypothesis’, which has evidence from deep generative models in many domains, such as language models, and machine vision models. Ref. [13] provides and interesting discussion of linearity of representations along with many references to papers that discuss the phenomenon in various data domains. The consequences of this for any downstream tasks are more difficult to reason about. It may be that this may help the latent representations satisfy the assumption of certain downstream analyses better (such as linearity assumptions). The fact that the representations are by construction Gaussian in distribution also contributes to this possible benefit.

Properties of the latent space

Like many modern dimension reduction techniques, the DeepSDF method can produce different latent space configurations across training runs while preserving the underlying relationships between specimens. This property is shared by other widely-used methods in biological data analysis, such as non-metric Multidimensional Scaling (NMDS), t-SNE, and UMAP. While methods like Principal Components Analysis (PCA) produce deterministic results, they achieve this by making strong assumptions about linearity and orthogonality that may not reflect biological reality. The non-deterministic nature of these more flexible methods is a reasonable trade-off for their ability to capture complex non-linear relationships in biological data. What matters is not the exact coordinates, but that the method reliably captures meaningful biological variation and relationships between specimens.

If a canonical representation is desired there are a number of options that involve rotating the latent space to a specific repeatable orientation. An example might be to rotate the latent space to align with the variation implied by phylogeny of the organisms under consideration, if available (see following section). Or it could be rotated to align linearly with any chosen set of summary variables, such as beak elongation, or trophic niche. All rotations are equally valid, but such a rotation could facilitate comparison between different models. It is important to note however, that vectors in the latent space are what is meaningful in this approach and this likely better reflects the biological reality, since evolution does not ’see’, or act, on variation aligned to our preconceived notions of interpretability (nor along axes of maximum variation, which is what PCA ultimately does).

Phylogenetic signal and latent space structure

A key challenge in analyzing morphological data is handling phylogenetic signal in multivariate analyses. Researchers often use phylogenetic PCA not to remove phylogenetic signal (which is impossible), but to redistribute it more evenly across axes rather than having it concentrated in the first few dimensions [25]. This redistribution is important because many downstream analyses use only the first few PC axes to represent morphological variation - if these axes concentrate phylogenetic signal, it can violate assumptions of subsequent comparative analyses.

Our analyses revealed an intriguing property of the DeepSDF latent space: it naturally distributes phylogenetic signal almost perfectly uniformly across all 64 dimensions, in contrast to standard PCA which concentrates signal in early axes (Fig 6). Even after phylogenetic PCA correction, some concentration of signal remains in PCA. The uniform distribution in DeepSDF is likely due to the spherical prior used in training, which preferences no directions in latent space over any other. Remarkably, this means DeepSDF achieves automatically what phylogenetic PCA attempts to do through explicit correction - preventing phylogenetic structure from being concentrated in dominant axes.

When explicitly aligning variables with phylogeny using PACA, we found that DeepSDF variables could concentrate phylogenetic signal more effectively than PCA variables. The first PACA axis from DeepSDF showed stronger phylogenetic signal (Blomberg’s K = 0.81) compared to PCA (K = 0.57). This suggests that while DeepSDF naturally distributes phylogenetic signal evenly, it preserves the underlying phylogenetic structure in a way that allows it to be concentrated along specific directions through rotation when desired. As Collyer and Adams note [25], this flexibility in representing phylogenetic structure - being able to either distribute it evenly or concentrate it through rotation - is valuable for downstream comparative analyses.

The ability of DeepSDF to maintain predictive power for trophic niches even with the first phylogenetically aligned component removed (balanced accuracy = 0.81 vs 0.78 for PCA) suggests it also captures meaningful non-phylogenetic variation in beak shape. This addresses a key concern raised by reviewers about whether the latent space merely reflects phylogeny. Instead, we find that DeepSDF learns a representation that naturally separates phylogenetic structure from other sources of variation, while preserving both. This property makes it particularly suitable for comparative analyses where researchers need to consider both phylogenetic and non-phylogenetic patterns.

This balance of even distribution and preservable structure aligns with recent work showing that deep generative models tend to encode interpretable concepts linearly in their latent spaces [13], as also discussed above. The spherical prior encourages even distribution of variation across dimensions, while the continuous nature of the decoder allows preservation of phylogenetic structure that can be recovered through rotation. These properties emerged naturally from the training process rather than being explicitly enforced, suggesting DeepSDF has learned a biologically meaningful representation of morphological variation.

Why use a generative model?

We have established that the representations learned by a deep generative model are at least as good as previous methods, at least for one ecologically meaningful task, suggesting they contain ecological ’signal’. But if a method such as the seemingly much simpler PCA can do as or almost as good, why should we consider using generative models instead? Generative models have a number of advantages from both a conceptual and a methodological perspective. From a conceptual perspective, as we discussed in the Introduction, there is a elegance to the idea of utilizing a model that attempts to learn a generative process for the data. In some sense this mirrors how we think the real biological features that our data measures have come into being. It also allows us to process the raw data much less before we analyze it, such that outputs of our analysis remain ’closer’ to the real biological objects of interest. That is, despite the potential complexity of the underlying generative model we are fitting, there are fewer layers of abstraction we have to go through to get our data into an analyzable form.

Methodologically, a major advantage is the ability to quickly and easily generate data from the model in the original data mode, for visual validation or other purposes, such as synthetic data creation.

Another substantial advantage is the reusability of the model. That is, the model as trained can be taken and fine-tuned using new datasets. The depth of fine-tuning can also be manipulated. It is possible for example to keep the decoder function fixed and optimize latent codes for new data into the existing latent space. On the other hand, it is also possible to fine-tune the decoder neural network weights as well, allowing new data to update the latent space itself. While not yet a true foundation model, this work demonstrates the potential for developing shared, reusable AI models in organismal biology. Foundation models typically require massive, diverse datasets and extensive validation across multiple tasks [28,29]. Our current implementation, trained on bird beaks alone, represents an initial exploration of how such approaches might eventually be developed for morphological analysis. The model is publicly available and can be fine-tuned on new data, allowing us to explore the challenges and opportunities of building shared morphological models. As the diversity and scale of 3D morphological datasets grows, and methods for aligning different types of morphological structures improve, this type of approach could potentially evolve toward true foundation models. We can see the transformative potential of such models in molecular biology, where tools like AlphaFold [30] have revolutionized protein structure prediction. This approach is in stark contrast to most existing types of models in organismal biology, which are trained or estimated by individual researchers and are not generally reusable. Unlike our generative AI model, incorporating new data into these traditional models typically requires retraining the model from scratch, making it much more difficult to accumulate improvements over time. However, significant work remains to develop appropriate architectures, training approaches, and validation methods for AI-based morphological foundation models.

Several expert reviewers of this paper brought up the issue of applying this generative AI method to smaller or more constrained datasets, as they were concerned that the method would require large general datasets to be useful. The answer to this is two-fold. For one, though we trained the model presented here on over 2000 bird beaks, we believe the model will still perform adequately on much smaller datasets of upwards of several hundred 3D meshes. To some extent, this will depend on how heterogeneous the datasets are. Datasets with fairly constrained variation will likely require fewer samples to capture the main patterns in the data. On the other hand, small datasets can be accommodated, as discussed above, by using the fine-tuning approach. Now that this model has been trained on a large dataset, it can be reused through fine-tuning on smaller datasets. This is one reason we have made the pretrained model available through the R package ‘fibre‘. Anyone can take our model, and a set of new bird beaks meshes, and fine-tune the model on the new dataset, taking advantage of what the model has already learned from the larger dataset. This will not just embed the new data into the latent space already learned, but also adjust the underlying model to take into account the new information from the new data, thus further improving performance on any downstream tasks. In fact the benefits of using pretrained models to improve performance on small datasets for various downstream tasks such as classification or regression is well-known and widely utilized within machine learning, and is often referred to as ’semi-supervised learning’ [31] or ’transfer learning’. Importantly this does not require access to the original bird beak meshes, which is an enormous and unweildy dataset that takes a fair time to download (in contrast our pretrained model is stored in a file just over 4mb). Of course, this will require care that any new bird beak 3D mesh data is preprocessed in the same manner that we processed our data. This can be achieved by modifying the code we provided publicly. The first author of this paper can also be contacted for help in this task. It is likely the model presented here may need to be further developed before it can become a truly useful foundation model, but we hope that what we present here is a useful starting point.

Future work

The method shown here represents a promising new approach for analyzing 3D morphological data, though significant development work remains. There are many downstream tasks that could be enabled by the flexibility of the model, as well as many ways the model could be improved or augmented. Here we talk about some of these possibilities.

The dataset

We used a set of 3D scans of museum specimen birds that was described in [23]. The data comes in the form of 3D meshes, produced by scanning the front of the head of bird specimens using a 3D scanner, and is publicly available at www.markmybird.org.

The signed distance function

The signed distance function of a 3 dimensional shape is a unique function that allows the reconstruction of the shape from 3D coordinates. The function takes a three dimensional vector, x , representing a coordinate in 3D space as input, and it outputs the signed distance between that coordinate and the closest point on the surface of the 3D shape, s. We refer to this function as:

(1)

The sign of the output represents whether the coordinate is inside or outside the shape (negative being inside, and positive outside). The full 3D surface of the shape is then implicitly represented by the zero isosurface of the function, that is, the infinitesimal region in which the function output goes from negative to positive (or from inside to outside the shape, ). Park et al. (2019) present a method to estimate such a function using a deep neural network to approximate the function itself, and a Monte Carlo sample of the signed distance field of the 3D shape(s) to serve as data for the optimization of the function parameters. The function can be augmented with a set of learned latent vectors that represent each of a set of individual 3D shapes that can be trained simultaneously. A coherent structure of the latent space is encouraged with a simple prior distribution that maintains compactness and sphericity.

In practice, the signed distance function for the 3D surface of bird bill i () is approximated by a function with parameters θ:

(2)

where is an n -dimensional vector that will be optimized to represent the unique shape of bird bill i.

The parameters of the function are optimized by comparing its output to a dataset composed of a spatial sample of 3D coordinates per bird bill, paired with their true SDF value, sampled over all target bird bill shapes indexed by i (S):

(3)

where is the total number of bird species and is the total number of coordinates sampled per bird species.

Model specifics

The goal of the model is to approximate the following posterior distribution:

(4)

Assuming that deviations of the estimated SDF from the true SDF follow a Gaussian distribution, and that the latent vectors z have a prior probability that is multivariate normal distribution with diagonal covariance matrix (e.g., spherical) we can express the model as:

(5)

In general the above posterior distribution is intractable but can be approximated efficiently in several ways. In deep learning applications a variation approximation can be used to estimate the full posterior (such as in, e.g., variational autoencoders). Here, however, training is simplified by only estimating the maximum a posteriori (MAP) of the distribution. In the following we assume accepts a matrix input z with n columns of rowwise concatenated vectors , and likewise x is a matrix with the 3 columns and the same number of rows as z, and it outputs a column vector of approximate SDF values with a length equal to the number of rows of z and x. s is a column vector of true SDF values. Given this we have:

(6)

Maximizing z gives:

(7)

Therefore we maximize the MAP by mimimizing the penalized likelihood expression

(8)

L is the standard sum of squared errors between the true SDF values and those predicted by the function , known as the reconstruction loss. The penalty term is just the squared deviations of z from zero (across all n dimensions), multiplied by a parameter , very similar to the L2 loss term over coefficients in a ridge regression. λ is a hyper-parameter of the model that can be changed to adjust the relative influence of the reconstruction and the prior to the loss.

In practice it is desirable to make sure the model correctly estimates the SDF near the surface of the 3D shape, since it is only the zero isosurface that ultimately determines the shape, and so finding very accurately where the SDF turns from positive to negative is the most important task. Therefore, in practice, the model is trained to focus on reconstructing the SDF close to the surface by clamping the observed and predicted SDF within a small range around zero, which is controlled by a hyper-parameter δ (which was set to 0.1 in this study). Therefore, instead of using the Gaussian L function, we use the following reconstruction loss instead:

(9)

where . Note this is possible because the formulation of Q derived above is not dependent on the Gaussian form and holds for any factorizable reconstruction loss function, as shown in Park et al. (2019). This penalized likelihood can be efficiently optimized, jointly over the z vectors and the parameters θ of the function using the stochastic gradient descent algorithm.

Autodecoder architecture

Note that using a fixed z based on MAP estimation, instead of a full posterior fit using a variational approximation, was shown to improve training results by Park et al. (2019). The overall structure of this model is what is known as an ’autodecoder’ in the machine learning literature. This distinguishes it from autoencoder and variational autoencoder architectures, which are more commonly used in generative AI modelling. Park et al. (2019) suggest a major advantage of a decoder-only model architecture is that z can be estimated using partial batches of independent SDF samples from any number of 3D meshes simultaneously, whereas models with autoencoders must process a set of an entire 3D meshes at once, because the encoder part of the model needs to capture the global structure of each of the meshes. The disadvantages of the decoder-only architecture is that we lose the benefit of amortized inference for large datasets, and we lose the ability to model uncertainty in the conditional probability distributions (and as a side-effect lose the power of variational autoencoders to find a minimal low-dimensional manifold to explain the data). The first issue is not a problem in our case because the full dataset of ~2,000 bird beaks is relatively small for a machine learning dataset and so amortized inference is not necessary. We mitigate the second problem by using a second-stage conditional variational autoencoder (VAE) model that we fit on the estimated z vectors, which is inspired by recent work on ’two-stage VAEs’ [38,39], which allows us to produce a full probabilitic generative model (see Trophic Niche Conditional VAE section for details).

Trophic niche conditional VAE

In order to create a fully probabilistic generative model for bird beaks, we fit a second-stage variational autoencoder, partially based on the ideas of ’two-stage VAEs’ [38,39]. Essentially we used the estimated latent vector from the DeepSDF model as input to a conditional variational autoencoder (VAE).

VAE models in general are a highly popular and powerful method in generative A.I. that we very briefly describe here. Variational Autoencoders (VAEs) are a class of deep generative models which seek to capture the underlying probabilistic distribution of complex, high-dimensional datasets [44]. VAEs consist of two main components: an encoder network, which maps the input data to a latent lower-dimensional space, and a decoder network, which reconstructs the original data from this latent representation. The encoder network approximates the posterior distribution of the latent variables, given the input data. This is typically modeled as a multivariate Gaussian distribution, parameterized by a mean vector and a covariance matrix which are outputs of the encoder network. The decoder network then generates data by sampling points from this distribution and mapping them back into the original high-dimensional space. The aim is to learn a set of latent variables that can accurately and efficiently represent the input data.

The key feature of VAEs is their use of a variational loss function during training, the evidence lower bound (ELBO) on the log-likelihood of the data:

(10)

which is a sum of two terms: a reconstruction loss and the regularization term, also known as the Kullback-Leibler (KL) divergence. The reconstruction loss quantifies how well the decoder is able to reconstruct the original data from the latent representation. The KL divergence term acts as a regularization component, which measures the divergence between the encoder’s distribution and a prior distribution, typically assumed to be a standard multivariate Gaussian. The regularization term ensures the latent space has good properties, encouraging the model to use all dimensions of the latent space and avoid overfitting.

In the version of VAEs as detailed in Dai and Wipf (2019), a trainable Gamma parameter is introduced in the reconstruction loss term, which we follow in this study. This Gamma parameter effectively controls the balance between the reconstruction loss and the KL divergence. By learning a suitable value for this parameter, the VAE is better equipped to capture the structure of the data and adjust the strength of the regularization based on the complexity of the dataset, potentially leading to more robust and interpretable results. A conditional VAE simply adds conditioning information to the encoder and decoder, to allow the distribution to depend on some additional factor, in this case, the trophic niche of the birds.

Fitting this secondary condition VAE allowed us to incorporate some uncertainty into the generative process, as well as estimate a lower-dimensional manifold embedded in the latent space on which the bird beaks seemed to lie. It also dealt with a common problem in generative models, known as the ’prior hole’ problem, which can also be an issue in VAEs [45,46]. Generally speaking this problem is caused by the training process encouraging conformity to a prior distribution but being unable to fully enforce it. This means the estimated latent vectors of the data (or ’aggregate posterior’) typically do not match the prior and thus are not fully independently Gaussian. This becomes a problem for generation because generation usually involves sampling a latent vector from the prior distribution and then running this through the decoder. If the estimated latent vectors deviate from the prior substantially then the generated samples will not be representative. Indeed, in our case, we found the latent vectors estimated for the bird beaks deviated substantially from an independent Gaussian, specifically most latent dimensions showed substantially thicker tails than expected under a Gaussian distribution (S1A Fig). However Dai and Wipf (2019) showed that fitting a second VAE on the results of a first-stage VAE resulted in much better conformity to the desired prior. Here we find that this is also the case when fitting a VAE on the results of our autodecoder architecture, the second-stage VAE estimated latent space conformed strongly to an independent Gaussian distribution (S1B Fig). This allowed us to create a model where we can generate hypothetical samples of bird beaks by sampling latent vectors from an independent Gaussian distribution, running them through the VAE decoder, then running the results of this, in turn, through the DeepSDF decoder to get the final bird beak mesh.

We additionally made the second-stage VAE a conditional VAE (CVAE), that was conditioned on the trophic niche of the bird to which each beak belonged. This allows the model to generate from the conditional distribution of bird beaks given a trophic niche, which was is complimentary to our Trophic Niche classifier analysis (see Downstream Analysis: Trophic Niche Prediction section for details).

Downstream analysis

Trophic niche prediction.

To additionally test the ecological relevance of the estimated beak morphology latent vectors, we trained a Random Forest classifier [51] to predict a bird species’ trophic niche from their 64 estimated latent morphology variables. Before training classification models, 89 of the 2,021 datapoints were excluded from the dataset after discovering potential issues in their meshes (mostly badly misaligned lower beaks).

We first removed 20% of the beaks to serve as a hold-out test dataset, with sampling stratified by trophic niche, before training models as described below.

For the training data, 1 out of 10 trophic niche categories were assigned to each bird species using the AVONET dataset [48]. We trained the model using the ’tidymodels’ package in R [52]. Overall, trophic niche classes were highly imbalanced in our dataset (Table 1), which is well-known to cause issues in classification models if not dealt with appropriately. We addressed this using case weights in the tidymodels framework. For each observation i in trophic niche class j, we assigned a weight w_i = 1/n_j, where n_j is the sample size of class j. This weighting scheme ensures each trophic niche class contributes equally to model training regardless of sample size.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Sample sizes for different trophic niches in the bird beak dataset. The data is highly imbalanced.

https://doi.org/10.1371/journal.pcbi.1012887.t001

The weights were incorporated using the case_weights() specification in tidymodels, and all model tuning and evaluation maintained these weights through cross-validation. Both Random Forest and multinomial regression models used these same weights, ensuring fair comparison between methods.

We tuned several hyperparameters of the Random Forest model based on a 20 monte carlo cross validation splits, each of which used a random 20% of the training data for model validation. We used the R package tidymodels for model fitting. We tuned the trees, mtry, and min_n parameters, which control the numbers of trees used to build an ensemble, the numbers of variables to include in each individual tree, and the minimum number of data points allowed in a terminal grouping in each tree, respectively.

We also trained a Random Forest classifier on 64 PCA axes calculated from landmarks (same as those used in [23]) after using the same alignment procedure used for the DeepSDF analysis. This provided a direct comparison to our DeepSDF variables by matching dimensionality and preprocessing steps. We used identical case weights and cross-validation procedures as described above.

We also ran a regularized multinomial regression for comparison, using the glmnet package through tidymodels. Like our Random Forest models, observations were weighted by the inverse of their class sample size, and elastic net regularization was used with tuned alpha (mixing) and lambda (overall regularization strength) parameters.

All models were evaluated on the original held-out test dataset using accuracy and balanced accuracy as metrics.