GoKawiilOverview of UCE
UCE is a machine learning model for mapping single-cell gene expression profiles into a universal embedding space, denoted as \({\mathcal{U}}\). In this space, each cell c i is represented as a d emb -dimensional vector, where d emb = 1,280. For all analyses we use the 1,280 dimensional UCE representation and not a two-dimensional embedding such as that generated from t-distributed stochastic neighbour embedding or UMAP. Users of UCE should rely on the full 1,280-dimensional representations to judge distances between cells, rather than data visualization using these methods, especially when they expect to see strong biological differences.
The model takes as input a dataset \({\mathcal{D}}\) with N cells \({\{{{\bf{c}}}_{i}\}}_{i=1}^{N}\). Cells in \({\mathcal{D}}\) can be drawn from one or more distinct scRNA-seq experiments. Each cell c i in \({\mathcal{D}}\) is described by a gene expression vector \({{\bf{x}}}^{i}\in {{\mathbb{N}}}^{{K}_{i}}\), where K i is the number of genes measured in c i and can differ across \({\mathcal{D}}\). The gene expression vectors \({{\bf{x}}}^{i}\in {{\mathbb{N}}}^{{K}_{i}}\) are not subset to those with high variance. UCE defines a function \({f}_{u}:{\{{{\rm{{\mathbb{N}}}}}^{{K}_{i}}\to {{\rm{{\mathbb{R}}}}}^{{d}_{\mathrm{emb}}}\}}_{i=1}^{N}\) that maps each gene expression vector xi to its cell embedding vector hi.
Model input: gene representation
The expression of gene g in cell c i is denoted by \({x}_{g}^{i}\), where g represents any protein-coding gene. The corresponding token embedding p g is a pretrained embedding for the protein(s) encoded by the gene g. These embeddings are derived from a pretrained protein language model that takes an amino acid sequence as input and returns a d p -dimensional embedding vector as output. To create P g , we take the average of all proteins coded by gene g. In the context of UCE, we can formulate this as a dictionary that maps each gene g to a d p -dimensional protein embedding vector. Specifically, we used the ESM2 model, which yields embeddings of size d p = 5,120 (refs. 32,33).
Protein language models are chosen for gene representation because they can generate universal representations of any protein sequence. Thus, for new species, all that is required is to have the amino acid sequences of that species’ protein coding genes. These genes do not need to have solved structures and orthology does not need to be calculated between them and the existing training data. Model training ablations demonstrate that, among a class of different protein language models, UCE models trained with ESM2-15B performed the best (Supplementary Fig. 8). Model ablation experiments further demonstrate that the use of protein embeddings to tokenize genes has substantial benefits for the more poorly represented species in the training corpus, as a small ablation model trained with protein embeddings outperformed a model with randomly initialized embeddings on all species except for human (Supplementary Fig. 8). Moreover, the use of protein embeddings enables a unique capability: embedding new species not found in the training data (Fig. 2).
Model input: cell representation
For each cell c i in the input dataset \({\mathcal{D}}\), we identify two distinct sets of protein-coding genes: the expressed genes \({G}_{i}^{+}\) and the non-expressed genes \({G}_{i}^{-}\). These sets are defined as follows:
$${G}_{i}^{+}=\{g| {x}_{g}^{i} > 0\}$$ (1)
$${G}_{i}^{-}=\{g| {x}_{g}^{i}=0\}$$ (2)
... continue reading