GoKawiilThe Malva platform
Malva provides a unified interface to search millions of cells by any RNA sequence. Malva integrates five modular components that operate independently but communicate through a central orchestrator (Extended Data Fig. 1b). When a user submits a query via the public RESTful API—a gene symbol, DNA sequence, or natural language request such as “cells expressing markers of neurodegeneration”—it is routed by a central orchestrator. Natural language queries first pass through an integrated language model that translates biological concepts into gene sets. The search engine then decomposes sequences into k-mers and queries our distributed index, leveraging pre-computed results for common genes. Malva Index returns cell identifiers with expression levels or coverage profiles, which are then enriched by a metadata service that appends standardized cell-level (for example, cell-type annotation) and sample-level (for example, tissue, disease and developmental stage) annotations. This enables users to filter results or perform targeted searches on specific sample subsets. The platform can also expand datasets available in Malva Index: a crawler monitors public repositories (GEO, SRA, HCA Data Portal and ENA) for new datasets, which are asynchronously processed into separate Malva Index instances and progressively merged into the searchable corpus. Load balancing across multiple workers ensures consistent performance even as hundreds of users query simultaneously. Results return in standard formats (JSON, h5ad) ready for downstream analysis in familiar tools such as scanpy, with full metadata to enable further filtering and aggregation. The server code operates on a standard, open-source stack of tools (Flask v3.1.0, numpy v2.2.6, pandas v2.3.2, SQLite v3.49.1 and polars v1.27.1).
Data sources, crawling and availability
Malva Index incorporates public single-cell and spatial transcriptomic datasets from major repositories (Human Cell Atlas, GEO, SRA, ENA, ArrayExpress and CNGBdb). Data are ingested into Malva Index via the malvacrawl service. The service maintains a registry of datasets identified primarily from the Human Cell Atlas Data Portal, supplemented by automated parsing of GEO and ENA metadata feeds, and manual ingestion of selected identifiers from other platforms (such as ArrayExpress or CNGB). Each candidate dataset is assigned an internal UUID (universally unique identifier) and checked against existing entries by GEO or ENA accession to prevent duplication. Because deposited metadata are frequently incomplete or inconsistent (for example, the sequencing technology or barcode format is often not specified), malvacrawl utilizes a two-stage annotation strategy. First, a large language model (instruction-tuned Llama 3.1) performs an initial parse of the dataset description, extracting candidate technology, organism and tissue labels. Second, these candidates are manually curated and mapped to a controlled ontology using text2term v4.5.0 (see ‘Metadata standardization’). For raw data acquisition, malvacrawl attempts to download FASTQ files via SRA prefetch, HTTP, or FTP in order of preference, using sra-tools (v3.4.1). For datasets archived as BAM files, we converted them back to FASTQ using bamtofastq (v1.4.1). Barcode and sequence parsing during preprocessing relied on dnaio (v1.2.4). For barcoded technologies (primarily 10x genomics), paired-end data are expected; therefore, if only a single FASTQ file is available, or only BAM files are archived, these BAM files are downloaded and converted back to FASTQ using standard 10x conversion utilities. The technology is identified either from deposited metadata, from the structure of the R1 read (barcode length and whitelist match rate), or, when ambiguous, via manual inspection. Datasets that are not compatible with Malva’s whitelist-based barcode system (see ‘Barcode preprocessing’) are deferred for future integration. For non-barcoded, plate-based assays (for example, Smart-seq2), each FASTQ file comprises one cell, and datasets are ingested in ‘bulk’ mode as per-cell indices, and later merged into a sample-level index that preserves cellular resolution.
Currently spanning ~60 million cells from thousands of experiments, Index includes diverse technologies (10x Chromium, Visium, Open-ST, Stereo-seq, Smart-seq2) across multiple organs, species, and conditions. The platform runs on-premise infrastructure with query limits to ensure consistent performance for all users.
Querying
Malva maintains local databases to translate user queries into searchable sequences. When users query gene symbols or Ensembl IDs, these are mapped to genomic coordinates on standard reference genomes (hg38 for human, mm10 for mouse) with default repeat masking55. For expression queries, we retrieve full transcript sequences from Ensembl. For coverage queries that visualize read distribution along genes, we use the genomic coordinates to define the region of interest and return k-mer matches at each position.
Several specialized databases support specific semantic biological queries. To support these via natural language, we integrated Llama 3.1 via Ollama as a translation layer between user prompts and gene sets56. The language model is optionally augmented with a Retrieval-Augmented Generation (RAG) database built from MSigDB57, which includes curated gene sets from Reactome, Gene Ontology (GO) and other pathway databases. When users submit queries such as “inflammatory response genes in macrophages”, the system retrieves relevant gene sets from MSigDB and uses the language model to refine the selection based on the specific query context. This approach provides a convenient interface for exploring biological concepts without requiring users to manually compile gene lists. The resulting gene sets are converted to sequences and searched across Malva Index like any other query.
Pre-computed gene expression matrices
To accelerate common gene-level queries, we pre-compute and store gene expression matrices for frequently accessed gene sets. For each species in Malva Index, we compile comprehensive transcript catalogues from Ensembl, including all protein-coding transcripts, non-coding RNAs, and 3′ UTR sequences. We then query Malva Index with these reference sequences to generate pseudocount matrices for every indexed dataset.
... continue reading