GoKawiilInfino details architecture for building search index directly on object storage
Infino describes how it built a retrieval engine whose full-text, vector and SQL search index lives on S3, GCS or Azure Blob Storage rather than local disk. The company organizes data into Parquet-based 'superfiles' grouped into 'supertables,' each with a manifest and term index, to reduce the number of random reads needed when querying object storage.