As Org Social grows, users follow more feeds, and individual social.org files accumulate hundreds of posts over time. Traditional approaches that download entire feeds sequentially create two major bottlenecks:
Bandwidth waste: Downloading complete files when users only need recent posts Time inefficiency: Sequential downloads that block the user interface
This article explores how Org-social.el 2.3+ (Official client) solves both problems with a sophisticated combination of concurrent queue processing and HTTP Range-based partial fetching while maintaining complete compatibility with all servers.
The Challenge
flowchart TB A[User Opens Timeline] --> B[20 Feeds to Download] B --> C[Traditional Approach: Sequential Downloads] C --> D[Feed 1: 27KB 150 posts] D --> E[Feed 2: 15KB 80 posts] E --> F[Feed 3: 12KB 60 posts] F --> G[... 17 more feeds] G --> H[Total: ~300KB and ~1500 posts] H --> I[Filter to last 14 days] I --> J[Actually needed: ~10 posts=first page] style C fill:#ffcccc,color:black style H fill:#ffcccc,color:black style J fill:#ccffcc,color:black
Downloading 300KB and processing 1500 posts to get 10 posts... It is not good!
Optimization
Org-social.el implements a sophisticated three-layer approach:
Layer 1: Concurrent Queue Processing
A process queue is a data structure that manages tasks to be executed. In Org Social, each feed to download is added to the queue as a pending task. The system then processes these tasks concurrently (multiple at the same time) using a worker pool—a limited number of threads that execute downloads in parallel.
... continue reading