Data Pipeline

The backbone of Convex’s data estate is the Federal Reserve Economic Data (FRED) API, maintained by the St. Louis Fed. We ingest 107 time series spanning the full macro spectrum — from overnight rates to housing starts. Every series has documented metadata, update frequency, and seasonal adjustment status.

Quality controls: All incoming data points pass outlier-bounds validation. Missing values are handled via forward-fill (for irregular release schedules) or interpolation (for known gaps). Source-level errors are tracked and surfaced in pipeline observability.

The GDELT Project (Global Database of Events, Language, and Tone) provides real-time event monitoring of global media. Convex ingests GDELT event data through per-pillar search queries across 6 geopolitical categories.

Noise filtering: Raw GDELT output is extremely noisy. We apply a strict 80+ trusted domain whitelist (Reuters, AP, AFP, BBC, FT, Economist, Bloomberg, government agencies, think tanks, defence outlets) that filters approximately 95% of GDELT noise while retaining high-quality event signals.

Events are queried in batches (3 parallel queries with inter-batch delays) with rate limiting (max 1 request/second, exponential backoff on 429/5xx responses).

Convex monitors 46 editorially diverse news sources via RSS, categorised into 7 editorial leans. This diversity is intentional — editorial convergence across normally-opposed sources is itself a powerful signal. Each source has a known editorial lean, update frequency, and authority profile.

The Commodity Futures Trading Commission (CFTC) publishes weekly Commitments of Traders reports showing how commercial hedgers, non-commercial speculators, and other traders are positioned in major futures contracts. Convex ingests this via the CFTC Socrata Open Data API.

For each contract, we track non-commercial long/short positions, net speculative positioning (long minus short), commercial net positioning, and open interest. We then compute 52-week percentile ranks for net speculative positioning — readings above the 90th or below the 10th percentile trigger "peak positioning" events that signal potential contrarian setups.

Convex tracks active prediction markets via the Polymarket Gamma APIacross 9 tag categories: politics, crypto, economics, geopolitics, Fed policy, inflation, recession, interest rates, and trade. Market prices represent crowd-aggregated implicit probabilities.

Prediction market data serves two purposes: providing an external market-implied probability anchor for comparison against our Bayesian scenario estimates, and surfacing emerging consensus on policy shifts before they appear in traditional data.

Data is cached with 5-minute revalidation. When a tracked scenario has an explicit Polymarket slug mapping, the market-implied probability appears alongside our model estimate.

Raw events from GDELT and RSS feeds pass through a multi-gate pipeline before reaching publication. This ensures only genuinely significant, non-duplicate, quality-validated content is published.

Beyond raw data ingestion, Convex computes 10 derived composite signals that synthesize multiple FRED series into higher-level macro narratives. These are pre-computed and injected as context into every analysis pipeline — providing the AI research desk with structured, quantitative context rather than raw numbers.