Data layer
Built our own indexer MVPs, then bought market data and migrated in a week.
- Stack
- RustYellowstone gRPCTimescaleDBClickHouseMobula
Market data is the foundation of a trading app: prices, candles, token metadata and launchpad activity across every chain we support. We built our own indexers, proved them out, and then made the call not to own that layer.
The Solana MVP streams transactions over Yellowstone gRPC and turns raw DEX and launchpad instructions into clean trades, candles and token metrics, in bronze, silver and gold stages.
Decisions
Buy the data layer, build the product
We built working MVPs of our own indexers on Solana and EVM. But a modular data layer is a product in itself, and we didn't have the people to run it and Boba at the same time. So I chose to buy market data and keep the team on what users actually see.
Codex was the first pick, because it covered the most chains out of the box. It also gave us consistent problems, so I moved us to Mobula: a one-week migration across the whole stack with zero downtime. Along the way, I personally added Robinhood Chain.
In the MVP: two databases, each doing what it's best at
Live trading views need upserts, point lookups, joins and read-after-write: that's Postgres. Full history needs append-only writes and heavy compression: that's ClickHouse, at about 10×.
Disk pressure never came from Timescale; it came from keeping unbounded history there. A six-hour hot window cuts a ~9GB/day firehose to about 2.3GB in Postgres, and everything older lives in ClickHouse.
What I built
- A hot/cold storage split: six-hour hot windows in TimescaleDB, a ClickHouse cold tier with dual writes, and read routers that merge the two for trades and candles.
- Memory work: jemalloc, per-pool memory gauges and per-replica eviction.
- Capped a runaway Postgres pool (~57 connections down to 8) and batched writes.
- Fixed fake wicks in OHLCV candles and added ingest bandwidth tracking.