TruthCrawl Distributed Node Network

TruthCrawl operates a distributed, decentralized network of crawl and index nodes. Each node independently crawls the web, contributes pages to the shared index, and replicates index data across the network — ensuring resilience, geographic diversity, and censorship resistance.

How Nodes Work

• Crawling — Each node runs up to 15 concurrent crawl workers, respecting robots.txt and per-domain rate limits (minimum 1.5s delay).
• Indexing — Pages are indexed locally using a BM25F inverted index with field-weighted term frequencies across title, body, headings, URL, and description.
• Replication — Index data is encrypted with ML-KEM-1024 post-quantum cryptography and replicated to the coordinator node.
• TruthRank — Each node computes authority scores based on inbound link graphs, contributing to the global TruthRank signal.

Run a Node

You can contribute to the TruthCrawl network by running your own node. Nodes earn trust over time through uptime, crawl volume, and index quality. Join the network to help build a more open, independent web index.

Node Requirements

• Linux-based server (Raspberry Pi or equivalent minimum)
• Stable internet connection
• At least 10 GB available storage
• Docker or Node.js 20+ runtime

Visit the API documentation for node registration details, or contact contact@ras-pi.com to get started.