Backend Engine

Overview

The Backend Engine is the heart of the Strategy Execution Platform, responsible for processing intents from submission to completion. This section provides a comprehensive walkthrough of the core components and the lifecycle of an intent through the system.

Core Components

Intent Subsystem (platform/core/intent/)

• Intent Manager (manager.py): The public-facing entry point for all intents. It receives Intent objects from the API, performs initial validation, assigns a priority score using the MLPrioritizer, and enqueues the intent for processing.

• Intent Validator (validator.py): Performs deep, stateful validation of an Intent. It checks constraints against current portfolio state, asset availability, and venue-specific rules.

• Intent Processor (processor.py): A distributed component powered by Ray.io. It takes a validated Intent and decomposes it into smaller, executable sub-intents if necessary.

• ML Prioritizer (prioritizer.py): Uses an ONNX model to assign a priority score to incoming intents based on market conditions, urgency, and potential profitability.

Execution Subsystem (platform/core/execution/)

• Execution Planner (planner.py): Consumes a processed Intent and generates a concrete ExecutionPlan. Its primary responsibility is to determine the optimal execution path by querying the Rust core’s optimize_route function.

• Execution Orchestrator: This logical component drives the execution of a ExecutionPlan. It iterates through the plan’s steps, makes the necessary calls to VenueAdapters, and emits events corresponding to the execution progress.

• Venue Manager (venue_manager.py): Manages and provides access to different VenueAdapter instances, abstracting away the specifics of each trading venue.

Market and Venue Adapters (platform/core/market/)

• VenueAdapter (adapter.py): An abstract base class defining the contract for all venue integrations (e.g., get_price, submit_order).

• UniswapV3Adapter (uniswap_v3.py): A concrete implementation of the VenueAdapter for interacting with Uniswap V3 pools.

State and Streaming Subsystems

• Event Stream (platform/streaming/event_stream.py): A wrapper around the NATS client that provides a clean publish and subscribe interface for the rest of the application.

• State Coordinator (platform/state/coordinator.py): The guardian of system state. It subscribes to all domain events from the EventStream and persists them to TimescaleDB while updating Redis read models.

Performance Core (Rust) (src/)

• PyO3 Bindings (lib.rs): The main entry point that defines the Python module (platform_rust) and registers all exposed Rust functions and classes.

• Execution Engine (execution.rs): Contains the high-performance optimize_route function. It maintains an in-memory graph of liquidity pools and uses Dijkstra’s algorithm to find the most efficient swap paths.

• Chain Monitor (chain_monitor.rs): Includes functions like decode_transaction that can parse raw, RLP-encoded blockchain transactions with high efficiency.

• Market Data (market_data.rs): Provides utilities for processing market data, such as aggregate_order_books, which can merge data from multiple venues.

The Lifecycle of an Intent

This section provides a narrative walkthrough of the system’s core function: processing a single trading Intent from its creation to its final, settled state.

Stage 1: The Spark - Defining and Understanding the Intent Model

Everything begins with the Intent. It is a rich Pydantic model that captures the complete strategic goal of the user.

Code Reference: platform/types/intent.py

# A simplified representation of the Intent model
class Intent(BaseModel):
    id: UUID = Field(default_factory=uuid4)
    strategy_id: UUID
    type: IntentType  # e.g., ACQUIRE, DISPOSE
 
    # What to trade
    assets: List[AssetSpec]
 
    # How to trade
    constraints: IntentConstraints
 
    # Current state
    status: IntentStatus = IntentStatus.PENDING
 
    # ML features for advanced processing
    ml_features: Optional[MLFeatures] = None

Key Fields:

• type: IntentType: An enum defining the high-level action. ACQUIRE means increasing a position, while DISPOSE means decreasing it.

• assets: List[AssetSpec]: Defines the assets involved. An AssetSpec can specify a concrete amount or a relative percentage.

• constraints: IntentConstraints: This is a critical sub-model that defines the rules of engagement for the execution. It includes:

  • max_slippage: The maximum price deviation the user will tolerate
  • time_window_ms: A deadline for execution
  • execution_style: A hint to the planner (e.g., AGGRESSIVE for speed, PASSIVE to minimize market impact)
  • allowed_venues: An explicit list of venues to use (or exclude)

Example Intent (in JSON format):

{
  "strategy_id": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "intent_type": "acquire",
  "assets": [
    {
      "asset": {
        "symbol": "WETH",
        "address": "0x82aF49447D8a07e3bd95BD0d56f35241523fBab1",
        "decimals": 18,
        "chain_id": 42161
      },
      "amount": "1.5"
    },
    {
      "asset": {
        "symbol": "USDC",
        "address": "0xaf88d065e77c8cC2239327C5EDb3A432268e5831",
        "decimals": 6,
        "chain_id": 42161
      }
    }
  ],
  "constraints": {
    "max_slippage": "0.005",
    "time_window_ms": 300000,
    "execution_style": "aggressive"
  }
}

Stage 2: Submission via the API

The journey begins when a client sends an HTTP POST request to the /intents endpoint.

Code Reference: platform/api/intent.py

The FastAPI route handler receives the raw JSON, parses it into a Pydantic Intent object (which provides automatic data validation), and immediately hands it off to the IntentManager. It then returns an IntentReceipt to the client with a 202 “Accepted” status code.

Stage 3: The Gatekeeper - Validation and Prioritization

Once the IntentManager receives the intent, it performs two critical pre-processing steps.

1. Deep Validation (IntentValidator): It runs a series of checks that go beyond the basic schema validation performed by Pydantic:

   • Does the strategy’s portfolio actually have enough USDC to acquire 1.5 WETH?
   • Are the specified venues (allowed_venues) supported on Arbitrum?
   • Is the requested max_slippage reasonable for the assets involved?

2. ML-driven Prioritization (MLPrioritizer): Not all intents are created equal. An intent to liquidate a position during a market crash is more urgent than a routine DCA purchase. The MLPrioritizer uses a pre-trained machine learning model to assign a priority score (1-10) to the intent.

If the intent passes validation, the IntentManager publishes the first event in its lifecycle, IntentSubmitted, to the NATS event stream.

Stage 4: The Strategist - Planning and Optimization

The ExecutionPlanner service listens for IntentSubmitted events. Its sole purpose is to transform the abstract goal of the Intent into a concrete, step-by-step ExecutionPlan.

How it Works:

1. Decomposition: First, it determines if the intent needs to be broken down. A large order might be decomposed into smaller “child” orders to be executed over time, minimizing market impact.

2. Route Optimization: This is the most critical step. For a swap intent, the planner needs to find the most efficient way to trade Asset A for Asset B. It does this by calling the high-performance optimize_route function in the Rust core.

3. Plan Creation: The planner receives the optimal route back from Rust and constructs an ExecutionPlan object. This object is a simple list of steps.

Finally, the ExecutionPlanner publishes a PlanCreated event containing this plan.

Stage 5: The Conductor - Orchestration and Execution

The ExecutionOrchestrator is the workhorse of the system. It listens for PlanCreated events and is responsible for carrying out the plan.

It iterates through each step in the plan and performs the necessary actions by calling the appropriate VenueAdapter. For our swap example:

1. It retrieves the UniswapV3Adapter
2. It calls adapter.build_swap_tx(...), passing in the details from the plan step
3. The adapter constructs the raw blockchain transaction needed to perform the swap
4. The orchestrator signs this transaction
5. It broadcasts the signed transaction to the Arbitrum network
6. It immediately publishes an ExecutionStepSubmitted event with the transaction hash
7. It then waits for the transaction to be mined and confirmed on-chain
8. Once it receives the transaction receipt, it publishes a final status event

Stage 6: The Scribe - State Coordination and Finality

Throughout this entire process, the StateCoordinator runs quietly in the background, listening to every event published on the NATS stream.

• When it hears IntentSubmitted, it creates a new entry for the intent in its Redis read model
• When it hears PlanCreated, it updates the Redis entry to PROCESSING and links the plan ID
• When it hears ExecutionStepSubmitted, it adds the transaction hash to the Redis model
• When it hears ExecutionCompleted, it marks the Redis entry as COMPLETED

Crucially, in parallel, it also appends a serialized copy of every single one of these events to the permanent, immutable log in TimescaleDB.

Data Models and Flows

Core Data Models (Pydantic)

These Python models, defined in platform/types/, form the backbone of the backend logic.

• Intent (intent.py): The central object representing a user’s trading goal
• Asset (common.py): A canonical representation of a tradable asset
• Event (events.py): The envelope for all persisted events

Key Data Flows

Flow 1: Intent Submission and Execution (Happy Path)

This flow describes the journey of an Intent from submission to completion.

The Performance Core: Inside the Rust Engine

The Bridge: How Python and Rust Communicate (PyO3)

The magic that connects our flexible Python orchestrator and our high-performance Rust engine is PyO3. It allows us to:

1. Write functions and structs in Rust
2. Add simple #[pyfunction] or #[pyclass] macros
3. Compile the Rust code into a native Python module (platform_rust.so)
4. Import and call these Rust functions from Python with near-zero overhead

Code Reference: The main bridge definition is in src/lib.rs, which exposes modules like execution. The Python-side shim that loads and calls these functions is in platform/rust_bindings.py.

Finding the Best Path: The Route Optimization Engine

The most critical piece of the Rust core is the ExecutionEngine.

Code Reference: src/execution.rs

It maintains an in-memory graph where nodes are tokens and edges are liquidity pools. When Python calls optimize_route, the Rust engine performs a graph traversal using Dijkstra’s algorithm to find the path that maximizes the output token amount for a given input.

Low-Level Power: Transaction Decoding and Chain Monitoring

Other performance-critical tasks handled by Rust include:

• Transaction Decoding (src/chain_monitor.rs): Parsing raw, RLP-encoded blockchain transactions is much faster in Rust than in Python
• Market Data Aggregation (src/market_data.rs): Aggregating order book data from multiple streaming sources into a single, consolidated view

Next Steps

• Frontend Engineers: Continue to Frontend Dashboard to understand the UI architecture
• Quantitative Developers: Jump to Development Guide to learn how to build trading strategies
• System Architects: Explore Integration & Operations for advanced concepts