Events Pipeline

Pipeline Overview

┌──────────────┐     ┌──────────────┐     ┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  1. CAPTURE  │────>│  2. BUFFER   │────>│  3. INGEST   │────>│  4. STORE    │────>│  5. COMPUTE  │
│  Browser SDK │     │  Client-side │     │  API endpoint│     │  Event store │     │  Metrics     │
│  / Server    │     │  batch queue │     │  validation  │     │  (append)    │     │  aggregation │
└──────────────┘     └──────────────┘     └──────────────┘     └──────────────┘     └──────────────┘

---

1. Capture

Events originate from two sources:

Frontend Events

Captured by a lightweight tracking SDK embedded in the player-facing site. See Data Model → Event Catalog for the full list of events, triggers, and descriptions.

// SDK interface
tracker.track('game_click', {
  gameId: 'abc123',
  surface: 'home',
  position: 5
})

tracker.track('gameplay_start', { gameId: 'abc123' })
tracker.track('gameplay_stop', { gameId: 'abc123' })

Frontend events use client-side Date.now() as timestamp.

Note: game_like and game_dislike are not sent through the event ingestion pipeline. They use dedicated feedback endpoints — see API Reference → Feedback.

Server Events

Generated automatically by the backend when serving responses (game_impression, game_page_view). These use server-side Date.now() as timestamp.

---

2. Buffer (Client-Side)

The tracking SDK buffers events before sending to reduce network overhead.

Batching Rules

Rule	Value
Flush interval	Every 5 seconds
Max batch size	50 events
Flush on unload	Yes — via navigator.sendBeacon on visibilitychange / beforeunload
Retry on failure	1 retry with exponential backoff (2s, then drop)
Offline	Events are dropped (no local queue in MVP)

Session Management

• A session ID is generated per browser tab and included with every event batch
• Session ID is a UUID stored in sessionStorage (dies with tab)
• A device fingerprint or anonymous ID (stored in localStorage) links sessions to a returning user

---

3. Ingest

Events arrive at POST /api/v1/events (see API Reference).

Validation

Each event is validated before acceptance:

Check	Rule
Type	Must be a known event type
Required fields	gameId present for game events, categoryId for category events
Timestamp	Must be ISO 8601 or Unix ms. Must be within 24h of server time
Session ID	Must be present
Deduplication	Same sessionId + type + gameId + timestamp within 1s is dropped

Invalid events are silently dropped. The response returns the count of accepted events.

Rate Limiting

Scope	Limit
Per IP	100 requests/min
Per session	500 events/min
Batch size	Max 50 events per request

---

4. Store

Accepted events are written to an append-only event store.

Event Record Schema

See Database → Analytics Domain for the full Prisma schema and field definitions.

Storage Strategy

Phase	Store	Notes
MVP	PostgreSQL table with partitioning by date	Simple, queryable, good enough for initial traffic
Growth	Move to ClickHouse or TimescaleDB	Columnar storage optimized for time-series aggregation
Scale	Kafka/Redpanda -> ClickHouse	Event streaming with buffered writes

See Scaling for migration triggers.

Partitioning & Indexes

Events are partitioned by month for fast queries and easy retention. See Database → Event Table Partitioning for the SQL and Database → Indexes Strategy for the full index map.

---

5. Compute (Metrics Aggregation)

Raw events are aggregated into metrics. Two computation modes exist:

Pre-computed Aggregates (Hot Path)

Run on a schedule to keep dashboard metrics fast.

Aggregate	Granularity	Schedule
Daily plays per game	1 day	Every hour
Daily unique players per game (DAU)	1 day	Every hour
Impressions per game	1 day	Every hour
Playtime per session	Per session	On session end or daily sweep

Stored in the DailyMetric table — see Database → Analytics Domain for the full schema.

On-demand Queries (Cold Path)

Complex metrics computed on request from the admin dashboard:

Metric	Query Pattern
CTR	clicks / impressions from daily aggregates
Retention D1	Join unique players day J vs day J+1
Retention D7	Join unique players day J vs days J+1..J+7
Engaged players	Count sessions with playtime >= 3min / total sessions
Returning users	Players active before the analysis period who are also active within it
Conversion	loadingEnds / pageViews from daily aggregates

These queries hit the daily metrics table, not raw events, so they remain fast even at scale.

---

Playtime Computation

Playtime is the most complex metric. It requires reconstructing a session timeline from multiple event types.

Algorithm

Input: All events for a (sessionId, gameId) pair, ordered by timestamp

1. Find game_loading_end → mark game as loaded
2. Track focus windows:
   - game_focused_start → game is focused
   - game_focused_stop  → game is unfocused
3. Track gameplay windows:
   - gameplay_start → gameplay is active
   - gameplay_stop  → gameplay is inactive
4. Active playtime = sum of intervals where:
   - game is loaded AND
   - game is focused AND
   - gameplay is active
5. Handle edge cases:
   - Missing gameplay_stop → cap at session end or 30min max
   - Missing game_focused_stop → cap at next event or 5min max
   - Overlapping events → use latest state

Trigger

• Primary: Computed when a session ends (tab closes / gameplay_stop with no follow-up)
• Fallback: Daily sweep job processes incomplete sessions older than 6 hours

---

Data Flow Diagram

┌─────────┐                              ┌──────────┐
│ Browser  │──game_click──────────┐       │  Server  │
│          │──gameplay_start──────┤       │          │
│          │──gameplay_stop───────┤       │          │──game_impression
│          │──game_loading_*──────┤       │          │──game_page_view
│          │──game_focused_*──────┤       │          │
│          │──show_ad─────────────┤       └────┬─────┘
│          │──category_click──────┤            │
└──────────┘                      │            │
                                  ▼            ▼
                          ┌──────────────────────┐
                          │   POST /api/v1/events │
                          │   (validation layer)  │
                          └──────────┬───────────┘
                                     │
                                     ▼
                          ┌──────────────────────┐
                          │    Events Table       │
                          │  (partitioned by      │
                          │   month, indexed)     │
                          └──────────┬───────────┘
                                     │
                          ┌──────────┴───────────┐
                          │                      │
                          ▼                      ▼
                 ┌─────────────────┐   ┌─────────────────┐
                 │  Hourly Agg Job │   │  On-demand query │
                 │  (daily_metrics)│   │  (admin API)     │
                 └────────┬────────┘   └─────────────────┘
                          │
                          ▼
                 ┌─────────────────┐
                 │  Daily Metrics  │
                 │  Table          │
                 └────────┬────────┘
                          │
                 ┌────────┴────────┐
                 │                 │
                 ▼                 ▼
         ┌────────────┐   ┌────────────────┐
         │ Ranking    │   │ Admin Dashboard │
         │ Algorithm  │   │ (metrics view) │
         └────────────┘   └────────────────┘

---

Retention Policy

See Scaling → Partition / Retention Strategy for the full retention policy across all growth phases.