Architecture

A dashboard rule becomes browser tracking configuration; a funnel turns recorded events into an observed drop-off report. This page explains the systems behind that workflow.

OwlEye has three client surfaces—a browser SDK, a team console, and public integration APIs—served by one Rust backend. The backend separates analytics facts in ClickHouse from transactional account and configuration data in SQLite.

System overview

Browser SDK       Team console     Integrations
Tracking ID       Session cookie   Scoped secret key
     │                  │                 │
     └──────────────────┼─────────────────┘
                        ▼
               Cloudflare → Caddy
                        │
                        ▼
                 Rust / Axum API
             access · validation · quotas
                 │              │
        config & usage     analytics facts
                 │              │
                 ▼              ▼
               SQLite       ClickHouse
                 ▲              ▲
                 └──────┬───────┘
                        │
                 Background jobs
           retention · billing · notifications

The public Astro website and documentation are served separately from the analytics API. The repository includes Cloudflare Workers delivery for the website and console, and a native deployment layout with the API behind Caddy on EC2 and ClickHouse on a separate instance. Visitors and the console access analytics through the API rather than connecting directly to either database.

Component	Responsibility
Browser SDK	Automatic page views and SPA navigation, explicit events, optional interaction rules and performance collection
Console	App configuration, team access, reports, rules, API keys, and AI interactions
Rust API	Authentication, authorization, origin policy, validation, privacy transforms, quotas, queries, and ingestion
SQLite	Users, organizations, memberships, sessions, sites, rules, keys, subscriptions, usage reservations, prompt allocations, and job/outbox state
ClickHouse	Page views, custom events, rule events, performance measurements
Background jobs	Retention, account cleanup, billing usage delivery, notifications

Event ingestion: browser to database

1. Collect. The SDK constructs a bounded event with a public Tracking ID. Optional modules add rules or performance collection; they do not require cookies or browser storage.
2. Resolve and authorize. The API resolves the active app, applies its origin policy and tracking settings, and checks rate limits. The SDK applies GPC choices before sending; the API does not filter DNT or GPC headers. Server ingestion uses a separate route requiring a site-scoped events:write key.
3. Validate and transform. The API validates the payload, derives pseudonymous visitor/session cohorts, adds available location context, and stamps the applicable retention policy.
4. Reserve allowance. SQLite records the quota reservation and event IDs used to reconcile usage. A request can accept fewer events than it submitted when the monthly allowance runs out.
5. Write. A bounded in-process queue (512 pending request batches per API process) combines events across requests. The worker flushes when it reaches 5,000 rows or one second after the first pending event, whichever happens first. A request batch stays together, so the row threshold is a trigger rather than a strict maximum. A full queue returns HTTP 503; it does not acknowledge events as stored. The writer sends explicit JSONEachRow columns with async_insert=0 and waits for the insert result.
6. Acknowledge and account. The API returns accepted event IDs after successful insertion. It commits usage and updates the event catalog; durable usage state supports reconciliation if accounting cannot finish immediately.

An HTTP 202 alone does not prove an event was stored: an exhausted allowance can produce accepted: 0. A paused app returns HTTP 403. Read the response’s accepted, dropped, and status fields. The server ingestion reference lists the public request and response contract.

Backpressure and delivery limits

The API queue is bounded and held in process memory; it is not a durable message broker. A full queue returns 503 instead of growing without limit. Requests wait for insertion completion, with a bounded timeout. An interrupted request can have an uncertain outcome if ClickHouse persisted the write before the caller received the response.

Once an event batch reaches the ClickHouse writer, transient connection failures, timeouts, and HTTP 408/429/500/502/503/504 responses get up to three retries after the first attempt, with 1, 2, and 4-second delays. Each destination table reuses the same serialized rows and insert deduplication token. Startup enables a 10,000-block deduplication window on both event tables. A successful destination is not replayed when the other destination fails. The queue remains bounded; requests and shutdown draining wait at most 180 seconds. Authorization, payload validation, quota rejections, and other permanent errors are not retried.

These are in-process storage retries, not a durable outbox or client-request idempotency. An API crash can lose queued work, exhausted retries can still have an uncertain outcome, and requests that never reach the API cannot be recovered here. Deduplication depends on retaining the block log during retries; do not disable it or run independent bulk writers that exhaust its window.

The browser SDK sends events immediately with credential-free fetch and keepalive: true. Visible-page duration closes on visibilitychange when the page becomes hidden, with pagehide as a fallback; it does not rely on beforeunload. Delivery is bounded and best-effort, without a persistent retry queue. The SDK limits its own in-flight bodies to 60,000 bytes and eight requests across entrypoints, but other code on the page shares browser keepalive limits. These limits and lifecycle listeners cannot guarantee delivery when offline or when a process is killed. JSON sendBeacon is not used: Beacon includes credentials and cannot opt out, conflicts with the public API’s credential-free cross-origin policy, and shares the keepalive quota. Do not assume exactly-once delivery or blindly retry events as though the API offered client idempotency. The service preserves event evidence for billing reconciliation; that is separate from an integrator’s retry policy.

Analytics reads and authorization

The console authenticates with secure, HttpOnly session cookies. For each protected app request, the API verifies current organization/app access before querying data. Role checks also gate configuration changes; possession of a public Tracking ID does not grant read access.

Reports use server-authored, app-scoped queries against ClickHouse. Configuration and permissions come from SQLite. Retention and privacy rules limit the data that can be returned. An empty query stays empty, and a failed query stays an error; neither becomes synthetic traffic.

Public integrations use scoped developer keys for the documented stats, events, and AI endpoints. These server-to-server APIs are distinct from the console’s session-authenticated management routes. See API authentication and keys for the supported integration surface.

AI request flow

Question + selected app
         ↓
Access and allowance checks
         ↓
Bounded analytics catalog → model interprets the question
         ↓
Validated report definition (or a clarification)
         ↓
Server-authored ClickHouse queries
         ↓
Aggregate supporting data → model explanation
         ↓
Answer + supporting evidence in the console

The backend calls the model provider through OpenRouter. The model does not receive database credentials, execute arbitrary SQL, or choose another app. The server validates report definitions and enforces access, date ranges, retained history, and privacy thresholds. Catalogs and aggregate reports are bounded; raw event payloads and visitor IDs are not the model’s query interface.

Prompt access and allocations live in SQLite. Failed requests and clarifications do not consume a completed-answer credit. AI analytics assistant explains supported questions and the information sent to the provider.

Background work and external services

Background tasks run alongside the API process. Retention and deletion workflows coordinate control-plane state with ClickHouse cleanup. Ingestion/deletion fences prevent in-flight batches from reintroducing rows after the corresponding local deletion completes.

SQLite outboxes track billing usage and notifications so delivery can be retried. Dodo handles payment-provider operations; email delivery handles sign-in and notifications. Uptime workers are currently disabled. These dependencies have separate failure paths from reading an already stored analytics report.

Privacy boundary

Public visitor tracking writes no cookies, local storage, or session storage. Event requests omit browser credentials. Five bounded UTM fields are automatic in SDK 1.0.3; arbitrary query parameters and fragments remain excluded by default.

The API uses the client address transiently for location lookup and pseudonymous cohort derivation; the raw address is not an analytics fact column. Cohorts are site-scoped estimates, not verified people or cross-device identities. Forwarded addresses are trusted only when the immediate peer is a configured trusted proxy. Console session cookies belong to the authenticated team application, not the visitor SDK.

See privacy and lifecycle for collection decisions and data handling.

Deployment and durability

The documented native deployment layout uses one API process with SQLite on persistent disk and a separate ClickHouse server. Local queues, schedulers, and ingestion/deletion coordination belong to that process. Adding API replicas requires a coordinated design for shared state, job ownership, and fencing; it is not just a matter of starting another copy behind a load balancer.

API binary updates do not replace persistent database files. Backups are still required before upgrades and migrations: SQLite backups protect accounts and configuration; ClickHouse backups protect analytics history. Exporting logs to CloudWatch does not back up either database.

To connect this data flow to your site, start with the quickstart, then configure dashboard-managed rules and a funnel.