Error telemetry: browser and backend errors

Errors land on a property (see the overview for the concept model). A property's ksts_ key is write-only and public by design.

The ingest endpoint is https://kamustatus-ingest.fly.dev (canonical name ingest.kamustatus.com once DNS is live). The wire contract is versioned in the kamustatus repo at packages/telemetry/PROTOCOL.md: POST {ingest}/i/{key}, Content-Type: text/plain (a CORS simple request), JSON envelope {"v":1,"sdk","release","sid","dropped","events":[...]}. Browsers must send an Origin matching the property's origin; server-side reporters send no Origin header, which ingest accepts by design.

Step 1 — create the property

In the dashboard: Monitor → Uptime → your project → Properties → add (name + origin, e.g. https://myapp.fly.dev). The property's Install tab shows ready-made snippets including the key.

Or from a terminal / agent, with the kamu CLI:

kamu status properties add <project-id> --name myapp --origin https://myapp.fly.dev
kamu status properties list <project-id>
kamu status properties show <property-id>     # prints the ksts_ key + script tag

Step 2a — browser (websites, dashboards)

One script tag. The SDK auto-captures uncaught errors, unhandled rejections, resource-load failures, pageviews and sessions. No cookies, no persistent identifiers (no consent banner needed by design).

<script async src="https://kamustatus-ingest.fly.dev/t.js"
        data-key="ksts_YOUR_KEY" data-release="1.2.3"></script>

data-release is optional but recommended — set it from your build so error groups are attributable to a release.

Step 2b — backend (server-side errors)

Backends use a small dependency-free reporter module copied into the service (deliberately vendored per service: no shared package, no build coupling). Canonical copies:

• Deno / TypeScript: kotisivukamu repo, e.g. api/src/lib/kamustatus-reporter.ts (also in kamuhub billing-api/, notify/). Full source below.
• Go: kamuhub repo, bff/kamustatus.go (also agent/kamustatus.go). Same behavior; call reportError(err, where) or reportErrorf(name, msg, stack, where) for recovered panics / direct 5xxs.

Design contract (both variants): fire-and-forget and deliberately silent — telemetry must never throw into a request path, never block a response, and never turn an ingest outage into a customer-visible failure. Batched (flush at 20 events or 5 s on an unref'd timer), rate-limited to 100 events/min with a dropped counter, message/stack truncated, query strings stripped from URLs (they can carry tokens).

Hook it in twice:

import { installGlobalErrorReporting, reportError } from "./lib/kamustatus-reporter.ts";

installGlobalErrorReporting();      // at startup: uncaught errors + rejections

app.onError((err, c) => {           // framework error handler (Hono shown)
  reportError(err, c.req.url);
  return c.text("Internal Server Error", 500);
});

Workers additionally report permanently failed jobs, e.g. graphile-worker:

runner.events.on("job:failed", ({ job, error }) => {
  reportError(error, `/jobs/${job.task_identifier}`);
});

Deno reporter — full source

Copy this file verbatim into the service (conventionally src/lib/kamustatus-reporter.ts). If you change it, propagate the change to every copy — they are kept byte-identical.

// KamuStatus server-side error reporter.
//
// It speaks the same v1 wire contract as the browser SDK (see kamustatus
// packages/telemetry/PROTOCOL.md): POST {url}/i/{key}, Content-Type text/plain,
// JSON body. Server-side reporters send no Origin header, which ingest accepts
// by design.
//
// Deliberately dependency-free and deliberately silent: telemetry must never
// throw into a request path, never block a response, and never turn an ingest
// outage into a customer-visible failure.

const SDK = "kamustatus-server/0.1.0";
const FLUSH_INTERVAL_MS = 5_000;
const FLUSH_AT_EVENTS = 20;
const MAX_EVENTS_PER_MINUTE = 100;
const MAX_MSG = 2048;
const MAX_STACK = 8192;

interface KamuStatusErrorEvent {
  t: "error";
  ts: number;
  url: string;
  name: string;
  msg: string;
  stack: string | null;
  src: null;
}

const INGEST_URL = (Deno.env.get("KAMUSTATUS_INGEST_URL") ?? "").replace(
  /\/+$/,
  "",
);
const INGEST_KEY = Deno.env.get("KAMUSTATUS_INGEST_KEY") ?? "";
const ENABLED = INGEST_URL !== "" && INGEST_KEY !== "";
const RELEASE = Deno.env.get("VERSION") || null;

// Events need an absolute url. On Fly the app name is the honest identity of
// the process; locally we fall back to localhost so dev noise is obvious.
const FLY_APP = Deno.env.get("FLY_APP_NAME") ?? "";
const BASE_URL = FLY_APP ? `https://${FLY_APP}.fly.dev` : "http://localhost";

// One sid per process, mirroring the browser SDK's one-per-page-load.
const SID = crypto.randomUUID().replaceAll("-", "").slice(0, 16);

let buffer: KamuStatusErrorEvent[] = [];
let dropped = 0;
let timer: ReturnType<typeof setTimeout> | undefined;
let windowStart = 0;
let windowCount = 0;
let warned = false;

function warnOnce(message: string): void {
  if (warned) return;
  warned = true;
  console.warn(`[kamustatus] ${message}`);
}

// Query strings can carry tokens and personal data, and the wire contract
// strips them for the browser too.
function normalizeUrl(where: string): string {
  const raw = where.startsWith("http://") || where.startsWith("https://")
    ? where
    : `${BASE_URL}${where.startsWith("/") ? where : `/${where}`}`;
  try {
    const u = new URL(raw);
    return `${u.origin}${u.pathname}`;
  } catch {
    return BASE_URL;
  }
}

/**
 * Buffer one error for delivery to KamuStatus. Fire-and-forget: it returns
 * immediately and never throws, whatever `err` is.
 *
 * `where` is a request URL or a path; the query string is stripped.
 */
export function reportError(err: unknown, where: string): void {
  if (!ENABLED) {
    warnOnce("KAMUSTATUS_INGEST_URL/KEY unset, error reporting disabled");
    return;
  }

  const now = Date.now();
  if (now - windowStart >= 60_000) {
    windowStart = now;
    windowCount = 0;
  }
  if (windowCount >= MAX_EVENTS_PER_MINUTE) {
    dropped++;
    return;
  }
  windowCount++;

  const e = err instanceof Error ? err : undefined;
  buffer.push({
    t: "error",
    ts: now,
    url: normalizeUrl(where),
    name: e?.name ?? "Error",
    msg: (e?.message ?? String(err)).slice(0, MAX_MSG),
    stack: e?.stack ? e.stack.slice(0, MAX_STACK) : null,
    src: null,
  });

  if (buffer.length >= FLUSH_AT_EVENTS) {
    flush();
    return;
  }
  if (timer === undefined) {
    const t = setTimeout(flush, FLUSH_INTERVAL_MS);
    timer = t;
    // Never hold a worker or a shutting-down server open for telemetry.
    Deno.unrefTimer(t);
  }
}

function flush(): void {
  if (timer !== undefined) {
    clearTimeout(timer);
    timer = undefined;
  }
  if (buffer.length === 0 && dropped === 0) return;

  const body = JSON.stringify({
    v: 1,
    sdk: SDK,
    release: RELEASE,
    sid: SID,
    dropped,
    events: buffer,
  });
  buffer = [];
  dropped = 0;

  fetch(`${INGEST_URL}/i/${INGEST_KEY}`, {
    method: "POST",
    headers: { "Content-Type": "text/plain" },
    body,
  })
    .then((res) => res.body?.cancel())
    .catch(() => warnOnce("ingest unreachable, dropping error events"));
}

/**
 * Report uncaught errors and unhandled rejections. Listeners only observe --
 * nothing is preventDefault()ed, so the process keeps whatever exit semantics
 * it already had (which matters for the workers, where a crash must still
 * crash).
 */
export function installGlobalErrorReporting(): void {
  globalThis.addEventListener("unhandledrejection", (event) => {
    reportError(event.reason, "/unhandledrejection");
  });
  globalThis.addEventListener("error", (event) => {
    reportError(event.error ?? event.message, "/uncaught");
  });
}

Step 3 — configuration

Two env vars, in fly.toml [env] (public by design, see above):

[env]
  KAMUSTATUS_INGEST_URL = "https://kamustatus-ingest.fly.dev"
  KAMUSTATUS_INGEST_KEY = "ksts_YOUR_KEY"

Also make sure VERSION is set at deploy time so events carry a release tag. If either var is unset the reporter logs one warning and disables itself, so local development needs no setup.

Verify

Trigger a test error and look under Monitor → project → Errors. It should appear within seconds. Then wire alert rules so new and regressed error groups notify you.