1 Commits

Author SHA1 Message Date
Carl-Gerhard Lindesvärd
bc3d7b7ea8 docs: add guides 2025-12-15 10:14:40 +01:00
101 changed files with 2020 additions and 6381 deletions

View File

@@ -98,10 +98,6 @@ You can find the how to [here](https://openpanel.dev/docs/self-hosting/self-host
### Start
```bash
pnpm install
cp .env.example .env
echo "API_URL=http://localhost:3333" > apps/start/.env
pnpm dock:up
pnpm codegen
pnpm migrate:deploy # once to setup the db
@@ -114,4 +110,4 @@ You can now access the following:
- API: https://api.localhost:3333
- Bullboard (queue): http://localhost:9999
- `pnpm dock:ch` to access clickhouse terminal
- `pnpm dock:redis` to access redis terminal
- `pnpm dock:redis` to access redis terminal

View File

@@ -1,7 +1,6 @@
import type { FastifyReply, FastifyRequest } from 'fastify';
import { assocPath, pathOr, pick } from 'ramda';
import { HttpError } from '@/utils/errors';
import { generateId, slug } from '@openpanel/common';
import { generateDeviceId, parseUserAgent } from '@openpanel/common/server';
import { getProfileById, getSalts, upsertProfile } from '@openpanel/db';
@@ -188,16 +187,9 @@ export async function handler(
break;
}
case 'identify': {
const payload = request.body.payload;
const geo = await getGeoLocation(ip);
if (!payload.profileId) {
throw new HttpError('Missing profileId', {
status: 400,
});
}
await identify({
payload,
payload: request.body.payload,
projectId,
geo,
ua,

View File

@@ -216,4 +216,3 @@ tracker = OpenPanel::SDK::Tracker.new(
)
```

View File

@@ -1,101 +0,0 @@
---
title: Migration from v1 to v2
description: Finally we feel ready to release v2 for all self-hostings. This is a big one!
---
## What's New in v2
- **Redesigned dashboard** - New UI built with Tanstack
- **Revenue tracking** - Track revenue alongside your analytics
- **Sessions** - View individual user sessions
- **Real-time view** - Live event stream
- **Customizable dashboards** - Grafana-style widget layouts
- **Improved report builder** - Faster and more flexible
- **General improvements** - We have also made a bunch of bug fixes, minor improvements and much more
## Migrating from v1
### Ensure you're on the self-hosting branch
Sometimes we add new helper scripts and what not. Always make sure you're on the latest commit before continuing.
```bash
cd ./self-hosting
git fetch origin
git checkout self-hosting
git pull origin self-hosting
```
### Envs
Since we have migrated to tanstack from nextjs we first need to update our envs. We have added a dedicated page for the [environment variables here](/docs/self-hosting/environment-variables).
```js title=".env"
NEXT_PUBLIC_DASHBOARD_URL="..." // [!code --]
NEXT_PUBLIC_API_URL="..." // [!code --]
NEXT_PUBLIC_SELF_HOSTED="..." // [!code --]
DASHBOARD_URL="..." // [!code ++]
API_URL="..." // [!code ++]
SELF_HOSTED="..." // [!code ++]
```
### Clickhouse 24 -> 25
We have updated Clickhouse to 25, this is important to not skip, otherwise your OpenPanel instance wont work.
You should edit your `./self-hosting/docker-compose.yml`
```js title="./self-hosting/docker-compose.yml"
services:
op-ch:
image: clickhouse/clickhouse-server:24.3.2-alpine // [!code --]
image: clickhouse/clickhouse-server:25.10.2.65 // [!code ++]
```
Since version 25 clickhouse enabled default user setup, this means that we need to disable it to avoid connection issues. With this setting we can still access our clickhouse instance (internally) without having a user.
```
services:
op-ch:
environment:
- CLICKHOUSE_SKIP_USER_SETUP=1
```
### Use our latest docker images
Last thing to do is to start using our latest docker images.
> Note: Before you might have been using the latest tag, which is not recommended. Change it to the actual latest version instead.
```js title="./self-hosting/docker-compose.yml"
services:
op-api:
image: lindesvard/openpanel-api:latest // [!code --]
image: lindesvard/openpanel-api:2.0.0 // [!code ++]
op-worker:
image: lindesvard/openpanel-worker:latest // [!code --]
image: lindesvard/openpanel-worker:2.0.0 // [!code ++]
op-dashboard:
image: lindesvard/openpanel-dashboard:latest // [!code --]
image: lindesvard/openpanel-dashboard:2.0.0 // [!code ++]
```
### Done?
When you're done with above steps you should need to restart all services. This will take quite some time depending on your hardware and how many events you have. Since we have made significant changes to the database schema and data we need to run migrations.
```bash
./stop
./start
```
## Using Coolify?
If you're using Coolify and running OpenPanel v1 you'll need to apply the above changes. You can take a look at our [Coolify PR](https://github.com/coollabsio/coolify/pull/7653) which shows what you need to change.
## Any issues with migrations?
If you stumble upon any issues during migrations, please reach out to us on [Discord](https://discord.gg/openpanel) and we'll try our best to help you out.

View File

@@ -3,20 +3,6 @@ title: Changelog for self-hosting
description: This is a list of changes that have been made to the self-hosting setup.
---
## 2.0.0
We have released the first stable version of OpenPanel v2. This is a big one!
Read more about it in our [migration guide](/docs/migration/migrate-v1-to-v2).
TLDR;
- Clickhouse upgraded from 24.3.2-alpine to 25.10.2.65
- Add `CLICKHOUSE_SKIP_USER_SETUP=1` to op-ch service
- `NEXT_PUBLIC_DASHBOARD_URL` -> `DASHBOARD_URL`
- `NEXT_PUBLIC_API_URL` -> `API_URL`
- `NEXT_PUBLIC_SELF_HOSTED` -> `SELF_HOSTED`
## 1.2.0
We have renamed `SELF_HOSTED` to `NEXT_PUBLIC_SELF_HOSTED`. It's important to rename this env before your upgrade to this version.
@@ -44,7 +30,7 @@ If you upgrading from a previous version, you'll need to edit your `.env` file i
### Removed Clickhouse Keeper
In 0.0.6 we introduced a cluster mode for Clickhouse. This was a mistake and we have removed it.
In 0.0.6 we introduced a cluster mode for Clickhouse. This was a misstake and we have removed it.
Remove op-zk from services and volumes

View File

@@ -109,8 +109,8 @@ Coolify automatically handles these variables:
- `DATABASE_URL`: PostgreSQL connection string
- `REDIS_URL`: Redis connection string
- `CLICKHOUSE_URL`: ClickHouse connection string
- `API_URL`: API endpoint URL (set via `SERVICE_FQDN_OPAPI`)
- `DASHBOARD_URL`: Dashboard URL (set via `SERVICE_FQDN_OPDASHBOARD`)
- `NEXT_PUBLIC_API_URL`: API endpoint URL (set via `SERVICE_FQDN_OPAPI`)
- `NEXT_PUBLIC_DASHBOARD_URL`: Dashboard URL (set via `SERVICE_FQDN_OPDASHBOARD`)
- `COOKIE_SECRET`: Automatically generated secret
You can configure optional variables like `ALLOW_REGISTRATION`, `RESEND_API_KEY`, `OPENAI_API_KEY`, etc. through Coolify's environment variable interface.

View File

@@ -126,7 +126,7 @@ If you want to use specific image versions, edit the `docker-compose.yml` file a
```yaml
op-api:
image: lindesvard/openpanel-api:2.0.0 # Specify version
image: lindesvard/openpanel-api:v1.0.0 # Specify version
```
### Scaling Workers

View File

@@ -54,8 +54,8 @@ Edit the `.env` file or environment variables in Dokploy. You **must** set these
```bash
# Required: Set these to your actual domain
API_URL=https://yourdomain.com/api
DASHBOARD_URL=https://yourdomain.com
NEXT_PUBLIC_API_URL=https://yourdomain.com/api
NEXT_PUBLIC_DASHBOARD_URL=https://yourdomain.com
# Database Configuration (automatically set by Dokploy)
OPENPANEL_POSTGRES_DB=openpanel-db
@@ -71,7 +71,7 @@ OPENPANEL_EMAIL_SENDER=noreply@yourdomain.com
```
<Callout type="warn">
⚠️ **Critical**: Unlike Coolify, Dokploy does not support `SERVICE_FQDN_*` variables. You **must** hardcode `API_URL` and `DASHBOARD_URL` with your actual domain values.
⚠️ **Critical**: Unlike Coolify, Dokploy does not support `SERVICE_FQDN_*` variables. You **must** hardcode `NEXT_PUBLIC_API_URL` and `NEXT_PUBLIC_DASHBOARD_URL` with your actual domain values.
</Callout>
</Step>
@@ -133,8 +133,8 @@ If you're using Cloudflare in front of Dokploy, remember to purge the Cloudflare
For Dokploy, you **must** hardcode these variables (unlike Coolify, Dokploy doesn't support `SERVICE_FQDN_*` variables):
- `API_URL` - Full API URL (e.g., `https://analytics.example.com/api`)
- `DASHBOARD_URL` - Full Dashboard URL (e.g., `https://analytics.example.com`)
- `NEXT_PUBLIC_API_URL` - Full API URL (e.g., `https://analytics.example.com/api`)
- `NEXT_PUBLIC_DASHBOARD_URL` - Full Dashboard URL (e.g., `https://analytics.example.com`)
Dokploy automatically sets:
- `OPENPANEL_POSTGRES_DB` - PostgreSQL database name
@@ -166,9 +166,9 @@ If API requests fail after deployment:
1. **Verify environment variables**:
```bash
# Check that API_URL is set correctly
docker exec <op-api-container> env | grep API_URL
docker exec <op-dashboard-container> env | grep API_URL
# Check that NEXT_PUBLIC_API_URL is set correctly
docker exec <op-api-container> env | grep NEXT_PUBLIC_API_URL
docker exec <op-dashboard-container> env | grep NEXT_PUBLIC_API_URL
```
2. **Check "Strip external path" setting**:
@@ -188,7 +188,7 @@ If account creation fails:
# In Dokploy, view logs for op-api service
```
2. Verify `API_URL` matches your domain:
2. Verify `NEXT_PUBLIC_API_URL` matches your domain:
- Should be `https://yourdomain.com/api`
- Not `http://localhost:3000` or similar
@@ -240,7 +240,7 @@ The Dokploy template differs from Coolify in these ways:
1. **Environment Variables**:
- Dokploy does not support `SERVICE_FQDN_*` variables
- Must hardcode `API_URL` and `DASHBOARD_URL`
- Must hardcode `NEXT_PUBLIC_API_URL` and `NEXT_PUBLIC_DASHBOARD_URL`
2. **Domain Configuration**:
- Must manually configure domain paths

View File

@@ -116,7 +116,7 @@ Remove `convert_any_join` from ClickHouse settings. Used for compatibility with
## Application URLs
### API_URL
### NEXT_PUBLIC_API_URL
**Type**: `string`
**Required**: Yes
@@ -126,10 +126,10 @@ Public API URL exposed to the browser. Used by the dashboard frontend and API se
**Example**:
```bash
API_URL=https://analytics.example.com/api
NEXT_PUBLIC_API_URL=https://analytics.example.com/api
```
### DASHBOARD_URL
### NEXT_PUBLIC_DASHBOARD_URL
**Type**: `string`
**Required**: Yes
@@ -139,7 +139,7 @@ Public dashboard URL exposed to the browser. Used by the dashboard frontend and
**Example**:
```bash
DASHBOARD_URL=https://analytics.example.com
NEXT_PUBLIC_DASHBOARD_URL=https://analytics.example.com
```
### API_CORS_ORIGINS
@@ -368,7 +368,7 @@ SLACK_STATE_SECRET=your-state-secret
## Self-hosting
### SELF_HOSTED
### NEXT_PUBLIC_SELF_HOSTED
**Type**: `boolean`
**Required**: No
@@ -378,7 +378,7 @@ Enable self-hosted mode. Set to `true` or `1` to enable self-hosting features. U
**Example**:
```bash
SELF_HOSTED=true
NEXT_PUBLIC_SELF_HOSTED=true
```
## Worker & Queue
@@ -784,8 +784,8 @@ For a basic self-hosted installation, these variables are required:
- `DATABASE_URL` - PostgreSQL connection
- `REDIS_URL` - Redis connection
- `CLICKHOUSE_URL` - ClickHouse connection
- `API_URL` - API endpoint URL
- `DASHBOARD_URL` - Dashboard URL
- `NEXT_PUBLIC_API_URL` - API endpoint URL
- `NEXT_PUBLIC_DASHBOARD_URL` - Dashboard URL
- `COOKIE_SECRET` - Session encryption secret
### Optional but Recommended

View File

@@ -163,7 +163,7 @@ For complete AI configuration details, see the [Environment Variables documentat
If you use a managed Redis service, you may need to set the `notify-keyspace-events` manually.
Without this setting we won't be able to listen for expired keys which we use for calculating currently active visitors.
Without this setting we wont be able to listen for expired keys which we use for caluclating currently active vistors.
> You will see a warning in the logs if this needs to be set manually.

View File

@@ -6,7 +6,6 @@ difficulty: intermediate
timeToComplete: 10
date: 2025-12-15
lastUpdated: 2025-12-15
team: OpenPanel Team
steps:
- name: "Add the dependency"
anchor: "install"

View File

@@ -6,7 +6,6 @@ difficulty: beginner
timeToComplete: 7
date: 2025-12-15
lastUpdated: 2025-12-15
team: OpenPanel Team
steps:
- name: "Install the SDK"
anchor: "install"

File diff suppressed because one or more lines are too long

View File

@@ -121,7 +121,7 @@ export default async function Page({
/>
<div className="row gap-4 items-center mt-8">
<div className="size-10 center-center bg-black rounded-full">
{author?.image ? (
{author.image ? (
<Image
className="size-10 object-cover rounded-full"
src={author.image}
@@ -134,7 +134,7 @@ export default async function Page({
)}
</div>
<div className="col flex-1">
<p className="font-medium">{author?.name || 'OpenPanel Team'}</p>
<p className="font-medium">{author.name}</p>
<div className="row gap-4 items-center">
<p className="text-muted-foreground text-sm">
{guide?.data.date.toLocaleDateString()}

View File

@@ -27,6 +27,10 @@ interface IPInfoResponse {
latitude: number | undefined;
longitude: number | undefined;
};
isp: string | null;
asn: string | null;
organization: string | null;
hostname: string | null;
isLocalhost: boolean;
isPrivate: boolean;
}
@@ -86,6 +90,84 @@ function isPrivateIP(ip: string): boolean {
return false;
}
async function getIPInfo(ip: string): Promise<IPInfo> {
if (!ip || ip === '127.0.0.1' || ip === '::1') {
return {
ip,
location: {
country: undefined,
city: undefined,
region: undefined,
latitude: undefined,
longitude: undefined,
},
isp: null,
asn: null,
organization: null,
hostname: null,
};
}
// Get geolocation
const geo = await getGeoLocation(ip);
// Get ISP/ASN info
let isp: string | null = null;
let asn: string | null = null;
let organization: string | null = null;
if (!isPrivateIP(ip)) {
try {
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 3000);
const response = await fetch(
`https://ip-api.com/json/${ip}?fields=isp,as,org,query,reverse`,
{
signal: controller.signal,
},
);
clearTimeout(timeout);
if (response.ok) {
const data = await response.json();
if (data.status !== 'fail') {
isp = data.isp || null;
asn = data.as ? `AS${data.as.split(' ')[0]}` : null;
organization = data.org || null;
}
}
} catch {
// Ignore errors
}
}
// Reverse DNS lookup for hostname
let hostname: string | null = null;
try {
const hostnames = await dns.reverse(ip);
hostname = hostnames[0] || null;
} catch {
// Ignore errors
}
return {
ip,
location: {
country: geo.country,
city: geo.city,
region: geo.region,
latitude: geo.latitude,
longitude: geo.longitude,
},
isp,
asn,
organization,
hostname,
};
}
export async function GET(request: Request) {
const { searchParams } = new URL(request.url);
const ipParam = searchParams.get('ip');
@@ -127,17 +209,12 @@ export async function GET(request: Request) {
}
try {
const geo = await fetch('https://api.openpanel.dev/misc/geo', {
headers: request.headers,
})
.then((res) => res.json())
.then((data) => data.selected.geo);
const info = await getIPInfo(ipToLookup);
const isLocalhost = ipToLookup === '127.0.0.1' || ipToLookup === '::1';
const isPrivate = isPrivateIP(ipToLookup);
const response: IPInfoResponse = {
location: geo,
ip: ipToLookup,
...info,
isLocalhost,
isPrivate,
};

View File

@@ -12,11 +12,7 @@ import {
import type { Metadata } from 'next';
import { notFound } from 'next/navigation';
type PageProps = {
params: Promise<{ slug: string[] }>;
};
export default async function Page(props: PageProps) {
export default async function Page(props: PageProps<'/docs/[[...slug]]'>) {
const params = await props.params;
const page = source.getPage(params.slug);
if (!page) notFound();
@@ -43,7 +39,9 @@ export async function generateStaticParams() {
return source.generateParams();
}
export async function generateMetadata(props: PageProps): Promise<Metadata> {
export async function generateMetadata(
props: PageProps<'/docs/[[...slug]]'>,
): Promise<Metadata> {
const params = await props.params;
const page = source.getPage(params.slug);
if (!page) notFound();

View File

@@ -1,8 +1,8 @@
import { baseOptions } from '@/lib/layout.shared';
import { source } from '@/lib/source';
import { DocsLayout } from 'fumadocs-ui/layouts/docs';
import { baseOptions } from '@/lib/layout.shared';
export default function Layout({ children }: { children: React.ReactNode }) {
export default function Layout({ children }: LayoutProps<'/docs'>) {
return (
<DocsLayout tree={source.pageTree} {...baseOptions()}>
{children}

View File

@@ -28,7 +28,7 @@ export const viewport: Viewport = {
export const metadata: Metadata = getRootMetadata();
export default function Layout({ children }: { children: React.ReactNode }) {
export default function Layout({ children }: LayoutProps<'/'>) {
return (
<html
lang="en"

View File

@@ -84,34 +84,6 @@ async function getOgData(
data?.data.description || 'Whooops, could not find this page',
};
}
case 'tools': {
if (segments.length > 1) {
const tool = segments[1];
switch (tool) {
case 'ip-lookup':
return {
title: 'IP Lookup Tool',
description:
'Find detailed information about any IP address including geolocation, ISP, and network details.',
};
case 'url-checker':
return {
title: 'URL Checker',
description:
'Analyze any website for SEO, social media, technical, and security information. Get comprehensive insights about any URL.',
};
default:
return {
title: 'Tools',
description: 'Free web tools for developers and website owners',
};
}
}
return {
title: 'Tools',
description: 'Free web tools for developers and website owners',
};
}
default: {
const data = await pageSource.getPage(segments);
return {

View File

@@ -1,18 +1,11 @@
import { url } from '@/lib/layout.shared';
import {
articleSource,
compareSource,
guideSource,
pageSource,
source,
} from '@/lib/source';
import { articleSource, compareSource, pageSource, source } from '@/lib/source';
import type { MetadataRoute } from 'next';
export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
const articles = await articleSource.getPages();
const docs = await source.getPages();
const pages = await pageSource.getPages();
const guides = await guideSource.getPages();
return [
{
url: url('/'),
@@ -56,12 +49,6 @@ export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
changeFrequency: 'yearly' as const,
priority: 0.5,
})),
...guides.map((item) => ({
url: url(item.url),
lastModified: item.data.date,
changeFrequency: 'monthly' as const,
priority: 0.5,
})),
...docs.map((item) => ({
url: url(item.url),
changeFrequency: 'monthly' as const,

View File

@@ -275,6 +275,49 @@ export default function IPLookupPage() {
</div>
)}
{/* Network Information */}
{(result.isp ||
result.asn ||
result.organization ||
result.hostname) && (
<div>
<h3 className="text-lg font-semibold mb-4 flex items-center gap-2">
<Network className="size-5" />
Network Information
</h3>
<div className="grid grid-cols-1 md:grid-cols-2 gap-4">
{result.isp && (
<InfoCard
icon={<Building2 className="size-5" />}
label="ISP"
value={result.isp}
/>
)}
{result.asn && (
<InfoCard
icon={<Network className="size-5" />}
label="ASN"
value={result.asn}
/>
)}
{result.organization && (
<InfoCard
icon={<Building2 className="size-5" />}
label="Organization"
value={result.organization}
/>
)}
{result.hostname && (
<InfoCard
icon={<Server className="size-5" />}
label="Hostname"
value={result.hostname}
/>
)}
</div>
</div>
)}
{/* Map Preview */}
{result.location.latitude && result.location.longitude && (
<div>

View File

@@ -27,7 +27,7 @@ export function useColumns() {
accessorKey: 'name',
header: 'Name',
cell({ row }) {
const { name, path, duration, properties, revenue } = row.original;
const { name, path, duration, properties } = row.original;
const renderName = () => {
if (name === 'screen_view') {
if (path.includes('/')) {
@@ -42,10 +42,6 @@ export function useColumns() {
);
}
if (name === 'revenue' && revenue) {
return `${name} (${number.currency(revenue / 100)})`;
}
return name.replace(/_/g, ' ');
};

View File

@@ -1,229 +0,0 @@
import { countries } from '@/translations/countries';
import type { RouterOutputs } from '@/trpc/client';
import { cn } from '@/utils/cn';
import type { InsightPayload } from '@openpanel/validation';
import { ArrowDown, ArrowUp, FilterIcon, RotateCcwIcon } from 'lucide-react';
import { last } from 'ramda';
import { useState } from 'react';
import { SerieIcon } from '../report-chart/common/serie-icon';
import { Badge } from '../ui/badge';
function formatWindowKind(windowKind: string): string {
switch (windowKind) {
case 'yesterday':
return 'Yesterday';
case 'rolling_7d':
return '7 Days';
case 'rolling_30d':
return '30 Days';
}
return windowKind;
}
interface InsightCardProps {
insight: RouterOutputs['insight']['list'][number];
className?: string;
onFilter?: () => void;
}
export function InsightCard({
insight,
className,
onFilter,
}: InsightCardProps) {
const payload = insight.payload;
const dimensions = payload?.dimensions;
const availableMetrics = Object.entries(payload?.metrics ?? {});
// Pick what to display: prefer share if available (geo/devices), else primaryMetric
const [metricIndex, setMetricIndex] = useState(
availableMetrics.findIndex(([key]) => key === payload?.primaryMetric),
);
const currentMetricKey = availableMetrics[metricIndex][0];
const currentMetricEntry = availableMetrics[metricIndex][1];
const metricUnit = currentMetricEntry?.unit;
const currentValue = currentMetricEntry?.current ?? null;
const compareValue = currentMetricEntry?.compare ?? null;
const direction = currentMetricEntry?.direction ?? 'flat';
const isIncrease = direction === 'up';
const isDecrease = direction === 'down';
const deltaText =
metricUnit === 'ratio'
? `${Math.abs((currentMetricEntry?.delta ?? 0) * 100).toFixed(1)}pp`
: `${Math.abs((currentMetricEntry?.changePct ?? 0) * 100).toFixed(1)}%`;
// Format metric values
const formatValue = (value: number | null): string => {
if (value == null) return '-';
if (metricUnit === 'ratio') return `${(value * 100).toFixed(1)}%`;
return Math.round(value).toLocaleString();
};
// Get the metric label
const metricKeyToLabel = (key: string) =>
key === 'share' ? 'Share' : key === 'pageviews' ? 'Pageviews' : 'Sessions';
const metricLabel = metricKeyToLabel(currentMetricKey);
const renderTitle = () => {
if (
dimensions[0]?.key === 'country' ||
dimensions[0]?.key === 'referrer_name' ||
dimensions[0]?.key === 'device'
) {
return (
<span className="capitalize flex items-center gap-2">
<SerieIcon name={dimensions[0]?.value} /> {insight.displayName}
</span>
);
}
if (insight.displayName.startsWith('http')) {
return (
<span className="flex items-center gap-2">
<SerieIcon
name={dimensions[0]?.displayName ?? dimensions[0]?.value}
/>
<span className="line-clamp-2">{dimensions[1]?.displayName}</span>
</span>
);
}
return insight.displayName;
};
return (
<div
className={cn(
'card p-4 h-full flex flex-col hover:bg-def-50 transition-colors group/card',
className,
)}
>
<div
className={cn(
'row justify-between h-4 items-center',
onFilter && 'group-hover/card:hidden',
)}
>
<Badge variant="outline" className="-ml-2">
{formatWindowKind(insight.windowKind)}
</Badge>
{/* Severity: subtle dot instead of big pill */}
{insight.severityBand && (
<div className="flex items-center gap-1 shrink-0">
<span
className={cn(
'h-2 w-2 rounded-full',
insight.severityBand === 'severe'
? 'bg-red-500'
: insight.severityBand === 'moderate'
? 'bg-yellow-500'
: 'bg-blue-500',
)}
/>
<span className="text-[11px] text-muted-foreground capitalize">
{insight.severityBand}
</span>
</div>
)}
</div>
{onFilter && (
<div className="row group-hover/card:flex hidden h-4 justify-between gap-2">
{availableMetrics.length > 1 ? (
<button
type="button"
className="text-[11px] text-muted-foreground capitalize flex items-center gap-1"
onClick={() =>
setMetricIndex((metricIndex + 1) % availableMetrics.length)
}
>
<RotateCcwIcon className="size-2" />
Show{' '}
{metricKeyToLabel(
availableMetrics[
(metricIndex + 1) % availableMetrics.length
][0],
)}
</button>
) : (
<div />
)}
<button
type="button"
className="text-[11px] text-muted-foreground capitalize flex items-center gap-1"
onClick={onFilter}
>
Filter <FilterIcon className="size-2" />
</button>
</div>
)}
<div className="font-semibold text-sm leading-snug line-clamp-2 mt-2">
{renderTitle()}
</div>
{/* Metric row */}
<div className="mt-auto pt-2">
<div className="flex items-end justify-between gap-3">
<div className="min-w-0">
<div className="text-[11px] text-muted-foreground mb-1">
{metricLabel}
</div>
<div className="col gap-1">
<div className="text-2xl font-semibold tracking-tight">
{formatValue(currentValue)}
</div>
{/* Inline compare, smaller */}
{compareValue != null && (
<div className="text-xs text-muted-foreground">
vs {formatValue(compareValue)}
</div>
)}
</div>
</div>
{/* Delta chip */}
<DeltaChip
isIncrease={isIncrease}
isDecrease={isDecrease}
deltaText={deltaText}
/>
</div>
</div>
</div>
);
}
function DeltaChip({
isIncrease,
isDecrease,
deltaText,
}: {
isIncrease: boolean;
isDecrease: boolean;
deltaText: string;
}) {
return (
<div
className={cn(
'flex items-center gap-1 rounded-full px-2 py-1 text-sm font-semibold',
isIncrease
? 'bg-emerald-500/10 text-emerald-600 dark:text-emerald-400'
: isDecrease
? 'bg-red-500/10 text-red-600 dark:text-red-400'
: 'bg-muted text-muted-foreground',
)}
>
{isIncrease ? (
<ArrowUp size={16} className="shrink-0" />
) : isDecrease ? (
<ArrowDown size={16} className="shrink-0" />
) : null}
<span>{deltaText}</span>
</div>
);
}

View File

@@ -1,75 +0,0 @@
import { useEventQueryFilters } from '@/hooks/use-event-query-filters';
import { useTRPC } from '@/integrations/trpc/react';
import { useQuery } from '@tanstack/react-query';
import { InsightCard } from '../insights/insight-card';
import { Skeleton } from '../skeleton';
import {
Carousel,
CarouselContent,
CarouselItem,
CarouselNext,
CarouselPrevious,
} from '../ui/carousel';
interface OverviewInsightsProps {
projectId: string;
}
export default function OverviewInsights({ projectId }: OverviewInsightsProps) {
const trpc = useTRPC();
const [filters, setFilter] = useEventQueryFilters();
const { data: insights, isLoading } = useQuery(
trpc.insight.list.queryOptions({
projectId,
limit: 20,
}),
);
if (isLoading) {
const keys = Array.from({ length: 4 }, (_, i) => `insight-skeleton-${i}`);
return (
<div className="col-span-6">
<Carousel opts={{ align: 'start' }} className="w-full">
<CarouselContent className="-ml-4">
{keys.map((key) => (
<CarouselItem
key={key}
className="pl-4 basis-full sm:basis-1/2 lg:basis-1/4"
>
<Skeleton className="h-36 w-full" />
</CarouselItem>
))}
</CarouselContent>
</Carousel>
</div>
);
}
if (!insights || insights.length === 0) return null;
return (
<div className="col-span-6 -mx-4">
<Carousel opts={{ align: 'start' }} className="w-full group">
<CarouselContent className="mr-4">
{insights.map((insight) => (
<CarouselItem
key={insight.id}
className="pl-4 basis-full sm:basis-1/2 lg:basis-1/4"
>
<InsightCard
insight={insight}
onFilter={() => {
insight.payload.dimensions.forEach((dim) => {
void setFilter(dim.key, dim.value, 'is');
});
}}
/>
</CarouselItem>
))}
</CarouselContent>
<CarouselPrevious className="!opacity-0 pointer-events-none transition-opacity group-hover:!opacity-100 group-hover:pointer-events-auto group-focus:opacity-100 group-focus:pointer-events-auto" />
<CarouselNext className="!opacity-0 pointer-events-none transition-opacity group-hover:!opacity-100 group-hover:pointer-events-auto group-focus:opacity-100 group-focus:pointer-events-auto" />
</Carousel>
</div>
);
}

View File

@@ -17,7 +17,6 @@ import {
LayoutPanelTopIcon,
PlusIcon,
SparklesIcon,
TrendingUpDownIcon,
UndoDotIcon,
UsersIcon,
WallpaperIcon,
@@ -40,18 +39,13 @@ export default function SidebarProjectMenu({
}: SidebarProjectMenuProps) {
return (
<>
<div className="mb-2 font-medium text-muted-foreground">Analytics</div>
<div className="mb-2 font-medium text-muted-foreground">Insights</div>
<SidebarLink icon={WallpaperIcon} label="Overview" href={'/'} />
<SidebarLink
icon={LayoutPanelTopIcon}
label="Dashboards"
href={'/dashboards'}
/>
<SidebarLink
icon={TrendingUpDownIcon}
label="Insights"
href={'/insights'}
/>
<SidebarLink icon={LayersIcon} label="Pages" href={'/pages'} />
<SidebarLink icon={Globe2Icon} label="Realtime" href={'/realtime'} />
<SidebarLink icon={GanttChartIcon} label="Events" href={'/events'} />

View File

@@ -123,7 +123,7 @@ export function SidebarContainer({
</div>
<div
className={cn([
'flex flex-grow col gap-1 overflow-auto p-4 hide-scrollbar',
'flex flex-grow col gap-1 overflow-auto p-4',
"[&_a[data-status='active']]:bg-def-200",
])}
>

View File

@@ -208,7 +208,7 @@ const CarouselPrevious = React.forwardRef<
variant={variant}
size={size}
className={cn(
'absolute h-10 w-10 rounded-full hover:scale-100 hover:translate-y-[-50%] transition-all duration-200',
'absolute h-10 w-10 rounded-full hover:scale-100 hover:translate-y-[-50%] transition-all duration-200',
orientation === 'horizontal'
? 'left-6 top-1/2 -translate-y-1/2'
: '-top-12 left-1/2 -translate-x-1/2 rotate-90',

View File

@@ -17,6 +17,5 @@ export function useAppContext() {
apiUrl: params.apiUrl,
dashboardUrl: params.dashboardUrl,
isSelfHosted: params.isSelfHosted,
isMaintenance: params.isMaintenance ?? false,
};
}

View File

@@ -38,7 +38,6 @@ import { Route as AppOrganizationIdProjectIdReportsRouteImport } from './routes/
import { Route as AppOrganizationIdProjectIdReferencesRouteImport } from './routes/_app.$organizationId.$projectId.references'
import { Route as AppOrganizationIdProjectIdRealtimeRouteImport } from './routes/_app.$organizationId.$projectId.realtime'
import { Route as AppOrganizationIdProjectIdPagesRouteImport } from './routes/_app.$organizationId.$projectId.pages'
import { Route as AppOrganizationIdProjectIdInsightsRouteImport } from './routes/_app.$organizationId.$projectId.insights'
import { Route as AppOrganizationIdProjectIdDashboardsRouteImport } from './routes/_app.$organizationId.$projectId.dashboards'
import { Route as AppOrganizationIdProjectIdChatRouteImport } from './routes/_app.$organizationId.$projectId.chat'
import { Route as AppOrganizationIdMembersTabsIndexRouteImport } from './routes/_app.$organizationId.members._tabs.index'
@@ -274,12 +273,6 @@ const AppOrganizationIdProjectIdPagesRoute =
path: '/pages',
getParentRoute: () => AppOrganizationIdProjectIdRoute,
} as any)
const AppOrganizationIdProjectIdInsightsRoute =
AppOrganizationIdProjectIdInsightsRouteImport.update({
id: '/insights',
path: '/insights',
getParentRoute: () => AppOrganizationIdProjectIdRoute,
} as any)
const AppOrganizationIdProjectIdDashboardsRoute =
AppOrganizationIdProjectIdDashboardsRouteImport.update({
id: '/dashboards',
@@ -502,7 +495,6 @@ export interface FileRoutesByFullPath {
'/$organizationId/': typeof AppOrganizationIdIndexRoute
'/$organizationId/$projectId/chat': typeof AppOrganizationIdProjectIdChatRoute
'/$organizationId/$projectId/dashboards': typeof AppOrganizationIdProjectIdDashboardsRoute
'/$organizationId/$projectId/insights': typeof AppOrganizationIdProjectIdInsightsRoute
'/$organizationId/$projectId/pages': typeof AppOrganizationIdProjectIdPagesRoute
'/$organizationId/$projectId/realtime': typeof AppOrganizationIdProjectIdRealtimeRoute
'/$organizationId/$projectId/references': typeof AppOrganizationIdProjectIdReferencesRoute
@@ -560,7 +552,6 @@ export interface FileRoutesByTo {
'/$organizationId': typeof AppOrganizationIdIndexRoute
'/$organizationId/$projectId/chat': typeof AppOrganizationIdProjectIdChatRoute
'/$organizationId/$projectId/dashboards': typeof AppOrganizationIdProjectIdDashboardsRoute
'/$organizationId/$projectId/insights': typeof AppOrganizationIdProjectIdInsightsRoute
'/$organizationId/$projectId/pages': typeof AppOrganizationIdProjectIdPagesRoute
'/$organizationId/$projectId/realtime': typeof AppOrganizationIdProjectIdRealtimeRoute
'/$organizationId/$projectId/references': typeof AppOrganizationIdProjectIdReferencesRoute
@@ -618,7 +609,6 @@ export interface FileRoutesById {
'/_app/$organizationId/': typeof AppOrganizationIdIndexRoute
'/_app/$organizationId/$projectId/chat': typeof AppOrganizationIdProjectIdChatRoute
'/_app/$organizationId/$projectId/dashboards': typeof AppOrganizationIdProjectIdDashboardsRoute
'/_app/$organizationId/$projectId/insights': typeof AppOrganizationIdProjectIdInsightsRoute
'/_app/$organizationId/$projectId/pages': typeof AppOrganizationIdProjectIdPagesRoute
'/_app/$organizationId/$projectId/realtime': typeof AppOrganizationIdProjectIdRealtimeRoute
'/_app/$organizationId/$projectId/references': typeof AppOrganizationIdProjectIdReferencesRoute
@@ -687,7 +677,6 @@ export interface FileRouteTypes {
| '/$organizationId/'
| '/$organizationId/$projectId/chat'
| '/$organizationId/$projectId/dashboards'
| '/$organizationId/$projectId/insights'
| '/$organizationId/$projectId/pages'
| '/$organizationId/$projectId/realtime'
| '/$organizationId/$projectId/references'
@@ -745,7 +734,6 @@ export interface FileRouteTypes {
| '/$organizationId'
| '/$organizationId/$projectId/chat'
| '/$organizationId/$projectId/dashboards'
| '/$organizationId/$projectId/insights'
| '/$organizationId/$projectId/pages'
| '/$organizationId/$projectId/realtime'
| '/$organizationId/$projectId/references'
@@ -802,7 +790,6 @@ export interface FileRouteTypes {
| '/_app/$organizationId/'
| '/_app/$organizationId/$projectId/chat'
| '/_app/$organizationId/$projectId/dashboards'
| '/_app/$organizationId/$projectId/insights'
| '/_app/$organizationId/$projectId/pages'
| '/_app/$organizationId/$projectId/realtime'
| '/_app/$organizationId/$projectId/references'
@@ -1098,13 +1085,6 @@ declare module '@tanstack/react-router' {
preLoaderRoute: typeof AppOrganizationIdProjectIdPagesRouteImport
parentRoute: typeof AppOrganizationIdProjectIdRoute
}
'/_app/$organizationId/$projectId/insights': {
id: '/_app/$organizationId/$projectId/insights'
path: '/insights'
fullPath: '/$organizationId/$projectId/insights'
preLoaderRoute: typeof AppOrganizationIdProjectIdInsightsRouteImport
parentRoute: typeof AppOrganizationIdProjectIdRoute
}
'/_app/$organizationId/$projectId/dashboards': {
id: '/_app/$organizationId/$projectId/dashboards'
path: '/dashboards'
@@ -1548,7 +1528,6 @@ const AppOrganizationIdProjectIdSettingsRouteWithChildren =
interface AppOrganizationIdProjectIdRouteChildren {
AppOrganizationIdProjectIdChatRoute: typeof AppOrganizationIdProjectIdChatRoute
AppOrganizationIdProjectIdDashboardsRoute: typeof AppOrganizationIdProjectIdDashboardsRoute
AppOrganizationIdProjectIdInsightsRoute: typeof AppOrganizationIdProjectIdInsightsRoute
AppOrganizationIdProjectIdPagesRoute: typeof AppOrganizationIdProjectIdPagesRoute
AppOrganizationIdProjectIdRealtimeRoute: typeof AppOrganizationIdProjectIdRealtimeRoute
AppOrganizationIdProjectIdReferencesRoute: typeof AppOrganizationIdProjectIdReferencesRoute
@@ -1569,8 +1548,6 @@ const AppOrganizationIdProjectIdRouteChildren: AppOrganizationIdProjectIdRouteCh
AppOrganizationIdProjectIdChatRoute: AppOrganizationIdProjectIdChatRoute,
AppOrganizationIdProjectIdDashboardsRoute:
AppOrganizationIdProjectIdDashboardsRoute,
AppOrganizationIdProjectIdInsightsRoute:
AppOrganizationIdProjectIdInsightsRoute,
AppOrganizationIdProjectIdPagesRoute: AppOrganizationIdProjectIdPagesRoute,
AppOrganizationIdProjectIdRealtimeRoute:
AppOrganizationIdProjectIdRealtimeRoute,

View File

@@ -34,7 +34,6 @@ interface MyRouterContext {
apiUrl: string;
dashboardUrl: string;
isSelfHosted: boolean;
isMaintenance: boolean;
}
export const Route = createRootRouteWithContext<MyRouterContext>()({

View File

@@ -3,7 +3,6 @@ import {
OverviewFiltersButtons,
} from '@/components/overview/filters/overview-filters-buttons';
import { LiveCounter } from '@/components/overview/live-counter';
import OverviewInsights from '@/components/overview/overview-insights';
import { OverviewInterval } from '@/components/overview/overview-interval';
import OverviewMetrics from '@/components/overview/overview-metrics';
import { OverviewRange } from '@/components/overview/overview-range';
@@ -51,7 +50,6 @@ function ProjectDashboard() {
</div>
<div className="grid grid-cols-6 gap-4 p-4 pt-0">
<OverviewMetrics projectId={projectId} />
<OverviewInsights projectId={projectId} />
<OverviewTopSources projectId={projectId} />
<OverviewTopPages projectId={projectId} />
<OverviewTopDevices projectId={projectId} />

View File

@@ -1,431 +0,0 @@
import { FullPageEmptyState } from '@/components/full-page-empty-state';
import { InsightCard } from '@/components/insights/insight-card';
import { PageContainer } from '@/components/page-container';
import { PageHeader } from '@/components/page-header';
import { Skeleton } from '@/components/skeleton';
import {
Carousel,
CarouselContent,
CarouselItem,
CarouselNext,
CarouselPrevious,
} from '@/components/ui/carousel';
import { Input } from '@/components/ui/input';
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from '@/components/ui/select';
import { TableButtons } from '@/components/ui/table';
import { useTRPC } from '@/integrations/trpc/react';
import { cn } from '@/utils/cn';
import { PAGE_TITLES, createProjectTitle } from '@/utils/title';
import { useQuery } from '@tanstack/react-query';
import { createFileRoute, useNavigate } from '@tanstack/react-router';
import { parseAsString, parseAsStringEnum, useQueryState } from 'nuqs';
import { useMemo } from 'react';
export const Route = createFileRoute(
'/_app/$organizationId/$projectId/insights',
)({
component: Component,
head: () => {
return {
meta: [
{
title: createProjectTitle(PAGE_TITLES.INSIGHTS),
},
],
};
},
});
type SortOption =
| 'impact-desc'
| 'impact-asc'
| 'severity-desc'
| 'severity-asc'
| 'recent';
function getModuleDisplayName(moduleKey: string): string {
const displayNames: Record<string, string> = {
geo: 'Geographic',
devices: 'Devices',
referrers: 'Referrers',
'entry-pages': 'Entry Pages',
'page-trends': 'Page Trends',
'exit-pages': 'Exit Pages',
'traffic-anomalies': 'Anomalies',
};
return displayNames[moduleKey] || moduleKey.replace('-', ' ');
}
function Component() {
const { projectId } = Route.useParams();
const trpc = useTRPC();
const { data: insights, isLoading } = useQuery(
trpc.insight.listAll.queryOptions({
projectId,
limit: 500,
}),
);
const navigate = useNavigate();
const [search, setSearch] = useQueryState(
'search',
parseAsString.withDefault(''),
);
const [moduleFilter, setModuleFilter] = useQueryState(
'module',
parseAsString.withDefault('all'),
);
const [windowKindFilter, setWindowKindFilter] = useQueryState(
'window',
parseAsStringEnum([
'all',
'yesterday',
'rolling_7d',
'rolling_30d',
]).withDefault('all'),
);
const [severityFilter, setSeverityFilter] = useQueryState(
'severity',
parseAsStringEnum(['all', 'severe', 'moderate', 'low', 'none']).withDefault(
'all',
),
);
const [directionFilter, setDirectionFilter] = useQueryState(
'direction',
parseAsStringEnum(['all', 'up', 'down', 'flat']).withDefault('all'),
);
const [sortBy, setSortBy] = useQueryState(
'sort',
parseAsStringEnum<SortOption>([
'impact-desc',
'impact-asc',
'severity-desc',
'severity-asc',
'recent',
]).withDefault('impact-desc'),
);
const filteredAndSorted = useMemo(() => {
if (!insights) return [];
const filtered = insights.filter((insight) => {
// Search filter
if (search) {
const searchLower = search.toLowerCase();
const matchesTitle = insight.title.toLowerCase().includes(searchLower);
const matchesSummary = insight.summary
?.toLowerCase()
.includes(searchLower);
const matchesDimension = insight.dimensionKey
.toLowerCase()
.includes(searchLower);
if (!matchesTitle && !matchesSummary && !matchesDimension) {
return false;
}
}
// Module filter
if (moduleFilter !== 'all' && insight.moduleKey !== moduleFilter) {
return false;
}
// Window kind filter
if (
windowKindFilter !== 'all' &&
insight.windowKind !== windowKindFilter
) {
return false;
}
// Severity filter
if (severityFilter !== 'all') {
if (severityFilter === 'none' && insight.severityBand) return false;
if (
severityFilter !== 'none' &&
insight.severityBand !== severityFilter
)
return false;
}
// Direction filter
if (directionFilter !== 'all' && insight.direction !== directionFilter) {
return false;
}
return true;
});
// Sort (create new array to avoid mutation)
const sorted = [...filtered].sort((a, b) => {
switch (sortBy) {
case 'impact-desc':
return (b.impactScore ?? 0) - (a.impactScore ?? 0);
case 'impact-asc':
return (a.impactScore ?? 0) - (b.impactScore ?? 0);
case 'severity-desc': {
const severityOrder: Record<string, number> = {
severe: 3,
moderate: 2,
low: 1,
};
const aSev = severityOrder[a.severityBand ?? ''] ?? 0;
const bSev = severityOrder[b.severityBand ?? ''] ?? 0;
return bSev - aSev;
}
case 'severity-asc': {
const severityOrder: Record<string, number> = {
severe: 3,
moderate: 2,
low: 1,
};
const aSev = severityOrder[a.severityBand ?? ''] ?? 0;
const bSev = severityOrder[b.severityBand ?? ''] ?? 0;
return aSev - bSev;
}
case 'recent':
return (
new Date(b.firstDetectedAt ?? 0).getTime() -
new Date(a.firstDetectedAt ?? 0).getTime()
);
default:
return 0;
}
});
return sorted;
}, [
insights,
search,
moduleFilter,
windowKindFilter,
severityFilter,
directionFilter,
sortBy,
]);
// Group insights by module
const groupedByModule = useMemo(() => {
const groups = new Map<string, typeof filteredAndSorted>();
for (const insight of filteredAndSorted) {
const existing = groups.get(insight.moduleKey) ?? [];
existing.push(insight);
groups.set(insight.moduleKey, existing);
}
// Sort modules by impact (referrers first, then by average impact score)
return Array.from(groups.entries()).sort(
([keyA, insightsA], [keyB, insightsB]) => {
// Referrers always first
if (keyA === 'referrers') return -1;
if (keyB === 'referrers') return 1;
// Calculate average impact for each module
const avgImpactA =
insightsA.reduce((sum, i) => sum + (i.impactScore ?? 0), 0) /
insightsA.length;
const avgImpactB =
insightsB.reduce((sum, i) => sum + (i.impactScore ?? 0), 0) /
insightsB.length;
// Sort by average impact (high to low)
return avgImpactB - avgImpactA;
},
);
}, [filteredAndSorted]);
if (isLoading) {
return (
<PageContainer>
<PageHeader title="Insights" className="mb-8" />
<div className="space-y-8">
{Array.from({ length: 3 }, (_, i) => `section-${i}`).map((key) => (
<div key={key} className="space-y-4">
<Skeleton className="h-8 w-32" />
<Carousel opts={{ align: 'start' }} className="w-full">
<CarouselContent className="-ml-4">
{Array.from({ length: 4 }, (_, i) => `skeleton-${i}`).map(
(cardKey) => (
<CarouselItem
key={cardKey}
className="pl-4 basis-full sm:basis-1/2 lg:basis-1/3 xl:basis-1/4"
>
<Skeleton className="h-48 w-full" />
</CarouselItem>
),
)}
</CarouselContent>
</Carousel>
</div>
))}
</div>
</PageContainer>
);
}
return (
<PageContainer>
<PageHeader
title="Insights"
description="Discover trends and changes in your analytics"
className="mb-8"
/>
<TableButtons className="mb-8">
<Input
placeholder="Search insights..."
value={search ?? ''}
onChange={(e) => void setSearch(e.target.value || null)}
className="max-w-xs"
/>
<Select
value={windowKindFilter ?? 'all'}
onValueChange={(v) =>
void setWindowKindFilter(v as typeof windowKindFilter)
}
>
<SelectTrigger className="w-[140px]">
<SelectValue placeholder="Time Window" />
</SelectTrigger>
<SelectContent>
<SelectItem value="all">All Windows</SelectItem>
<SelectItem value="yesterday">Yesterday</SelectItem>
<SelectItem value="rolling_7d">7 Days</SelectItem>
<SelectItem value="rolling_30d">30 Days</SelectItem>
</SelectContent>
</Select>
<Select
value={severityFilter ?? 'all'}
onValueChange={(v) =>
void setSeverityFilter(v as typeof severityFilter)
}
>
<SelectTrigger className="w-[140px]">
<SelectValue placeholder="Severity" />
</SelectTrigger>
<SelectContent>
<SelectItem value="all">All Severity</SelectItem>
<SelectItem value="severe">Severe</SelectItem>
<SelectItem value="moderate">Moderate</SelectItem>
<SelectItem value="low">Low</SelectItem>
<SelectItem value="none">No Severity</SelectItem>
</SelectContent>
</Select>
<Select
value={directionFilter ?? 'all'}
onValueChange={(v) =>
void setDirectionFilter(v as typeof directionFilter)
}
>
<SelectTrigger className="w-[140px]">
<SelectValue placeholder="Direction" />
</SelectTrigger>
<SelectContent>
<SelectItem value="all">All Directions</SelectItem>
<SelectItem value="up">Increasing</SelectItem>
<SelectItem value="down">Decreasing</SelectItem>
<SelectItem value="flat">Flat</SelectItem>
</SelectContent>
</Select>
<Select
value={sortBy ?? 'impact-desc'}
onValueChange={(v) => void setSortBy(v as SortOption)}
>
<SelectTrigger className="w-[160px]">
<SelectValue placeholder="Sort by" />
</SelectTrigger>
<SelectContent>
<SelectItem value="impact-desc">Impact (High Low)</SelectItem>
<SelectItem value="impact-asc">Impact (Low High)</SelectItem>
<SelectItem value="severity-desc">Severity (High Low)</SelectItem>
<SelectItem value="severity-asc">Severity (Low High)</SelectItem>
<SelectItem value="recent">Most Recent</SelectItem>
</SelectContent>
</Select>
</TableButtons>
{filteredAndSorted.length === 0 && !isLoading && (
<FullPageEmptyState
title="No insights found"
description={
search || moduleFilter !== 'all' || windowKindFilter !== 'all'
? 'Try adjusting your filters to see more insights.'
: 'Insights will appear here as trends are detected in your analytics.'
}
/>
)}
{groupedByModule.length > 0 && (
<div className="space-y-8">
{groupedByModule.map(([moduleKey, moduleInsights]) => (
<div key={moduleKey} className="space-y-4">
<div className="flex items-center justify-between">
<h2 className="text-lg font-semibold capitalize">
{getModuleDisplayName(moduleKey)}
</h2>
<span className="text-sm text-muted-foreground">
{moduleInsights.length}{' '}
{moduleInsights.length === 1 ? 'insight' : 'insights'}
</span>
</div>
<div className="-mx-8">
<Carousel
opts={{ align: 'start', dragFree: true }}
className="w-full group"
>
<CarouselContent className="mx-4 mr-8">
{moduleInsights.map((insight, index) => (
<CarouselItem
key={insight.id}
className={cn(
'pl-4 basis-full sm:basis-1/2 lg:basis-1/3 xl:basis-1/4',
)}
>
<InsightCard
insight={insight}
onFilter={(() => {
const filterString = insight.payload?.dimensions
.map(
(dim) =>
`${dim.key},is,${encodeURIComponent(dim.value)}`,
)
.join(';');
if (filterString) {
return () => {
navigate({
to: '/$organizationId/$projectId',
from: Route.fullPath,
search: {
f: filterString,
},
});
};
}
return undefined;
})()}
/>
</CarouselItem>
))}
</CarouselContent>
<CarouselPrevious className="opacity-0 [&:disabled]:opacity-0 pointer-events-none transition-opacity group-hover:opacity-100 group-hover:pointer-events-auto left-3" />
<CarouselNext className="opacity-0 [&:disabled]:opacity-0 pointer-events-none transition-opacity group-hover:opacity-100 group-hover:pointer-events-auto right-3" />
</Carousel>
</div>
</div>
))}
</div>
)}
{filteredAndSorted.length > 0 && (
<div className="mt-8 text-sm text-muted-foreground text-center">
Showing {filteredAndSorted.length} of {insights?.length ?? 0} insights
</div>
)}
</PageContainer>
);
}

View File

@@ -1,10 +1,5 @@
import { FullPageEmptyState } from '@/components/full-page-empty-state';
import { Sidebar } from '@/components/sidebar';
import { Button, LinkButton, buttonVariants } from '@/components/ui/button';
import { useAppContext } from '@/hooks/use-app-context';
import { cn } from '@/utils/cn';
import { Outlet, createFileRoute, redirect } from '@tanstack/react-router';
import { ConstructionIcon } from 'lucide-react';
export const Route = createFileRoute('/_app')({
beforeLoad: async ({ context }) => {
@@ -16,28 +11,6 @@ export const Route = createFileRoute('/_app')({
});
function AppLayout() {
const { isMaintenance } = useAppContext();
if (isMaintenance) {
return (
<FullPageEmptyState
icon={ConstructionIcon}
className="min-h-screen"
title="Maintenance mode"
description="We are currently performing maintenance on the system. Please check back later."
>
<a
href="https://status.openpanel.dev/"
className={cn(buttonVariants())}
target="_blank"
rel="noopener noreferrer"
>
Check out our status page
</a>
</FullPageEmptyState>
);
}
return (
<div className="flex h-screen w-full">
<Sidebar />

View File

@@ -8,7 +8,6 @@ export const getServerEnvs = createServerFn().handler(async () => {
process.env.DASHBOARD_URL || process.env.NEXT_PUBLIC_DASHBOARD_URL,
),
isSelfHosted: process.env.SELF_HOSTED !== undefined,
isMaintenance: process.env.MAINTENANCE === '1',
};
return envs;

View File

@@ -90,7 +90,6 @@ export const PAGE_TITLES = {
CHAT: 'AI Assistant',
REALTIME: 'Realtime',
REFERENCES: 'References',
INSIGHTS: 'Insights',
// Profiles
PROFILES: 'Profiles',
PROFILE_EVENTS: 'Profile events',

View File

@@ -34,11 +34,6 @@ export async function bootCron() {
type: 'flushSessions',
pattern: 1000 * 10,
},
{
name: 'insightsDaily',
type: 'insightsDaily',
pattern: '0 2 * * *',
},
];
if (process.env.SELF_HOSTED && process.env.NODE_ENV === 'production') {

View File

@@ -7,7 +7,6 @@ import {
cronQueue,
eventsGroupQueues,
importQueue,
insightsQueue,
miscQueue,
notificationQueue,
queueLogger,
@@ -22,7 +21,6 @@ import { Worker as GroupWorker } from 'groupmq';
import { cronJob } from './jobs/cron';
import { incomingEvent } from './jobs/events.incoming-event';
import { importJob } from './jobs/import';
import { insightsProjectJob } from './jobs/insights';
import { miscJob } from './jobs/misc';
import { notificationJob } from './jobs/notification';
import { sessionsJob } from './jobs/sessions';
@@ -51,15 +49,7 @@ function getEnabledQueues(): QueueName[] {
logger.info('No ENABLED_QUEUES specified, starting all queues', {
totalEventShards: EVENTS_GROUP_QUEUES_SHARDS,
});
return [
'events',
'sessions',
'cron',
'notification',
'misc',
'import',
'insights',
];
return ['events', 'sessions', 'cron', 'notification', 'misc', 'import'];
}
const queues = enabledQueuesEnv
@@ -127,7 +117,7 @@ export async function bootWorkers() {
const worker = new GroupWorker<EventsQueuePayloadIncomingEvent['payload']>({
queue,
concurrency,
logger: process.env.NODE_ENV === 'production' ? queueLogger : undefined,
logger: queueLogger,
blockingTimeoutSec: Number.parseFloat(
process.env.EVENT_BLOCKING_TIMEOUT_SEC || '1',
),
@@ -197,17 +187,6 @@ export async function bootWorkers() {
logger.info('Started worker for import', { concurrency });
}
// Start insights worker
if (enabledQueues.includes('insights')) {
const concurrency = getConcurrencyFor('insights', 5);
const insightsWorker = new Worker(insightsQueue.name, insightsProjectJob, {
...workerOptions,
concurrency,
});
workers.push(insightsWorker);
logger.info('Started worker for insights', { concurrency });
}
if (workers.length === 0) {
logger.warn(
'No workers started. Check ENABLED_QUEUES environment variable.',

View File

@@ -6,7 +6,6 @@ import {
cronQueue,
eventsGroupQueues,
importQueue,
insightsQueue,
miscQueue,
notificationQueue,
sessionsQueue,
@@ -43,7 +42,6 @@ async function start() {
new BullMQAdapter(notificationQueue),
new BullMQAdapter(miscQueue),
new BullMQAdapter(importQueue),
new BullMQAdapter(insightsQueue),
],
serverAdapter: serverAdapter,
});

View File

@@ -1,10 +1,6 @@
import { TABLE_NAMES, chQuery } from '@openpanel/db';
export async function ping() {
if (process.env.DISABLE_PING) {
return;
}
const [res] = await chQuery<{ count: number }>(
`SELECT COUNT(*) as count FROM ${TABLE_NAMES.events}`,
);

View File

@@ -6,7 +6,6 @@ import type { CronQueuePayload } from '@openpanel/queue';
import { jobdeleteProjects } from './cron.delete-projects';
import { ping } from './cron.ping';
import { salt } from './cron.salt';
import { insightsDailyJob } from './insights';
export async function cronJob(job: Job<CronQueuePayload>) {
switch (job.data.type) {
@@ -28,8 +27,5 @@ export async function cronJob(job: Job<CronQueuePayload>) {
case 'deleteProjects': {
return await jobdeleteProjects(job);
}
case 'insightsDaily': {
return await insightsDailyJob(job);
}
}
}

View File

@@ -96,6 +96,7 @@ export async function createSessionEnd(
...payload,
properties: {
...payload.properties,
...(session?.properties ?? {}),
__bounce: session.is_bounce,
},
name: 'session_end',

View File

@@ -319,6 +319,7 @@ describe('incomingEvent', () => {
utm_content: '',
utm_medium: '',
revenue: 0,
properties: {},
project_id: projectId,
device_id: 'last-device-123',
profile_id: 'profile-123',

View File

@@ -1,72 +0,0 @@
import { ch } from '@openpanel/db/src/clickhouse/client';
import {
createEngine,
devicesModule,
entryPagesModule,
geoModule,
insightStore,
pageTrendsModule,
referrersModule,
} from '@openpanel/db/src/services/insights';
import type {
CronQueuePayload,
InsightsQueuePayloadProject,
} from '@openpanel/queue';
import { insightsQueue } from '@openpanel/queue';
import type { Job } from 'bullmq';
const defaultEngineConfig = {
keepTopNPerModuleWindow: 20,
closeStaleAfterDays: 7,
dimensionBatchSize: 50,
globalThresholds: {
minTotal: 200,
minAbsDelta: 80,
minPct: 0.15,
},
};
export async function insightsDailyJob(job: Job<CronQueuePayload>) {
const projectIds = await insightStore.listProjectIdsForCadence('daily');
const date = new Date().toISOString().slice(0, 10);
for (const projectId of projectIds) {
await insightsQueue.add(
'insightsProject',
{
type: 'insightsProject',
payload: { projectId, date },
},
{
jobId: `daily:${date}:${projectId}`, // idempotent
},
);
}
}
export async function insightsProjectJob(
job: Job<InsightsQueuePayloadProject>,
) {
const { projectId, date } = job.data.payload;
const engine = createEngine({
store: insightStore,
modules: [
referrersModule,
entryPagesModule,
pageTrendsModule,
geoModule,
devicesModule,
],
db: ch,
config: defaultEngineConfig,
});
const projectCreatedAt = await insightStore.getProjectCreatedAt(projectId);
await engine.runProject({
projectId,
cadence: 'daily',
now: new Date(date),
projectCreatedAt,
});
}

10
packages/auth/nextjs.ts Normal file
View File

@@ -0,0 +1,10 @@
import { unstable_cache } from 'next/cache';
import { cookies } from 'next/headers';
import { validateSessionToken } from './src/session';
export const auth = async () => {
const token = (await cookies().get('session')?.value) ?? null;
return cachedAuth(token);
};
export const cachedAuth = unstable_cache(validateSessionToken);

View File

@@ -22,6 +22,7 @@
"typescript": "catalog:"
},
"peerDependencies": {
"next": "14.2.1",
"react": "catalog:"
}
}
}

View File

@@ -245,259 +245,3 @@ export function getDefaultIntervalByDates(
return null;
}
export const countries = {
AF: 'Afghanistan',
AL: 'Albania',
DZ: 'Algeria',
AS: 'American Samoa',
AD: 'Andorra',
AO: 'Angola',
AI: 'Anguilla',
AQ: 'Antarctica',
AG: 'Antigua and Barbuda',
AR: 'Argentina',
AM: 'Armenia',
AW: 'Aruba',
AU: 'Australia',
AT: 'Austria',
AZ: 'Azerbaijan',
BS: 'Bahamas',
BH: 'Bahrain',
BD: 'Bangladesh',
BB: 'Barbados',
BY: 'Belarus',
BE: 'Belgium',
BZ: 'Belize',
BJ: 'Benin',
BM: 'Bermuda',
BT: 'Bhutan',
BO: 'Bolivia',
BQ: 'Bonaire, Sint Eustatius and Saba',
BA: 'Bosnia and Herzegovina',
BW: 'Botswana',
BV: 'Bouvet Island',
BR: 'Brazil',
IO: 'British Indian Ocean Territory',
BN: 'Brunei Darussalam',
BG: 'Bulgaria',
BF: 'Burkina Faso',
BI: 'Burundi',
CV: 'Cabo Verde',
KH: 'Cambodia',
CM: 'Cameroon',
CA: 'Canada',
KY: 'Cayman Islands',
CF: 'Central African Republic',
TD: 'Chad',
CL: 'Chile',
CN: 'China',
CX: 'Christmas Island',
CC: 'Cocos (Keeling) Islands',
CO: 'Colombia',
KM: 'Comoros',
CD: 'Congo (Democratic Republic)',
CG: 'Congo',
CK: 'Cook Islands',
CR: 'Costa Rica',
HR: 'Croatia',
CU: 'Cuba',
CW: 'Curaçao',
CY: 'Cyprus',
CZ: 'Czechia',
CI: "Côte d'Ivoire",
DK: 'Denmark',
DJ: 'Djibouti',
DM: 'Dominica',
DO: 'Dominican Republic',
EC: 'Ecuador',
EG: 'Egypt',
SV: 'El Salvador',
GQ: 'Equatorial Guinea',
ER: 'Eritrea',
EE: 'Estonia',
SZ: 'Eswatina',
ET: 'Ethiopia',
FK: 'Falkland Islands',
FO: 'Faroe Islands',
FJ: 'Fiji',
FI: 'Finland',
FR: 'France',
GF: 'French Guiana',
PF: 'French Polynesia',
TF: 'French Southern Territories',
GA: 'Gabon',
GM: 'Gambia',
GE: 'Georgia',
DE: 'Germany',
GH: 'Ghana',
GI: 'Gibraltar',
GR: 'Greece',
GL: 'Greenland',
GD: 'Grenada',
GP: 'Guadeloupe',
GU: 'Guam',
GT: 'Guatemala',
GG: 'Guernsey',
GN: 'Guinea',
GW: 'Guinea-Bissau',
GY: 'Guyana',
HT: 'Haiti',
HM: 'Heard Island and McDonald Islands',
VA: 'Holy See',
HN: 'Honduras',
HK: 'Hong Kong',
HU: 'Hungary',
IS: 'Iceland',
IN: 'India',
ID: 'Indonesia',
IR: 'Iran',
IQ: 'Iraq',
IE: 'Ireland',
IM: 'Isle of Man',
IL: 'Israel',
IT: 'Italy',
JM: 'Jamaica',
JP: 'Japan',
JE: 'Jersey',
JO: 'Jordan',
KZ: 'Kazakhstan',
KE: 'Kenya',
KI: 'Kiribati',
KP: "Korea (Democratic People's Republic)",
KR: 'Korea (Republic)',
KW: 'Kuwait',
KG: 'Kyrgyzstan',
LA: "Lao People's Democratic Republic",
LV: 'Latvia',
LB: 'Lebanon',
LS: 'Lesotho',
LR: 'Liberia',
LY: 'Libya',
LI: 'Liechtenstein',
LT: 'Lithuania',
LU: 'Luxembourg',
MO: 'Macao',
MG: 'Madagascar',
MW: 'Malawi',
MY: 'Malaysia',
MV: 'Maldives',
ML: 'Mali',
MT: 'Malta',
MH: 'Marshall Islands',
MQ: 'Martinique',
MR: 'Mauritania',
MU: 'Mauritius',
YT: 'Mayotte',
MX: 'Mexico',
FM: 'Micronesia',
MD: 'Moldova',
MC: 'Monaco',
MN: 'Mongolia',
ME: 'Montenegro',
MS: 'Montserrat',
MA: 'Morocco',
MZ: 'Mozambique',
MM: 'Myanmar',
NA: 'Namibia',
NR: 'Nauru',
NP: 'Nepal',
NL: 'Netherlands',
NC: 'New Caledonia',
NZ: 'New Zealand',
NI: 'Nicaragua',
NE: 'Niger',
NG: 'Nigeria',
NU: 'Niue',
NF: 'Norfolk Island',
MP: 'Northern Mariana Islands',
NO: 'Norway',
OM: 'Oman',
PK: 'Pakistan',
PW: 'Palau',
PS: 'Palestine, State of',
PA: 'Panama',
PG: 'Papua New Guinea',
PY: 'Paraguay',
PE: 'Peru',
PH: 'Philippines',
PN: 'Pitcairn',
PL: 'Poland',
PT: 'Portugal',
PR: 'Puerto Rico',
QA: 'Qatar',
MK: 'Republic of North Macedonia',
RO: 'Romania',
RU: 'Russian Federation',
RW: 'Rwanda',
RE: 'Réunion',
BL: 'Saint Barthélemy',
SH: 'Saint Helena, Ascension and Tristan da Cunha',
KN: 'Saint Kitts and Nevis',
LC: 'Saint Lucia',
MF: 'Saint Martin (French part)',
PM: 'Saint Pierre and Miquelon',
VC: 'Saint Vincent and the Grenadines',
WS: 'Samoa',
SM: 'San Marino',
ST: 'Sao Tome and Principe',
SA: 'Saudi Arabia',
SN: 'Senegal',
RS: 'Serbia',
SC: 'Seychelles',
SL: 'Sierra Leone',
SG: 'Singapore',
SX: 'Sint Maarten (Dutch part)',
SK: 'Slovakia',
SI: 'Slovenia',
SB: 'Solomon Islands',
SO: 'Somalia',
ZA: 'South Africa',
GS: 'South Georgia and the South Sandwich Islands',
SS: 'South Sudan',
ES: 'Spain',
LK: 'Sri Lanka',
SD: 'Sudan',
SR: 'Suriname',
SJ: 'Svalbard and Jan Mayen',
SE: 'Sweden',
CH: 'Switzerland',
SY: 'Syrian Arab Republic',
TW: 'Taiwan',
TJ: 'Tajikistan',
TZ: 'Tanzania, United Republic of',
TH: 'Thailand',
TL: 'Timor-Leste',
TG: 'Togo',
TK: 'Tokelau',
TO: 'Tonga',
TT: 'Trinidad and Tobago',
TN: 'Tunisia',
TR: 'Turkey',
TM: 'Turkmenistan',
TC: 'Turks and Caicos Islands',
TV: 'Tuvalu',
UG: 'Uganda',
UA: 'Ukraine',
AE: 'United Arab Emirates',
GB: 'United Kingdom',
US: 'United States',
UM: 'United States Minor Outlying Islands',
UY: 'Uruguay',
UZ: 'Uzbekistan',
VU: 'Vanuatu',
VE: 'Venezuela',
VN: 'Viet Nam',
VG: 'Virgin Islands (British)',
VI: 'Virgin Islands (U.S.)',
WF: 'Wallis and Futuna',
EH: 'Western Sahara',
YE: 'Yemen',
ZM: 'Zambia',
ZW: 'Zimbabwe',
AX: 'Åland Islands',
} as const;
export function getCountry(code?: string) {
return countries[code as keyof typeof countries];
}

View File

@@ -100,9 +100,6 @@ async function createOldSessions() {
if (!row || row.count === '0') {
return null;
}
if (row.created_at.startsWith('1970')) {
return null;
}
return new Date(row.created_at);
} catch (e) {
return defaultDate;

View File

@@ -1,302 +0,0 @@
import fs from 'node:fs';
import path from 'node:path';
import {
chMigrationClient,
createTable,
moveDataBetweenTables,
renameTable,
runClickhouseMigrationCommands,
} from '../src/clickhouse/migration';
import { getIsCluster } from './helpers';
export async function up() {
const isClustered = getIsCluster();
const sqls: string[] = [];
const eventTables = createTable({
name: 'events_new_20251123',
columns: [
'`id` UUID DEFAULT generateUUIDv4()',
'`name` LowCardinality(String)',
'`sdk_name` LowCardinality(String)',
'`sdk_version` LowCardinality(String)',
'`device_id` String CODEC(ZSTD(3))',
'`profile_id` String CODEC(ZSTD(3))',
'`project_id` String CODEC(ZSTD(3))',
'`session_id` String CODEC(LZ4)',
'`path` String CODEC(ZSTD(3))',
'`origin` String CODEC(ZSTD(3))',
'`referrer` String CODEC(ZSTD(3))',
'`referrer_name` String CODEC(ZSTD(3))',
'`referrer_type` LowCardinality(String)',
'`revenue` UInt64',
'`duration` UInt64 CODEC(Delta(4), LZ4)',
'`properties` Map(String, String) CODEC(ZSTD(3))',
'`created_at` DateTime64(3) CODEC(DoubleDelta, ZSTD(3))',
'`country` LowCardinality(FixedString(2))',
'`city` String',
'`region` LowCardinality(String)',
'`longitude` Nullable(Float32) CODEC(Gorilla, LZ4)',
'`latitude` Nullable(Float32) CODEC(Gorilla, LZ4)',
'`os` LowCardinality(String)',
'`os_version` LowCardinality(String)',
'`browser` LowCardinality(String)',
'`browser_version` LowCardinality(String)',
'`device` LowCardinality(String)',
'`brand` LowCardinality(String)',
'`model` LowCardinality(String)',
'`imported_at` Nullable(DateTime) CODEC(Delta(4), LZ4)',
],
indices: [
'INDEX idx_name name TYPE bloom_filter GRANULARITY 1',
"INDEX idx_properties_bounce properties['__bounce'] TYPE set(3) GRANULARITY 1",
'INDEX idx_origin origin TYPE bloom_filter(0.05) GRANULARITY 1',
'INDEX idx_path path TYPE bloom_filter(0.01) GRANULARITY 1',
],
// New ORDER BY: project_id, toDate(created_at), created_at, name
// Removed profile_id, added created_at for better ordering within same day
orderBy: ['project_id', 'toDate(created_at)', 'created_at', 'name'],
partitionBy: 'toYYYYMM(created_at)',
settings: {
index_granularity: 8192,
// For lightweight updates
enable_block_offset_column: 1,
enable_block_number_column: 1,
},
distributionHash:
'cityHash64(project_id, toString(toStartOfHour(created_at)))',
replicatedVersion: '1',
isClustered,
});
// Step 1: Create temporary tables with new ORDER BY keys
// Events table with new ORDER BY
sqls.push(...eventTables);
const sessionTables = createTable({
name: 'sessions_new_20251123',
engine: 'VersionedCollapsingMergeTree(sign, version)',
columns: [
'`id` String',
'`project_id` String CODEC(ZSTD(3))',
'`profile_id` String CODEC(ZSTD(3))',
'`device_id` String CODEC(ZSTD(3))',
'`created_at` DateTime64(3) CODEC(DoubleDelta, ZSTD(3))',
'`ended_at` DateTime64(3) CODEC(DoubleDelta, ZSTD(3))',
'`is_bounce` Bool',
'`entry_origin` LowCardinality(String)',
'`entry_path` String CODEC(ZSTD(3))',
'`exit_origin` LowCardinality(String)',
'`exit_path` String CODEC(ZSTD(3))',
'`screen_view_count` Int32',
'`revenue` Float64',
'`event_count` Int32',
'`duration` UInt32',
'`country` LowCardinality(FixedString(2))',
'`region` LowCardinality(String)',
'`city` String',
'`longitude` Nullable(Float32) CODEC(Gorilla, LZ4)',
'`latitude` Nullable(Float32) CODEC(Gorilla, LZ4)',
'`device` LowCardinality(String)',
'`brand` LowCardinality(String)',
'`model` LowCardinality(String)',
'`browser` LowCardinality(String)',
'`browser_version` LowCardinality(String)',
'`os` LowCardinality(String)',
'`os_version` LowCardinality(String)',
'`utm_medium` String CODEC(ZSTD(3))',
'`utm_source` String CODEC(ZSTD(3))',
'`utm_campaign` String CODEC(ZSTD(3))',
'`utm_content` String CODEC(ZSTD(3))',
'`utm_term` String CODEC(ZSTD(3))',
'`referrer` String CODEC(ZSTD(3))',
'`referrer_name` String CODEC(ZSTD(3))',
'`referrer_type` LowCardinality(String)',
'`sign` Int8',
'`version` UInt64',
],
// New ORDER BY: project_id, toDate(created_at), created_at, id
// Removed profile_id, reordered to match query patterns (date first, then id)
orderBy: ['project_id', 'toDate(created_at)', 'created_at'],
partitionBy: 'toYYYYMM(created_at)',
settings: {
index_granularity: 8192,
},
distributionHash:
'cityHash64(project_id, toString(toStartOfHour(created_at)))',
replicatedVersion: '1',
isClustered,
});
// Sessions table with new ORDER BY
sqls.push(...sessionTables);
const firstEventDateResponse = await chMigrationClient.query({
query: 'SELECT min(created_at) as created_at FROM events',
format: 'JSONEachRow',
});
const firstEventDateJson = await firstEventDateResponse.json<{
created_at: string;
}>();
if (
firstEventDateJson[0]?.created_at &&
!firstEventDateJson[0]?.created_at.startsWith('1970')
) {
const firstEventDate = new Date(firstEventDateJson[0]?.created_at);
// Step 2: Copy data from old tables to new tables (partitioned by month for efficiency)
// Set endDate to first of next month to ensure we capture all data in the current month
const endDate = new Date();
endDate.setMonth(endDate.getMonth() + 1);
endDate.setDate(1);
sqls.push(
...moveDataBetweenTables({
from: 'events',
to: 'events_new_20251123',
batch: {
startDate: firstEventDate,
endDate: endDate,
column: 'toDate(created_at)',
interval: 'month',
transform: (date: Date) => {
const year = date.getFullYear();
const month = String(date.getMonth() + 1).padStart(2, '0');
return `${year}-${month}-01`;
},
},
}),
);
}
const firstSessionDateResponse = await chMigrationClient.query({
query: 'SELECT min(created_at) as created_at FROM sessions',
format: 'JSONEachRow',
});
const firstSessionDateJson = await firstSessionDateResponse.json<{
created_at: string;
}>();
if (
firstSessionDateJson[0]?.created_at &&
!firstSessionDateJson[0]?.created_at.startsWith('1970')
) {
const firstSessionDate = new Date(
firstSessionDateJson[0]?.created_at ?? '',
);
// Set endDate to first of next month to ensure we capture all data in the current month
const endDate = new Date();
endDate.setMonth(endDate.getMonth() + 1);
endDate.setDate(1);
sqls.push(
...moveDataBetweenTables({
from: 'sessions',
to: 'sessions_new_20251123',
columns: [
'id',
'project_id',
'profile_id',
'device_id',
'created_at',
'ended_at',
'is_bounce',
'entry_origin',
'entry_path',
'exit_origin',
'exit_path',
'screen_view_count',
'revenue',
'event_count',
'duration',
'country',
'region',
'city',
'longitude',
'latitude',
'device',
'brand',
'model',
'browser',
'browser_version',
'os',
'os_version',
'utm_medium',
'utm_source',
'utm_campaign',
'utm_content',
'utm_term',
'referrer',
'referrer_name',
'referrer_type',
'sign',
'version',
],
batch: {
startDate: firstSessionDate,
endDate: endDate,
column: 'toDate(created_at)',
interval: 'month',
transform: (date: Date) => {
const year = date.getFullYear();
const month = String(date.getMonth() + 1).padStart(2, '0');
return `${year}-${month}-01`;
},
},
}),
);
}
sqls.push(
...renameTable({ from: 'events', to: 'events_20251123', isClustered }),
);
sqls.push(
...renameTable({ from: 'sessions', to: 'sessions_20251123', isClustered }),
);
if (isClustered && sessionTables[1] && eventTables[1]) {
sqls.push(
// Drop temporary DISTRIBUTED tables (will be recreated)
`DROP TABLE IF EXISTS events_new_20251123 ON CLUSTER '{cluster}'`,
`DROP TABLE IF EXISTS sessions_new_20251123 ON CLUSTER '{cluster}'`,
// Rename new tables to correct names
`RENAME TABLE events_new_20251123_replicated TO events_replicated ON CLUSTER '{cluster}'`,
`RENAME TABLE sessions_new_20251123_replicated TO sessions_replicated ON CLUSTER '{cluster}'`,
// Create new distributed tables
eventTables[1].replaceAll('events_new_20251123', 'events'), // creates a new distributed table
sessionTables[1].replaceAll('sessions_new_20251123', 'sessions'), // creates a new distributed table
);
} else {
sqls.push(
...renameTable({
from: 'events_new_20251123',
to: 'events',
isClustered,
}),
);
sqls.push(
...renameTable({
from: 'sessions_new_20251123',
to: 'sessions',
isClustered,
}),
);
}
fs.writeFileSync(
path.join(__filename.replace('.ts', '.sql')),
sqls
.map((sql) =>
sql
.trim()
.replace(/;$/, '')
.replace(/\n{2,}/g, '\n')
.concat(';'),
)
.join('\n\n---\n\n'),
);
if (!process.argv.includes('--dry')) {
await runClickhouseMigrationCommands(sqls);
}
}

View File

@@ -30,7 +30,3 @@ export function getIsSelfHosting() {
export function getIsDry() {
return process.argv.includes('--dry');
}
export function getShouldIgnoreRecord() {
return process.argv.includes('--no-record');
}

View File

@@ -10,7 +10,6 @@ import {
getIsCluster,
getIsDry,
getIsSelfHosting,
getShouldIgnoreRecord,
printBoxMessage,
} from './helpers';
@@ -56,12 +55,8 @@ async function migrate() {
]);
if (!getIsSelfHosting()) {
if (!getIsDry()) {
printBoxMessage('🕒 Migrations starts in 10 seconds', []);
await new Promise((resolve) => setTimeout(resolve, 10000));
} else {
printBoxMessage('🕒 Migrations starts now (dry run)', []);
}
printBoxMessage('🕒 Migrations starts in 10 seconds', []);
await new Promise((resolve) => setTimeout(resolve, 10000));
}
if (migration) {
@@ -86,7 +81,7 @@ async function runMigration(migrationsDir: string, file: string) {
try {
const migration = await import(path.join(migrationsDir, file));
await migration.up();
if (!getIsDry() && !getShouldIgnoreRecord()) {
if (!getIsDry()) {
await db.codeMigration.upsert({
where: {
name: file,

View File

@@ -28,5 +28,4 @@ export * from './src/types';
export * from './src/clickhouse/query-builder';
export * from './src/services/import.service';
export * from './src/services/overview.service';
export * from './src/services/insights';
export * from './src/session-context';

View File

@@ -1,57 +0,0 @@
-- CreateEnum
CREATE TYPE "public"."InsightState" AS ENUM ('active', 'suppressed', 'closed');
-- CreateTable
CREATE TABLE "public"."project_insights" (
"id" UUID NOT NULL DEFAULT gen_random_uuid(),
"projectId" TEXT NOT NULL,
"moduleKey" TEXT NOT NULL,
"dimensionKey" TEXT NOT NULL,
"windowKind" TEXT NOT NULL,
"state" "public"."InsightState" NOT NULL DEFAULT 'active',
"title" TEXT NOT NULL,
"summary" TEXT,
"payload" JSONB,
"currentValue" DOUBLE PRECISION,
"compareValue" DOUBLE PRECISION,
"changePct" DOUBLE PRECISION,
"direction" TEXT,
"impactScore" DOUBLE PRECISION NOT NULL DEFAULT 0,
"severityBand" TEXT,
"version" INTEGER NOT NULL DEFAULT 1,
"threadId" UUID NOT NULL DEFAULT gen_random_uuid(),
"windowStart" TIMESTAMP(3),
"windowEnd" TIMESTAMP(3),
"firstDetectedAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"lastUpdatedAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"lastSeenAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT "project_insights_pkey" PRIMARY KEY ("id")
);
-- CreateTable
CREATE TABLE "public"."insight_events" (
"id" UUID NOT NULL DEFAULT gen_random_uuid(),
"insightId" UUID NOT NULL,
"eventKind" TEXT NOT NULL,
"changeFrom" JSONB,
"changeTo" JSONB,
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT "insight_events_pkey" PRIMARY KEY ("id")
);
-- CreateIndex
CREATE INDEX "project_insights_projectId_impactScore_idx" ON "public"."project_insights"("projectId", "impactScore" DESC);
-- CreateIndex
CREATE INDEX "project_insights_projectId_moduleKey_windowKind_state_idx" ON "public"."project_insights"("projectId", "moduleKey", "windowKind", "state");
-- CreateIndex
CREATE UNIQUE INDEX "project_insights_projectId_moduleKey_dimensionKey_windowKin_key" ON "public"."project_insights"("projectId", "moduleKey", "dimensionKey", "windowKind", "state");
-- CreateIndex
CREATE INDEX "insight_events_insightId_createdAt_idx" ON "public"."insight_events"("insightId", "createdAt");
-- AddForeignKey
ALTER TABLE "public"."insight_events" ADD CONSTRAINT "insight_events_insightId_fkey" FOREIGN KEY ("insightId") REFERENCES "public"."project_insights"("id") ON DELETE CASCADE ON UPDATE CASCADE;

View File

@@ -1,9 +0,0 @@
/*
Warnings:
- Made the column `payload` on table `project_insights` required. This step will fail if there are existing NULL values in that column.
*/
-- AlterTable
ALTER TABLE "public"."project_insights" ALTER COLUMN "payload" SET NOT NULL,
ALTER COLUMN "payload" SET DEFAULT '{}';

View File

@@ -1,13 +0,0 @@
/*
Warnings:
- You are about to drop the column `changePct` on the `project_insights` table. All the data in the column will be lost.
- You are about to drop the column `compareValue` on the `project_insights` table. All the data in the column will be lost.
- You are about to drop the column `currentValue` on the `project_insights` table. All the data in the column will be lost.
*/
-- AlterTable
ALTER TABLE "public"."project_insights" DROP COLUMN "changePct",
DROP COLUMN "compareValue",
DROP COLUMN "currentValue",
ADD COLUMN "displayName" TEXT NOT NULL DEFAULT '';

View File

@@ -497,58 +497,3 @@ model Import {
@@map("imports")
}
enum InsightState {
active
suppressed
closed
}
model ProjectInsight {
id String @id @default(dbgenerated("gen_random_uuid()")) @db.Uuid
projectId String
moduleKey String // e.g. "referrers", "entry-pages"
dimensionKey String // e.g. "referrer:instagram", "page:/pricing"
windowKind String // "yesterday" | "rolling_7d" | "rolling_30d"
state InsightState @default(active)
title String
summary String?
displayName String @default("")
/// [IPrismaProjectInsightPayload]
payload Json @default("{}") // Rendered insight payload (typed)
direction String? // "up" | "down" | "flat"
impactScore Float @default(0)
severityBand String? // "low" | "moderate" | "severe"
version Int @default(1)
threadId String @default(dbgenerated("gen_random_uuid()")) @db.Uuid
windowStart DateTime?
windowEnd DateTime?
firstDetectedAt DateTime @default(now())
lastUpdatedAt DateTime @default(now()) @updatedAt
lastSeenAt DateTime @default(now())
events InsightEvent[]
@@unique([projectId, moduleKey, dimensionKey, windowKind, state])
@@index([projectId, impactScore(sort: Desc)])
@@index([projectId, moduleKey, windowKind, state])
@@map("project_insights")
}
model InsightEvent {
id String @id @default(dbgenerated("gen_random_uuid()")) @db.Uuid
insightId String @db.Uuid
insight ProjectInsight @relation(fields: [insightId], references: [id], onDelete: Cascade)
eventKind String // "created" | "updated" | "severity_up" | "direction_flip" | "closed" | etc
changeFrom Json?
changeTo Json?
createdAt DateTime @default(now())
@@index([insightId, createdAt])
@@map("insight_events")
}

View File

@@ -68,30 +68,6 @@ export class ProfileBuffer extends BaseBuffer {
const existingProfile = await this.fetchProfile(profile, logger);
// Delete any properties that are not server related if we have a non-server profile
if (
existingProfile?.properties.device !== 'server' &&
profile.properties.device === 'server'
) {
profile.properties = omit(
[
'city',
'country',
'region',
'longitude',
'latitude',
'os',
'osVersion',
'browser',
'device',
'isServer',
'os_version',
'browser_version',
],
profile.properties,
);
}
const mergedProfile: IClickhouseProfile = existingProfile
? deepMergeObjects(existingProfile, omit(['created_at'], profile))
: profile;

View File

@@ -1,5 +1,6 @@
import { type Redis, getRedisCache } from '@openpanel/redis';
import { toDots } from '@openpanel/common';
import { getSafeJson } from '@openpanel/json';
import { assocPath, clone } from 'ramda';
import { TABLE_NAMES, ch } from '../clickhouse/client';
@@ -90,6 +91,10 @@ export class SessionBuffer extends BaseBuffer {
session: newSession,
});
}
newSession.properties = toDots({
...(event.properties || {}),
...(newSession.properties || {}),
});
const addedRevenue = event.name === 'revenue' ? (event.revenue ?? 0) : 0;
newSession.revenue = (newSession.revenue ?? 0) + addedRevenue;
@@ -163,6 +168,7 @@ export class SessionBuffer extends BaseBuffer {
: '',
sign: 1,
version: 1,
properties: toDots(event.properties || {}),
},
];
}

View File

@@ -217,7 +217,6 @@ export function moveDataBetweenTables({
from,
to,
batch,
columns,
}: {
from: string;
to: string;
@@ -228,15 +227,11 @@ export function moveDataBetweenTables({
endDate?: Date;
startDate?: Date;
};
columns?: string[];
}): string[] {
const sqls: string[] = [];
// Build the SELECT clause
const selectClause = columns && columns.length > 0 ? columns.join(', ') : '*';
if (!batch) {
return [`INSERT INTO ${to} SELECT ${selectClause} FROM ${from}`];
return [`INSERT INTO ${to} SELECT * FROM ${from}`];
}
// Start from today and go back 3 years
@@ -252,109 +247,32 @@ export function moveDataBetweenTables({
let currentDate = endDate;
const interval = batch.interval || 'day';
// Helper function to get the start of the week (Monday) for a given date
const getWeekStart = (date: Date): Date => {
const d = new Date(date);
const day = d.getDay();
const diff = d.getDate() - day + (day === 0 ? -6 : 1); // Adjust to Monday
d.setDate(diff);
d.setHours(0, 0, 0, 0); // Normalize to start of day
return d;
};
// Helper function to compare dates based on interval
const shouldContinue = (
current: Date,
start: Date,
intervalType: string,
): boolean => {
if (intervalType === 'month') {
// For months, compare by year and month
// Continue if current month is >= start month
const currentYear = current.getFullYear();
const currentMonth = current.getMonth();
const startYear = start.getFullYear();
const startMonth = start.getMonth();
return (
currentYear > startYear ||
(currentYear === startYear && currentMonth >= startMonth)
);
}
if (intervalType === 'week') {
// For weeks, compare by week start dates
const currentWeekStart = getWeekStart(current);
const startWeekStart = getWeekStart(start);
return currentWeekStart >= startWeekStart;
}
return current > start;
};
while (shouldContinue(currentDate, startDate, interval)) {
while (currentDate > startDate) {
const previousDate = new Date(currentDate);
switch (interval) {
case 'month':
previousDate.setMonth(previousDate.getMonth() - 1);
// If we've gone below startDate's month, adjust to start of startDate's month
// This ensures we generate SQL for the month containing startDate
if (
previousDate.getFullYear() < startDate.getFullYear() ||
(previousDate.getFullYear() === startDate.getFullYear() &&
previousDate.getMonth() < startDate.getMonth())
) {
previousDate.setFullYear(startDate.getFullYear());
previousDate.setMonth(startDate.getMonth());
previousDate.setDate(1);
}
break;
case 'week': {
case 'week':
previousDate.setDate(previousDate.getDate() - 7);
// If we've gone below startDate's week, adjust to start of startDate's week
const startWeekStart = getWeekStart(startDate);
const prevWeekStart = getWeekStart(previousDate);
if (prevWeekStart < startWeekStart) {
previousDate.setTime(startWeekStart.getTime());
// Ensure we don't go below startDate
if (previousDate < startDate) {
previousDate.setTime(startDate.getTime());
}
break;
}
// day
default:
previousDate.setDate(previousDate.getDate() - 1);
break;
}
// For monthly/weekly intervals with transform, upperBoundDate should be currentDate
// because currentDate already represents the start of the period we're processing
// The WHERE clause uses > previousDate AND <= currentDate to get exactly one period
let upperBoundDate = currentDate;
// Don't exceed the endDate
if (upperBoundDate > endDate) {
upperBoundDate = endDate;
}
const sql = `INSERT INTO ${to}
SELECT ${selectClause} FROM ${from}
SELECT * FROM ${from}
WHERE ${batch.column} > '${batch.transform ? batch.transform(previousDate) : formatClickhouseDate(previousDate, true)}'
AND ${batch.column} <= '${batch.transform ? batch.transform(upperBoundDate) : formatClickhouseDate(upperBoundDate, true)}'`;
AND ${batch.column} <= '${batch.transform ? batch.transform(currentDate) : formatClickhouseDate(currentDate, true)}'`;
sqls.push(sql);
// For monthly/weekly intervals, stop if we've reached the start period
if (interval === 'month') {
const prevYear = previousDate.getFullYear();
const prevMonth = previousDate.getMonth();
const startYear = startDate.getFullYear();
const startMonth = startDate.getMonth();
if (prevYear === startYear && prevMonth === startMonth) {
break;
}
} else if (interval === 'week') {
const prevWeekStart = getWeekStart(previousDate);
const startWeekStart = getWeekStart(startDate);
if (prevWeekStart.getTime() === startWeekStart.getTime()) {
break;
}
}
currentDate = previousDate;
}

View File

@@ -43,7 +43,7 @@ class Expression {
}
export class Query<T = any> {
private _select: (string | Expression)[] = [];
private _select: string[] = [];
private _except: string[] = [];
private _from?: string | Expression;
private _where: WhereCondition[] = [];
@@ -81,19 +81,17 @@ export class Query<T = any> {
// Select methods
select<U>(
columns: (string | Expression | null | undefined | false)[],
columns: (string | null | undefined | false)[],
type: 'merge' | 'replace' = 'replace',
): Query<U> {
if (this._skipNext) return this as unknown as Query<U>;
if (type === 'merge') {
this._select = [
...this._select,
...columns.filter((col): col is string | Expression => Boolean(col)),
...columns.filter((col): col is string => Boolean(col)),
];
} else {
this._select = columns.filter((col): col is string | Expression =>
Boolean(col),
);
this._select = columns.filter((col): col is string => Boolean(col));
}
return this as unknown as Query<U>;
}
@@ -374,14 +372,7 @@ export class Query<T = any> {
if (this._select.length > 0) {
parts.push(
'SELECT',
this._select
// Important: Expressions are treated as raw SQL; do not run escapeDate()
// on them, otherwise any embedded date strings get double-escaped
// (e.g. ''2025-12-16 23:59:59'') which ClickHouse rejects.
.map((col) =>
col instanceof Expression ? col.toString() : this.escapeDate(col),
)
.join(', '),
this._select.map((col) => this.escapeDate(col)).join(', '),
);
} else {
parts.push('SELECT *');

View File

@@ -42,11 +42,11 @@ const getPrismaClient = () => {
operation === 'update' ||
operation === 'delete'
) {
// logger.info('Prisma operation', {
// operation,
// args,
// model,
// });
logger.info('Prisma operation', {
operation,
args,
model,
});
}
return query(args);
},

View File

@@ -267,22 +267,43 @@ export function getChartSql({
'COUNT(*)::float / COUNT(DISTINCT profile_id)::float as count';
}
const mathFunction = {
property_sum: 'sum',
property_average: 'avg',
property_max: 'max',
property_min: 'min',
}[event.segment as string];
if (mathFunction && event.property) {
const propertyKey = getSelectPropertyKey(event.property);
if (isNumericColumn(event.property)) {
sb.select.count = `${mathFunction}(${propertyKey}) as count`;
sb.where.property = `${propertyKey} IS NOT NULL`;
if (event.segment === 'property_sum' && event.property) {
if (event.property === 'revenue') {
sb.select.count = 'sum(revenue) as count';
sb.where.property = 'revenue > 0';
} else {
sb.select.count = `${mathFunction}(toFloat64OrNull(${propertyKey})) as count`;
sb.where.property = `${propertyKey} IS NOT NULL AND notEmpty(${propertyKey})`;
sb.select.count = `sum(toFloat64(${getSelectPropertyKey(event.property)})) as count`;
sb.where.property = `${getSelectPropertyKey(event.property)} IS NOT NULL AND notEmpty(${getSelectPropertyKey(event.property)})`;
}
}
if (event.segment === 'property_average' && event.property) {
if (event.property === 'revenue') {
sb.select.count = 'avg(revenue) as count';
sb.where.property = 'revenue > 0';
} else {
sb.select.count = `avg(toFloat64(${getSelectPropertyKey(event.property)})) as count`;
sb.where.property = `${getSelectPropertyKey(event.property)} IS NOT NULL AND notEmpty(${getSelectPropertyKey(event.property)})`;
}
}
if (event.segment === 'property_max' && event.property) {
if (event.property === 'revenue') {
sb.select.count = 'max(revenue) as count';
sb.where.property = 'revenue > 0';
} else {
sb.select.count = `max(toFloat64(${getSelectPropertyKey(event.property)})) as count`;
sb.where.property = `${getSelectPropertyKey(event.property)} IS NOT NULL AND notEmpty(${getSelectPropertyKey(event.property)})`;
}
}
if (event.segment === 'property_min' && event.property) {
if (event.property === 'revenue') {
sb.select.count = 'min(revenue) as count';
sb.where.property = 'revenue > 0';
} else {
sb.select.count = `min(toFloat64(${getSelectPropertyKey(event.property)})) as count`;
sb.where.property = `${getSelectPropertyKey(event.property)} IS NOT NULL AND notEmpty(${getSelectPropertyKey(event.property)})`;
}
}

View File

@@ -131,6 +131,7 @@ export function transformSessionToEvent(
duration: 0,
revenue: session.revenue,
properties: {
...session.properties,
is_bounce: session.is_bounce,
__query: {
utm_medium: session.utm_medium,
@@ -591,9 +592,6 @@ export async function getEventList(options: GetEventListOptions) {
if (select.sdkVersion) {
sb.select.sdkVersion = 'sdk_version';
}
if (select.revenue) {
sb.select.revenue = 'revenue';
}
if (profileId) {
sb.where.deviceId = `(device_id IN (SELECT device_id as did FROM ${TABLE_NAMES.events} WHERE project_id = ${sqlstring.escape(projectId)} AND device_id != '' AND profile_id = ${sqlstring.escape(profileId)} group by did) OR profile_id = ${sqlstring.escape(profileId)})`;
@@ -630,7 +628,8 @@ export async function getEventList(options: GetEventListOptions) {
}
}
sb.orderBy.created_at = 'created_at DESC';
sb.orderBy.created_at =
'toDate(created_at) DESC, created_at DESC, profile_id DESC, name DESC';
if (custom) {
custom(sb);

View File

@@ -1,68 +0,0 @@
import crypto from 'node:crypto';
import type { ClickHouseClient } from '@clickhouse/client';
import {
type Query,
clix as originalClix,
} from '../../clickhouse/query-builder';
/**
* Creates a cached wrapper around clix that automatically caches query results
* based on query hash. This eliminates duplicate queries within the same module/window context.
*
* @param client - ClickHouse client
* @param cache - Optional cache Map to store query results
* @param timezone - Timezone for queries (defaults to UTC)
* @returns A function that creates cached Query instances (compatible with clix API)
*/
export function createCachedClix(
client: ClickHouseClient,
cache?: Map<string, any>,
timezone?: string,
) {
function clixCached(): Query {
const query = originalClix(client, timezone);
const queryTimezone = timezone ?? 'UTC';
// Override execute() method to add caching
const originalExecute = query.execute.bind(query);
query.execute = async () => {
// Build the query SQL string
const querySQL = query.toSQL();
// Create cache key from query SQL + timezone
const cacheKey = crypto
.createHash('sha256')
.update(`${querySQL}|${queryTimezone}`)
.digest('hex');
// Check cache first
if (cache?.has(cacheKey)) {
return cache.get(cacheKey);
}
// Execute query
const result = await originalExecute();
// Cache the result
if (cache) {
cache.set(cacheKey, result);
}
return result;
};
return query;
}
// Copy static methods from original clix
clixCached.exp = originalClix.exp;
clixCached.date = originalClix.date;
clixCached.datetime = originalClix.datetime;
clixCached.dynamicDatetime = originalClix.dynamicDatetime;
clixCached.toStartOf = originalClix.toStartOf;
clixCached.toStartOfInterval = originalClix.toStartOfInterval;
clixCached.toInterval = originalClix.toInterval;
clixCached.toDate = originalClix.toDate;
return clixCached;
}

View File

@@ -1,303 +0,0 @@
import { createCachedClix } from './cached-clix';
import { materialDecision } from './material';
import { defaultImpactScore, severityBand } from './scoring';
import type {
Cadence,
ComputeContext,
ComputeResult,
InsightModule,
InsightStore,
WindowKind,
} from './types';
import { resolveWindow } from './windows';
const DEFAULT_WINDOWS: WindowKind[] = [
'yesterday',
'rolling_7d',
'rolling_30d',
];
export interface EngineConfig {
keepTopNPerModuleWindow: number; // e.g. 5
closeStaleAfterDays: number; // e.g. 7
dimensionBatchSize: number; // e.g. 50
globalThresholds: {
minTotal: number; // e.g. 200
minAbsDelta: number; // e.g. 80
minPct: number; // e.g. 0.15
};
}
/** Simple gating to cut noise; modules can override via thresholds. */
function passesThresholds(
r: ComputeResult,
mod: InsightModule,
cfg: EngineConfig,
): boolean {
const t = mod.thresholds ?? {};
const minTotal = t.minTotal ?? cfg.globalThresholds.minTotal;
const minAbsDelta = t.minAbsDelta ?? cfg.globalThresholds.minAbsDelta;
const minPct = t.minPct ?? cfg.globalThresholds.minPct;
const cur = r.currentValue ?? 0;
const cmp = r.compareValue ?? 0;
const total = cur + cmp;
const absDelta = Math.abs(cur - cmp);
const pct = Math.abs(r.changePct ?? 0);
if (total < minTotal) return false;
if (absDelta < minAbsDelta) return false;
if (pct < minPct) return false;
return true;
}
function chunk<T>(arr: T[], size: number): T[][] {
if (size <= 0) return [arr];
const out: T[][] = [];
for (let i = 0; i < arr.length; i += size) out.push(arr.slice(i, i + size));
return out;
}
export function createEngine(args: {
store: InsightStore;
modules: InsightModule[];
db: any;
logger?: Pick<Console, 'info' | 'warn' | 'error'>;
config: EngineConfig;
}) {
const { store, modules, db, config } = args;
const logger = args.logger ?? console;
function isProjectOldEnoughForWindow(
projectCreatedAt: Date | null | undefined,
baselineStart: Date,
): boolean {
if (!projectCreatedAt) return true; // best-effort; don't block if unknown
return projectCreatedAt.getTime() <= baselineStart.getTime();
}
async function runProject(opts: {
projectId: string;
cadence: Cadence;
now: Date;
projectCreatedAt?: Date | null;
}): Promise<void> {
const { projectId, cadence, now, projectCreatedAt } = opts;
const projLogger = logger;
const eligible = modules.filter((m) => m.cadence.includes(cadence));
for (const mod of eligible) {
const windows = mod.windows ?? DEFAULT_WINDOWS;
for (const windowKind of windows) {
let window: ReturnType<typeof resolveWindow>;
let ctx: ComputeContext;
try {
window = resolveWindow(windowKind, now);
if (
!isProjectOldEnoughForWindow(projectCreatedAt, window.baselineStart)
) {
continue;
}
// Initialize cache for this module+window combination.
// Cache is automatically garbage collected when context goes out of scope.
const cache = new Map<string, any>();
ctx = {
projectId,
window,
db,
now,
logger: projLogger,
clix: createCachedClix(db, cache),
};
} catch (e) {
projLogger.error('[insights] failed to create compute context', {
projectId,
module: mod.key,
windowKind,
err: e,
});
continue;
}
// 1) enumerate dimensions
let dims: string[] = [];
try {
dims = mod.enumerateDimensions
? await mod.enumerateDimensions(ctx)
: [];
} catch (e) {
// Important: enumeration failures should not abort the whole project run.
// Also avoid lifecycle close/suppression when we didn't actually evaluate dims.
projLogger.error('[insights] module enumerateDimensions failed', {
projectId,
module: mod.key,
windowKind,
err: e,
});
continue;
}
const maxDims = mod.thresholds?.maxDims ?? 25;
if (dims.length > maxDims) dims = dims.slice(0, maxDims);
if (dims.length === 0) {
// Still do lifecycle close / suppression based on "nothing emitted"
await store.closeMissingActiveInsights({
projectId,
moduleKey: mod.key,
windowKind,
seenDimensionKeys: [],
now,
staleDays: config.closeStaleAfterDays,
});
await store.applySuppression({
projectId,
moduleKey: mod.key,
windowKind,
keepTopN: config.keepTopNPerModuleWindow,
now,
});
continue;
}
// 2) compute in batches
const seen: string[] = [];
const dimBatches = chunk(dims, config.dimensionBatchSize);
for (const batch of dimBatches) {
let results: ComputeResult[] = [];
try {
results = await mod.computeMany(ctx, batch);
} catch (e) {
projLogger.error('[insights] module computeMany failed', {
projectId,
module: mod.key,
windowKind,
err: e,
});
continue;
}
for (const r of results) {
if (!r?.ok) continue;
if (!r.dimensionKey) continue;
// 3) gate noise
if (!passesThresholds(r, mod, config)) continue;
// 4) score
const impact = mod.score
? mod.score(r, ctx)
: defaultImpactScore(r);
const sev = severityBand(r.changePct);
// 5) dedupe/material change requires loading prev identity
const prev = await store.getActiveInsightByIdentity({
projectId,
moduleKey: mod.key,
dimensionKey: r.dimensionKey,
windowKind,
});
const decision = materialDecision(prev, {
changePct: r.changePct,
direction: r.direction,
});
// 6) render
const card = mod.render(r, ctx);
// 7) upsert
const persisted = await store.upsertInsight({
projectId,
moduleKey: mod.key,
dimensionKey: r.dimensionKey,
window,
card,
metrics: {
direction: r.direction,
impactScore: impact,
severityBand: sev,
},
now,
decision,
prev,
});
seen.push(r.dimensionKey);
// 8) events only when material
if (!prev) {
await store.insertEvent({
projectId,
insightId: persisted.id,
moduleKey: mod.key,
dimensionKey: r.dimensionKey,
windowKind,
eventKind: 'created',
changeFrom: null,
changeTo: {
title: card.title,
changePct: r.changePct,
direction: r.direction,
impact,
severityBand: sev,
},
now,
});
} else if (decision.material) {
const eventKind =
decision.reason === 'direction_flip'
? 'direction_flip'
: decision.reason === 'severity_change'
? sev && prev.severityBand && sev > prev.severityBand
? 'severity_up'
: 'severity_down'
: 'updated';
await store.insertEvent({
projectId,
insightId: persisted.id,
moduleKey: mod.key,
dimensionKey: r.dimensionKey,
windowKind,
eventKind,
changeFrom: {
direction: prev.direction,
impactScore: prev.impactScore,
severityBand: prev.severityBand,
},
changeTo: {
changePct: r.changePct,
direction: r.direction,
impactScore: impact,
severityBand: sev,
},
now,
});
}
}
}
// 10) lifecycle: close missing insights for this module/window
await store.closeMissingActiveInsights({
projectId,
moduleKey: mod.key,
windowKind,
seenDimensionKeys: seen,
now,
staleDays: config.closeStaleAfterDays,
});
// 11) suppression: keep top N
await store.applySuppression({
projectId,
moduleKey: mod.key,
windowKind,
keepTopN: config.keepTopNPerModuleWindow,
now,
});
}
}
}
return { runProject };
}

View File

@@ -1,8 +0,0 @@
export * from './types';
export * from './windows';
export * from './scoring';
export * from './material';
export * from './engine';
export * from './store';
export * from './utils';
export * from './modules';

View File

@@ -1,43 +0,0 @@
import { severityBand as band } from './scoring';
import type { MaterialDecision, PersistedInsight } from './types';
export function materialDecision(
prev: PersistedInsight | null,
next: {
changePct?: number;
direction?: 'up' | 'down' | 'flat';
},
): MaterialDecision {
const nextBand = band(next.changePct);
if (!prev) {
return { material: true, reason: 'created', newSeverityBand: nextBand };
}
// direction flip is always meaningful
const prevDir = (prev.direction ?? 'flat') as any;
const nextDir = next.direction ?? 'flat';
if (prevDir !== nextDir && (nextDir === 'up' || nextDir === 'down')) {
return {
material: true,
reason: 'direction_flip',
newSeverityBand: nextBand,
};
}
// severity band change
const prevBand = (prev.severityBand ?? null) as any;
if (prevBand !== nextBand && nextBand !== null) {
return {
material: true,
reason: 'severity_change',
newSeverityBand: nextBand,
};
}
// Otherwise: treat as non-material (silent refresh). You can add deadband crossing here if you store prior changePct.
return {
material: false,
reason: 'none',
newSeverityBand: prevBand ?? nextBand,
};
}

View File

@@ -1,275 +0,0 @@
import { TABLE_NAMES, formatClickhouseDate } from '../../../clickhouse/client';
import type {
ComputeContext,
ComputeResult,
InsightModule,
RenderedCard,
} from '../types';
import {
buildLookupMap,
computeChangePct,
computeDirection,
computeMedian,
getEndOfDay,
getWeekday,
selectTopDimensions,
} from '../utils';
async function fetchDeviceAggregates(ctx: ComputeContext): Promise<{
currentMap: Map<string, number>;
baselineMap: Map<string, number>;
totalCurrent: number;
totalBaseline: number;
}> {
if (ctx.window.kind === 'yesterday') {
const [currentResults, baselineResults, totals] = await Promise.all([
ctx
.clix()
.select<{ device: string; cnt: number }>(['device', 'count(*) as cnt'])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.start,
getEndOfDay(ctx.window.end),
])
.groupBy(['device'])
.execute(),
ctx
.clix()
.select<{ date: string; device: string; cnt: number }>([
'toDate(created_at) as date',
'device',
'count(*) as cnt',
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.baselineEnd),
])
.groupBy(['date', 'device'])
.execute(),
ctx
.clix()
.select<{ cur_total: number }>([
ctx.clix.exp(
`countIf(created_at BETWEEN '${formatClickhouseDate(ctx.window.start)}' AND '${formatClickhouseDate(getEndOfDay(ctx.window.end))}') as cur_total`,
),
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.end),
])
.execute(),
]);
const currentMap = buildLookupMap(currentResults, (r) => r.device);
const targetWeekday = getWeekday(ctx.window.start);
const aggregated = new Map<string, { date: string; cnt: number }[]>();
for (const r of baselineResults) {
if (!aggregated.has(r.device)) {
aggregated.set(r.device, []);
}
const entries = aggregated.get(r.device)!;
const existing = entries.find((e) => e.date === r.date);
if (existing) {
existing.cnt += Number(r.cnt ?? 0);
} else {
entries.push({ date: r.date, cnt: Number(r.cnt ?? 0) });
}
}
const baselineMap = new Map<string, number>();
for (const [deviceType, entries] of aggregated) {
const sameWeekdayValues = entries
.filter((e) => getWeekday(new Date(e.date)) === targetWeekday)
.map((e) => e.cnt)
.sort((a, b) => a - b);
if (sameWeekdayValues.length > 0) {
baselineMap.set(deviceType, computeMedian(sameWeekdayValues));
}
}
const totalCurrent = totals[0]?.cur_total ?? 0;
const totalBaseline =
baselineMap.size > 0
? Array.from(baselineMap.values()).reduce((sum, val) => sum + val, 0)
: 0;
return { currentMap, baselineMap, totalCurrent, totalBaseline };
}
const curStart = formatClickhouseDate(ctx.window.start);
const curEnd = formatClickhouseDate(getEndOfDay(ctx.window.end));
const baseStart = formatClickhouseDate(ctx.window.baselineStart);
const baseEnd = formatClickhouseDate(getEndOfDay(ctx.window.baselineEnd));
const [results, totals] = await Promise.all([
ctx
.clix()
.select<{ device: string; cur: number; base: number }>([
'device',
ctx.clix.exp(
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur`,
),
ctx.clix.exp(
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base`,
),
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.end),
])
.groupBy(['device'])
.execute(),
ctx
.clix()
.select<{ cur_total: number; base_total: number }>([
ctx.clix.exp(
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur_total`,
),
ctx.clix.exp(
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base_total`,
),
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.end),
])
.execute(),
]);
const currentMap = buildLookupMap(
results,
(r) => r.device,
(r) => Number(r.cur ?? 0),
);
const baselineMap = buildLookupMap(
results,
(r) => r.device,
(r) => Number(r.base ?? 0),
);
const totalCurrent = totals[0]?.cur_total ?? 0;
const totalBaseline = totals[0]?.base_total ?? 0;
return { currentMap, baselineMap, totalCurrent, totalBaseline };
}
export const devicesModule: InsightModule = {
key: 'devices',
cadence: ['daily'],
thresholds: { minTotal: 100, minAbsDelta: 0, minPct: 0.08, maxDims: 5 },
async enumerateDimensions(ctx) {
const { currentMap, baselineMap } = await fetchDeviceAggregates(ctx);
const topDims = selectTopDimensions(
currentMap,
baselineMap,
this.thresholds?.maxDims ?? 5,
);
return topDims.map((dim) => `device:${dim}`);
},
async computeMany(ctx, dimensionKeys): Promise<ComputeResult[]> {
const { currentMap, baselineMap, totalCurrent, totalBaseline } =
await fetchDeviceAggregates(ctx);
const results: ComputeResult[] = [];
for (const dimKey of dimensionKeys) {
if (!dimKey.startsWith('device:')) continue;
const deviceType = dimKey.replace('device:', '');
const currentValue = currentMap.get(deviceType) ?? 0;
const compareValue = baselineMap.get(deviceType) ?? 0;
const currentShare = totalCurrent > 0 ? currentValue / totalCurrent : 0;
const compareShare = totalBaseline > 0 ? compareValue / totalBaseline : 0;
const shareShiftPp = (currentShare - compareShare) * 100;
const changePct = computeChangePct(currentValue, compareValue);
const direction = computeDirection(changePct);
results.push({
ok: true,
dimensionKey: dimKey,
currentValue,
compareValue,
changePct,
direction,
extra: {
shareShiftPp,
currentShare,
compareShare,
},
});
}
return results;
},
render(result, ctx): RenderedCard {
const device = result.dimensionKey.replace('device:', '');
const changePct = result.changePct ?? 0;
const isIncrease = changePct >= 0;
const sessionsCurrent = result.currentValue ?? 0;
const sessionsCompare = result.compareValue ?? 0;
const shareCurrent = Number(result.extra?.currentShare ?? 0);
const shareCompare = Number(result.extra?.compareShare ?? 0);
return {
title: `${device} ${isIncrease ? '↑' : '↓'} ${Math.abs(changePct * 100).toFixed(0)}%`,
summary: `${ctx.window.label}. Device traffic change.`,
displayName: device,
payload: {
kind: 'insight_v1',
dimensions: [{ key: 'device', value: device, displayName: device }],
primaryMetric: 'sessions',
metrics: {
sessions: {
current: sessionsCurrent,
compare: sessionsCompare,
delta: sessionsCurrent - sessionsCompare,
changePct: sessionsCompare > 0 ? (result.changePct ?? 0) : null,
direction: result.direction ?? 'flat',
unit: 'count',
},
share: {
current: shareCurrent,
compare: shareCompare,
delta: shareCurrent - shareCompare,
changePct:
shareCompare > 0
? (shareCurrent - shareCompare) / shareCompare
: null,
direction:
shareCurrent - shareCompare > 0.0005
? 'up'
: shareCurrent - shareCompare < -0.0005
? 'down'
: 'flat',
unit: 'ratio',
},
},
extra: {
// keep module-specific flags/fields if needed later
},
},
};
},
};

View File

@@ -1,287 +0,0 @@
import { TABLE_NAMES, formatClickhouseDate } from '../../../clickhouse/client';
import type {
ComputeContext,
ComputeResult,
InsightModule,
RenderedCard,
} from '../types';
import {
buildLookupMap,
computeChangePct,
computeDirection,
computeWeekdayMedians,
getEndOfDay,
getWeekday,
selectTopDimensions,
} from '../utils';
const DELIMITER = '|||';
async function fetchEntryPageAggregates(ctx: ComputeContext): Promise<{
currentMap: Map<string, number>;
baselineMap: Map<string, number>;
totalCurrent: number;
totalBaseline: number;
}> {
if (ctx.window.kind === 'yesterday') {
const [currentResults, baselineResults, totals] = await Promise.all([
ctx
.clix()
.select<{ entry_origin: string; entry_path: string; cnt: number }>([
'entry_origin',
'entry_path',
'count(*) as cnt',
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.start,
getEndOfDay(ctx.window.end),
])
.groupBy(['entry_origin', 'entry_path'])
.execute(),
ctx
.clix()
.select<{
date: string;
entry_origin: string;
entry_path: string;
cnt: number;
}>([
'toDate(created_at) as date',
'entry_origin',
'entry_path',
'count(*) as cnt',
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.baselineEnd),
])
.groupBy(['date', 'entry_origin', 'entry_path'])
.execute(),
ctx
.clix()
.select<{ cur_total: number }>([
ctx.clix.exp(
`countIf(created_at BETWEEN '${formatClickhouseDate(ctx.window.start)}' AND '${formatClickhouseDate(getEndOfDay(ctx.window.end))}') as cur_total`,
),
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.end),
])
.execute(),
]);
const currentMap = buildLookupMap(
currentResults,
(r) => `${r.entry_origin || ''}${DELIMITER}${r.entry_path || '/'}`,
);
const targetWeekday = getWeekday(ctx.window.start);
const baselineMap = computeWeekdayMedians(
baselineResults,
targetWeekday,
(r) => `${r.entry_origin || ''}${DELIMITER}${r.entry_path || '/'}`,
);
const totalCurrent = totals[0]?.cur_total ?? 0;
const totalBaseline = Array.from(baselineMap.values()).reduce(
(sum, val) => sum + val,
0,
);
return { currentMap, baselineMap, totalCurrent, totalBaseline };
}
const curStart = formatClickhouseDate(ctx.window.start);
const curEnd = formatClickhouseDate(getEndOfDay(ctx.window.end));
const baseStart = formatClickhouseDate(ctx.window.baselineStart);
const baseEnd = formatClickhouseDate(getEndOfDay(ctx.window.baselineEnd));
const [results, totals] = await Promise.all([
ctx
.clix()
.select<{
entry_origin: string;
entry_path: string;
cur: number;
base: number;
}>([
'entry_origin',
'entry_path',
ctx.clix.exp(
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur`,
),
ctx.clix.exp(
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base`,
),
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.end),
])
.groupBy(['entry_origin', 'entry_path'])
.execute(),
ctx
.clix()
.select<{ cur_total: number; base_total: number }>([
ctx.clix.exp(
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur_total`,
),
ctx.clix.exp(
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base_total`,
),
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.end),
])
.execute(),
]);
const currentMap = buildLookupMap(
results,
(r) => `${r.entry_origin || ''}${DELIMITER}${r.entry_path || '/'}`,
(r) => Number(r.cur ?? 0),
);
const baselineMap = buildLookupMap(
results,
(r) => `${r.entry_origin || ''}${DELIMITER}${r.entry_path || '/'}`,
(r) => Number(r.base ?? 0),
);
const totalCurrent = totals[0]?.cur_total ?? 0;
const totalBaseline = totals[0]?.base_total ?? 0;
return { currentMap, baselineMap, totalCurrent, totalBaseline };
}
export const entryPagesModule: InsightModule = {
key: 'entry-pages',
cadence: ['daily'],
thresholds: { minTotal: 100, minAbsDelta: 30, minPct: 0.2, maxDims: 100 },
async enumerateDimensions(ctx) {
const { currentMap, baselineMap } = await fetchEntryPageAggregates(ctx);
const topDims = selectTopDimensions(
currentMap,
baselineMap,
this.thresholds?.maxDims ?? 100,
);
return topDims.map((dim) => `entry:${dim}`);
},
async computeMany(ctx, dimensionKeys): Promise<ComputeResult[]> {
const { currentMap, baselineMap, totalCurrent, totalBaseline } =
await fetchEntryPageAggregates(ctx);
const results: ComputeResult[] = [];
for (const dimKey of dimensionKeys) {
if (!dimKey.startsWith('entry:')) continue;
const originPath = dimKey.replace('entry:', '');
const currentValue = currentMap.get(originPath) ?? 0;
const compareValue = baselineMap.get(originPath) ?? 0;
const currentShare = totalCurrent > 0 ? currentValue / totalCurrent : 0;
const compareShare = totalBaseline > 0 ? compareValue / totalBaseline : 0;
const shareShiftPp = (currentShare - compareShare) * 100;
const changePct = computeChangePct(currentValue, compareValue);
const direction = computeDirection(changePct);
results.push({
ok: true,
dimensionKey: dimKey,
currentValue,
compareValue,
changePct,
direction,
extra: {
shareShiftPp,
currentShare,
compareShare,
isNew: compareValue === 0 && currentValue > 0,
},
});
}
return results;
},
render(result, ctx): RenderedCard {
const originPath = result.dimensionKey.replace('entry:', '');
const [origin, path] = originPath.split(DELIMITER);
const displayValue = origin ? `${origin}${path}` : path || '/';
const pct = ((result.changePct ?? 0) * 100).toFixed(1);
const isIncrease = (result.changePct ?? 0) >= 0;
const isNew = result.extra?.isNew as boolean | undefined;
const title = isNew
? `New entry page: ${displayValue}`
: `Entry page ${displayValue} ${isIncrease ? '↑' : '↓'} ${Math.abs(Number(pct))}%`;
const sessionsCurrent = result.currentValue ?? 0;
const sessionsCompare = result.compareValue ?? 0;
const shareCurrent = Number(result.extra?.currentShare ?? 0);
const shareCompare = Number(result.extra?.compareShare ?? 0);
return {
title,
summary: `${ctx.window.label}. Sessions ${sessionsCurrent} vs ${sessionsCompare}.`,
displayName: displayValue,
payload: {
kind: 'insight_v1',
dimensions: [
{ key: 'origin', value: origin ?? '', displayName: origin ?? '' },
{ key: 'path', value: path ?? '', displayName: path ?? '' },
],
primaryMetric: 'sessions',
metrics: {
sessions: {
current: sessionsCurrent,
compare: sessionsCompare,
delta: sessionsCurrent - sessionsCompare,
changePct: sessionsCompare > 0 ? (result.changePct ?? 0) : null,
direction: result.direction ?? 'flat',
unit: 'count',
},
share: {
current: shareCurrent,
compare: shareCompare,
delta: shareCurrent - shareCompare,
changePct:
shareCompare > 0
? (shareCurrent - shareCompare) / shareCompare
: null,
direction:
shareCurrent - shareCompare > 0.0005
? 'up'
: shareCurrent - shareCompare < -0.0005
? 'down'
: 'flat',
unit: 'ratio',
},
},
extra: {
isNew: result.extra?.isNew,
},
},
};
},
};

View File

@@ -1,271 +0,0 @@
import { getCountry } from '@openpanel/constants';
import { TABLE_NAMES, formatClickhouseDate } from '../../../clickhouse/client';
import type {
ComputeContext,
ComputeResult,
InsightModule,
RenderedCard,
} from '../types';
import {
buildLookupMap,
computeChangePct,
computeDirection,
computeWeekdayMedians,
getEndOfDay,
getWeekday,
selectTopDimensions,
} from '../utils';
async function fetchGeoAggregates(ctx: ComputeContext): Promise<{
currentMap: Map<string, number>;
baselineMap: Map<string, number>;
totalCurrent: number;
totalBaseline: number;
}> {
if (ctx.window.kind === 'yesterday') {
const [currentResults, baselineResults, totals] = await Promise.all([
ctx
.clix()
.select<{ country: string; cnt: number }>([
'country',
'count(*) as cnt',
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.start,
getEndOfDay(ctx.window.end),
])
.groupBy(['country'])
.execute(),
ctx
.clix()
.select<{ date: string; country: string; cnt: number }>([
'toDate(created_at) as date',
'country',
'count(*) as cnt',
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.baselineEnd),
])
.groupBy(['date', 'country'])
.execute(),
ctx
.clix()
.select<{ cur_total: number }>([
ctx.clix.exp(
`countIf(created_at BETWEEN '${formatClickhouseDate(ctx.window.start)}' AND '${formatClickhouseDate(getEndOfDay(ctx.window.end))}') as cur_total`,
),
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.end),
])
.execute(),
]);
const currentMap = buildLookupMap(
currentResults,
(r) => r.country || 'unknown',
);
const targetWeekday = getWeekday(ctx.window.start);
const baselineMap = computeWeekdayMedians(
baselineResults,
targetWeekday,
(r) => r.country || 'unknown',
);
const totalCurrent = totals[0]?.cur_total ?? 0;
const totalBaseline = Array.from(baselineMap.values()).reduce(
(sum, val) => sum + val,
0,
);
return { currentMap, baselineMap, totalCurrent, totalBaseline };
}
const curStart = formatClickhouseDate(ctx.window.start);
const curEnd = formatClickhouseDate(getEndOfDay(ctx.window.end));
const baseStart = formatClickhouseDate(ctx.window.baselineStart);
const baseEnd = formatClickhouseDate(getEndOfDay(ctx.window.baselineEnd));
const [results, totals] = await Promise.all([
ctx
.clix()
.select<{ country: string; cur: number; base: number }>([
'country',
ctx.clix.exp(
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur`,
),
ctx.clix.exp(
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base`,
),
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.end),
])
.groupBy(['country'])
.execute(),
ctx
.clix()
.select<{ cur_total: number; base_total: number }>([
ctx.clix.exp(
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur_total`,
),
ctx.clix.exp(
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base_total`,
),
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.end),
])
.execute(),
]);
const currentMap = buildLookupMap(
results,
(r) => r.country || 'unknown',
(r) => Number(r.cur ?? 0),
);
const baselineMap = buildLookupMap(
results,
(r) => r.country || 'unknown',
(r) => Number(r.base ?? 0),
);
const totalCurrent = totals[0]?.cur_total ?? 0;
const totalBaseline = totals[0]?.base_total ?? 0;
return { currentMap, baselineMap, totalCurrent, totalBaseline };
}
export const geoModule: InsightModule = {
key: 'geo',
cadence: ['daily'],
thresholds: { minTotal: 100, minAbsDelta: 0, minPct: 0.08, maxDims: 30 },
async enumerateDimensions(ctx) {
const { currentMap, baselineMap } = await fetchGeoAggregates(ctx);
const topDims = selectTopDimensions(
currentMap,
baselineMap,
this.thresholds?.maxDims ?? 30,
);
return topDims.map((dim) => `country:${dim}`);
},
async computeMany(ctx, dimensionKeys): Promise<ComputeResult[]> {
const { currentMap, baselineMap, totalCurrent, totalBaseline } =
await fetchGeoAggregates(ctx);
const results: ComputeResult[] = [];
for (const dimKey of dimensionKeys) {
if (!dimKey.startsWith('country:')) continue;
const country = dimKey.replace('country:', '');
const currentValue = currentMap.get(country) ?? 0;
const compareValue = baselineMap.get(country) ?? 0;
const currentShare = totalCurrent > 0 ? currentValue / totalCurrent : 0;
const compareShare = totalBaseline > 0 ? compareValue / totalBaseline : 0;
const shareShiftPp = (currentShare - compareShare) * 100;
const changePct = computeChangePct(currentValue, compareValue);
const direction = computeDirection(changePct);
results.push({
ok: true,
dimensionKey: dimKey,
currentValue,
compareValue,
changePct,
direction,
extra: {
shareShiftPp,
currentShare,
compareShare,
isNew: compareValue === 0 && currentValue > 0,
},
});
}
return results;
},
render(result, ctx): RenderedCard {
const country = result.dimensionKey.replace('country:', '');
const changePct = result.changePct ?? 0;
const isIncrease = changePct >= 0;
const isNew = result.extra?.isNew as boolean | undefined;
const displayName = getCountry(country);
const title = isNew
? `New traffic from: ${displayName}`
: `${displayName} ${isIncrease ? '↑' : '↓'} ${Math.abs(changePct * 100).toFixed(0)}%`;
const sessionsCurrent = result.currentValue ?? 0;
const sessionsCompare = result.compareValue ?? 0;
const shareCurrent = Number(result.extra?.currentShare ?? 0);
const shareCompare = Number(result.extra?.compareShare ?? 0);
return {
title,
summary: `${ctx.window.label}. Traffic change from ${displayName}.`,
displayName,
payload: {
kind: 'insight_v1',
dimensions: [
{ key: 'country', value: country, displayName: displayName },
],
primaryMetric: 'sessions',
metrics: {
sessions: {
current: sessionsCurrent,
compare: sessionsCompare,
delta: sessionsCurrent - sessionsCompare,
changePct: sessionsCompare > 0 ? (result.changePct ?? 0) : null,
direction: result.direction ?? 'flat',
unit: 'count',
},
share: {
current: shareCurrent,
compare: shareCompare,
delta: shareCurrent - shareCompare,
changePct:
shareCompare > 0
? (shareCurrent - shareCompare) / shareCompare
: null,
direction:
shareCurrent - shareCompare > 0.0005
? 'up'
: shareCurrent - shareCompare < -0.0005
? 'down'
: 'flat',
unit: 'ratio',
},
},
extra: {
isNew: result.extra?.isNew,
},
},
};
},
};

View File

@@ -1,5 +0,0 @@
export { referrersModule } from './referrers.module';
export { entryPagesModule } from './entry-pages.module';
export { pageTrendsModule } from './page-trends.module';
export { geoModule } from './geo.module';
export { devicesModule } from './devices.module';

View File

@@ -1,298 +0,0 @@
import { TABLE_NAMES, formatClickhouseDate } from '../../../clickhouse/client';
import type {
ComputeContext,
ComputeResult,
InsightModule,
RenderedCard,
} from '../types';
import {
buildLookupMap,
computeChangePct,
computeDirection,
computeWeekdayMedians,
getEndOfDay,
getWeekday,
selectTopDimensions,
} from '../utils';
const DELIMITER = '|||';
async function fetchPageTrendAggregates(ctx: ComputeContext): Promise<{
currentMap: Map<string, number>;
baselineMap: Map<string, number>;
totalCurrent: number;
totalBaseline: number;
}> {
if (ctx.window.kind === 'yesterday') {
const [currentResults, baselineResults, totals] = await Promise.all([
ctx
.clix()
.select<{ origin: string; path: string; cnt: number }>([
'origin',
'path',
'count(*) as cnt',
])
.from(TABLE_NAMES.events)
.where('project_id', '=', ctx.projectId)
.where('name', '=', 'screen_view')
.where('created_at', 'BETWEEN', [
ctx.window.start,
getEndOfDay(ctx.window.end),
])
.groupBy(['origin', 'path'])
.execute(),
ctx
.clix()
.select<{ date: string; origin: string; path: string; cnt: number }>([
'toDate(created_at) as date',
'origin',
'path',
'count(*) as cnt',
])
.from(TABLE_NAMES.events)
.where('project_id', '=', ctx.projectId)
.where('name', '=', 'screen_view')
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.baselineEnd),
])
.groupBy(['date', 'origin', 'path'])
.execute(),
ctx
.clix()
.select<{ cur_total: number }>([
ctx.clix.exp(
`countIf(created_at BETWEEN '${formatClickhouseDate(ctx.window.start)}' AND '${formatClickhouseDate(getEndOfDay(ctx.window.end))}') as cur_total`,
),
])
.from(TABLE_NAMES.events)
.where('project_id', '=', ctx.projectId)
.where('name', '=', 'screen_view')
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.end),
])
.execute(),
]);
const currentMap = buildLookupMap(
currentResults,
(r) => `${r.origin || ''}${DELIMITER}${r.path || '/'}`,
);
const targetWeekday = getWeekday(ctx.window.start);
const baselineMap = computeWeekdayMedians(
baselineResults,
targetWeekday,
(r) => `${r.origin || ''}${DELIMITER}${r.path || '/'}`,
);
const totalCurrent = totals[0]?.cur_total ?? 0;
const totalBaseline = Array.from(baselineMap.values()).reduce(
(sum, val) => sum + val,
0,
);
return { currentMap, baselineMap, totalCurrent, totalBaseline };
}
const curStart = formatClickhouseDate(ctx.window.start);
const curEnd = formatClickhouseDate(getEndOfDay(ctx.window.end));
const baseStart = formatClickhouseDate(ctx.window.baselineStart);
const baseEnd = formatClickhouseDate(getEndOfDay(ctx.window.baselineEnd));
const [results, totals] = await Promise.all([
ctx
.clix()
.select<{ origin: string; path: string; cur: number; base: number }>([
'origin',
'path',
ctx.clix.exp(
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur`,
),
ctx.clix.exp(
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base`,
),
])
.from(TABLE_NAMES.events)
.where('project_id', '=', ctx.projectId)
.where('name', '=', 'screen_view')
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.end),
])
.groupBy(['origin', 'path'])
.execute(),
ctx
.clix()
.select<{ cur_total: number; base_total: number }>([
ctx.clix.exp(
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur_total`,
),
ctx.clix.exp(
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base_total`,
),
])
.from(TABLE_NAMES.events)
.where('project_id', '=', ctx.projectId)
.where('name', '=', 'screen_view')
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.end),
])
.execute(),
]);
const currentMap = buildLookupMap(
results,
(r) => `${r.origin || ''}${DELIMITER}${r.path || '/'}`,
(r) => Number(r.cur ?? 0),
);
const baselineMap = buildLookupMap(
results,
(r) => `${r.origin || ''}${DELIMITER}${r.path || '/'}`,
(r) => Number(r.base ?? 0),
);
const totalCurrent = totals[0]?.cur_total ?? 0;
const totalBaseline = totals[0]?.base_total ?? 0;
return { currentMap, baselineMap, totalCurrent, totalBaseline };
}
export const pageTrendsModule: InsightModule = {
key: 'page-trends',
cadence: ['daily'],
// Share-based thresholds (values in basis points: 100 = 1%)
// minTotal: require at least 0.5% combined share (current + baseline)
// minAbsDelta: require at least 0.5 percentage point shift
// minPct: require at least 25% relative change in share
thresholds: { minTotal: 50, minAbsDelta: 50, minPct: 0.25, maxDims: 100 },
async enumerateDimensions(ctx) {
const { currentMap, baselineMap } = await fetchPageTrendAggregates(ctx);
const topDims = selectTopDimensions(
currentMap,
baselineMap,
this.thresholds?.maxDims ?? 100,
);
return topDims.map((dim) => `page:${dim}`);
},
async computeMany(ctx, dimensionKeys): Promise<ComputeResult[]> {
const { currentMap, baselineMap, totalCurrent, totalBaseline } =
await fetchPageTrendAggregates(ctx);
const results: ComputeResult[] = [];
for (const dimKey of dimensionKeys) {
if (!dimKey.startsWith('page:')) continue;
const originPath = dimKey.replace('page:', '');
const pageviewsCurrent = currentMap.get(originPath) ?? 0;
const pageviewsCompare = baselineMap.get(originPath) ?? 0;
const currentShare =
totalCurrent > 0 ? pageviewsCurrent / totalCurrent : 0;
const compareShare =
totalBaseline > 0 ? pageviewsCompare / totalBaseline : 0;
// Use share values in basis points (100 = 1%) for thresholding
// This makes thresholds intuitive: minAbsDelta=50 means 0.5pp shift
const currentShareBp = currentShare * 10000;
const compareShareBp = compareShare * 10000;
const shareShiftPp = (currentShare - compareShare) * 100;
// changePct is relative change in share, not absolute pageviews
const shareChangePct = computeChangePct(currentShare, compareShare);
const direction = computeDirection(shareChangePct);
results.push({
ok: true,
dimensionKey: dimKey,
// Use share in basis points for threshold checks
currentValue: currentShareBp,
compareValue: compareShareBp,
changePct: shareChangePct,
direction,
extra: {
// Keep absolute values for display
pageviewsCurrent,
pageviewsCompare,
shareShiftPp,
currentShare,
compareShare,
isNew: pageviewsCompare === 0 && pageviewsCurrent > 0,
},
});
}
return results;
},
render(result, ctx): RenderedCard {
const originPath = result.dimensionKey.replace('page:', '');
const [origin, path] = originPath.split(DELIMITER);
const displayValue = origin ? `${origin}${path}` : path || '/';
// Get absolute pageviews from extra (currentValue/compareValue are now share-based)
const pageviewsCurrent = Number(result.extra?.pageviewsCurrent ?? 0);
const pageviewsCompare = Number(result.extra?.pageviewsCompare ?? 0);
const shareCurrent = Number(result.extra?.currentShare ?? 0);
const shareCompare = Number(result.extra?.compareShare ?? 0);
const shareShiftPp = Number(result.extra?.shareShiftPp ?? 0);
const isNew = result.extra?.isNew as boolean | undefined;
// Display share shift in percentage points
const isIncrease = shareShiftPp >= 0;
const shareShiftDisplay = Math.abs(shareShiftPp).toFixed(1);
const title = isNew
? `New page getting views: ${displayValue}`
: `Page ${displayValue} share ${isIncrease ? '↑' : '↓'} ${shareShiftDisplay}pp`;
return {
title,
summary: `${ctx.window.label}. Share ${(shareCurrent * 100).toFixed(1)}% vs ${(shareCompare * 100).toFixed(1)}%.`,
displayName: displayValue,
payload: {
kind: 'insight_v1',
dimensions: [
{ key: 'origin', value: origin ?? '', displayName: origin ?? '' },
{ key: 'path', value: path ?? '', displayName: path ?? '' },
],
primaryMetric: 'share',
metrics: {
pageviews: {
current: pageviewsCurrent,
compare: pageviewsCompare,
delta: pageviewsCurrent - pageviewsCompare,
changePct:
pageviewsCompare > 0
? (pageviewsCurrent - pageviewsCompare) / pageviewsCompare
: null,
direction:
pageviewsCurrent > pageviewsCompare
? 'up'
: pageviewsCurrent < pageviewsCompare
? 'down'
: 'flat',
unit: 'count',
},
share: {
current: shareCurrent,
compare: shareCompare,
delta: shareCurrent - shareCompare,
changePct: result.changePct ?? null, // This is now share-based
direction: result.direction ?? 'flat',
unit: 'ratio',
},
},
extra: {
isNew: result.extra?.isNew,
shareShiftPp,
},
},
};
},
};

View File

@@ -1,275 +0,0 @@
import { TABLE_NAMES, formatClickhouseDate } from '../../../clickhouse/client';
import type {
ComputeContext,
ComputeResult,
InsightModule,
RenderedCard,
} from '../types';
import {
buildLookupMap,
computeChangePct,
computeDirection,
computeWeekdayMedians,
getEndOfDay,
getWeekday,
selectTopDimensions,
} from '../utils';
async function fetchReferrerAggregates(ctx: ComputeContext): Promise<{
currentMap: Map<string, number>;
baselineMap: Map<string, number>;
totalCurrent: number;
totalBaseline: number;
}> {
if (ctx.window.kind === 'yesterday') {
const [currentResults, baselineResults, totals] = await Promise.all([
ctx
.clix()
.select<{ referrer_name: string; cnt: number }>([
'referrer_name',
'count(*) as cnt',
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.start,
getEndOfDay(ctx.window.end),
])
.groupBy(['referrer_name'])
.execute(),
ctx
.clix()
.select<{ date: string; referrer_name: string; cnt: number }>([
'toDate(created_at) as date',
'referrer_name',
'count(*) as cnt',
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.baselineEnd),
])
.groupBy(['date', 'referrer_name'])
.execute(),
ctx
.clix()
.select<{ cur_total: number }>([
ctx.clix.exp(
`countIf(created_at BETWEEN '${formatClickhouseDate(ctx.window.start)}' AND '${formatClickhouseDate(getEndOfDay(ctx.window.end))}') as cur_total`,
),
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.end),
])
.execute(),
]);
const currentMap = buildLookupMap(
currentResults,
(r) => r.referrer_name || 'direct',
);
const targetWeekday = getWeekday(ctx.window.start);
const baselineMap = computeWeekdayMedians(
baselineResults,
targetWeekday,
(r) => r.referrer_name || 'direct',
);
const totalCurrent = totals[0]?.cur_total ?? 0;
const totalBaseline = Array.from(baselineMap.values()).reduce(
(sum, val) => sum + val,
0,
);
return { currentMap, baselineMap, totalCurrent, totalBaseline };
}
const curStart = formatClickhouseDate(ctx.window.start);
const curEnd = formatClickhouseDate(getEndOfDay(ctx.window.end));
const baseStart = formatClickhouseDate(ctx.window.baselineStart);
const baseEnd = formatClickhouseDate(getEndOfDay(ctx.window.baselineEnd));
const [results, totals] = await Promise.all([
ctx
.clix()
.select<{ referrer_name: string; cur: number; base: number }>([
'referrer_name',
ctx.clix.exp(
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur`,
),
ctx.clix.exp(
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base`,
),
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.end),
])
.groupBy(['referrer_name'])
.execute(),
ctx
.clix()
.select<{ cur_total: number; base_total: number }>([
ctx.clix.exp(
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur_total`,
),
ctx.clix.exp(
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base_total`,
),
])
.from(TABLE_NAMES.sessions)
.where('project_id', '=', ctx.projectId)
.where('sign', '=', 1)
.where('created_at', 'BETWEEN', [
ctx.window.baselineStart,
getEndOfDay(ctx.window.end),
])
.execute(),
]);
const currentMap = buildLookupMap(
results,
(r) => r.referrer_name || 'direct',
(r) => Number(r.cur ?? 0),
);
const baselineMap = buildLookupMap(
results,
(r) => r.referrer_name || 'direct',
(r) => Number(r.base ?? 0),
);
const totalCurrent = totals[0]?.cur_total ?? 0;
const totalBaseline = totals[0]?.base_total ?? 0;
return { currentMap, baselineMap, totalCurrent, totalBaseline };
}
export const referrersModule: InsightModule = {
key: 'referrers',
cadence: ['daily'],
thresholds: { minTotal: 100, minAbsDelta: 20, minPct: 0.15, maxDims: 50 },
async enumerateDimensions(ctx) {
const { currentMap, baselineMap } = await fetchReferrerAggregates(ctx);
const topDims = selectTopDimensions(
currentMap,
baselineMap,
this.thresholds?.maxDims ?? 50,
);
return topDims.map((dim) => `referrer:${dim}`);
},
async computeMany(ctx, dimensionKeys): Promise<ComputeResult[]> {
const { currentMap, baselineMap, totalCurrent, totalBaseline } =
await fetchReferrerAggregates(ctx);
const results: ComputeResult[] = [];
for (const dimKey of dimensionKeys) {
if (!dimKey.startsWith('referrer:')) continue;
const referrerName = dimKey.replace('referrer:', '');
const currentValue = currentMap.get(referrerName) ?? 0;
const compareValue = baselineMap.get(referrerName) ?? 0;
const currentShare = totalCurrent > 0 ? currentValue / totalCurrent : 0;
const compareShare = totalBaseline > 0 ? compareValue / totalBaseline : 0;
const shareShiftPp = (currentShare - compareShare) * 100;
const changePct = computeChangePct(currentValue, compareValue);
const direction = computeDirection(changePct);
results.push({
ok: true,
dimensionKey: dimKey,
currentValue,
compareValue,
changePct,
direction,
extra: {
shareShiftPp,
currentShare,
compareShare,
isNew: compareValue === 0 && currentValue > 0,
isGone: currentValue === 0 && compareValue > 0,
},
});
}
return results;
},
render(result, ctx): RenderedCard {
const referrer = result.dimensionKey.replace('referrer:', '');
const pct = ((result.changePct ?? 0) * 100).toFixed(1);
const isIncrease = (result.changePct ?? 0) >= 0;
const isNew = result.extra?.isNew as boolean | undefined;
const title = isNew
? `New traffic source: ${referrer}`
: `Traffic from ${referrer} ${isIncrease ? '↑' : '↓'} ${Math.abs(Number(pct))}%`;
const sessionsCurrent = result.currentValue ?? 0;
const sessionsCompare = result.compareValue ?? 0;
const shareCurrent = Number(result.extra?.currentShare ?? 0);
const shareCompare = Number(result.extra?.compareShare ?? 0);
return {
title,
summary: `${ctx.window.label}. Sessions ${sessionsCurrent} vs ${sessionsCompare}.`,
displayName: referrer,
payload: {
kind: 'insight_v1',
dimensions: [
{
key: 'referrer_name',
value: referrer,
displayName: referrer,
},
],
primaryMetric: 'sessions',
metrics: {
sessions: {
current: sessionsCurrent,
compare: sessionsCompare,
delta: sessionsCurrent - sessionsCompare,
changePct: sessionsCompare > 0 ? (result.changePct ?? 0) : null,
direction: result.direction ?? 'flat',
unit: 'count',
},
share: {
current: shareCurrent,
compare: shareCompare,
delta: shareCurrent - shareCompare,
changePct:
shareCompare > 0
? (shareCurrent - shareCompare) / shareCompare
: null,
direction:
shareCurrent - shareCompare > 0.0005
? 'up'
: shareCurrent - shareCompare < -0.0005
? 'down'
: 'flat',
unit: 'ratio',
},
},
extra: {
isNew: result.extra?.isNew,
isGone: result.extra?.isGone,
},
},
};
},
};

View File

@@ -1,18 +0,0 @@
import type { ComputeResult } from './types';
export function defaultImpactScore(r: ComputeResult): number {
const vol = (r.currentValue ?? 0) + (r.compareValue ?? 0);
const pct = Math.abs(r.changePct ?? 0);
// stable-ish: bigger change + bigger volume => higher impact
return Math.log1p(vol) * (pct * 100);
}
export function severityBand(
changePct?: number | null,
): 'low' | 'moderate' | 'severe' | null {
const p = Math.abs(changePct ?? 0);
if (p < 0.1) return null;
if (p < 0.5) return 'low';
if (p < 1) return 'moderate';
return 'severe';
}

View File

@@ -1,343 +0,0 @@
import { Prisma, db } from '../../prisma-client';
import type {
Cadence,
InsightStore,
PersistedInsight,
RenderedCard,
WindowKind,
WindowRange,
} from './types';
export const insightStore: InsightStore = {
async listProjectIdsForCadence(cadence: Cadence): Promise<string[]> {
const projects = await db.project.findMany({
where: {
deleteAt: null,
eventsCount: { gt: 10_000 },
updatedAt: { gt: new Date(Date.now() - 1000 * 60 * 60 * 24) },
organization: {
subscriptionStatus: 'active',
},
},
select: { id: true },
});
return projects.map((p) => p.id);
},
async getProjectCreatedAt(projectId: string): Promise<Date | null> {
const project = await db.project.findFirst({
where: { id: projectId, deleteAt: null },
select: { createdAt: true },
});
return project?.createdAt ?? null;
},
async getActiveInsightByIdentity({
projectId,
moduleKey,
dimensionKey,
windowKind,
}): Promise<PersistedInsight | null> {
const insight = await db.projectInsight.findFirst({
where: {
projectId,
moduleKey,
dimensionKey,
windowKind,
state: 'active',
},
});
if (!insight) return null;
return {
id: insight.id,
projectId: insight.projectId,
moduleKey: insight.moduleKey,
dimensionKey: insight.dimensionKey,
windowKind: insight.windowKind as WindowKind,
state: insight.state as 'active' | 'suppressed' | 'closed',
version: insight.version,
impactScore: insight.impactScore,
lastSeenAt: insight.lastSeenAt,
lastUpdatedAt: insight.lastUpdatedAt,
direction: insight.direction,
severityBand: insight.severityBand,
};
},
async upsertInsight({
projectId,
moduleKey,
dimensionKey,
window,
card,
metrics,
now,
decision,
prev,
}): Promise<PersistedInsight> {
const baseData = {
projectId,
moduleKey,
dimensionKey,
windowKind: window.kind,
state: prev?.state === 'closed' ? 'active' : (prev?.state ?? 'active'),
title: card.title,
summary: card.summary ?? null,
displayName: card.displayName,
payload: card.payload,
direction: metrics.direction ?? null,
impactScore: metrics.impactScore,
severityBand: metrics.severityBand ?? null,
version: prev ? (decision.material ? prev.version + 1 : prev.version) : 1,
windowStart: window.start,
windowEnd: window.end,
lastSeenAt: now,
lastUpdatedAt: now,
};
// Try to find existing insight first
const existing = prev
? await db.projectInsight.findFirst({
where: {
projectId,
moduleKey,
dimensionKey,
windowKind: window.kind,
state: prev.state,
},
})
: null;
let insight: any;
if (existing) {
// Update existing
insight = await db.projectInsight.update({
where: { id: existing.id },
data: {
...baseData,
threadId: existing.threadId, // Preserve threadId
},
});
} else {
// Create new - need to check if there's a closed/suppressed one to reopen
const closed = await db.projectInsight.findFirst({
where: {
projectId,
moduleKey,
dimensionKey,
windowKind: window.kind,
state: { in: ['closed', 'suppressed'] },
},
orderBy: { lastUpdatedAt: 'desc' },
});
if (closed) {
// Reopen and update
insight = await db.projectInsight.update({
where: { id: closed.id },
data: {
...baseData,
state: 'active',
threadId: closed.threadId, // Preserve threadId
},
});
} else {
// Create new
insight = await db.projectInsight.create({
data: {
...baseData,
firstDetectedAt: now,
},
});
}
}
return {
id: insight.id,
projectId: insight.projectId,
moduleKey: insight.moduleKey,
dimensionKey: insight.dimensionKey,
windowKind: insight.windowKind as WindowKind,
state: insight.state as 'active' | 'suppressed' | 'closed',
version: insight.version,
impactScore: insight.impactScore,
lastSeenAt: insight.lastSeenAt,
lastUpdatedAt: insight.lastUpdatedAt,
direction: insight.direction,
severityBand: insight.severityBand,
};
},
async insertEvent({
projectId,
insightId,
moduleKey,
dimensionKey,
windowKind,
eventKind,
changeFrom,
changeTo,
now,
}): Promise<void> {
await db.insightEvent.create({
data: {
insightId,
eventKind,
changeFrom: changeFrom
? (changeFrom as Prisma.InputJsonValue)
: Prisma.DbNull,
changeTo: changeTo
? (changeTo as Prisma.InputJsonValue)
: Prisma.DbNull,
createdAt: now,
},
});
},
async closeMissingActiveInsights({
projectId,
moduleKey,
windowKind,
seenDimensionKeys,
now,
staleDays,
}): Promise<number> {
const staleDate = new Date(now);
staleDate.setDate(staleDate.getDate() - staleDays);
const result = await db.projectInsight.updateMany({
where: {
projectId,
moduleKey,
windowKind,
state: 'active',
lastSeenAt: { lt: staleDate },
dimensionKey: { notIn: seenDimensionKeys },
},
data: {
state: 'closed',
lastUpdatedAt: now,
},
});
return result.count;
},
async applySuppression({
projectId,
moduleKey,
windowKind,
keepTopN,
now,
}): Promise<{ suppressed: number; unsuppressed: number }> {
// Get all active insights for this module/window, ordered by impactScore desc
const insights = await db.projectInsight.findMany({
where: {
projectId,
moduleKey,
windowKind,
state: { in: ['active', 'suppressed'] },
},
orderBy: { impactScore: 'desc' },
});
if (insights.length === 0) {
return { suppressed: 0, unsuppressed: 0 };
}
let suppressed = 0;
let unsuppressed = 0;
// For "yesterday" insights, suppress any that are stale (windowEnd is not actually yesterday)
// This prevents showing confusing insights like "Yesterday traffic dropped" when it's from 2+ days ago
if (windowKind === 'yesterday') {
const yesterday = new Date(now);
yesterday.setUTCHours(0, 0, 0, 0);
yesterday.setUTCDate(yesterday.getUTCDate() - 1);
const yesterdayTime = yesterday.getTime();
for (const insight of insights) {
// If windowEnd is null, consider it stale
const isStale = insight.windowEnd
? new Date(insight.windowEnd).setUTCHours(0, 0, 0, 0) !==
yesterdayTime
: true;
if (isStale && insight.state === 'active') {
await db.projectInsight.update({
where: { id: insight.id },
data: { state: 'suppressed', lastUpdatedAt: now },
});
suppressed++;
}
}
// Filter to only non-stale insights for top-N logic
const freshInsights = insights.filter((insight) => {
if (!insight.windowEnd) return false;
const windowEndTime = new Date(insight.windowEnd).setUTCHours(
0,
0,
0,
0,
);
return windowEndTime === yesterdayTime;
});
const topN = freshInsights.slice(0, keepTopN);
const belowN = freshInsights.slice(keepTopN);
for (const insight of belowN) {
if (insight.state === 'active') {
await db.projectInsight.update({
where: { id: insight.id },
data: { state: 'suppressed', lastUpdatedAt: now },
});
suppressed++;
}
}
for (const insight of topN) {
if (insight.state === 'suppressed') {
await db.projectInsight.update({
where: { id: insight.id },
data: { state: 'active', lastUpdatedAt: now },
});
unsuppressed++;
}
}
return { suppressed, unsuppressed };
}
// For non-yesterday windows, apply standard top-N suppression
const topN = insights.slice(0, keepTopN);
const belowN = insights.slice(keepTopN);
// Suppress those below top N
for (const insight of belowN) {
if (insight.state === 'active') {
await db.projectInsight.update({
where: { id: insight.id },
data: { state: 'suppressed', lastUpdatedAt: now },
});
suppressed++;
}
}
// Unsuppress those in top N
for (const insight of topN) {
if (insight.state === 'suppressed') {
await db.projectInsight.update({
where: { id: insight.id },
data: { state: 'active', lastUpdatedAt: now },
});
unsuppressed++;
}
}
return { suppressed, unsuppressed };
},
};

View File

@@ -1,191 +0,0 @@
import type {
InsightDimension,
InsightMetricEntry,
InsightMetricKey,
InsightPayload,
} from '@openpanel/validation';
export type Cadence = 'daily';
export type WindowKind = 'yesterday' | 'rolling_7d' | 'rolling_30d';
export interface WindowRange {
kind: WindowKind;
start: Date; // inclusive
end: Date; // inclusive (or exclusive, but be consistent)
baselineStart: Date;
baselineEnd: Date;
label: string; // e.g. "Yesterday" / "Last 7 days"
}
export interface ComputeContext {
projectId: string;
window: WindowRange;
db: any; // your DB client
now: Date;
logger: Pick<Console, 'info' | 'warn' | 'error'>;
/**
* Cached clix function that automatically caches query results based on query hash.
* This eliminates duplicate queries within the same module+window context.
* Use this instead of importing clix directly to benefit from automatic caching.
*/
clix: ReturnType<typeof import('./cached-clix').createCachedClix>;
}
export interface ComputeResult {
ok: boolean;
dimensionKey: string; // e.g. "referrer:instagram" / "page:/pricing"
currentValue?: number;
compareValue?: number;
changePct?: number; // -0.15 = -15%
direction?: 'up' | 'down' | 'flat';
extra?: Record<string, unknown>; // share delta pp, rank, sparkline, etc.
}
// Types imported from @openpanel/validation:
// - InsightMetricKey
// - InsightMetricEntry
// - InsightDimension
// - InsightPayload
/**
* Render should be deterministic and safe to call multiple times.
* Returns the shape that matches ProjectInsight create input.
* The payload contains all metric data and display metadata.
*/
export interface RenderedCard {
title: string;
summary?: string;
displayName: string;
payload: InsightPayload; // Contains dimensions, primaryMetric, metrics, extra
}
/** Optional per-module thresholds (the engine can still apply global defaults) */
export interface ModuleThresholds {
minTotal?: number; // min current+baseline
minAbsDelta?: number; // min abs(current-compare)
minPct?: number; // min abs(changePct)
maxDims?: number; // cap enumerateDimensions
}
export interface InsightModule {
key: string;
cadence: Cadence[];
/** Optional per-module override; engine applies a default if omitted. */
windows?: WindowKind[];
thresholds?: ModuleThresholds;
enumerateDimensions?(ctx: ComputeContext): Promise<string[]>;
/** Preferred path: batch compute many dimensions in one go. */
computeMany(
ctx: ComputeContext,
dimensionKeys: string[],
): Promise<ComputeResult[]>;
/** Must not do DB reads; just format output. */
render(result: ComputeResult, ctx: ComputeContext): RenderedCard;
/** Score decides what to show (top-N). */
score?(result: ComputeResult, ctx: ComputeContext): number;
/** Optional: compute "drivers" for AI explain step */
drivers?(
result: ComputeResult,
ctx: ComputeContext,
): Promise<Record<string, unknown>>;
}
/** Insight row shape returned from persistence (minimal fields engine needs). */
export interface PersistedInsight {
id: string;
projectId: string;
moduleKey: string;
dimensionKey: string;
windowKind: WindowKind;
state: 'active' | 'suppressed' | 'closed';
version: number;
impactScore: number;
lastSeenAt: Date;
lastUpdatedAt: Date;
direction?: string | null;
severityBand?: string | null;
}
/** Material change decision used for events/notifications. */
export type MaterialReason =
| 'created'
| 'direction_flip'
| 'severity_change'
| 'cross_deadband'
| 'reopened'
| 'none';
export interface MaterialDecision {
material: boolean;
reason: MaterialReason;
newSeverityBand?: 'low' | 'moderate' | 'severe' | null;
}
/**
* Persistence interface: implement with Postgres.
* Keep engine independent of query builder choice.
*/
export interface InsightStore {
listProjectIdsForCadence(cadence: Cadence): Promise<string[]>;
/** Used by the engine/worker to decide if a window has enough baseline history. */
getProjectCreatedAt(projectId: string): Promise<Date | null>;
getActiveInsightByIdentity(args: {
projectId: string;
moduleKey: string;
dimensionKey: string;
windowKind: WindowKind;
}): Promise<PersistedInsight | null>;
upsertInsight(args: {
projectId: string;
moduleKey: string;
dimensionKey: string;
window: WindowRange;
card: RenderedCard;
metrics: {
direction?: 'up' | 'down' | 'flat';
impactScore: number;
severityBand?: string | null;
};
now: Date;
decision: MaterialDecision;
prev: PersistedInsight | null;
}): Promise<PersistedInsight>;
insertEvent(args: {
projectId: string;
insightId: string;
moduleKey: string;
dimensionKey: string;
windowKind: WindowKind;
eventKind:
| 'created'
| 'updated'
| 'severity_up'
| 'severity_down'
| 'direction_flip'
| 'closed'
| 'reopened'
| 'suppressed'
| 'unsuppressed';
changeFrom?: Record<string, unknown> | null;
changeTo?: Record<string, unknown> | null;
now: Date;
}): Promise<void>;
/** Mark insights as not seen this run if you prefer lifecycle via closeMissing() */
closeMissingActiveInsights(args: {
projectId: string;
moduleKey: string;
windowKind: WindowKind;
seenDimensionKeys: string[];
now: Date;
staleDays: number; // close if not seen for X days
}): Promise<number>; // count closed
/** Enforce top-N display by suppressing below-threshold insights. */
applySuppression(args: {
projectId: string;
moduleKey: string;
windowKind: WindowKind;
keepTopN: number;
now: Date;
}): Promise<{ suppressed: number; unsuppressed: number }>;
}

View File

@@ -1,151 +0,0 @@
/**
* Shared utilities for insight modules
*/
/**
* Get UTC weekday (0 = Sunday, 6 = Saturday)
*/
export function getWeekday(date: Date): number {
return date.getUTCDay();
}
/**
* Compute median of a sorted array of numbers
*/
export function computeMedian(sortedValues: number[]): number {
if (sortedValues.length === 0) return 0;
const mid = Math.floor(sortedValues.length / 2);
return sortedValues.length % 2 === 0
? ((sortedValues[mid - 1] ?? 0) + (sortedValues[mid] ?? 0)) / 2
: (sortedValues[mid] ?? 0);
}
/**
* Compute weekday medians from daily breakdown data.
* Groups by dimension, filters to matching weekday, computes median per dimension.
*
* @param data - Array of { date, dimension, cnt } rows
* @param targetWeekday - Weekday to filter to (0-6)
* @param getDimension - Function to extract normalized dimension from row
* @returns Map of dimension -> median value
*/
export function computeWeekdayMedians<T>(
data: T[],
targetWeekday: number,
getDimension: (row: T) => string,
): Map<string, number> {
// Group by dimension, filtered to target weekday
const byDimension = new Map<string, number[]>();
for (const row of data) {
const rowWeekday = getWeekday(new Date((row as any).date));
if (rowWeekday !== targetWeekday) continue;
const dim = getDimension(row);
const values = byDimension.get(dim) ?? [];
values.push(Number((row as any).cnt ?? 0));
byDimension.set(dim, values);
}
// Compute median per dimension
const result = new Map<string, number>();
for (const [dim, values] of byDimension) {
values.sort((a, b) => a - b);
result.set(dim, computeMedian(values));
}
return result;
}
/**
* Compute change percentage between current and compare values
*/
export function computeChangePct(
currentValue: number,
compareValue: number,
): number {
return compareValue > 0
? (currentValue - compareValue) / compareValue
: currentValue > 0
? 1
: 0;
}
/**
* Determine direction based on change percentage
*/
export function computeDirection(
changePct: number,
threshold = 0.05,
): 'up' | 'down' | 'flat' {
return changePct > threshold
? 'up'
: changePct < -threshold
? 'down'
: 'flat';
}
/**
* Get end of day timestamp (23:59:59.999) for a given date.
* Used to ensure BETWEEN queries include the full day.
*/
export function getEndOfDay(date: Date): Date {
const end = new Date(date);
end.setUTCHours(23, 59, 59, 999);
return end;
}
/**
* Build a lookup map from query results.
* Aggregates counts by key, handling duplicate keys by summing values.
*
* @param results - Array of result rows
* @param getKey - Function to extract the key from each row
* @param getCount - Function to extract the count from each row (defaults to 'cnt' field)
* @returns Map of key -> aggregated count
*/
export function buildLookupMap<T>(
results: T[],
getKey: (row: T) => string,
getCount: (row: T) => number = (row) => Number((row as any).cnt ?? 0),
): Map<string, number> {
const map = new Map<string, number>();
for (const row of results) {
const key = getKey(row);
const cnt = getCount(row);
map.set(key, (map.get(key) ?? 0) + cnt);
}
return map;
}
/**
* Select top-N dimensions by ranking on greatest(current, baseline).
* This preserves union behavior: dimensions with high values in either period are included.
*
* @param currentMap - Map of dimension -> current value
* @param baselineMap - Map of dimension -> baseline value
* @param maxDims - Maximum number of dimensions to return
* @returns Array of dimension keys, ranked by greatest(current, baseline)
*/
export function selectTopDimensions(
currentMap: Map<string, number>,
baselineMap: Map<string, number>,
maxDims: number,
): string[] {
// Merge all dimensions from both maps
const allDims = new Set<string>();
for (const dim of currentMap.keys()) allDims.add(dim);
for (const dim of baselineMap.keys()) allDims.add(dim);
// Rank by greatest(current, baseline)
const ranked = Array.from(allDims)
.map((dim) => ({
dim,
maxValue: Math.max(currentMap.get(dim) ?? 0, baselineMap.get(dim) ?? 0),
}))
.sort((a, b) => b.maxValue - a.maxValue)
.slice(0, maxDims)
.map((x) => x.dim);
return ranked;
}

View File

@@ -1,59 +0,0 @@
import type { WindowKind, WindowRange } from './types';
function atUtcMidnight(d: Date) {
const x = new Date(d);
x.setUTCHours(0, 0, 0, 0);
return x;
}
function addDays(d: Date, days: number) {
const x = new Date(d);
x.setUTCDate(x.getUTCDate() + days);
return x;
}
/**
* Convention: end is inclusive (end of day). If you prefer exclusive, adapt consistently.
*/
export function resolveWindow(kind: WindowKind, now: Date): WindowRange {
const today0 = atUtcMidnight(now);
const yesterday0 = addDays(today0, -1);
if (kind === 'yesterday') {
const start = yesterday0;
const end = yesterday0;
// Baseline: median of last 4 same weekdays -> engine/module implements the median.
// Here we just define the candidate range; module queries last 28 days and filters weekday.
const baselineStart = addDays(yesterday0, -28);
const baselineEnd = addDays(yesterday0, -1);
return { kind, start, end, baselineStart, baselineEnd, label: 'Yesterday' };
}
if (kind === 'rolling_7d') {
const end = yesterday0;
const start = addDays(end, -6); // 7 days inclusive
const baselineEnd = addDays(start, -1);
const baselineStart = addDays(baselineEnd, -6);
return {
kind,
start,
end,
baselineStart,
baselineEnd,
label: 'Last 7 days',
};
}
// rolling_30d
{
const end = yesterday0;
const start = addDays(end, -29);
const baselineEnd = addDays(start, -1);
const baselineStart = addDays(baselineEnd, -29);
return {
kind,
start,
end,
baselineStart,
baselineEnd,
label: 'Last 30 days',
};
}
}

View File

@@ -7,28 +7,6 @@ import { TABLE_NAMES, ch } from '../clickhouse/client';
import { clix } from '../clickhouse/query-builder';
import { getEventFiltersWhereClause } from './chart.service';
// Constants
const ROLLUP_DATE_PREFIX = '1970-01-01';
const COLUMN_PREFIX_MAP: Record<string, string> = {
region: 'country',
city: 'country',
browser_version: 'browser',
os_version: 'os',
};
// Types
type MetricsRow = {
bounce_rate: number;
unique_visitors: number;
total_sessions: number;
avg_session_duration: number;
total_screen_views: number;
views_per_session: number;
};
type MetricsSeriesRow = MetricsRow & { date: string; total_revenue: number };
export const zGetMetricsInput = z.object({
projectId: z.string(),
filters: z.array(z.any()),
@@ -107,112 +85,11 @@ export type IGetTopGenericInput = z.infer<typeof zGetTopGenericInput> & {
export class OverviewService {
constructor(private client: typeof ch) {}
// Helper methods
private isRollupRow(date: string): boolean {
return date.startsWith(ROLLUP_DATE_PREFIX);
}
private getFillConfig(interval: string, startDate: string, endDate: string) {
const useDateOnly = ['month', 'week'].includes(interval);
return {
from: clix.toStartOf(
clix.datetime(startDate, useDateOnly ? 'toDate' : 'toDateTime'),
interval as any,
),
to: clix.datetime(endDate, useDateOnly ? 'toDate' : 'toDateTime'),
step: clix.toInterval('1', interval as any),
};
}
private createRevenueQuery({
projectId,
startDate,
endDate,
interval,
timezone,
filters,
}: {
projectId: string;
startDate: string;
endDate: string;
interval: string;
timezone: string;
filters: IChartEventFilter[];
}) {
return clix(this.client, timezone)
.select<{ date: string; total_revenue: number }>([
`${clix.toStartOf('created_at', interval as any, timezone)} AS date`,
'sum(revenue) AS total_revenue',
])
.from(TABLE_NAMES.events)
.where('project_id', '=', projectId)
.where('name', '=', 'revenue')
.where('revenue', '>', 0)
.where('created_at', 'BETWEEN', [
clix.datetime(startDate, 'toDateTime'),
clix.datetime(endDate, 'toDateTime'),
])
.rawWhere(this.getRawWhereClause('events', filters))
.groupBy(['date'])
.rollup()
.transform({
date: (item) => new Date(item.date).toISOString(),
});
}
private mergeRevenueIntoSeries<T extends { date: string }>(
series: T[],
revenueData: { date: string; total_revenue: number }[],
): (T & { total_revenue: number })[] {
const revenueByDate = new Map(
revenueData
.filter((r) => !this.isRollupRow(r.date))
.map((r) => [r.date, r.total_revenue]),
);
return series.map((row) => ({
...row,
total_revenue: revenueByDate.get(row.date) ?? 0,
}));
}
private getOverallRevenue(
revenueData: { date: string; total_revenue: number }[],
): number {
return (
revenueData.find((r) => this.isRollupRow(r.date))?.total_revenue ?? 0
);
}
private withDistinctSessionsIfNeeded<T>(
query: ReturnType<typeof clix>,
params: {
filters: IChartEventFilter[];
projectId: string;
startDate: string;
endDate: string;
timezone: string;
},
): ReturnType<typeof clix> {
if (!this.isPageFilter(params.filters)) {
query.rawWhere(this.getRawWhereClause('sessions', params.filters));
return query;
}
return clix(this.client, params.timezone)
.with('distinct_sessions', this.getDistinctSessions(params))
.merge(query)
.where(
'id',
'IN',
clix.exp('(SELECT session_id FROM distinct_sessions)'),
);
}
isPageFilter(filters: IChartEventFilter[]) {
return filters.some((filter) => filter.name === 'path' && filter.value);
}
async getMetrics({
getMetrics({
projectId,
filters,
startDate,
@@ -239,42 +116,170 @@ export class OverviewService {
views_per_session: number;
total_revenue: number;
}[];
}> {
return this.isPageFilter(filters)
? this.getMetricsWithPageFilter({
projectId,
filters,
startDate,
endDate,
interval,
timezone,
})
: this.getMetricsFromSessions({
projectId,
filters,
startDate,
endDate,
interval,
timezone,
});
}
private async getMetricsFromSessions({
projectId,
filters,
startDate,
endDate,
interval,
timezone,
}: IGetMetricsInput): Promise<{
metrics: MetricsRow & { total_revenue: number };
series: MetricsSeriesRow[];
}> {
const where = this.getRawWhereClause('sessions', filters);
const fillConfig = this.getFillConfig(interval, startDate, endDate);
if (this.isPageFilter(filters)) {
// Session aggregation with bounce rates
const sessionAggQuery = clix(this.client, timezone)
.select([
`${clix.toStartOf('created_at', interval, timezone)} AS date`,
'round((countIf(is_bounce = 1 AND sign = 1) * 100.) / countIf(sign = 1), 2) AS bounce_rate',
'sum(revenue * sign) AS total_revenue',
])
.from(TABLE_NAMES.sessions, true)
.where('sign', '=', 1)
.where('project_id', '=', projectId)
.where('created_at', 'BETWEEN', [
clix.datetime(startDate, 'toDateTime'),
clix.datetime(endDate, 'toDateTime'),
])
.rawWhere(where)
.groupBy(['date'])
.rollup()
.orderBy('date', 'ASC');
// Session metrics query
const sessionQuery = clix(this.client, timezone)
// Overall unique visitors
const overallUniqueVisitorsQuery = clix(this.client, timezone)
.select([
'uniq(profile_id) AS unique_visitors',
'uniq(session_id) AS total_sessions',
])
.from(TABLE_NAMES.events)
.where('project_id', '=', projectId)
.where('name', '=', 'screen_view')
.where('created_at', 'BETWEEN', [
clix.datetime(startDate, 'toDateTime'),
clix.datetime(endDate, 'toDateTime'),
])
.rawWhere(this.getRawWhereClause('events', filters));
// Use toDate for month/week intervals, toDateTime for others
const rollupDate =
interval === 'month' || interval === 'week'
? clix.date('1970-01-01')
: clix.datetime('1970-01-01 00:00:00');
return clix(this.client, timezone)
.with('session_agg', sessionAggQuery)
.with(
'overall_bounce_rate',
clix(this.client, timezone)
.select(['bounce_rate'])
.from('session_agg')
.where('date', '=', rollupDate),
)
.with(
'overall_total_revenue',
clix(this.client, timezone)
.select(['total_revenue'])
.from('session_agg')
.where('date', '=', rollupDate),
)
.with(
'daily_stats',
clix(this.client, timezone)
.select(['date', 'bounce_rate', 'total_revenue'])
.from('session_agg')
.where('date', '!=', rollupDate),
)
.with('overall_unique_visitors', overallUniqueVisitorsQuery)
.select<{
date: string;
bounce_rate: number;
unique_visitors: number;
total_sessions: number;
avg_session_duration: number;
total_screen_views: number;
views_per_session: number;
total_revenue: number;
overall_unique_visitors: number;
overall_total_sessions: number;
overall_bounce_rate: number;
overall_total_revenue: number;
}>([
`${clix.toStartOf('e.created_at', interval)} AS date`,
'ds.bounce_rate as bounce_rate',
'uniq(e.profile_id) AS unique_visitors',
'uniq(e.session_id) AS total_sessions',
'round(avgIf(duration, duration > 0), 2) / 1000 AS _avg_session_duration',
'if(isNaN(_avg_session_duration), 0, _avg_session_duration) AS avg_session_duration',
'count(*) AS total_screen_views',
'round((count(*) * 1.) / uniq(e.session_id), 2) AS views_per_session',
'ds.total_revenue AS total_revenue',
'(SELECT unique_visitors FROM overall_unique_visitors) AS overall_unique_visitors',
'(SELECT total_sessions FROM overall_unique_visitors) AS overall_total_sessions',
'(SELECT bounce_rate FROM overall_bounce_rate) AS overall_bounce_rate',
'(SELECT total_revenue FROM overall_total_revenue) AS overall_total_revenue',
])
.from(`${TABLE_NAMES.events} AS e`)
.leftJoin(
'daily_stats AS ds',
`${clix.toStartOf('e.created_at', interval)} = ds.date`,
)
.where('e.project_id', '=', projectId)
.where('e.name', '=', 'screen_view')
.where('e.created_at', 'BETWEEN', [
clix.datetime(startDate, 'toDateTime'),
clix.datetime(endDate, 'toDateTime'),
])
.rawWhere(this.getRawWhereClause('events', filters))
.groupBy(['date', 'ds.bounce_rate', 'ds.total_revenue'])
.orderBy('date', 'ASC')
.fill(
clix.toStartOf(
clix.datetime(
startDate,
['month', 'week'].includes(interval) ? 'toDate' : 'toDateTime',
),
interval,
),
clix.datetime(
endDate,
['month', 'week'].includes(interval) ? 'toDate' : 'toDateTime',
),
clix.toInterval('1', interval),
)
.transform({
date: (item) => new Date(item.date).toISOString(),
})
.execute()
.then((res) => {
const anyRowWithData = res.find(
(item) =>
item.overall_bounce_rate !== null ||
item.overall_total_sessions !== null ||
item.overall_unique_visitors !== null ||
item.overall_total_revenue !== null,
);
return {
metrics: {
bounce_rate: anyRowWithData?.overall_bounce_rate ?? 0,
unique_visitors: anyRowWithData?.overall_unique_visitors ?? 0,
total_sessions: anyRowWithData?.overall_total_sessions ?? 0,
avg_session_duration: average(
res.map((item) => item.avg_session_duration),
),
total_screen_views: sum(
res.map((item) => item.total_screen_views),
),
views_per_session: average(
res.map((item) => item.views_per_session),
),
total_revenue: anyRowWithData?.overall_total_revenue ?? 0,
},
series: res.map(
omit([
'overall_bounce_rate',
'overall_unique_visitors',
'overall_total_sessions',
'overall_total_revenue',
]),
),
};
});
}
const query = clix(this.client, timezone)
.select<{
date: string;
bounce_rate: number;
@@ -283,8 +288,9 @@ export class OverviewService {
avg_session_duration: number;
total_screen_views: number;
views_per_session: number;
total_revenue: number;
}>([
`${clix.toStartOf('created_at', interval as any, timezone)} AS date`,
`${clix.toStartOf('created_at', interval, timezone)} AS date`,
'round(sum(sign * is_bounce) * 100.0 / sum(sign), 2) as bounce_rate',
'uniqIf(profile_id, sign > 0) AS unique_visitors',
'sum(sign) AS total_sessions',
@@ -292,6 +298,7 @@ export class OverviewService {
'if(isNaN(_avg_session_duration), 0, _avg_session_duration) AS avg_session_duration',
'sum(sign * screen_view_count) AS total_screen_views',
'round(sum(sign * screen_view_count) * 1.0 / sum(sign), 2) AS views_per_session',
'sum(revenue * sign) AS total_revenue',
])
.from('sessions')
.where('created_at', 'BETWEEN', [
@@ -304,200 +311,41 @@ export class OverviewService {
.having('sum(sign)', '>', 0)
.rollup()
.orderBy('date', 'ASC')
.fill(fillConfig.from, fillConfig.to, fillConfig.step)
.fill(
clix.toStartOf(
clix.datetime(
startDate,
['month', 'week'].includes(interval) ? 'toDate' : 'toDateTime',
),
interval,
),
clix.datetime(
endDate,
['month', 'week'].includes(interval) ? 'toDate' : 'toDateTime',
),
clix.toInterval('1', interval),
)
.transform({
date: (item) => new Date(item.date).toISOString(),
});
// Revenue query
const revenueQuery = this.createRevenueQuery({
projectId,
startDate,
endDate,
interval,
timezone,
filters,
return query.execute().then((res) => {
// First row is the rollup row containing the total values
return {
metrics: {
bounce_rate: res[0]?.bounce_rate ?? 0,
unique_visitors: res[0]?.unique_visitors ?? 0,
total_sessions: res[0]?.total_sessions ?? 0,
avg_session_duration: res[0]?.avg_session_duration ?? 0,
total_screen_views: res[0]?.total_screen_views ?? 0,
views_per_session: res[0]?.views_per_session ?? 0,
total_revenue: res[0]?.total_revenue ?? 0,
},
series: res
.slice(1)
.map(omit(['overall_bounce_rate', 'overall_unique_visitors'])),
};
});
// Execute both queries in parallel and merge results
const [sessionRes, revenueRes] = await Promise.all([
sessionQuery.execute(),
revenueQuery.execute(),
]);
const overallRevenue = this.getOverallRevenue(revenueRes);
const series = this.mergeRevenueIntoSeries(sessionRes.slice(1), revenueRes);
return {
metrics: {
bounce_rate: sessionRes[0]?.bounce_rate ?? 0,
unique_visitors: sessionRes[0]?.unique_visitors ?? 0,
total_sessions: sessionRes[0]?.total_sessions ?? 0,
avg_session_duration: sessionRes[0]?.avg_session_duration ?? 0,
total_screen_views: sessionRes[0]?.total_screen_views ?? 0,
views_per_session: sessionRes[0]?.views_per_session ?? 0,
total_revenue: overallRevenue,
},
series,
};
}
private async getMetricsWithPageFilter({
projectId,
filters,
startDate,
endDate,
interval,
timezone,
}: IGetMetricsInput): Promise<{
metrics: MetricsRow & { total_revenue: number };
series: MetricsSeriesRow[];
}> {
const where = this.getRawWhereClause('sessions', filters);
const fillConfig = this.getFillConfig(interval, startDate, endDate);
// Session aggregation with bounce rates
const sessionAggQuery = clix(this.client, timezone)
.select([
`${clix.toStartOf('created_at', interval as any, timezone)} AS date`,
'round((countIf(is_bounce = 1 AND sign = 1) * 100.) / countIf(sign = 1), 2) AS bounce_rate',
])
.from(TABLE_NAMES.sessions, true)
.where('sign', '=', 1)
.where('project_id', '=', projectId)
.where('created_at', 'BETWEEN', [
clix.datetime(startDate, 'toDateTime'),
clix.datetime(endDate, 'toDateTime'),
])
.rawWhere(where)
.groupBy(['date'])
.rollup()
.orderBy('date', 'ASC');
// Overall unique visitors
const overallUniqueVisitorsQuery = clix(this.client, timezone)
.select([
'uniq(profile_id) AS unique_visitors',
'uniq(session_id) AS total_sessions',
])
.from(TABLE_NAMES.events)
.where('project_id', '=', projectId)
.where('name', '=', 'screen_view')
.where('created_at', 'BETWEEN', [
clix.datetime(startDate, 'toDateTime'),
clix.datetime(endDate, 'toDateTime'),
])
.rawWhere(this.getRawWhereClause('events', filters));
// Use toDate for month/week intervals, toDateTime for others
const rollupDate =
interval === 'month' || interval === 'week'
? clix.date(ROLLUP_DATE_PREFIX)
: clix.datetime(`${ROLLUP_DATE_PREFIX} 00:00:00`);
// Main metrics query (without revenue)
const mainQuery = clix(this.client, timezone)
.with('session_agg', sessionAggQuery)
.with(
'overall_bounce_rate',
clix(this.client, timezone)
.select(['bounce_rate'])
.from('session_agg')
.where('date', '=', rollupDate),
)
.with(
'daily_session_stats',
clix(this.client, timezone)
.select(['date', 'bounce_rate'])
.from('session_agg')
.where('date', '!=', rollupDate),
)
.with('overall_unique_visitors', overallUniqueVisitorsQuery)
.select<{
date: string;
bounce_rate: number;
unique_visitors: number;
total_sessions: number;
avg_session_duration: number;
total_screen_views: number;
views_per_session: number;
overall_unique_visitors: number;
overall_total_sessions: number;
overall_bounce_rate: number;
}>([
`${clix.toStartOf('e.created_at', interval as any)} AS date`,
'dss.bounce_rate as bounce_rate',
'uniq(e.profile_id) AS unique_visitors',
'uniq(e.session_id) AS total_sessions',
'round(avgIf(duration, duration > 0), 2) / 1000 AS _avg_session_duration',
'if(isNaN(_avg_session_duration), 0, _avg_session_duration) AS avg_session_duration',
'count(*) AS total_screen_views',
'round((count(*) * 1.) / uniq(e.session_id), 2) AS views_per_session',
'(SELECT unique_visitors FROM overall_unique_visitors) AS overall_unique_visitors',
'(SELECT total_sessions FROM overall_unique_visitors) AS overall_total_sessions',
'(SELECT bounce_rate FROM overall_bounce_rate) AS overall_bounce_rate',
])
.from(`${TABLE_NAMES.events} AS e`)
.leftJoin(
'daily_session_stats AS dss',
`${clix.toStartOf('e.created_at', interval as any)} = dss.date`,
)
.where('e.project_id', '=', projectId)
.where('e.name', '=', 'screen_view')
.where('e.created_at', 'BETWEEN', [
clix.datetime(startDate, 'toDateTime'),
clix.datetime(endDate, 'toDateTime'),
])
.rawWhere(this.getRawWhereClause('events', filters))
.groupBy(['date', 'dss.bounce_rate'])
.orderBy('date', 'ASC')
.fill(fillConfig.from, fillConfig.to, fillConfig.step)
.transform({
date: (item) => new Date(item.date).toISOString(),
});
// Revenue query
const revenueQuery = this.createRevenueQuery({
projectId,
startDate,
endDate,
interval,
timezone,
filters,
});
// Execute both queries in parallel and merge results
const [mainRes, revenueRes] = await Promise.all([
mainQuery.execute(),
revenueQuery.execute(),
]);
const overallRevenue = this.getOverallRevenue(revenueRes);
const series = this.mergeRevenueIntoSeries(mainRes, revenueRes);
const anyRowWithData = mainRes.find(
(item) =>
item.overall_bounce_rate !== null ||
item.overall_total_sessions !== null ||
item.overall_unique_visitors !== null,
);
return {
metrics: {
bounce_rate: anyRowWithData?.overall_bounce_rate ?? 0,
unique_visitors: anyRowWithData?.overall_unique_visitors ?? 0,
total_sessions: anyRowWithData?.overall_total_sessions ?? 0,
avg_session_duration: average(
mainRes.map((item) => item.avg_session_duration),
),
total_screen_views: sum(mainRes.map((item) => item.total_screen_views)),
views_per_session: average(
mainRes.map((item) => item.views_per_session),
),
total_revenue: overallRevenue,
},
series,
};
}
getRawWhereClause(type: 'events' | 'sessions', filters: IChartEventFilter[]) {
@@ -520,6 +368,12 @@ export class OverviewService {
}
return item;
}),
// .filter((item) => {
// if (this.isPageFilter(filters) && type === 'sessions') {
// return item.name !== 'entry_path' && item.name !== 'entry_origin';
// }
// return true;
// }),
);
return Object.values(where).join(' AND ');
@@ -560,6 +414,7 @@ export class OverviewService {
'entry_path',
'entry_origin',
'coalesce(round(countIf(is_bounce = 1 AND sign = 1) * 100.0 / countIf(sign = 1), 2), 0) as bounce_rate',
'sum(revenue * sign) as revenue',
])
.from(TABLE_NAMES.sessions, true)
.where('sign', '=', 1)
@@ -591,6 +446,7 @@ export class OverviewService {
'p.avg_duration',
'p.count as sessions',
'b.bounce_rate',
'coalesce(b.revenue, 0) as revenue',
])
.from('page_stats p', false)
.leftJoin(
@@ -613,6 +469,16 @@ export class OverviewService {
limit = 10,
timezone,
}: IGetTopEntryExitInput) {
const where = this.getRawWhereClause('sessions', filters);
const distinctSessionQuery = this.getDistinctSessions({
projectId,
filters,
startDate,
endDate,
timezone,
});
const offset = (cursor - 1) * limit;
const query = clix(this.client, timezone)
@@ -637,19 +503,25 @@ export class OverviewService {
clix.datetime(startDate, 'toDateTime'),
clix.datetime(endDate, 'toDateTime'),
])
.rawWhere(where)
.groupBy([`${mode}_origin`, `${mode}_path`])
.having('sum(sign)', '>', 0)
.orderBy('sessions', 'DESC')
.limit(limit)
.offset(offset);
const mainQuery = this.withDistinctSessionsIfNeeded(query, {
projectId,
filters,
startDate,
endDate,
timezone,
});
let mainQuery = query;
if (this.isPageFilter(filters)) {
mainQuery = clix(this.client, timezone)
.with('distinct_sessions', distinctSessionQuery)
.merge(query)
.where(
'id',
'IN',
clix.exp('(SELECT session_id FROM distinct_sessions)'),
);
}
return mainQuery.execute();
}
@@ -688,7 +560,28 @@ export class OverviewService {
limit = 10,
timezone,
}: IGetTopGenericInput) {
const prefixColumn = COLUMN_PREFIX_MAP[column] ?? null;
const distinctSessionQuery = this.getDistinctSessions({
projectId,
filters,
startDate,
endDate,
timezone,
});
const prefixColumn = (() => {
switch (column) {
case 'region':
return 'country';
case 'city':
return 'country';
case 'browser_version':
return 'browser';
case 'os_version':
return 'os';
}
return null;
})();
const offset = (cursor - 1) * limit;
const query = clix(this.client, timezone)
@@ -719,15 +612,24 @@ export class OverviewService {
.limit(limit)
.offset(offset);
const mainQuery = this.withDistinctSessionsIfNeeded(query, {
projectId,
filters,
startDate,
endDate,
timezone,
});
let mainQuery = query;
return mainQuery.execute();
if (this.isPageFilter(filters)) {
mainQuery = clix(this.client, timezone)
.with('distinct_sessions', distinctSessionQuery)
.merge(query)
.where(
'id',
'IN',
clix.exp('(SELECT session_id FROM distinct_sessions)'),
);
} else {
mainQuery.rawWhere(this.getRawWhereClause('sessions', filters));
}
const res = await mainQuery.execute();
return res;
}
}

View File

@@ -1,5 +1,6 @@
import { cacheable } from '@openpanel/redis';
import type { IChartEventFilter } from '@openpanel/validation';
import { uniq } from 'ramda';
import sqlstring from 'sqlstring';
import {
TABLE_NAMES,
@@ -52,6 +53,7 @@ export type IClickhouseSession = {
revenue: number;
sign: 1 | 0;
version: number;
properties: Record<string, string>;
};
export interface IServiceSession {
@@ -90,6 +92,7 @@ export interface IServiceSession {
utmContent: string;
utmTerm: string;
revenue: number;
properties: Record<string, string>;
profile?: IServiceProfile;
}
@@ -141,6 +144,7 @@ export function transformSession(session: IClickhouseSession): IServiceSession {
utmContent: session.utm_content,
utmTerm: session.utm_term,
revenue: session.revenue,
properties: session.properties,
profile: undefined,
};
}
@@ -196,13 +200,12 @@ export async function getSessionList({
if (cursor) {
const cAt = sqlstring.escape(cursor.createdAt);
// TODO: remove id from cursor
const cId = sqlstring.escape(cursor.id);
sb.where.cursor = `created_at < toDateTime64(${cAt}, 3)`;
sb.where.cursor = `(created_at < toDateTime64(${cAt}, 3) OR (created_at = toDateTime64(${cAt}, 3) AND id < ${cId}))`;
sb.where.cursorWindow = `created_at >= toDateTime64(${cAt}, 3) - INTERVAL ${dateIntervalInDays} DAY`;
sb.orderBy.created_at = 'created_at DESC';
sb.orderBy.created_at = 'toDate(created_at) DESC, created_at DESC, id DESC';
} else {
sb.orderBy.created_at = 'created_at DESC';
sb.orderBy.created_at = 'toDate(created_at) DESC, created_at DESC, id DESC';
sb.where.created_at = `created_at > now() - INTERVAL ${dateIntervalInDays} DAY`;
}

View File

@@ -180,11 +180,11 @@ export function sessionConsistency() {
// For write operations with session: cache WAL LSN after write
if (isWriteOperation(operation)) {
// logger.info('Prisma operation', {
// operation,
// args,
// model,
// });
logger.info('Prisma operation', {
operation,
args,
model,
});
const result = await query(args);

View File

@@ -3,7 +3,6 @@ import type {
IIntegrationConfig,
INotificationRuleConfig,
IProjectFilters,
InsightPayload,
} from '@openpanel/validation';
import type {
IClickhouseBotEvent,
@@ -19,7 +18,6 @@ declare global {
type IPrismaIntegrationConfig = IIntegrationConfig;
type IPrismaNotificationPayload = INotificationPayload;
type IPrismaProjectFilters = IProjectFilters[];
type IPrismaProjectInsightPayload = InsightPayload;
type IPrismaClickhouseEvent = IClickhouseEvent;
type IPrismaClickhouseProfile = IClickhouseProfile;
type IPrismaClickhouseBotEvent = IClickhouseBotEvent;

View File

@@ -111,18 +111,13 @@ export type CronQueuePayloadProject = {
type: 'deleteProjects';
payload: undefined;
};
export type CronQueuePayloadInsightsDaily = {
type: 'insightsDaily';
payload: undefined;
};
export type CronQueuePayload =
| CronQueuePayloadSalt
| CronQueuePayloadFlushEvents
| CronQueuePayloadFlushSessions
| CronQueuePayloadFlushProfiles
| CronQueuePayloadPing
| CronQueuePayloadProject
| CronQueuePayloadInsightsDaily;
| CronQueuePayloadProject;
export type MiscQueuePayloadTrialEndingSoon = {
type: 'trialEndingSoon';
@@ -151,7 +146,7 @@ export const eventsGroupQueues = Array.from({
}).map(
(_, index, list) =>
new GroupQueue<EventsQueuePayloadIncomingEvent['payload']>({
logger: process.env.NODE_ENV === 'production' ? queueLogger : undefined,
logger: queueLogger,
namespace: getQueueName(
list.length === 1 ? 'group_events' : `group_events_${index}`,
),
@@ -240,21 +235,6 @@ export const importQueue = new Queue<ImportQueuePayload>(
},
);
export type InsightsQueuePayloadProject = {
type: 'insightsProject';
payload: { projectId: string; date: string };
};
export const insightsQueue = new Queue<InsightsQueuePayloadProject>(
getQueueName('insights'),
{
connection: getRedisQueue(),
defaultJobOptions: {
removeOnComplete: 100,
},
},
);
export function addTrialEndingSoonJob(organizationId: string, delay: number) {
return miscQueue.add(
'misc',

View File

@@ -1,6 +1,6 @@
{
"name": "@openpanel/astro",
"version": "1.0.6-local",
"version": "1.0.5-local",
"config": {
"transformPackageJson": false,
"transformEnvs": true
@@ -14,7 +14,7 @@
"files": ["src", "index.ts"],
"keywords": ["astro-component"],
"dependencies": {
"@openpanel/web": "workspace:1.0.6-local"
"@openpanel/web": "workspace:1.0.5-local"
},
"devDependencies": {
"astro": "^5.7.7"

View File

@@ -33,7 +33,7 @@ const methods: { name: OpenPanelMethodNames; value: unknown }[] = [
value: {
...options,
sdk: 'astro',
sdkVersion: '1.0.6',
sdkVersion: '1.0.5',
},
},
];

View File

@@ -1,13 +1,13 @@
{
"name": "@openpanel/express",
"version": "1.0.4-local",
"version": "1.0.3-local",
"module": "index.ts",
"scripts": {
"build": "rm -rf dist && tsup",
"typecheck": "tsc --noEmit"
},
"dependencies": {
"@openpanel/sdk": "workspace:1.0.3-local",
"@openpanel/sdk": "workspace:1.0.2-local",
"@openpanel/common": "workspace:*"
},
"peerDependencies": {

View File

@@ -1,13 +1,13 @@
{
"name": "@openpanel/nextjs",
"version": "1.1.2-local",
"version": "1.1.1-local",
"module": "index.ts",
"scripts": {
"build": "rm -rf dist && tsup",
"typecheck": "tsc --noEmit"
},
"dependencies": {
"@openpanel/web": "workspace:1.0.6-local"
"@openpanel/web": "workspace:1.0.5-local"
},
"peerDependencies": {
"next": "^12.0.0 || ^13.0.0 || ^14.0.0 || ^15.0.0 || ^16.0.0",

View File

@@ -1,13 +1,13 @@
{
"name": "@openpanel/react-native",
"version": "1.0.4-local",
"version": "1.0.3-local",
"module": "index.ts",
"scripts": {
"build": "rm -rf dist && tsup",
"typecheck": "tsc --noEmit"
},
"dependencies": {
"@openpanel/sdk": "workspace:1.0.3-local"
"@openpanel/sdk": "workspace:1.0.2-local"
},
"devDependencies": {
"@openpanel/tsconfig": "workspace:*",

View File

@@ -1,6 +1,6 @@
{
"name": "@openpanel/sdk",
"version": "1.0.3-local",
"version": "1.0.2-local",
"module": "index.ts",
"scripts": {
"build": "rm -rf dist && tsup",

View File

@@ -68,7 +68,6 @@ export type OpenPanelOptions = {
waitForProfile?: boolean;
filter?: (payload: TrackHandlerPayload) => boolean;
disabled?: boolean;
debug?: boolean;
};
export class OpenPanel {
@@ -130,7 +129,6 @@ export class OpenPanel {
}
async track(name: string, properties?: TrackProperties) {
this.log('track event', name, properties);
return this.send({
type: 'track',
payload: {
@@ -145,7 +143,6 @@ export class OpenPanel {
}
async identify(payload: IdentifyPayload) {
this.log('identify user', payload);
if (payload.profileId) {
this.profileId = payload.profileId;
this.flush();
@@ -165,10 +162,12 @@ export class OpenPanel {
}
}
/**
* @deprecated This method is deprecated and will be removed in a future version.
*/
async alias(payload: AliasPayload) {}
async alias(payload: AliasPayload) {
return this.send({
type: 'alias',
payload,
});
}
async increment(payload: IncrementPayload) {
return this.send({
@@ -225,10 +224,4 @@ export class OpenPanel {
});
this.queue = [];
}
log(...args: any[]) {
if (this.options.debug) {
console.log('[OpenPanel.dev]', ...args);
}
}
}

View File

@@ -1,13 +1,13 @@
{
"name": "@openpanel/web",
"version": "1.0.6-local",
"version": "1.0.5-local",
"module": "index.ts",
"scripts": {
"build": "rm -rf dist && tsup",
"typecheck": "tsc --noEmit"
},
"dependencies": {
"@openpanel/sdk": "workspace:1.0.3-local"
"@openpanel/sdk": "workspace:1.0.2-local"
},
"devDependencies": {
"@openpanel/tsconfig": "workspace:*",

View File

@@ -38,6 +38,8 @@ export class OpenPanel extends OpenPanelBase {
});
if (!this.isServer()) {
console.log('OpenPanel.dev - Initialized', this.options);
try {
const pending = sessionStorage.getItem('openpanel-pending-revenues');
if (pending) {
@@ -200,6 +202,7 @@ export class OpenPanel extends OpenPanelBase {
}
this.lastPath = path;
console.log('OpenPanel.dev - Track page view');
super.track('screen_view', {
...(properties ?? {}),
__path: path,

View File

@@ -5,7 +5,6 @@ import { clientRouter } from './routers/client';
import { dashboardRouter } from './routers/dashboard';
import { eventRouter } from './routers/event';
import { importRouter } from './routers/import';
import { insightRouter } from './routers/insight';
import { integrationRouter } from './routers/integration';
import { notificationRouter } from './routers/notification';
import { onboardingRouter } from './routers/onboarding';
@@ -48,7 +47,6 @@ export const appRouter = createTRPCRouter({
overview: overviewRouter,
realtime: realtimeRouter,
chat: chatRouter,
insight: insightRouter,
});
// export type definition of API

View File

@@ -10,10 +10,12 @@ import {
convertClickhouseDateToJs,
db,
eventService,
formatClickhouseDate,
getChartStartEndDate,
getConversionEventNames,
getEventList,
getEventMetasCached,
getEvents,
getSettingsForProject,
overviewService,
sessionService,
@@ -141,7 +143,6 @@ export const eventRouter = createTRPCRouter({
path: columnVisibility?.name ?? true,
duration: columnVisibility?.name ?? true,
projectId: false,
revenue: true,
},
});
@@ -220,7 +221,6 @@ export const eventRouter = createTRPCRouter({
path: columnVisibility?.name ?? true,
duration: columnVisibility?.name ?? true,
projectId: false,
revenue: true,
},
custom: (sb) => {
sb.where.name = `name IN (${filteredConversions.map((event) => sqlstring.escape(event.name)).join(',')})`;

View File

@@ -1,102 +0,0 @@
import { db } from '@openpanel/db';
import { z } from 'zod';
import { getProjectAccess } from '../access';
import { TRPCAccessError } from '../errors';
import { createTRPCRouter, protectedProcedure } from '../trpc';
export const insightRouter = createTRPCRouter({
list: protectedProcedure
.input(
z.object({
projectId: z.string(),
limit: z.number().min(1).max(100).optional().default(50),
}),
)
.query(async ({ input: { projectId, limit }, ctx }) => {
const access = await getProjectAccess({
userId: ctx.session.userId,
projectId,
});
if (!access) {
throw TRPCAccessError('You do not have access to this project');
}
// Fetch more insights than needed to account for deduplication
const allInsights = await db.projectInsight.findMany({
where: {
projectId,
state: 'active',
moduleKey: {
notIn: ['page-trends', 'entry-pages'],
},
},
orderBy: {
impactScore: 'desc',
},
take: limit * 3, // Fetch 3x to account for deduplication
});
// WindowKind priority: yesterday (1) > rolling_7d (2) > rolling_30d (3)
const windowKindPriority: Record<string, number> = {
yesterday: 1,
rolling_7d: 2,
rolling_30d: 3,
};
// Group by moduleKey + dimensionKey, keep only highest priority windowKind
const deduplicated = new Map<string, (typeof allInsights)[0]>();
for (const insight of allInsights) {
const key = `${insight.moduleKey}:${insight.dimensionKey}`;
const existing = deduplicated.get(key);
const currentPriority = windowKindPriority[insight.windowKind] ?? 999;
const existingPriority = existing
? (windowKindPriority[existing.windowKind] ?? 999)
: 999;
// Keep if no existing, or if current has higher priority (lower number)
if (!existing || currentPriority < existingPriority) {
deduplicated.set(key, insight);
}
}
// Convert back to array, sort by impactScore, and limit
const insights = Array.from(deduplicated.values())
.sort((a, b) => (b.impactScore ?? 0) - (a.impactScore ?? 0))
.slice(0, limit)
.map(({ impactScore, ...rest }) => rest); // Remove impactScore from response
return insights;
}),
listAll: protectedProcedure
.input(
z.object({
projectId: z.string(),
limit: z.number().min(1).max(500).optional().default(200),
}),
)
.query(async ({ input: { projectId, limit }, ctx }) => {
const access = await getProjectAccess({
userId: ctx.session.userId,
projectId,
});
if (!access) {
throw TRPCAccessError('You do not have access to this project');
}
const insights = await db.projectInsight.findMany({
where: {
projectId,
state: 'active',
},
orderBy: {
impactScore: 'desc',
},
take: limit,
});
return insights;
}),
});

View File

@@ -1,3 +1,2 @@
export * from './src/index';
export * from './src/types.validation';
export * from './src/types.insights';

View File

@@ -553,5 +553,3 @@ export const zCreateImport = z.object({
});
export type ICreateImport = z.infer<typeof zCreateImport>;
export * from './types.insights';

View File

@@ -1,43 +0,0 @@
export type InsightMetricKey = 'sessions' | 'pageviews' | 'share';
export type InsightMetricUnit = 'count' | 'ratio';
export interface InsightMetricEntry {
current: number;
compare: number;
delta: number;
changePct: number | null;
direction: 'up' | 'down' | 'flat';
unit: InsightMetricUnit;
}
export interface InsightDimension {
key: string;
value: string;
displayName?: string;
}
export interface InsightExtra {
[key: string]: unknown;
currentShare?: number;
compareShare?: number;
shareShiftPp?: number;
isNew?: boolean;
isGone?: boolean;
}
/**
* Shared payload shape for insights cards. This is embedded in DB rows and
* shipped to the frontend, so it must remain backwards compatible.
*/
export interface InsightPayload {
kind?: 'insight_v1';
dimensions: InsightDimension[];
primaryMetric: InsightMetricKey;
metrics: Partial<Record<InsightMetricKey, InsightMetricEntry>>;
/**
* Module-specific extra data.
*/
extra?: Record<string, unknown>;
}

2635
pnpm-lock.yaml generated

File diff suppressed because it is too large Load Diff

View File

@@ -7,10 +7,10 @@ packages:
# Define a catalog of version ranges.
catalog:
zod: ^3.24.2
react: ^19.2.3
"@types/react": ^19.2.3
"react-dom": ^19.2.3
"@types/react-dom": ^19.2.3
react: ^19.0.0
"@types/react": ^19.0.0
"react-dom": ^19.0.0
"@types/react-dom": ^19.0.0
"@types/node": ^24.7.1
typescript: ^5.9.3
groupmq: 1.1.1-next.2

View File

@@ -1,12 +1,12 @@
version: "3"
version: '3'
services:
op-proxy:
image: caddy:2-alpine
restart: always
ports:
- "80:80"
- "443:443"
- '80:80'
- '443:443'
volumes:
- op-proxy-data:/data
- op-proxy-config:/config
@@ -28,7 +28,7 @@ services:
volumes:
- op-db-data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
test: [ 'CMD-SHELL', 'pg_isready -U postgres' ]
interval: 10s
timeout: 5s
retries: 5
@@ -49,9 +49,9 @@ services:
restart: always
volumes:
- op-kv-data:/data
command: ["redis-server", "--maxmemory-policy", "noeviction"]
command: [ 'redis-server', '--maxmemory-policy', 'noeviction' ]
healthcheck:
test: ["CMD-SHELL", "redis-cli ping"]
test: [ 'CMD-SHELL', 'redis-cli ping' ]
interval: 10s
timeout: 5s
retries: 5
@@ -65,7 +65,7 @@ services:
# - 6379:6379
op-ch:
image: clickhouse/clickhouse-server:25.10.2.65
image: clickhouse/clickhouse-server:24.3.2-alpine
restart: always
volumes:
- op-ch-data:/var/lib/clickhouse
@@ -73,10 +73,8 @@ services:
- ./clickhouse/clickhouse-config.xml:/etc/clickhouse-server/config.d/op-config.xml:ro
- ./clickhouse/clickhouse-user-config.xml:/etc/clickhouse-server/users.d/op-user-config.xml:ro
- ./clickhouse/init-db.sh:/docker-entrypoint-initdb.d/init-db.sh:ro
environment:
- CLICKHOUSE_SKIP_USER_SETUP=1
healthcheck:
test: ["CMD-SHELL", 'clickhouse-client --query "SELECT 1"']
test: [ 'CMD-SHELL', 'clickhouse-client --query "SELECT 1"' ]
interval: 10s
timeout: 5s
retries: 5
@@ -91,7 +89,7 @@ services:
max-file: "3"
op-api:
image: lindesvard/openpanel-api:2.0.0
image: lindesvard/openpanel-api:latest
restart: always
command: >
sh -c "
@@ -101,7 +99,7 @@ services:
pnpm start
"
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:3000/healthcheck || exit 1"]
test: [ 'CMD-SHELL', 'curl -f http://localhost:3000/healthcheck || exit 1' ]
interval: 10s
timeout: 5s
retries: 5
@@ -121,7 +119,7 @@ services:
max-file: "3"
op-dashboard:
image: lindesvard/openpanel-dashboard:2.0.0
image: lindesvard/openpanel-dashboard:latest
restart: always
depends_on:
op-api:
@@ -129,8 +127,7 @@ services:
env_file:
- .env
healthcheck:
test:
["CMD-SHELL", "curl -f http://localhost:3000/api/healthcheck || exit 1"]
test: [ 'CMD-SHELL', 'curl -f http://localhost:3000/api/healthcheck || exit 1' ]
interval: 10s
timeout: 5s
retries: 5
@@ -141,7 +138,7 @@ services:
max-file: "3"
op-worker:
image: lindesvard/openpanel-worker:2.0.0
image: lindesvard/openpanel-worker:latest
restart: always
depends_on:
op-api:
@@ -149,7 +146,7 @@ services:
env_file:
- .env
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:3000/healthcheck || exit 1"]
test: [ 'CMD-SHELL', 'curl -f http://localhost:3000/healthcheck || exit 1' ]
interval: 10s
timeout: 5s
retries: 5
@@ -174,4 +171,4 @@ volumes:
op-proxy-data:
driver: local
op-proxy-config:
driver: local
driver: local

Some files were not shown because too many files have changed in this diff Show More