Compare commits
1 Commits
feature/in
...
feature/gu
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bc3d7b7ea8 |
@@ -98,10 +98,6 @@ You can find the how to [here](https://openpanel.dev/docs/self-hosting/self-host
|
||||
### Start
|
||||
|
||||
```bash
|
||||
pnpm install
|
||||
cp .env.example .env
|
||||
echo "API_URL=http://localhost:3333" > apps/start/.env
|
||||
|
||||
pnpm dock:up
|
||||
pnpm codegen
|
||||
pnpm migrate:deploy # once to setup the db
|
||||
@@ -114,4 +110,4 @@ You can now access the following:
|
||||
- API: https://api.localhost:3333
|
||||
- Bullboard (queue): http://localhost:9999
|
||||
- `pnpm dock:ch` to access clickhouse terminal
|
||||
- `pnpm dock:redis` to access redis terminal
|
||||
- `pnpm dock:redis` to access redis terminal
|
||||
@@ -1,7 +1,6 @@
|
||||
import type { FastifyReply, FastifyRequest } from 'fastify';
|
||||
import { assocPath, pathOr, pick } from 'ramda';
|
||||
|
||||
import { HttpError } from '@/utils/errors';
|
||||
import { generateId, slug } from '@openpanel/common';
|
||||
import { generateDeviceId, parseUserAgent } from '@openpanel/common/server';
|
||||
import { getProfileById, getSalts, upsertProfile } from '@openpanel/db';
|
||||
@@ -188,16 +187,9 @@ export async function handler(
|
||||
break;
|
||||
}
|
||||
case 'identify': {
|
||||
const payload = request.body.payload;
|
||||
const geo = await getGeoLocation(ip);
|
||||
if (!payload.profileId) {
|
||||
throw new HttpError('Missing profileId', {
|
||||
status: 400,
|
||||
});
|
||||
}
|
||||
|
||||
await identify({
|
||||
payload,
|
||||
payload: request.body.payload,
|
||||
projectId,
|
||||
geo,
|
||||
ua,
|
||||
|
||||
@@ -216,4 +216,3 @@ tracker = OpenPanel::SDK::Tracker.new(
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
|
||||
@@ -1,101 +0,0 @@
|
||||
---
|
||||
title: Migration from v1 to v2
|
||||
description: Finally we feel ready to release v2 for all self-hostings. This is a big one!
|
||||
---
|
||||
|
||||
## What's New in v2
|
||||
|
||||
- **Redesigned dashboard** - New UI built with Tanstack
|
||||
- **Revenue tracking** - Track revenue alongside your analytics
|
||||
- **Sessions** - View individual user sessions
|
||||
- **Real-time view** - Live event stream
|
||||
- **Customizable dashboards** - Grafana-style widget layouts
|
||||
- **Improved report builder** - Faster and more flexible
|
||||
- **General improvements** - We have also made a bunch of bug fixes, minor improvements and much more
|
||||
|
||||
## Migrating from v1
|
||||
|
||||
### Ensure you're on the self-hosting branch
|
||||
|
||||
Sometimes we add new helper scripts and what not. Always make sure you're on the latest commit before continuing.
|
||||
|
||||
```bash
|
||||
cd ./self-hosting
|
||||
git fetch origin
|
||||
git checkout self-hosting
|
||||
git pull origin self-hosting
|
||||
```
|
||||
|
||||
### Envs
|
||||
|
||||
Since we have migrated to tanstack from nextjs we first need to update our envs. We have added a dedicated page for the [environment variables here](/docs/self-hosting/environment-variables).
|
||||
|
||||
```js title=".env"
|
||||
NEXT_PUBLIC_DASHBOARD_URL="..." // [!code --]
|
||||
NEXT_PUBLIC_API_URL="..." // [!code --]
|
||||
NEXT_PUBLIC_SELF_HOSTED="..." // [!code --]
|
||||
|
||||
DASHBOARD_URL="..." // [!code ++]
|
||||
API_URL="..." // [!code ++]
|
||||
SELF_HOSTED="..." // [!code ++]
|
||||
```
|
||||
|
||||
### Clickhouse 24 -> 25
|
||||
|
||||
We have updated Clickhouse to 25, this is important to not skip, otherwise your OpenPanel instance wont work.
|
||||
|
||||
You should edit your `./self-hosting/docker-compose.yml`
|
||||
|
||||
```js title="./self-hosting/docker-compose.yml"
|
||||
services:
|
||||
op-ch:
|
||||
image: clickhouse/clickhouse-server:24.3.2-alpine // [!code --]
|
||||
image: clickhouse/clickhouse-server:25.10.2.65 // [!code ++]
|
||||
```
|
||||
|
||||
Since version 25 clickhouse enabled default user setup, this means that we need to disable it to avoid connection issues. With this setting we can still access our clickhouse instance (internally) without having a user.
|
||||
|
||||
```
|
||||
services:
|
||||
op-ch:
|
||||
environment:
|
||||
- CLICKHOUSE_SKIP_USER_SETUP=1
|
||||
```
|
||||
|
||||
### Use our latest docker images
|
||||
|
||||
Last thing to do is to start using our latest docker images.
|
||||
|
||||
> Note: Before you might have been using the latest tag, which is not recommended. Change it to the actual latest version instead.
|
||||
|
||||
```js title="./self-hosting/docker-compose.yml"
|
||||
services:
|
||||
op-api:
|
||||
image: lindesvard/openpanel-api:latest // [!code --]
|
||||
image: lindesvard/openpanel-api:2.0.0 // [!code ++]
|
||||
|
||||
op-worker:
|
||||
image: lindesvard/openpanel-worker:latest // [!code --]
|
||||
image: lindesvard/openpanel-worker:2.0.0 // [!code ++]
|
||||
|
||||
op-dashboard:
|
||||
image: lindesvard/openpanel-dashboard:latest // [!code --]
|
||||
image: lindesvard/openpanel-dashboard:2.0.0 // [!code ++]
|
||||
```
|
||||
|
||||
### Done?
|
||||
|
||||
When you're done with above steps you should need to restart all services. This will take quite some time depending on your hardware and how many events you have. Since we have made significant changes to the database schema and data we need to run migrations.
|
||||
|
||||
```bash
|
||||
./stop
|
||||
./start
|
||||
```
|
||||
|
||||
## Using Coolify?
|
||||
|
||||
If you're using Coolify and running OpenPanel v1 you'll need to apply the above changes. You can take a look at our [Coolify PR](https://github.com/coollabsio/coolify/pull/7653) which shows what you need to change.
|
||||
|
||||
## Any issues with migrations?
|
||||
|
||||
If you stumble upon any issues during migrations, please reach out to us on [Discord](https://discord.gg/openpanel) and we'll try our best to help you out.
|
||||
@@ -3,20 +3,6 @@ title: Changelog for self-hosting
|
||||
description: This is a list of changes that have been made to the self-hosting setup.
|
||||
---
|
||||
|
||||
## 2.0.0
|
||||
|
||||
We have released the first stable version of OpenPanel v2. This is a big one!
|
||||
|
||||
Read more about it in our [migration guide](/docs/migration/migrate-v1-to-v2).
|
||||
|
||||
TLDR;
|
||||
|
||||
- Clickhouse upgraded from 24.3.2-alpine to 25.10.2.65
|
||||
- Add `CLICKHOUSE_SKIP_USER_SETUP=1` to op-ch service
|
||||
- `NEXT_PUBLIC_DASHBOARD_URL` -> `DASHBOARD_URL`
|
||||
- `NEXT_PUBLIC_API_URL` -> `API_URL`
|
||||
- `NEXT_PUBLIC_SELF_HOSTED` -> `SELF_HOSTED`
|
||||
|
||||
## 1.2.0
|
||||
|
||||
We have renamed `SELF_HOSTED` to `NEXT_PUBLIC_SELF_HOSTED`. It's important to rename this env before your upgrade to this version.
|
||||
@@ -44,7 +30,7 @@ If you upgrading from a previous version, you'll need to edit your `.env` file i
|
||||
|
||||
### Removed Clickhouse Keeper
|
||||
|
||||
In 0.0.6 we introduced a cluster mode for Clickhouse. This was a mistake and we have removed it.
|
||||
In 0.0.6 we introduced a cluster mode for Clickhouse. This was a misstake and we have removed it.
|
||||
|
||||
Remove op-zk from services and volumes
|
||||
|
||||
|
||||
@@ -109,8 +109,8 @@ Coolify automatically handles these variables:
|
||||
- `DATABASE_URL`: PostgreSQL connection string
|
||||
- `REDIS_URL`: Redis connection string
|
||||
- `CLICKHOUSE_URL`: ClickHouse connection string
|
||||
- `API_URL`: API endpoint URL (set via `SERVICE_FQDN_OPAPI`)
|
||||
- `DASHBOARD_URL`: Dashboard URL (set via `SERVICE_FQDN_OPDASHBOARD`)
|
||||
- `NEXT_PUBLIC_API_URL`: API endpoint URL (set via `SERVICE_FQDN_OPAPI`)
|
||||
- `NEXT_PUBLIC_DASHBOARD_URL`: Dashboard URL (set via `SERVICE_FQDN_OPDASHBOARD`)
|
||||
- `COOKIE_SECRET`: Automatically generated secret
|
||||
|
||||
You can configure optional variables like `ALLOW_REGISTRATION`, `RESEND_API_KEY`, `OPENAI_API_KEY`, etc. through Coolify's environment variable interface.
|
||||
|
||||
@@ -126,7 +126,7 @@ If you want to use specific image versions, edit the `docker-compose.yml` file a
|
||||
|
||||
```yaml
|
||||
op-api:
|
||||
image: lindesvard/openpanel-api:2.0.0 # Specify version
|
||||
image: lindesvard/openpanel-api:v1.0.0 # Specify version
|
||||
```
|
||||
|
||||
### Scaling Workers
|
||||
|
||||
@@ -54,8 +54,8 @@ Edit the `.env` file or environment variables in Dokploy. You **must** set these
|
||||
|
||||
```bash
|
||||
# Required: Set these to your actual domain
|
||||
API_URL=https://yourdomain.com/api
|
||||
DASHBOARD_URL=https://yourdomain.com
|
||||
NEXT_PUBLIC_API_URL=https://yourdomain.com/api
|
||||
NEXT_PUBLIC_DASHBOARD_URL=https://yourdomain.com
|
||||
|
||||
# Database Configuration (automatically set by Dokploy)
|
||||
OPENPANEL_POSTGRES_DB=openpanel-db
|
||||
@@ -71,7 +71,7 @@ OPENPANEL_EMAIL_SENDER=noreply@yourdomain.com
|
||||
```
|
||||
|
||||
<Callout type="warn">
|
||||
⚠️ **Critical**: Unlike Coolify, Dokploy does not support `SERVICE_FQDN_*` variables. You **must** hardcode `API_URL` and `DASHBOARD_URL` with your actual domain values.
|
||||
⚠️ **Critical**: Unlike Coolify, Dokploy does not support `SERVICE_FQDN_*` variables. You **must** hardcode `NEXT_PUBLIC_API_URL` and `NEXT_PUBLIC_DASHBOARD_URL` with your actual domain values.
|
||||
</Callout>
|
||||
</Step>
|
||||
|
||||
@@ -133,8 +133,8 @@ If you're using Cloudflare in front of Dokploy, remember to purge the Cloudflare
|
||||
|
||||
For Dokploy, you **must** hardcode these variables (unlike Coolify, Dokploy doesn't support `SERVICE_FQDN_*` variables):
|
||||
|
||||
- `API_URL` - Full API URL (e.g., `https://analytics.example.com/api`)
|
||||
- `DASHBOARD_URL` - Full Dashboard URL (e.g., `https://analytics.example.com`)
|
||||
- `NEXT_PUBLIC_API_URL` - Full API URL (e.g., `https://analytics.example.com/api`)
|
||||
- `NEXT_PUBLIC_DASHBOARD_URL` - Full Dashboard URL (e.g., `https://analytics.example.com`)
|
||||
|
||||
Dokploy automatically sets:
|
||||
- `OPENPANEL_POSTGRES_DB` - PostgreSQL database name
|
||||
@@ -166,9 +166,9 @@ If API requests fail after deployment:
|
||||
|
||||
1. **Verify environment variables**:
|
||||
```bash
|
||||
# Check that API_URL is set correctly
|
||||
docker exec <op-api-container> env | grep API_URL
|
||||
docker exec <op-dashboard-container> env | grep API_URL
|
||||
# Check that NEXT_PUBLIC_API_URL is set correctly
|
||||
docker exec <op-api-container> env | grep NEXT_PUBLIC_API_URL
|
||||
docker exec <op-dashboard-container> env | grep NEXT_PUBLIC_API_URL
|
||||
```
|
||||
|
||||
2. **Check "Strip external path" setting**:
|
||||
@@ -188,7 +188,7 @@ If account creation fails:
|
||||
# In Dokploy, view logs for op-api service
|
||||
```
|
||||
|
||||
2. Verify `API_URL` matches your domain:
|
||||
2. Verify `NEXT_PUBLIC_API_URL` matches your domain:
|
||||
- Should be `https://yourdomain.com/api`
|
||||
- Not `http://localhost:3000` or similar
|
||||
|
||||
@@ -240,7 +240,7 @@ The Dokploy template differs from Coolify in these ways:
|
||||
|
||||
1. **Environment Variables**:
|
||||
- Dokploy does not support `SERVICE_FQDN_*` variables
|
||||
- Must hardcode `API_URL` and `DASHBOARD_URL`
|
||||
- Must hardcode `NEXT_PUBLIC_API_URL` and `NEXT_PUBLIC_DASHBOARD_URL`
|
||||
|
||||
2. **Domain Configuration**:
|
||||
- Must manually configure domain paths
|
||||
|
||||
@@ -116,7 +116,7 @@ Remove `convert_any_join` from ClickHouse settings. Used for compatibility with
|
||||
|
||||
## Application URLs
|
||||
|
||||
### API_URL
|
||||
### NEXT_PUBLIC_API_URL
|
||||
|
||||
**Type**: `string`
|
||||
**Required**: Yes
|
||||
@@ -126,10 +126,10 @@ Public API URL exposed to the browser. Used by the dashboard frontend and API se
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
API_URL=https://analytics.example.com/api
|
||||
NEXT_PUBLIC_API_URL=https://analytics.example.com/api
|
||||
```
|
||||
|
||||
### DASHBOARD_URL
|
||||
### NEXT_PUBLIC_DASHBOARD_URL
|
||||
|
||||
**Type**: `string`
|
||||
**Required**: Yes
|
||||
@@ -139,7 +139,7 @@ Public dashboard URL exposed to the browser. Used by the dashboard frontend and
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
DASHBOARD_URL=https://analytics.example.com
|
||||
NEXT_PUBLIC_DASHBOARD_URL=https://analytics.example.com
|
||||
```
|
||||
|
||||
### API_CORS_ORIGINS
|
||||
@@ -368,7 +368,7 @@ SLACK_STATE_SECRET=your-state-secret
|
||||
|
||||
## Self-hosting
|
||||
|
||||
### SELF_HOSTED
|
||||
### NEXT_PUBLIC_SELF_HOSTED
|
||||
|
||||
**Type**: `boolean`
|
||||
**Required**: No
|
||||
@@ -378,7 +378,7 @@ Enable self-hosted mode. Set to `true` or `1` to enable self-hosting features. U
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
SELF_HOSTED=true
|
||||
NEXT_PUBLIC_SELF_HOSTED=true
|
||||
```
|
||||
|
||||
## Worker & Queue
|
||||
@@ -784,8 +784,8 @@ For a basic self-hosted installation, these variables are required:
|
||||
- `DATABASE_URL` - PostgreSQL connection
|
||||
- `REDIS_URL` - Redis connection
|
||||
- `CLICKHOUSE_URL` - ClickHouse connection
|
||||
- `API_URL` - API endpoint URL
|
||||
- `DASHBOARD_URL` - Dashboard URL
|
||||
- `NEXT_PUBLIC_API_URL` - API endpoint URL
|
||||
- `NEXT_PUBLIC_DASHBOARD_URL` - Dashboard URL
|
||||
- `COOKIE_SECRET` - Session encryption secret
|
||||
|
||||
### Optional but Recommended
|
||||
|
||||
@@ -163,7 +163,7 @@ For complete AI configuration details, see the [Environment Variables documentat
|
||||
|
||||
If you use a managed Redis service, you may need to set the `notify-keyspace-events` manually.
|
||||
|
||||
Without this setting we won't be able to listen for expired keys which we use for calculating currently active visitors.
|
||||
Without this setting we wont be able to listen for expired keys which we use for caluclating currently active vistors.
|
||||
|
||||
> You will see a warning in the logs if this needs to be set manually.
|
||||
|
||||
|
||||
@@ -6,7 +6,6 @@ difficulty: intermediate
|
||||
timeToComplete: 10
|
||||
date: 2025-12-15
|
||||
lastUpdated: 2025-12-15
|
||||
team: OpenPanel Team
|
||||
steps:
|
||||
- name: "Add the dependency"
|
||||
anchor: "install"
|
||||
|
||||
@@ -6,7 +6,6 @@ difficulty: beginner
|
||||
timeToComplete: 7
|
||||
date: 2025-12-15
|
||||
lastUpdated: 2025-12-15
|
||||
team: OpenPanel Team
|
||||
steps:
|
||||
- name: "Install the SDK"
|
||||
anchor: "install"
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -121,7 +121,7 @@ export default async function Page({
|
||||
/>
|
||||
<div className="row gap-4 items-center mt-8">
|
||||
<div className="size-10 center-center bg-black rounded-full">
|
||||
{author?.image ? (
|
||||
{author.image ? (
|
||||
<Image
|
||||
className="size-10 object-cover rounded-full"
|
||||
src={author.image}
|
||||
@@ -134,7 +134,7 @@ export default async function Page({
|
||||
)}
|
||||
</div>
|
||||
<div className="col flex-1">
|
||||
<p className="font-medium">{author?.name || 'OpenPanel Team'}</p>
|
||||
<p className="font-medium">{author.name}</p>
|
||||
<div className="row gap-4 items-center">
|
||||
<p className="text-muted-foreground text-sm">
|
||||
{guide?.data.date.toLocaleDateString()}
|
||||
|
||||
@@ -27,6 +27,10 @@ interface IPInfoResponse {
|
||||
latitude: number | undefined;
|
||||
longitude: number | undefined;
|
||||
};
|
||||
isp: string | null;
|
||||
asn: string | null;
|
||||
organization: string | null;
|
||||
hostname: string | null;
|
||||
isLocalhost: boolean;
|
||||
isPrivate: boolean;
|
||||
}
|
||||
@@ -86,6 +90,84 @@ function isPrivateIP(ip: string): boolean {
|
||||
return false;
|
||||
}
|
||||
|
||||
async function getIPInfo(ip: string): Promise<IPInfo> {
|
||||
if (!ip || ip === '127.0.0.1' || ip === '::1') {
|
||||
return {
|
||||
ip,
|
||||
location: {
|
||||
country: undefined,
|
||||
city: undefined,
|
||||
region: undefined,
|
||||
latitude: undefined,
|
||||
longitude: undefined,
|
||||
},
|
||||
isp: null,
|
||||
asn: null,
|
||||
organization: null,
|
||||
hostname: null,
|
||||
};
|
||||
}
|
||||
|
||||
// Get geolocation
|
||||
const geo = await getGeoLocation(ip);
|
||||
|
||||
// Get ISP/ASN info
|
||||
let isp: string | null = null;
|
||||
let asn: string | null = null;
|
||||
let organization: string | null = null;
|
||||
|
||||
if (!isPrivateIP(ip)) {
|
||||
try {
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), 3000);
|
||||
|
||||
const response = await fetch(
|
||||
`https://ip-api.com/json/${ip}?fields=isp,as,org,query,reverse`,
|
||||
{
|
||||
signal: controller.signal,
|
||||
},
|
||||
);
|
||||
|
||||
clearTimeout(timeout);
|
||||
|
||||
if (response.ok) {
|
||||
const data = await response.json();
|
||||
if (data.status !== 'fail') {
|
||||
isp = data.isp || null;
|
||||
asn = data.as ? `AS${data.as.split(' ')[0]}` : null;
|
||||
organization = data.org || null;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Ignore errors
|
||||
}
|
||||
}
|
||||
|
||||
// Reverse DNS lookup for hostname
|
||||
let hostname: string | null = null;
|
||||
try {
|
||||
const hostnames = await dns.reverse(ip);
|
||||
hostname = hostnames[0] || null;
|
||||
} catch {
|
||||
// Ignore errors
|
||||
}
|
||||
|
||||
return {
|
||||
ip,
|
||||
location: {
|
||||
country: geo.country,
|
||||
city: geo.city,
|
||||
region: geo.region,
|
||||
latitude: geo.latitude,
|
||||
longitude: geo.longitude,
|
||||
},
|
||||
isp,
|
||||
asn,
|
||||
organization,
|
||||
hostname,
|
||||
};
|
||||
}
|
||||
|
||||
export async function GET(request: Request) {
|
||||
const { searchParams } = new URL(request.url);
|
||||
const ipParam = searchParams.get('ip');
|
||||
@@ -127,17 +209,12 @@ export async function GET(request: Request) {
|
||||
}
|
||||
|
||||
try {
|
||||
const geo = await fetch('https://api.openpanel.dev/misc/geo', {
|
||||
headers: request.headers,
|
||||
})
|
||||
.then((res) => res.json())
|
||||
.then((data) => data.selected.geo);
|
||||
const info = await getIPInfo(ipToLookup);
|
||||
const isLocalhost = ipToLookup === '127.0.0.1' || ipToLookup === '::1';
|
||||
const isPrivate = isPrivateIP(ipToLookup);
|
||||
|
||||
const response: IPInfoResponse = {
|
||||
location: geo,
|
||||
ip: ipToLookup,
|
||||
...info,
|
||||
isLocalhost,
|
||||
isPrivate,
|
||||
};
|
||||
|
||||
@@ -12,11 +12,7 @@ import {
|
||||
import type { Metadata } from 'next';
|
||||
import { notFound } from 'next/navigation';
|
||||
|
||||
type PageProps = {
|
||||
params: Promise<{ slug: string[] }>;
|
||||
};
|
||||
|
||||
export default async function Page(props: PageProps) {
|
||||
export default async function Page(props: PageProps<'/docs/[[...slug]]'>) {
|
||||
const params = await props.params;
|
||||
const page = source.getPage(params.slug);
|
||||
if (!page) notFound();
|
||||
@@ -43,7 +39,9 @@ export async function generateStaticParams() {
|
||||
return source.generateParams();
|
||||
}
|
||||
|
||||
export async function generateMetadata(props: PageProps): Promise<Metadata> {
|
||||
export async function generateMetadata(
|
||||
props: PageProps<'/docs/[[...slug]]'>,
|
||||
): Promise<Metadata> {
|
||||
const params = await props.params;
|
||||
const page = source.getPage(params.slug);
|
||||
if (!page) notFound();
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
import { baseOptions } from '@/lib/layout.shared';
|
||||
import { source } from '@/lib/source';
|
||||
import { DocsLayout } from 'fumadocs-ui/layouts/docs';
|
||||
import { baseOptions } from '@/lib/layout.shared';
|
||||
|
||||
export default function Layout({ children }: { children: React.ReactNode }) {
|
||||
export default function Layout({ children }: LayoutProps<'/docs'>) {
|
||||
return (
|
||||
<DocsLayout tree={source.pageTree} {...baseOptions()}>
|
||||
{children}
|
||||
|
||||
@@ -28,7 +28,7 @@ export const viewport: Viewport = {
|
||||
|
||||
export const metadata: Metadata = getRootMetadata();
|
||||
|
||||
export default function Layout({ children }: { children: React.ReactNode }) {
|
||||
export default function Layout({ children }: LayoutProps<'/'>) {
|
||||
return (
|
||||
<html
|
||||
lang="en"
|
||||
|
||||
@@ -84,34 +84,6 @@ async function getOgData(
|
||||
data?.data.description || 'Whooops, could not find this page',
|
||||
};
|
||||
}
|
||||
case 'tools': {
|
||||
if (segments.length > 1) {
|
||||
const tool = segments[1];
|
||||
switch (tool) {
|
||||
case 'ip-lookup':
|
||||
return {
|
||||
title: 'IP Lookup Tool',
|
||||
description:
|
||||
'Find detailed information about any IP address including geolocation, ISP, and network details.',
|
||||
};
|
||||
case 'url-checker':
|
||||
return {
|
||||
title: 'URL Checker',
|
||||
description:
|
||||
'Analyze any website for SEO, social media, technical, and security information. Get comprehensive insights about any URL.',
|
||||
};
|
||||
default:
|
||||
return {
|
||||
title: 'Tools',
|
||||
description: 'Free web tools for developers and website owners',
|
||||
};
|
||||
}
|
||||
}
|
||||
return {
|
||||
title: 'Tools',
|
||||
description: 'Free web tools for developers and website owners',
|
||||
};
|
||||
}
|
||||
default: {
|
||||
const data = await pageSource.getPage(segments);
|
||||
return {
|
||||
|
||||
@@ -1,18 +1,11 @@
|
||||
import { url } from '@/lib/layout.shared';
|
||||
import {
|
||||
articleSource,
|
||||
compareSource,
|
||||
guideSource,
|
||||
pageSource,
|
||||
source,
|
||||
} from '@/lib/source';
|
||||
import { articleSource, compareSource, pageSource, source } from '@/lib/source';
|
||||
import type { MetadataRoute } from 'next';
|
||||
|
||||
export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
|
||||
const articles = await articleSource.getPages();
|
||||
const docs = await source.getPages();
|
||||
const pages = await pageSource.getPages();
|
||||
const guides = await guideSource.getPages();
|
||||
return [
|
||||
{
|
||||
url: url('/'),
|
||||
@@ -56,12 +49,6 @@ export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
|
||||
changeFrequency: 'yearly' as const,
|
||||
priority: 0.5,
|
||||
})),
|
||||
...guides.map((item) => ({
|
||||
url: url(item.url),
|
||||
lastModified: item.data.date,
|
||||
changeFrequency: 'monthly' as const,
|
||||
priority: 0.5,
|
||||
})),
|
||||
...docs.map((item) => ({
|
||||
url: url(item.url),
|
||||
changeFrequency: 'monthly' as const,
|
||||
|
||||
@@ -275,6 +275,49 @@ export default function IPLookupPage() {
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Network Information */}
|
||||
{(result.isp ||
|
||||
result.asn ||
|
||||
result.organization ||
|
||||
result.hostname) && (
|
||||
<div>
|
||||
<h3 className="text-lg font-semibold mb-4 flex items-center gap-2">
|
||||
<Network className="size-5" />
|
||||
Network Information
|
||||
</h3>
|
||||
<div className="grid grid-cols-1 md:grid-cols-2 gap-4">
|
||||
{result.isp && (
|
||||
<InfoCard
|
||||
icon={<Building2 className="size-5" />}
|
||||
label="ISP"
|
||||
value={result.isp}
|
||||
/>
|
||||
)}
|
||||
{result.asn && (
|
||||
<InfoCard
|
||||
icon={<Network className="size-5" />}
|
||||
label="ASN"
|
||||
value={result.asn}
|
||||
/>
|
||||
)}
|
||||
{result.organization && (
|
||||
<InfoCard
|
||||
icon={<Building2 className="size-5" />}
|
||||
label="Organization"
|
||||
value={result.organization}
|
||||
/>
|
||||
)}
|
||||
{result.hostname && (
|
||||
<InfoCard
|
||||
icon={<Server className="size-5" />}
|
||||
label="Hostname"
|
||||
value={result.hostname}
|
||||
/>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Map Preview */}
|
||||
{result.location.latitude && result.location.longitude && (
|
||||
<div>
|
||||
|
||||
@@ -27,7 +27,7 @@ export function useColumns() {
|
||||
accessorKey: 'name',
|
||||
header: 'Name',
|
||||
cell({ row }) {
|
||||
const { name, path, duration, properties, revenue } = row.original;
|
||||
const { name, path, duration, properties } = row.original;
|
||||
const renderName = () => {
|
||||
if (name === 'screen_view') {
|
||||
if (path.includes('/')) {
|
||||
@@ -42,10 +42,6 @@ export function useColumns() {
|
||||
);
|
||||
}
|
||||
|
||||
if (name === 'revenue' && revenue) {
|
||||
return `${name} (${number.currency(revenue / 100)})`;
|
||||
}
|
||||
|
||||
return name.replace(/_/g, ' ');
|
||||
};
|
||||
|
||||
|
||||
@@ -1,229 +0,0 @@
|
||||
import { countries } from '@/translations/countries';
|
||||
import type { RouterOutputs } from '@/trpc/client';
|
||||
import { cn } from '@/utils/cn';
|
||||
import type { InsightPayload } from '@openpanel/validation';
|
||||
import { ArrowDown, ArrowUp, FilterIcon, RotateCcwIcon } from 'lucide-react';
|
||||
import { last } from 'ramda';
|
||||
import { useState } from 'react';
|
||||
import { SerieIcon } from '../report-chart/common/serie-icon';
|
||||
import { Badge } from '../ui/badge';
|
||||
|
||||
function formatWindowKind(windowKind: string): string {
|
||||
switch (windowKind) {
|
||||
case 'yesterday':
|
||||
return 'Yesterday';
|
||||
case 'rolling_7d':
|
||||
return '7 Days';
|
||||
case 'rolling_30d':
|
||||
return '30 Days';
|
||||
}
|
||||
return windowKind;
|
||||
}
|
||||
|
||||
interface InsightCardProps {
|
||||
insight: RouterOutputs['insight']['list'][number];
|
||||
className?: string;
|
||||
onFilter?: () => void;
|
||||
}
|
||||
|
||||
export function InsightCard({
|
||||
insight,
|
||||
className,
|
||||
onFilter,
|
||||
}: InsightCardProps) {
|
||||
const payload = insight.payload;
|
||||
const dimensions = payload?.dimensions;
|
||||
const availableMetrics = Object.entries(payload?.metrics ?? {});
|
||||
|
||||
// Pick what to display: prefer share if available (geo/devices), else primaryMetric
|
||||
const [metricIndex, setMetricIndex] = useState(
|
||||
availableMetrics.findIndex(([key]) => key === payload?.primaryMetric),
|
||||
);
|
||||
const currentMetricKey = availableMetrics[metricIndex][0];
|
||||
const currentMetricEntry = availableMetrics[metricIndex][1];
|
||||
|
||||
const metricUnit = currentMetricEntry?.unit;
|
||||
const currentValue = currentMetricEntry?.current ?? null;
|
||||
const compareValue = currentMetricEntry?.compare ?? null;
|
||||
|
||||
const direction = currentMetricEntry?.direction ?? 'flat';
|
||||
const isIncrease = direction === 'up';
|
||||
const isDecrease = direction === 'down';
|
||||
|
||||
const deltaText =
|
||||
metricUnit === 'ratio'
|
||||
? `${Math.abs((currentMetricEntry?.delta ?? 0) * 100).toFixed(1)}pp`
|
||||
: `${Math.abs((currentMetricEntry?.changePct ?? 0) * 100).toFixed(1)}%`;
|
||||
|
||||
// Format metric values
|
||||
const formatValue = (value: number | null): string => {
|
||||
if (value == null) return '-';
|
||||
if (metricUnit === 'ratio') return `${(value * 100).toFixed(1)}%`;
|
||||
return Math.round(value).toLocaleString();
|
||||
};
|
||||
|
||||
// Get the metric label
|
||||
const metricKeyToLabel = (key: string) =>
|
||||
key === 'share' ? 'Share' : key === 'pageviews' ? 'Pageviews' : 'Sessions';
|
||||
|
||||
const metricLabel = metricKeyToLabel(currentMetricKey);
|
||||
|
||||
const renderTitle = () => {
|
||||
if (
|
||||
dimensions[0]?.key === 'country' ||
|
||||
dimensions[0]?.key === 'referrer_name' ||
|
||||
dimensions[0]?.key === 'device'
|
||||
) {
|
||||
return (
|
||||
<span className="capitalize flex items-center gap-2">
|
||||
<SerieIcon name={dimensions[0]?.value} /> {insight.displayName}
|
||||
</span>
|
||||
);
|
||||
}
|
||||
|
||||
if (insight.displayName.startsWith('http')) {
|
||||
return (
|
||||
<span className="flex items-center gap-2">
|
||||
<SerieIcon
|
||||
name={dimensions[0]?.displayName ?? dimensions[0]?.value}
|
||||
/>
|
||||
<span className="line-clamp-2">{dimensions[1]?.displayName}</span>
|
||||
</span>
|
||||
);
|
||||
}
|
||||
|
||||
return insight.displayName;
|
||||
};
|
||||
|
||||
return (
|
||||
<div
|
||||
className={cn(
|
||||
'card p-4 h-full flex flex-col hover:bg-def-50 transition-colors group/card',
|
||||
className,
|
||||
)}
|
||||
>
|
||||
<div
|
||||
className={cn(
|
||||
'row justify-between h-4 items-center',
|
||||
onFilter && 'group-hover/card:hidden',
|
||||
)}
|
||||
>
|
||||
<Badge variant="outline" className="-ml-2">
|
||||
{formatWindowKind(insight.windowKind)}
|
||||
</Badge>
|
||||
{/* Severity: subtle dot instead of big pill */}
|
||||
{insight.severityBand && (
|
||||
<div className="flex items-center gap-1 shrink-0">
|
||||
<span
|
||||
className={cn(
|
||||
'h-2 w-2 rounded-full',
|
||||
insight.severityBand === 'severe'
|
||||
? 'bg-red-500'
|
||||
: insight.severityBand === 'moderate'
|
||||
? 'bg-yellow-500'
|
||||
: 'bg-blue-500',
|
||||
)}
|
||||
/>
|
||||
<span className="text-[11px] text-muted-foreground capitalize">
|
||||
{insight.severityBand}
|
||||
</span>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
{onFilter && (
|
||||
<div className="row group-hover/card:flex hidden h-4 justify-between gap-2">
|
||||
{availableMetrics.length > 1 ? (
|
||||
<button
|
||||
type="button"
|
||||
className="text-[11px] text-muted-foreground capitalize flex items-center gap-1"
|
||||
onClick={() =>
|
||||
setMetricIndex((metricIndex + 1) % availableMetrics.length)
|
||||
}
|
||||
>
|
||||
<RotateCcwIcon className="size-2" />
|
||||
Show{' '}
|
||||
{metricKeyToLabel(
|
||||
availableMetrics[
|
||||
(metricIndex + 1) % availableMetrics.length
|
||||
][0],
|
||||
)}
|
||||
</button>
|
||||
) : (
|
||||
<div />
|
||||
)}
|
||||
<button
|
||||
type="button"
|
||||
className="text-[11px] text-muted-foreground capitalize flex items-center gap-1"
|
||||
onClick={onFilter}
|
||||
>
|
||||
Filter <FilterIcon className="size-2" />
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
<div className="font-semibold text-sm leading-snug line-clamp-2 mt-2">
|
||||
{renderTitle()}
|
||||
</div>
|
||||
|
||||
{/* Metric row */}
|
||||
<div className="mt-auto pt-2">
|
||||
<div className="flex items-end justify-between gap-3">
|
||||
<div className="min-w-0">
|
||||
<div className="text-[11px] text-muted-foreground mb-1">
|
||||
{metricLabel}
|
||||
</div>
|
||||
|
||||
<div className="col gap-1">
|
||||
<div className="text-2xl font-semibold tracking-tight">
|
||||
{formatValue(currentValue)}
|
||||
</div>
|
||||
|
||||
{/* Inline compare, smaller */}
|
||||
{compareValue != null && (
|
||||
<div className="text-xs text-muted-foreground">
|
||||
vs {formatValue(compareValue)}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Delta chip */}
|
||||
<DeltaChip
|
||||
isIncrease={isIncrease}
|
||||
isDecrease={isDecrease}
|
||||
deltaText={deltaText}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function DeltaChip({
|
||||
isIncrease,
|
||||
isDecrease,
|
||||
deltaText,
|
||||
}: {
|
||||
isIncrease: boolean;
|
||||
isDecrease: boolean;
|
||||
deltaText: string;
|
||||
}) {
|
||||
return (
|
||||
<div
|
||||
className={cn(
|
||||
'flex items-center gap-1 rounded-full px-2 py-1 text-sm font-semibold',
|
||||
isIncrease
|
||||
? 'bg-emerald-500/10 text-emerald-600 dark:text-emerald-400'
|
||||
: isDecrease
|
||||
? 'bg-red-500/10 text-red-600 dark:text-red-400'
|
||||
: 'bg-muted text-muted-foreground',
|
||||
)}
|
||||
>
|
||||
{isIncrease ? (
|
||||
<ArrowUp size={16} className="shrink-0" />
|
||||
) : isDecrease ? (
|
||||
<ArrowDown size={16} className="shrink-0" />
|
||||
) : null}
|
||||
<span>{deltaText}</span>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -1,75 +0,0 @@
|
||||
import { useEventQueryFilters } from '@/hooks/use-event-query-filters';
|
||||
import { useTRPC } from '@/integrations/trpc/react';
|
||||
import { useQuery } from '@tanstack/react-query';
|
||||
import { InsightCard } from '../insights/insight-card';
|
||||
import { Skeleton } from '../skeleton';
|
||||
import {
|
||||
Carousel,
|
||||
CarouselContent,
|
||||
CarouselItem,
|
||||
CarouselNext,
|
||||
CarouselPrevious,
|
||||
} from '../ui/carousel';
|
||||
|
||||
interface OverviewInsightsProps {
|
||||
projectId: string;
|
||||
}
|
||||
|
||||
export default function OverviewInsights({ projectId }: OverviewInsightsProps) {
|
||||
const trpc = useTRPC();
|
||||
const [filters, setFilter] = useEventQueryFilters();
|
||||
const { data: insights, isLoading } = useQuery(
|
||||
trpc.insight.list.queryOptions({
|
||||
projectId,
|
||||
limit: 20,
|
||||
}),
|
||||
);
|
||||
|
||||
if (isLoading) {
|
||||
const keys = Array.from({ length: 4 }, (_, i) => `insight-skeleton-${i}`);
|
||||
return (
|
||||
<div className="col-span-6">
|
||||
<Carousel opts={{ align: 'start' }} className="w-full">
|
||||
<CarouselContent className="-ml-4">
|
||||
{keys.map((key) => (
|
||||
<CarouselItem
|
||||
key={key}
|
||||
className="pl-4 basis-full sm:basis-1/2 lg:basis-1/4"
|
||||
>
|
||||
<Skeleton className="h-36 w-full" />
|
||||
</CarouselItem>
|
||||
))}
|
||||
</CarouselContent>
|
||||
</Carousel>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
if (!insights || insights.length === 0) return null;
|
||||
|
||||
return (
|
||||
<div className="col-span-6 -mx-4">
|
||||
<Carousel opts={{ align: 'start' }} className="w-full group">
|
||||
<CarouselContent className="mr-4">
|
||||
{insights.map((insight) => (
|
||||
<CarouselItem
|
||||
key={insight.id}
|
||||
className="pl-4 basis-full sm:basis-1/2 lg:basis-1/4"
|
||||
>
|
||||
<InsightCard
|
||||
insight={insight}
|
||||
onFilter={() => {
|
||||
insight.payload.dimensions.forEach((dim) => {
|
||||
void setFilter(dim.key, dim.value, 'is');
|
||||
});
|
||||
}}
|
||||
/>
|
||||
</CarouselItem>
|
||||
))}
|
||||
</CarouselContent>
|
||||
<CarouselPrevious className="!opacity-0 pointer-events-none transition-opacity group-hover:!opacity-100 group-hover:pointer-events-auto group-focus:opacity-100 group-focus:pointer-events-auto" />
|
||||
<CarouselNext className="!opacity-0 pointer-events-none transition-opacity group-hover:!opacity-100 group-hover:pointer-events-auto group-focus:opacity-100 group-focus:pointer-events-auto" />
|
||||
</Carousel>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -17,7 +17,6 @@ import {
|
||||
LayoutPanelTopIcon,
|
||||
PlusIcon,
|
||||
SparklesIcon,
|
||||
TrendingUpDownIcon,
|
||||
UndoDotIcon,
|
||||
UsersIcon,
|
||||
WallpaperIcon,
|
||||
@@ -40,18 +39,13 @@ export default function SidebarProjectMenu({
|
||||
}: SidebarProjectMenuProps) {
|
||||
return (
|
||||
<>
|
||||
<div className="mb-2 font-medium text-muted-foreground">Analytics</div>
|
||||
<div className="mb-2 font-medium text-muted-foreground">Insights</div>
|
||||
<SidebarLink icon={WallpaperIcon} label="Overview" href={'/'} />
|
||||
<SidebarLink
|
||||
icon={LayoutPanelTopIcon}
|
||||
label="Dashboards"
|
||||
href={'/dashboards'}
|
||||
/>
|
||||
<SidebarLink
|
||||
icon={TrendingUpDownIcon}
|
||||
label="Insights"
|
||||
href={'/insights'}
|
||||
/>
|
||||
<SidebarLink icon={LayersIcon} label="Pages" href={'/pages'} />
|
||||
<SidebarLink icon={Globe2Icon} label="Realtime" href={'/realtime'} />
|
||||
<SidebarLink icon={GanttChartIcon} label="Events" href={'/events'} />
|
||||
|
||||
@@ -123,7 +123,7 @@ export function SidebarContainer({
|
||||
</div>
|
||||
<div
|
||||
className={cn([
|
||||
'flex flex-grow col gap-1 overflow-auto p-4 hide-scrollbar',
|
||||
'flex flex-grow col gap-1 overflow-auto p-4',
|
||||
"[&_a[data-status='active']]:bg-def-200",
|
||||
])}
|
||||
>
|
||||
|
||||
@@ -208,7 +208,7 @@ const CarouselPrevious = React.forwardRef<
|
||||
variant={variant}
|
||||
size={size}
|
||||
className={cn(
|
||||
'absolute h-10 w-10 rounded-full hover:scale-100 hover:translate-y-[-50%] transition-all duration-200',
|
||||
'absolute h-10 w-10 rounded-full hover:scale-100 hover:translate-y-[-50%] transition-all duration-200',
|
||||
orientation === 'horizontal'
|
||||
? 'left-6 top-1/2 -translate-y-1/2'
|
||||
: '-top-12 left-1/2 -translate-x-1/2 rotate-90',
|
||||
|
||||
@@ -17,6 +17,5 @@ export function useAppContext() {
|
||||
apiUrl: params.apiUrl,
|
||||
dashboardUrl: params.dashboardUrl,
|
||||
isSelfHosted: params.isSelfHosted,
|
||||
isMaintenance: params.isMaintenance ?? false,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -38,7 +38,6 @@ import { Route as AppOrganizationIdProjectIdReportsRouteImport } from './routes/
|
||||
import { Route as AppOrganizationIdProjectIdReferencesRouteImport } from './routes/_app.$organizationId.$projectId.references'
|
||||
import { Route as AppOrganizationIdProjectIdRealtimeRouteImport } from './routes/_app.$organizationId.$projectId.realtime'
|
||||
import { Route as AppOrganizationIdProjectIdPagesRouteImport } from './routes/_app.$organizationId.$projectId.pages'
|
||||
import { Route as AppOrganizationIdProjectIdInsightsRouteImport } from './routes/_app.$organizationId.$projectId.insights'
|
||||
import { Route as AppOrganizationIdProjectIdDashboardsRouteImport } from './routes/_app.$organizationId.$projectId.dashboards'
|
||||
import { Route as AppOrganizationIdProjectIdChatRouteImport } from './routes/_app.$organizationId.$projectId.chat'
|
||||
import { Route as AppOrganizationIdMembersTabsIndexRouteImport } from './routes/_app.$organizationId.members._tabs.index'
|
||||
@@ -274,12 +273,6 @@ const AppOrganizationIdProjectIdPagesRoute =
|
||||
path: '/pages',
|
||||
getParentRoute: () => AppOrganizationIdProjectIdRoute,
|
||||
} as any)
|
||||
const AppOrganizationIdProjectIdInsightsRoute =
|
||||
AppOrganizationIdProjectIdInsightsRouteImport.update({
|
||||
id: '/insights',
|
||||
path: '/insights',
|
||||
getParentRoute: () => AppOrganizationIdProjectIdRoute,
|
||||
} as any)
|
||||
const AppOrganizationIdProjectIdDashboardsRoute =
|
||||
AppOrganizationIdProjectIdDashboardsRouteImport.update({
|
||||
id: '/dashboards',
|
||||
@@ -502,7 +495,6 @@ export interface FileRoutesByFullPath {
|
||||
'/$organizationId/': typeof AppOrganizationIdIndexRoute
|
||||
'/$organizationId/$projectId/chat': typeof AppOrganizationIdProjectIdChatRoute
|
||||
'/$organizationId/$projectId/dashboards': typeof AppOrganizationIdProjectIdDashboardsRoute
|
||||
'/$organizationId/$projectId/insights': typeof AppOrganizationIdProjectIdInsightsRoute
|
||||
'/$organizationId/$projectId/pages': typeof AppOrganizationIdProjectIdPagesRoute
|
||||
'/$organizationId/$projectId/realtime': typeof AppOrganizationIdProjectIdRealtimeRoute
|
||||
'/$organizationId/$projectId/references': typeof AppOrganizationIdProjectIdReferencesRoute
|
||||
@@ -560,7 +552,6 @@ export interface FileRoutesByTo {
|
||||
'/$organizationId': typeof AppOrganizationIdIndexRoute
|
||||
'/$organizationId/$projectId/chat': typeof AppOrganizationIdProjectIdChatRoute
|
||||
'/$organizationId/$projectId/dashboards': typeof AppOrganizationIdProjectIdDashboardsRoute
|
||||
'/$organizationId/$projectId/insights': typeof AppOrganizationIdProjectIdInsightsRoute
|
||||
'/$organizationId/$projectId/pages': typeof AppOrganizationIdProjectIdPagesRoute
|
||||
'/$organizationId/$projectId/realtime': typeof AppOrganizationIdProjectIdRealtimeRoute
|
||||
'/$organizationId/$projectId/references': typeof AppOrganizationIdProjectIdReferencesRoute
|
||||
@@ -618,7 +609,6 @@ export interface FileRoutesById {
|
||||
'/_app/$organizationId/': typeof AppOrganizationIdIndexRoute
|
||||
'/_app/$organizationId/$projectId/chat': typeof AppOrganizationIdProjectIdChatRoute
|
||||
'/_app/$organizationId/$projectId/dashboards': typeof AppOrganizationIdProjectIdDashboardsRoute
|
||||
'/_app/$organizationId/$projectId/insights': typeof AppOrganizationIdProjectIdInsightsRoute
|
||||
'/_app/$organizationId/$projectId/pages': typeof AppOrganizationIdProjectIdPagesRoute
|
||||
'/_app/$organizationId/$projectId/realtime': typeof AppOrganizationIdProjectIdRealtimeRoute
|
||||
'/_app/$organizationId/$projectId/references': typeof AppOrganizationIdProjectIdReferencesRoute
|
||||
@@ -687,7 +677,6 @@ export interface FileRouteTypes {
|
||||
| '/$organizationId/'
|
||||
| '/$organizationId/$projectId/chat'
|
||||
| '/$organizationId/$projectId/dashboards'
|
||||
| '/$organizationId/$projectId/insights'
|
||||
| '/$organizationId/$projectId/pages'
|
||||
| '/$organizationId/$projectId/realtime'
|
||||
| '/$organizationId/$projectId/references'
|
||||
@@ -745,7 +734,6 @@ export interface FileRouteTypes {
|
||||
| '/$organizationId'
|
||||
| '/$organizationId/$projectId/chat'
|
||||
| '/$organizationId/$projectId/dashboards'
|
||||
| '/$organizationId/$projectId/insights'
|
||||
| '/$organizationId/$projectId/pages'
|
||||
| '/$organizationId/$projectId/realtime'
|
||||
| '/$organizationId/$projectId/references'
|
||||
@@ -802,7 +790,6 @@ export interface FileRouteTypes {
|
||||
| '/_app/$organizationId/'
|
||||
| '/_app/$organizationId/$projectId/chat'
|
||||
| '/_app/$organizationId/$projectId/dashboards'
|
||||
| '/_app/$organizationId/$projectId/insights'
|
||||
| '/_app/$organizationId/$projectId/pages'
|
||||
| '/_app/$organizationId/$projectId/realtime'
|
||||
| '/_app/$organizationId/$projectId/references'
|
||||
@@ -1098,13 +1085,6 @@ declare module '@tanstack/react-router' {
|
||||
preLoaderRoute: typeof AppOrganizationIdProjectIdPagesRouteImport
|
||||
parentRoute: typeof AppOrganizationIdProjectIdRoute
|
||||
}
|
||||
'/_app/$organizationId/$projectId/insights': {
|
||||
id: '/_app/$organizationId/$projectId/insights'
|
||||
path: '/insights'
|
||||
fullPath: '/$organizationId/$projectId/insights'
|
||||
preLoaderRoute: typeof AppOrganizationIdProjectIdInsightsRouteImport
|
||||
parentRoute: typeof AppOrganizationIdProjectIdRoute
|
||||
}
|
||||
'/_app/$organizationId/$projectId/dashboards': {
|
||||
id: '/_app/$organizationId/$projectId/dashboards'
|
||||
path: '/dashboards'
|
||||
@@ -1548,7 +1528,6 @@ const AppOrganizationIdProjectIdSettingsRouteWithChildren =
|
||||
interface AppOrganizationIdProjectIdRouteChildren {
|
||||
AppOrganizationIdProjectIdChatRoute: typeof AppOrganizationIdProjectIdChatRoute
|
||||
AppOrganizationIdProjectIdDashboardsRoute: typeof AppOrganizationIdProjectIdDashboardsRoute
|
||||
AppOrganizationIdProjectIdInsightsRoute: typeof AppOrganizationIdProjectIdInsightsRoute
|
||||
AppOrganizationIdProjectIdPagesRoute: typeof AppOrganizationIdProjectIdPagesRoute
|
||||
AppOrganizationIdProjectIdRealtimeRoute: typeof AppOrganizationIdProjectIdRealtimeRoute
|
||||
AppOrganizationIdProjectIdReferencesRoute: typeof AppOrganizationIdProjectIdReferencesRoute
|
||||
@@ -1569,8 +1548,6 @@ const AppOrganizationIdProjectIdRouteChildren: AppOrganizationIdProjectIdRouteCh
|
||||
AppOrganizationIdProjectIdChatRoute: AppOrganizationIdProjectIdChatRoute,
|
||||
AppOrganizationIdProjectIdDashboardsRoute:
|
||||
AppOrganizationIdProjectIdDashboardsRoute,
|
||||
AppOrganizationIdProjectIdInsightsRoute:
|
||||
AppOrganizationIdProjectIdInsightsRoute,
|
||||
AppOrganizationIdProjectIdPagesRoute: AppOrganizationIdProjectIdPagesRoute,
|
||||
AppOrganizationIdProjectIdRealtimeRoute:
|
||||
AppOrganizationIdProjectIdRealtimeRoute,
|
||||
|
||||
@@ -34,7 +34,6 @@ interface MyRouterContext {
|
||||
apiUrl: string;
|
||||
dashboardUrl: string;
|
||||
isSelfHosted: boolean;
|
||||
isMaintenance: boolean;
|
||||
}
|
||||
|
||||
export const Route = createRootRouteWithContext<MyRouterContext>()({
|
||||
|
||||
@@ -3,7 +3,6 @@ import {
|
||||
OverviewFiltersButtons,
|
||||
} from '@/components/overview/filters/overview-filters-buttons';
|
||||
import { LiveCounter } from '@/components/overview/live-counter';
|
||||
import OverviewInsights from '@/components/overview/overview-insights';
|
||||
import { OverviewInterval } from '@/components/overview/overview-interval';
|
||||
import OverviewMetrics from '@/components/overview/overview-metrics';
|
||||
import { OverviewRange } from '@/components/overview/overview-range';
|
||||
@@ -51,7 +50,6 @@ function ProjectDashboard() {
|
||||
</div>
|
||||
<div className="grid grid-cols-6 gap-4 p-4 pt-0">
|
||||
<OverviewMetrics projectId={projectId} />
|
||||
<OverviewInsights projectId={projectId} />
|
||||
<OverviewTopSources projectId={projectId} />
|
||||
<OverviewTopPages projectId={projectId} />
|
||||
<OverviewTopDevices projectId={projectId} />
|
||||
|
||||
@@ -1,431 +0,0 @@
|
||||
import { FullPageEmptyState } from '@/components/full-page-empty-state';
|
||||
import { InsightCard } from '@/components/insights/insight-card';
|
||||
import { PageContainer } from '@/components/page-container';
|
||||
import { PageHeader } from '@/components/page-header';
|
||||
import { Skeleton } from '@/components/skeleton';
|
||||
import {
|
||||
Carousel,
|
||||
CarouselContent,
|
||||
CarouselItem,
|
||||
CarouselNext,
|
||||
CarouselPrevious,
|
||||
} from '@/components/ui/carousel';
|
||||
import { Input } from '@/components/ui/input';
|
||||
import {
|
||||
Select,
|
||||
SelectContent,
|
||||
SelectItem,
|
||||
SelectTrigger,
|
||||
SelectValue,
|
||||
} from '@/components/ui/select';
|
||||
import { TableButtons } from '@/components/ui/table';
|
||||
import { useTRPC } from '@/integrations/trpc/react';
|
||||
import { cn } from '@/utils/cn';
|
||||
import { PAGE_TITLES, createProjectTitle } from '@/utils/title';
|
||||
import { useQuery } from '@tanstack/react-query';
|
||||
import { createFileRoute, useNavigate } from '@tanstack/react-router';
|
||||
import { parseAsString, parseAsStringEnum, useQueryState } from 'nuqs';
|
||||
import { useMemo } from 'react';
|
||||
|
||||
export const Route = createFileRoute(
|
||||
'/_app/$organizationId/$projectId/insights',
|
||||
)({
|
||||
component: Component,
|
||||
head: () => {
|
||||
return {
|
||||
meta: [
|
||||
{
|
||||
title: createProjectTitle(PAGE_TITLES.INSIGHTS),
|
||||
},
|
||||
],
|
||||
};
|
||||
},
|
||||
});
|
||||
|
||||
type SortOption =
|
||||
| 'impact-desc'
|
||||
| 'impact-asc'
|
||||
| 'severity-desc'
|
||||
| 'severity-asc'
|
||||
| 'recent';
|
||||
|
||||
function getModuleDisplayName(moduleKey: string): string {
|
||||
const displayNames: Record<string, string> = {
|
||||
geo: 'Geographic',
|
||||
devices: 'Devices',
|
||||
referrers: 'Referrers',
|
||||
'entry-pages': 'Entry Pages',
|
||||
'page-trends': 'Page Trends',
|
||||
'exit-pages': 'Exit Pages',
|
||||
'traffic-anomalies': 'Anomalies',
|
||||
};
|
||||
return displayNames[moduleKey] || moduleKey.replace('-', ' ');
|
||||
}
|
||||
|
||||
function Component() {
|
||||
const { projectId } = Route.useParams();
|
||||
const trpc = useTRPC();
|
||||
const { data: insights, isLoading } = useQuery(
|
||||
trpc.insight.listAll.queryOptions({
|
||||
projectId,
|
||||
limit: 500,
|
||||
}),
|
||||
);
|
||||
const navigate = useNavigate();
|
||||
|
||||
const [search, setSearch] = useQueryState(
|
||||
'search',
|
||||
parseAsString.withDefault(''),
|
||||
);
|
||||
const [moduleFilter, setModuleFilter] = useQueryState(
|
||||
'module',
|
||||
parseAsString.withDefault('all'),
|
||||
);
|
||||
const [windowKindFilter, setWindowKindFilter] = useQueryState(
|
||||
'window',
|
||||
parseAsStringEnum([
|
||||
'all',
|
||||
'yesterday',
|
||||
'rolling_7d',
|
||||
'rolling_30d',
|
||||
]).withDefault('all'),
|
||||
);
|
||||
const [severityFilter, setSeverityFilter] = useQueryState(
|
||||
'severity',
|
||||
parseAsStringEnum(['all', 'severe', 'moderate', 'low', 'none']).withDefault(
|
||||
'all',
|
||||
),
|
||||
);
|
||||
const [directionFilter, setDirectionFilter] = useQueryState(
|
||||
'direction',
|
||||
parseAsStringEnum(['all', 'up', 'down', 'flat']).withDefault('all'),
|
||||
);
|
||||
const [sortBy, setSortBy] = useQueryState(
|
||||
'sort',
|
||||
parseAsStringEnum<SortOption>([
|
||||
'impact-desc',
|
||||
'impact-asc',
|
||||
'severity-desc',
|
||||
'severity-asc',
|
||||
'recent',
|
||||
]).withDefault('impact-desc'),
|
||||
);
|
||||
|
||||
const filteredAndSorted = useMemo(() => {
|
||||
if (!insights) return [];
|
||||
|
||||
const filtered = insights.filter((insight) => {
|
||||
// Search filter
|
||||
if (search) {
|
||||
const searchLower = search.toLowerCase();
|
||||
const matchesTitle = insight.title.toLowerCase().includes(searchLower);
|
||||
const matchesSummary = insight.summary
|
||||
?.toLowerCase()
|
||||
.includes(searchLower);
|
||||
const matchesDimension = insight.dimensionKey
|
||||
.toLowerCase()
|
||||
.includes(searchLower);
|
||||
if (!matchesTitle && !matchesSummary && !matchesDimension) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// Module filter
|
||||
if (moduleFilter !== 'all' && insight.moduleKey !== moduleFilter) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Window kind filter
|
||||
if (
|
||||
windowKindFilter !== 'all' &&
|
||||
insight.windowKind !== windowKindFilter
|
||||
) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Severity filter
|
||||
if (severityFilter !== 'all') {
|
||||
if (severityFilter === 'none' && insight.severityBand) return false;
|
||||
if (
|
||||
severityFilter !== 'none' &&
|
||||
insight.severityBand !== severityFilter
|
||||
)
|
||||
return false;
|
||||
}
|
||||
|
||||
// Direction filter
|
||||
if (directionFilter !== 'all' && insight.direction !== directionFilter) {
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
});
|
||||
|
||||
// Sort (create new array to avoid mutation)
|
||||
const sorted = [...filtered].sort((a, b) => {
|
||||
switch (sortBy) {
|
||||
case 'impact-desc':
|
||||
return (b.impactScore ?? 0) - (a.impactScore ?? 0);
|
||||
case 'impact-asc':
|
||||
return (a.impactScore ?? 0) - (b.impactScore ?? 0);
|
||||
case 'severity-desc': {
|
||||
const severityOrder: Record<string, number> = {
|
||||
severe: 3,
|
||||
moderate: 2,
|
||||
low: 1,
|
||||
};
|
||||
const aSev = severityOrder[a.severityBand ?? ''] ?? 0;
|
||||
const bSev = severityOrder[b.severityBand ?? ''] ?? 0;
|
||||
return bSev - aSev;
|
||||
}
|
||||
case 'severity-asc': {
|
||||
const severityOrder: Record<string, number> = {
|
||||
severe: 3,
|
||||
moderate: 2,
|
||||
low: 1,
|
||||
};
|
||||
const aSev = severityOrder[a.severityBand ?? ''] ?? 0;
|
||||
const bSev = severityOrder[b.severityBand ?? ''] ?? 0;
|
||||
return aSev - bSev;
|
||||
}
|
||||
case 'recent':
|
||||
return (
|
||||
new Date(b.firstDetectedAt ?? 0).getTime() -
|
||||
new Date(a.firstDetectedAt ?? 0).getTime()
|
||||
);
|
||||
default:
|
||||
return 0;
|
||||
}
|
||||
});
|
||||
|
||||
return sorted;
|
||||
}, [
|
||||
insights,
|
||||
search,
|
||||
moduleFilter,
|
||||
windowKindFilter,
|
||||
severityFilter,
|
||||
directionFilter,
|
||||
sortBy,
|
||||
]);
|
||||
|
||||
// Group insights by module
|
||||
const groupedByModule = useMemo(() => {
|
||||
const groups = new Map<string, typeof filteredAndSorted>();
|
||||
|
||||
for (const insight of filteredAndSorted) {
|
||||
const existing = groups.get(insight.moduleKey) ?? [];
|
||||
existing.push(insight);
|
||||
groups.set(insight.moduleKey, existing);
|
||||
}
|
||||
|
||||
// Sort modules by impact (referrers first, then by average impact score)
|
||||
return Array.from(groups.entries()).sort(
|
||||
([keyA, insightsA], [keyB, insightsB]) => {
|
||||
// Referrers always first
|
||||
if (keyA === 'referrers') return -1;
|
||||
if (keyB === 'referrers') return 1;
|
||||
|
||||
// Calculate average impact for each module
|
||||
const avgImpactA =
|
||||
insightsA.reduce((sum, i) => sum + (i.impactScore ?? 0), 0) /
|
||||
insightsA.length;
|
||||
const avgImpactB =
|
||||
insightsB.reduce((sum, i) => sum + (i.impactScore ?? 0), 0) /
|
||||
insightsB.length;
|
||||
|
||||
// Sort by average impact (high to low)
|
||||
return avgImpactB - avgImpactA;
|
||||
},
|
||||
);
|
||||
}, [filteredAndSorted]);
|
||||
|
||||
if (isLoading) {
|
||||
return (
|
||||
<PageContainer>
|
||||
<PageHeader title="Insights" className="mb-8" />
|
||||
<div className="space-y-8">
|
||||
{Array.from({ length: 3 }, (_, i) => `section-${i}`).map((key) => (
|
||||
<div key={key} className="space-y-4">
|
||||
<Skeleton className="h-8 w-32" />
|
||||
<Carousel opts={{ align: 'start' }} className="w-full">
|
||||
<CarouselContent className="-ml-4">
|
||||
{Array.from({ length: 4 }, (_, i) => `skeleton-${i}`).map(
|
||||
(cardKey) => (
|
||||
<CarouselItem
|
||||
key={cardKey}
|
||||
className="pl-4 basis-full sm:basis-1/2 lg:basis-1/3 xl:basis-1/4"
|
||||
>
|
||||
<Skeleton className="h-48 w-full" />
|
||||
</CarouselItem>
|
||||
),
|
||||
)}
|
||||
</CarouselContent>
|
||||
</Carousel>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
</PageContainer>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<PageContainer>
|
||||
<PageHeader
|
||||
title="Insights"
|
||||
description="Discover trends and changes in your analytics"
|
||||
className="mb-8"
|
||||
/>
|
||||
<TableButtons className="mb-8">
|
||||
<Input
|
||||
placeholder="Search insights..."
|
||||
value={search ?? ''}
|
||||
onChange={(e) => void setSearch(e.target.value || null)}
|
||||
className="max-w-xs"
|
||||
/>
|
||||
<Select
|
||||
value={windowKindFilter ?? 'all'}
|
||||
onValueChange={(v) =>
|
||||
void setWindowKindFilter(v as typeof windowKindFilter)
|
||||
}
|
||||
>
|
||||
<SelectTrigger className="w-[140px]">
|
||||
<SelectValue placeholder="Time Window" />
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
<SelectItem value="all">All Windows</SelectItem>
|
||||
<SelectItem value="yesterday">Yesterday</SelectItem>
|
||||
<SelectItem value="rolling_7d">7 Days</SelectItem>
|
||||
<SelectItem value="rolling_30d">30 Days</SelectItem>
|
||||
</SelectContent>
|
||||
</Select>
|
||||
<Select
|
||||
value={severityFilter ?? 'all'}
|
||||
onValueChange={(v) =>
|
||||
void setSeverityFilter(v as typeof severityFilter)
|
||||
}
|
||||
>
|
||||
<SelectTrigger className="w-[140px]">
|
||||
<SelectValue placeholder="Severity" />
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
<SelectItem value="all">All Severity</SelectItem>
|
||||
<SelectItem value="severe">Severe</SelectItem>
|
||||
<SelectItem value="moderate">Moderate</SelectItem>
|
||||
<SelectItem value="low">Low</SelectItem>
|
||||
<SelectItem value="none">No Severity</SelectItem>
|
||||
</SelectContent>
|
||||
</Select>
|
||||
<Select
|
||||
value={directionFilter ?? 'all'}
|
||||
onValueChange={(v) =>
|
||||
void setDirectionFilter(v as typeof directionFilter)
|
||||
}
|
||||
>
|
||||
<SelectTrigger className="w-[140px]">
|
||||
<SelectValue placeholder="Direction" />
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
<SelectItem value="all">All Directions</SelectItem>
|
||||
<SelectItem value="up">Increasing</SelectItem>
|
||||
<SelectItem value="down">Decreasing</SelectItem>
|
||||
<SelectItem value="flat">Flat</SelectItem>
|
||||
</SelectContent>
|
||||
</Select>
|
||||
<Select
|
||||
value={sortBy ?? 'impact-desc'}
|
||||
onValueChange={(v) => void setSortBy(v as SortOption)}
|
||||
>
|
||||
<SelectTrigger className="w-[160px]">
|
||||
<SelectValue placeholder="Sort by" />
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
<SelectItem value="impact-desc">Impact (High → Low)</SelectItem>
|
||||
<SelectItem value="impact-asc">Impact (Low → High)</SelectItem>
|
||||
<SelectItem value="severity-desc">Severity (High → Low)</SelectItem>
|
||||
<SelectItem value="severity-asc">Severity (Low → High)</SelectItem>
|
||||
<SelectItem value="recent">Most Recent</SelectItem>
|
||||
</SelectContent>
|
||||
</Select>
|
||||
</TableButtons>
|
||||
|
||||
{filteredAndSorted.length === 0 && !isLoading && (
|
||||
<FullPageEmptyState
|
||||
title="No insights found"
|
||||
description={
|
||||
search || moduleFilter !== 'all' || windowKindFilter !== 'all'
|
||||
? 'Try adjusting your filters to see more insights.'
|
||||
: 'Insights will appear here as trends are detected in your analytics.'
|
||||
}
|
||||
/>
|
||||
)}
|
||||
|
||||
{groupedByModule.length > 0 && (
|
||||
<div className="space-y-8">
|
||||
{groupedByModule.map(([moduleKey, moduleInsights]) => (
|
||||
<div key={moduleKey} className="space-y-4">
|
||||
<div className="flex items-center justify-between">
|
||||
<h2 className="text-lg font-semibold capitalize">
|
||||
{getModuleDisplayName(moduleKey)}
|
||||
</h2>
|
||||
<span className="text-sm text-muted-foreground">
|
||||
{moduleInsights.length}{' '}
|
||||
{moduleInsights.length === 1 ? 'insight' : 'insights'}
|
||||
</span>
|
||||
</div>
|
||||
<div className="-mx-8">
|
||||
<Carousel
|
||||
opts={{ align: 'start', dragFree: true }}
|
||||
className="w-full group"
|
||||
>
|
||||
<CarouselContent className="mx-4 mr-8">
|
||||
{moduleInsights.map((insight, index) => (
|
||||
<CarouselItem
|
||||
key={insight.id}
|
||||
className={cn(
|
||||
'pl-4 basis-full sm:basis-1/2 lg:basis-1/3 xl:basis-1/4',
|
||||
)}
|
||||
>
|
||||
<InsightCard
|
||||
insight={insight}
|
||||
onFilter={(() => {
|
||||
const filterString = insight.payload?.dimensions
|
||||
.map(
|
||||
(dim) =>
|
||||
`${dim.key},is,${encodeURIComponent(dim.value)}`,
|
||||
)
|
||||
.join(';');
|
||||
if (filterString) {
|
||||
return () => {
|
||||
navigate({
|
||||
to: '/$organizationId/$projectId',
|
||||
from: Route.fullPath,
|
||||
search: {
|
||||
f: filterString,
|
||||
},
|
||||
});
|
||||
};
|
||||
}
|
||||
return undefined;
|
||||
})()}
|
||||
/>
|
||||
</CarouselItem>
|
||||
))}
|
||||
</CarouselContent>
|
||||
<CarouselPrevious className="opacity-0 [&:disabled]:opacity-0 pointer-events-none transition-opacity group-hover:opacity-100 group-hover:pointer-events-auto left-3" />
|
||||
<CarouselNext className="opacity-0 [&:disabled]:opacity-0 pointer-events-none transition-opacity group-hover:opacity-100 group-hover:pointer-events-auto right-3" />
|
||||
</Carousel>
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{filteredAndSorted.length > 0 && (
|
||||
<div className="mt-8 text-sm text-muted-foreground text-center">
|
||||
Showing {filteredAndSorted.length} of {insights?.length ?? 0} insights
|
||||
</div>
|
||||
)}
|
||||
</PageContainer>
|
||||
);
|
||||
}
|
||||
@@ -1,10 +1,5 @@
|
||||
import { FullPageEmptyState } from '@/components/full-page-empty-state';
|
||||
import { Sidebar } from '@/components/sidebar';
|
||||
import { Button, LinkButton, buttonVariants } from '@/components/ui/button';
|
||||
import { useAppContext } from '@/hooks/use-app-context';
|
||||
import { cn } from '@/utils/cn';
|
||||
import { Outlet, createFileRoute, redirect } from '@tanstack/react-router';
|
||||
import { ConstructionIcon } from 'lucide-react';
|
||||
|
||||
export const Route = createFileRoute('/_app')({
|
||||
beforeLoad: async ({ context }) => {
|
||||
@@ -16,28 +11,6 @@ export const Route = createFileRoute('/_app')({
|
||||
});
|
||||
|
||||
function AppLayout() {
|
||||
const { isMaintenance } = useAppContext();
|
||||
|
||||
if (isMaintenance) {
|
||||
return (
|
||||
<FullPageEmptyState
|
||||
icon={ConstructionIcon}
|
||||
className="min-h-screen"
|
||||
title="Maintenance mode"
|
||||
description="We are currently performing maintenance on the system. Please check back later."
|
||||
>
|
||||
<a
|
||||
href="https://status.openpanel.dev/"
|
||||
className={cn(buttonVariants())}
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
>
|
||||
Check out our status page
|
||||
</a>
|
||||
</FullPageEmptyState>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="flex h-screen w-full">
|
||||
<Sidebar />
|
||||
|
||||
@@ -8,7 +8,6 @@ export const getServerEnvs = createServerFn().handler(async () => {
|
||||
process.env.DASHBOARD_URL || process.env.NEXT_PUBLIC_DASHBOARD_URL,
|
||||
),
|
||||
isSelfHosted: process.env.SELF_HOSTED !== undefined,
|
||||
isMaintenance: process.env.MAINTENANCE === '1',
|
||||
};
|
||||
|
||||
return envs;
|
||||
|
||||
@@ -90,7 +90,6 @@ export const PAGE_TITLES = {
|
||||
CHAT: 'AI Assistant',
|
||||
REALTIME: 'Realtime',
|
||||
REFERENCES: 'References',
|
||||
INSIGHTS: 'Insights',
|
||||
// Profiles
|
||||
PROFILES: 'Profiles',
|
||||
PROFILE_EVENTS: 'Profile events',
|
||||
|
||||
@@ -34,11 +34,6 @@ export async function bootCron() {
|
||||
type: 'flushSessions',
|
||||
pattern: 1000 * 10,
|
||||
},
|
||||
{
|
||||
name: 'insightsDaily',
|
||||
type: 'insightsDaily',
|
||||
pattern: '0 2 * * *',
|
||||
},
|
||||
];
|
||||
|
||||
if (process.env.SELF_HOSTED && process.env.NODE_ENV === 'production') {
|
||||
|
||||
@@ -7,7 +7,6 @@ import {
|
||||
cronQueue,
|
||||
eventsGroupQueues,
|
||||
importQueue,
|
||||
insightsQueue,
|
||||
miscQueue,
|
||||
notificationQueue,
|
||||
queueLogger,
|
||||
@@ -22,7 +21,6 @@ import { Worker as GroupWorker } from 'groupmq';
|
||||
import { cronJob } from './jobs/cron';
|
||||
import { incomingEvent } from './jobs/events.incoming-event';
|
||||
import { importJob } from './jobs/import';
|
||||
import { insightsProjectJob } from './jobs/insights';
|
||||
import { miscJob } from './jobs/misc';
|
||||
import { notificationJob } from './jobs/notification';
|
||||
import { sessionsJob } from './jobs/sessions';
|
||||
@@ -51,15 +49,7 @@ function getEnabledQueues(): QueueName[] {
|
||||
logger.info('No ENABLED_QUEUES specified, starting all queues', {
|
||||
totalEventShards: EVENTS_GROUP_QUEUES_SHARDS,
|
||||
});
|
||||
return [
|
||||
'events',
|
||||
'sessions',
|
||||
'cron',
|
||||
'notification',
|
||||
'misc',
|
||||
'import',
|
||||
'insights',
|
||||
];
|
||||
return ['events', 'sessions', 'cron', 'notification', 'misc', 'import'];
|
||||
}
|
||||
|
||||
const queues = enabledQueuesEnv
|
||||
@@ -127,7 +117,7 @@ export async function bootWorkers() {
|
||||
const worker = new GroupWorker<EventsQueuePayloadIncomingEvent['payload']>({
|
||||
queue,
|
||||
concurrency,
|
||||
logger: process.env.NODE_ENV === 'production' ? queueLogger : undefined,
|
||||
logger: queueLogger,
|
||||
blockingTimeoutSec: Number.parseFloat(
|
||||
process.env.EVENT_BLOCKING_TIMEOUT_SEC || '1',
|
||||
),
|
||||
@@ -197,17 +187,6 @@ export async function bootWorkers() {
|
||||
logger.info('Started worker for import', { concurrency });
|
||||
}
|
||||
|
||||
// Start insights worker
|
||||
if (enabledQueues.includes('insights')) {
|
||||
const concurrency = getConcurrencyFor('insights', 5);
|
||||
const insightsWorker = new Worker(insightsQueue.name, insightsProjectJob, {
|
||||
...workerOptions,
|
||||
concurrency,
|
||||
});
|
||||
workers.push(insightsWorker);
|
||||
logger.info('Started worker for insights', { concurrency });
|
||||
}
|
||||
|
||||
if (workers.length === 0) {
|
||||
logger.warn(
|
||||
'No workers started. Check ENABLED_QUEUES environment variable.',
|
||||
|
||||
@@ -6,7 +6,6 @@ import {
|
||||
cronQueue,
|
||||
eventsGroupQueues,
|
||||
importQueue,
|
||||
insightsQueue,
|
||||
miscQueue,
|
||||
notificationQueue,
|
||||
sessionsQueue,
|
||||
@@ -43,7 +42,6 @@ async function start() {
|
||||
new BullMQAdapter(notificationQueue),
|
||||
new BullMQAdapter(miscQueue),
|
||||
new BullMQAdapter(importQueue),
|
||||
new BullMQAdapter(insightsQueue),
|
||||
],
|
||||
serverAdapter: serverAdapter,
|
||||
});
|
||||
|
||||
@@ -1,10 +1,6 @@
|
||||
import { TABLE_NAMES, chQuery } from '@openpanel/db';
|
||||
|
||||
export async function ping() {
|
||||
if (process.env.DISABLE_PING) {
|
||||
return;
|
||||
}
|
||||
|
||||
const [res] = await chQuery<{ count: number }>(
|
||||
`SELECT COUNT(*) as count FROM ${TABLE_NAMES.events}`,
|
||||
);
|
||||
|
||||
@@ -6,7 +6,6 @@ import type { CronQueuePayload } from '@openpanel/queue';
|
||||
import { jobdeleteProjects } from './cron.delete-projects';
|
||||
import { ping } from './cron.ping';
|
||||
import { salt } from './cron.salt';
|
||||
import { insightsDailyJob } from './insights';
|
||||
|
||||
export async function cronJob(job: Job<CronQueuePayload>) {
|
||||
switch (job.data.type) {
|
||||
@@ -28,8 +27,5 @@ export async function cronJob(job: Job<CronQueuePayload>) {
|
||||
case 'deleteProjects': {
|
||||
return await jobdeleteProjects(job);
|
||||
}
|
||||
case 'insightsDaily': {
|
||||
return await insightsDailyJob(job);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -96,6 +96,7 @@ export async function createSessionEnd(
|
||||
...payload,
|
||||
properties: {
|
||||
...payload.properties,
|
||||
...(session?.properties ?? {}),
|
||||
__bounce: session.is_bounce,
|
||||
},
|
||||
name: 'session_end',
|
||||
|
||||
@@ -319,6 +319,7 @@ describe('incomingEvent', () => {
|
||||
utm_content: '',
|
||||
utm_medium: '',
|
||||
revenue: 0,
|
||||
properties: {},
|
||||
project_id: projectId,
|
||||
device_id: 'last-device-123',
|
||||
profile_id: 'profile-123',
|
||||
|
||||
@@ -1,72 +0,0 @@
|
||||
import { ch } from '@openpanel/db/src/clickhouse/client';
|
||||
import {
|
||||
createEngine,
|
||||
devicesModule,
|
||||
entryPagesModule,
|
||||
geoModule,
|
||||
insightStore,
|
||||
pageTrendsModule,
|
||||
referrersModule,
|
||||
} from '@openpanel/db/src/services/insights';
|
||||
import type {
|
||||
CronQueuePayload,
|
||||
InsightsQueuePayloadProject,
|
||||
} from '@openpanel/queue';
|
||||
import { insightsQueue } from '@openpanel/queue';
|
||||
import type { Job } from 'bullmq';
|
||||
|
||||
const defaultEngineConfig = {
|
||||
keepTopNPerModuleWindow: 20,
|
||||
closeStaleAfterDays: 7,
|
||||
dimensionBatchSize: 50,
|
||||
globalThresholds: {
|
||||
minTotal: 200,
|
||||
minAbsDelta: 80,
|
||||
minPct: 0.15,
|
||||
},
|
||||
};
|
||||
|
||||
export async function insightsDailyJob(job: Job<CronQueuePayload>) {
|
||||
const projectIds = await insightStore.listProjectIdsForCadence('daily');
|
||||
const date = new Date().toISOString().slice(0, 10);
|
||||
|
||||
for (const projectId of projectIds) {
|
||||
await insightsQueue.add(
|
||||
'insightsProject',
|
||||
{
|
||||
type: 'insightsProject',
|
||||
payload: { projectId, date },
|
||||
},
|
||||
{
|
||||
jobId: `daily:${date}:${projectId}`, // idempotent
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
export async function insightsProjectJob(
|
||||
job: Job<InsightsQueuePayloadProject>,
|
||||
) {
|
||||
const { projectId, date } = job.data.payload;
|
||||
const engine = createEngine({
|
||||
store: insightStore,
|
||||
modules: [
|
||||
referrersModule,
|
||||
entryPagesModule,
|
||||
pageTrendsModule,
|
||||
geoModule,
|
||||
devicesModule,
|
||||
],
|
||||
db: ch,
|
||||
config: defaultEngineConfig,
|
||||
});
|
||||
|
||||
const projectCreatedAt = await insightStore.getProjectCreatedAt(projectId);
|
||||
|
||||
await engine.runProject({
|
||||
projectId,
|
||||
cadence: 'daily',
|
||||
now: new Date(date),
|
||||
projectCreatedAt,
|
||||
});
|
||||
}
|
||||
10
packages/auth/nextjs.ts
Normal file
10
packages/auth/nextjs.ts
Normal file
@@ -0,0 +1,10 @@
|
||||
import { unstable_cache } from 'next/cache';
|
||||
import { cookies } from 'next/headers';
|
||||
import { validateSessionToken } from './src/session';
|
||||
|
||||
export const auth = async () => {
|
||||
const token = (await cookies().get('session')?.value) ?? null;
|
||||
return cachedAuth(token);
|
||||
};
|
||||
|
||||
export const cachedAuth = unstable_cache(validateSessionToken);
|
||||
@@ -22,6 +22,7 @@
|
||||
"typescript": "catalog:"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"next": "14.2.1",
|
||||
"react": "catalog:"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -245,259 +245,3 @@ export function getDefaultIntervalByDates(
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
export const countries = {
|
||||
AF: 'Afghanistan',
|
||||
AL: 'Albania',
|
||||
DZ: 'Algeria',
|
||||
AS: 'American Samoa',
|
||||
AD: 'Andorra',
|
||||
AO: 'Angola',
|
||||
AI: 'Anguilla',
|
||||
AQ: 'Antarctica',
|
||||
AG: 'Antigua and Barbuda',
|
||||
AR: 'Argentina',
|
||||
AM: 'Armenia',
|
||||
AW: 'Aruba',
|
||||
AU: 'Australia',
|
||||
AT: 'Austria',
|
||||
AZ: 'Azerbaijan',
|
||||
BS: 'Bahamas',
|
||||
BH: 'Bahrain',
|
||||
BD: 'Bangladesh',
|
||||
BB: 'Barbados',
|
||||
BY: 'Belarus',
|
||||
BE: 'Belgium',
|
||||
BZ: 'Belize',
|
||||
BJ: 'Benin',
|
||||
BM: 'Bermuda',
|
||||
BT: 'Bhutan',
|
||||
BO: 'Bolivia',
|
||||
BQ: 'Bonaire, Sint Eustatius and Saba',
|
||||
BA: 'Bosnia and Herzegovina',
|
||||
BW: 'Botswana',
|
||||
BV: 'Bouvet Island',
|
||||
BR: 'Brazil',
|
||||
IO: 'British Indian Ocean Territory',
|
||||
BN: 'Brunei Darussalam',
|
||||
BG: 'Bulgaria',
|
||||
BF: 'Burkina Faso',
|
||||
BI: 'Burundi',
|
||||
CV: 'Cabo Verde',
|
||||
KH: 'Cambodia',
|
||||
CM: 'Cameroon',
|
||||
CA: 'Canada',
|
||||
KY: 'Cayman Islands',
|
||||
CF: 'Central African Republic',
|
||||
TD: 'Chad',
|
||||
CL: 'Chile',
|
||||
CN: 'China',
|
||||
CX: 'Christmas Island',
|
||||
CC: 'Cocos (Keeling) Islands',
|
||||
CO: 'Colombia',
|
||||
KM: 'Comoros',
|
||||
CD: 'Congo (Democratic Republic)',
|
||||
CG: 'Congo',
|
||||
CK: 'Cook Islands',
|
||||
CR: 'Costa Rica',
|
||||
HR: 'Croatia',
|
||||
CU: 'Cuba',
|
||||
CW: 'Curaçao',
|
||||
CY: 'Cyprus',
|
||||
CZ: 'Czechia',
|
||||
CI: "Côte d'Ivoire",
|
||||
DK: 'Denmark',
|
||||
DJ: 'Djibouti',
|
||||
DM: 'Dominica',
|
||||
DO: 'Dominican Republic',
|
||||
EC: 'Ecuador',
|
||||
EG: 'Egypt',
|
||||
SV: 'El Salvador',
|
||||
GQ: 'Equatorial Guinea',
|
||||
ER: 'Eritrea',
|
||||
EE: 'Estonia',
|
||||
SZ: 'Eswatina',
|
||||
ET: 'Ethiopia',
|
||||
FK: 'Falkland Islands',
|
||||
FO: 'Faroe Islands',
|
||||
FJ: 'Fiji',
|
||||
FI: 'Finland',
|
||||
FR: 'France',
|
||||
GF: 'French Guiana',
|
||||
PF: 'French Polynesia',
|
||||
TF: 'French Southern Territories',
|
||||
GA: 'Gabon',
|
||||
GM: 'Gambia',
|
||||
GE: 'Georgia',
|
||||
DE: 'Germany',
|
||||
GH: 'Ghana',
|
||||
GI: 'Gibraltar',
|
||||
GR: 'Greece',
|
||||
GL: 'Greenland',
|
||||
GD: 'Grenada',
|
||||
GP: 'Guadeloupe',
|
||||
GU: 'Guam',
|
||||
GT: 'Guatemala',
|
||||
GG: 'Guernsey',
|
||||
GN: 'Guinea',
|
||||
GW: 'Guinea-Bissau',
|
||||
GY: 'Guyana',
|
||||
HT: 'Haiti',
|
||||
HM: 'Heard Island and McDonald Islands',
|
||||
VA: 'Holy See',
|
||||
HN: 'Honduras',
|
||||
HK: 'Hong Kong',
|
||||
HU: 'Hungary',
|
||||
IS: 'Iceland',
|
||||
IN: 'India',
|
||||
ID: 'Indonesia',
|
||||
IR: 'Iran',
|
||||
IQ: 'Iraq',
|
||||
IE: 'Ireland',
|
||||
IM: 'Isle of Man',
|
||||
IL: 'Israel',
|
||||
IT: 'Italy',
|
||||
JM: 'Jamaica',
|
||||
JP: 'Japan',
|
||||
JE: 'Jersey',
|
||||
JO: 'Jordan',
|
||||
KZ: 'Kazakhstan',
|
||||
KE: 'Kenya',
|
||||
KI: 'Kiribati',
|
||||
KP: "Korea (Democratic People's Republic)",
|
||||
KR: 'Korea (Republic)',
|
||||
KW: 'Kuwait',
|
||||
KG: 'Kyrgyzstan',
|
||||
LA: "Lao People's Democratic Republic",
|
||||
LV: 'Latvia',
|
||||
LB: 'Lebanon',
|
||||
LS: 'Lesotho',
|
||||
LR: 'Liberia',
|
||||
LY: 'Libya',
|
||||
LI: 'Liechtenstein',
|
||||
LT: 'Lithuania',
|
||||
LU: 'Luxembourg',
|
||||
MO: 'Macao',
|
||||
MG: 'Madagascar',
|
||||
MW: 'Malawi',
|
||||
MY: 'Malaysia',
|
||||
MV: 'Maldives',
|
||||
ML: 'Mali',
|
||||
MT: 'Malta',
|
||||
MH: 'Marshall Islands',
|
||||
MQ: 'Martinique',
|
||||
MR: 'Mauritania',
|
||||
MU: 'Mauritius',
|
||||
YT: 'Mayotte',
|
||||
MX: 'Mexico',
|
||||
FM: 'Micronesia',
|
||||
MD: 'Moldova',
|
||||
MC: 'Monaco',
|
||||
MN: 'Mongolia',
|
||||
ME: 'Montenegro',
|
||||
MS: 'Montserrat',
|
||||
MA: 'Morocco',
|
||||
MZ: 'Mozambique',
|
||||
MM: 'Myanmar',
|
||||
NA: 'Namibia',
|
||||
NR: 'Nauru',
|
||||
NP: 'Nepal',
|
||||
NL: 'Netherlands',
|
||||
NC: 'New Caledonia',
|
||||
NZ: 'New Zealand',
|
||||
NI: 'Nicaragua',
|
||||
NE: 'Niger',
|
||||
NG: 'Nigeria',
|
||||
NU: 'Niue',
|
||||
NF: 'Norfolk Island',
|
||||
MP: 'Northern Mariana Islands',
|
||||
NO: 'Norway',
|
||||
OM: 'Oman',
|
||||
PK: 'Pakistan',
|
||||
PW: 'Palau',
|
||||
PS: 'Palestine, State of',
|
||||
PA: 'Panama',
|
||||
PG: 'Papua New Guinea',
|
||||
PY: 'Paraguay',
|
||||
PE: 'Peru',
|
||||
PH: 'Philippines',
|
||||
PN: 'Pitcairn',
|
||||
PL: 'Poland',
|
||||
PT: 'Portugal',
|
||||
PR: 'Puerto Rico',
|
||||
QA: 'Qatar',
|
||||
MK: 'Republic of North Macedonia',
|
||||
RO: 'Romania',
|
||||
RU: 'Russian Federation',
|
||||
RW: 'Rwanda',
|
||||
RE: 'Réunion',
|
||||
BL: 'Saint Barthélemy',
|
||||
SH: 'Saint Helena, Ascension and Tristan da Cunha',
|
||||
KN: 'Saint Kitts and Nevis',
|
||||
LC: 'Saint Lucia',
|
||||
MF: 'Saint Martin (French part)',
|
||||
PM: 'Saint Pierre and Miquelon',
|
||||
VC: 'Saint Vincent and the Grenadines',
|
||||
WS: 'Samoa',
|
||||
SM: 'San Marino',
|
||||
ST: 'Sao Tome and Principe',
|
||||
SA: 'Saudi Arabia',
|
||||
SN: 'Senegal',
|
||||
RS: 'Serbia',
|
||||
SC: 'Seychelles',
|
||||
SL: 'Sierra Leone',
|
||||
SG: 'Singapore',
|
||||
SX: 'Sint Maarten (Dutch part)',
|
||||
SK: 'Slovakia',
|
||||
SI: 'Slovenia',
|
||||
SB: 'Solomon Islands',
|
||||
SO: 'Somalia',
|
||||
ZA: 'South Africa',
|
||||
GS: 'South Georgia and the South Sandwich Islands',
|
||||
SS: 'South Sudan',
|
||||
ES: 'Spain',
|
||||
LK: 'Sri Lanka',
|
||||
SD: 'Sudan',
|
||||
SR: 'Suriname',
|
||||
SJ: 'Svalbard and Jan Mayen',
|
||||
SE: 'Sweden',
|
||||
CH: 'Switzerland',
|
||||
SY: 'Syrian Arab Republic',
|
||||
TW: 'Taiwan',
|
||||
TJ: 'Tajikistan',
|
||||
TZ: 'Tanzania, United Republic of',
|
||||
TH: 'Thailand',
|
||||
TL: 'Timor-Leste',
|
||||
TG: 'Togo',
|
||||
TK: 'Tokelau',
|
||||
TO: 'Tonga',
|
||||
TT: 'Trinidad and Tobago',
|
||||
TN: 'Tunisia',
|
||||
TR: 'Turkey',
|
||||
TM: 'Turkmenistan',
|
||||
TC: 'Turks and Caicos Islands',
|
||||
TV: 'Tuvalu',
|
||||
UG: 'Uganda',
|
||||
UA: 'Ukraine',
|
||||
AE: 'United Arab Emirates',
|
||||
GB: 'United Kingdom',
|
||||
US: 'United States',
|
||||
UM: 'United States Minor Outlying Islands',
|
||||
UY: 'Uruguay',
|
||||
UZ: 'Uzbekistan',
|
||||
VU: 'Vanuatu',
|
||||
VE: 'Venezuela',
|
||||
VN: 'Viet Nam',
|
||||
VG: 'Virgin Islands (British)',
|
||||
VI: 'Virgin Islands (U.S.)',
|
||||
WF: 'Wallis and Futuna',
|
||||
EH: 'Western Sahara',
|
||||
YE: 'Yemen',
|
||||
ZM: 'Zambia',
|
||||
ZW: 'Zimbabwe',
|
||||
AX: 'Åland Islands',
|
||||
} as const;
|
||||
|
||||
export function getCountry(code?: string) {
|
||||
return countries[code as keyof typeof countries];
|
||||
}
|
||||
|
||||
@@ -100,9 +100,6 @@ async function createOldSessions() {
|
||||
if (!row || row.count === '0') {
|
||||
return null;
|
||||
}
|
||||
if (row.created_at.startsWith('1970')) {
|
||||
return null;
|
||||
}
|
||||
return new Date(row.created_at);
|
||||
} catch (e) {
|
||||
return defaultDate;
|
||||
|
||||
@@ -1,302 +0,0 @@
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import {
|
||||
chMigrationClient,
|
||||
createTable,
|
||||
moveDataBetweenTables,
|
||||
renameTable,
|
||||
runClickhouseMigrationCommands,
|
||||
} from '../src/clickhouse/migration';
|
||||
import { getIsCluster } from './helpers';
|
||||
|
||||
export async function up() {
|
||||
const isClustered = getIsCluster();
|
||||
|
||||
const sqls: string[] = [];
|
||||
|
||||
const eventTables = createTable({
|
||||
name: 'events_new_20251123',
|
||||
columns: [
|
||||
'`id` UUID DEFAULT generateUUIDv4()',
|
||||
'`name` LowCardinality(String)',
|
||||
'`sdk_name` LowCardinality(String)',
|
||||
'`sdk_version` LowCardinality(String)',
|
||||
'`device_id` String CODEC(ZSTD(3))',
|
||||
'`profile_id` String CODEC(ZSTD(3))',
|
||||
'`project_id` String CODEC(ZSTD(3))',
|
||||
'`session_id` String CODEC(LZ4)',
|
||||
'`path` String CODEC(ZSTD(3))',
|
||||
'`origin` String CODEC(ZSTD(3))',
|
||||
'`referrer` String CODEC(ZSTD(3))',
|
||||
'`referrer_name` String CODEC(ZSTD(3))',
|
||||
'`referrer_type` LowCardinality(String)',
|
||||
'`revenue` UInt64',
|
||||
'`duration` UInt64 CODEC(Delta(4), LZ4)',
|
||||
'`properties` Map(String, String) CODEC(ZSTD(3))',
|
||||
'`created_at` DateTime64(3) CODEC(DoubleDelta, ZSTD(3))',
|
||||
'`country` LowCardinality(FixedString(2))',
|
||||
'`city` String',
|
||||
'`region` LowCardinality(String)',
|
||||
'`longitude` Nullable(Float32) CODEC(Gorilla, LZ4)',
|
||||
'`latitude` Nullable(Float32) CODEC(Gorilla, LZ4)',
|
||||
'`os` LowCardinality(String)',
|
||||
'`os_version` LowCardinality(String)',
|
||||
'`browser` LowCardinality(String)',
|
||||
'`browser_version` LowCardinality(String)',
|
||||
'`device` LowCardinality(String)',
|
||||
'`brand` LowCardinality(String)',
|
||||
'`model` LowCardinality(String)',
|
||||
'`imported_at` Nullable(DateTime) CODEC(Delta(4), LZ4)',
|
||||
],
|
||||
indices: [
|
||||
'INDEX idx_name name TYPE bloom_filter GRANULARITY 1',
|
||||
"INDEX idx_properties_bounce properties['__bounce'] TYPE set(3) GRANULARITY 1",
|
||||
'INDEX idx_origin origin TYPE bloom_filter(0.05) GRANULARITY 1',
|
||||
'INDEX idx_path path TYPE bloom_filter(0.01) GRANULARITY 1',
|
||||
],
|
||||
// New ORDER BY: project_id, toDate(created_at), created_at, name
|
||||
// Removed profile_id, added created_at for better ordering within same day
|
||||
orderBy: ['project_id', 'toDate(created_at)', 'created_at', 'name'],
|
||||
partitionBy: 'toYYYYMM(created_at)',
|
||||
settings: {
|
||||
index_granularity: 8192,
|
||||
// For lightweight updates
|
||||
enable_block_offset_column: 1,
|
||||
enable_block_number_column: 1,
|
||||
},
|
||||
distributionHash:
|
||||
'cityHash64(project_id, toString(toStartOfHour(created_at)))',
|
||||
replicatedVersion: '1',
|
||||
isClustered,
|
||||
});
|
||||
|
||||
// Step 1: Create temporary tables with new ORDER BY keys
|
||||
// Events table with new ORDER BY
|
||||
sqls.push(...eventTables);
|
||||
|
||||
const sessionTables = createTable({
|
||||
name: 'sessions_new_20251123',
|
||||
engine: 'VersionedCollapsingMergeTree(sign, version)',
|
||||
columns: [
|
||||
'`id` String',
|
||||
'`project_id` String CODEC(ZSTD(3))',
|
||||
'`profile_id` String CODEC(ZSTD(3))',
|
||||
'`device_id` String CODEC(ZSTD(3))',
|
||||
'`created_at` DateTime64(3) CODEC(DoubleDelta, ZSTD(3))',
|
||||
'`ended_at` DateTime64(3) CODEC(DoubleDelta, ZSTD(3))',
|
||||
'`is_bounce` Bool',
|
||||
'`entry_origin` LowCardinality(String)',
|
||||
'`entry_path` String CODEC(ZSTD(3))',
|
||||
'`exit_origin` LowCardinality(String)',
|
||||
'`exit_path` String CODEC(ZSTD(3))',
|
||||
'`screen_view_count` Int32',
|
||||
'`revenue` Float64',
|
||||
'`event_count` Int32',
|
||||
'`duration` UInt32',
|
||||
'`country` LowCardinality(FixedString(2))',
|
||||
'`region` LowCardinality(String)',
|
||||
'`city` String',
|
||||
'`longitude` Nullable(Float32) CODEC(Gorilla, LZ4)',
|
||||
'`latitude` Nullable(Float32) CODEC(Gorilla, LZ4)',
|
||||
'`device` LowCardinality(String)',
|
||||
'`brand` LowCardinality(String)',
|
||||
'`model` LowCardinality(String)',
|
||||
'`browser` LowCardinality(String)',
|
||||
'`browser_version` LowCardinality(String)',
|
||||
'`os` LowCardinality(String)',
|
||||
'`os_version` LowCardinality(String)',
|
||||
'`utm_medium` String CODEC(ZSTD(3))',
|
||||
'`utm_source` String CODEC(ZSTD(3))',
|
||||
'`utm_campaign` String CODEC(ZSTD(3))',
|
||||
'`utm_content` String CODEC(ZSTD(3))',
|
||||
'`utm_term` String CODEC(ZSTD(3))',
|
||||
'`referrer` String CODEC(ZSTD(3))',
|
||||
'`referrer_name` String CODEC(ZSTD(3))',
|
||||
'`referrer_type` LowCardinality(String)',
|
||||
'`sign` Int8',
|
||||
'`version` UInt64',
|
||||
],
|
||||
// New ORDER BY: project_id, toDate(created_at), created_at, id
|
||||
// Removed profile_id, reordered to match query patterns (date first, then id)
|
||||
orderBy: ['project_id', 'toDate(created_at)', 'created_at'],
|
||||
partitionBy: 'toYYYYMM(created_at)',
|
||||
settings: {
|
||||
index_granularity: 8192,
|
||||
},
|
||||
distributionHash:
|
||||
'cityHash64(project_id, toString(toStartOfHour(created_at)))',
|
||||
replicatedVersion: '1',
|
||||
isClustered,
|
||||
});
|
||||
|
||||
// Sessions table with new ORDER BY
|
||||
sqls.push(...sessionTables);
|
||||
|
||||
const firstEventDateResponse = await chMigrationClient.query({
|
||||
query: 'SELECT min(created_at) as created_at FROM events',
|
||||
format: 'JSONEachRow',
|
||||
});
|
||||
const firstEventDateJson = await firstEventDateResponse.json<{
|
||||
created_at: string;
|
||||
}>();
|
||||
if (
|
||||
firstEventDateJson[0]?.created_at &&
|
||||
!firstEventDateJson[0]?.created_at.startsWith('1970')
|
||||
) {
|
||||
const firstEventDate = new Date(firstEventDateJson[0]?.created_at);
|
||||
// Step 2: Copy data from old tables to new tables (partitioned by month for efficiency)
|
||||
// Set endDate to first of next month to ensure we capture all data in the current month
|
||||
const endDate = new Date();
|
||||
endDate.setMonth(endDate.getMonth() + 1);
|
||||
endDate.setDate(1);
|
||||
|
||||
sqls.push(
|
||||
...moveDataBetweenTables({
|
||||
from: 'events',
|
||||
to: 'events_new_20251123',
|
||||
batch: {
|
||||
startDate: firstEventDate,
|
||||
endDate: endDate,
|
||||
column: 'toDate(created_at)',
|
||||
interval: 'month',
|
||||
transform: (date: Date) => {
|
||||
const year = date.getFullYear();
|
||||
const month = String(date.getMonth() + 1).padStart(2, '0');
|
||||
return `${year}-${month}-01`;
|
||||
},
|
||||
},
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
const firstSessionDateResponse = await chMigrationClient.query({
|
||||
query: 'SELECT min(created_at) as created_at FROM sessions',
|
||||
format: 'JSONEachRow',
|
||||
});
|
||||
const firstSessionDateJson = await firstSessionDateResponse.json<{
|
||||
created_at: string;
|
||||
}>();
|
||||
|
||||
if (
|
||||
firstSessionDateJson[0]?.created_at &&
|
||||
!firstSessionDateJson[0]?.created_at.startsWith('1970')
|
||||
) {
|
||||
const firstSessionDate = new Date(
|
||||
firstSessionDateJson[0]?.created_at ?? '',
|
||||
);
|
||||
// Set endDate to first of next month to ensure we capture all data in the current month
|
||||
const endDate = new Date();
|
||||
endDate.setMonth(endDate.getMonth() + 1);
|
||||
endDate.setDate(1);
|
||||
|
||||
sqls.push(
|
||||
...moveDataBetweenTables({
|
||||
from: 'sessions',
|
||||
to: 'sessions_new_20251123',
|
||||
columns: [
|
||||
'id',
|
||||
'project_id',
|
||||
'profile_id',
|
||||
'device_id',
|
||||
'created_at',
|
||||
'ended_at',
|
||||
'is_bounce',
|
||||
'entry_origin',
|
||||
'entry_path',
|
||||
'exit_origin',
|
||||
'exit_path',
|
||||
'screen_view_count',
|
||||
'revenue',
|
||||
'event_count',
|
||||
'duration',
|
||||
'country',
|
||||
'region',
|
||||
'city',
|
||||
'longitude',
|
||||
'latitude',
|
||||
'device',
|
||||
'brand',
|
||||
'model',
|
||||
'browser',
|
||||
'browser_version',
|
||||
'os',
|
||||
'os_version',
|
||||
'utm_medium',
|
||||
'utm_source',
|
||||
'utm_campaign',
|
||||
'utm_content',
|
||||
'utm_term',
|
||||
'referrer',
|
||||
'referrer_name',
|
||||
'referrer_type',
|
||||
'sign',
|
||||
'version',
|
||||
],
|
||||
batch: {
|
||||
startDate: firstSessionDate,
|
||||
endDate: endDate,
|
||||
column: 'toDate(created_at)',
|
||||
interval: 'month',
|
||||
transform: (date: Date) => {
|
||||
const year = date.getFullYear();
|
||||
const month = String(date.getMonth() + 1).padStart(2, '0');
|
||||
return `${year}-${month}-01`;
|
||||
},
|
||||
},
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
sqls.push(
|
||||
...renameTable({ from: 'events', to: 'events_20251123', isClustered }),
|
||||
);
|
||||
sqls.push(
|
||||
...renameTable({ from: 'sessions', to: 'sessions_20251123', isClustered }),
|
||||
);
|
||||
|
||||
if (isClustered && sessionTables[1] && eventTables[1]) {
|
||||
sqls.push(
|
||||
// Drop temporary DISTRIBUTED tables (will be recreated)
|
||||
`DROP TABLE IF EXISTS events_new_20251123 ON CLUSTER '{cluster}'`,
|
||||
`DROP TABLE IF EXISTS sessions_new_20251123 ON CLUSTER '{cluster}'`,
|
||||
// Rename new tables to correct names
|
||||
`RENAME TABLE events_new_20251123_replicated TO events_replicated ON CLUSTER '{cluster}'`,
|
||||
`RENAME TABLE sessions_new_20251123_replicated TO sessions_replicated ON CLUSTER '{cluster}'`,
|
||||
// Create new distributed tables
|
||||
eventTables[1].replaceAll('events_new_20251123', 'events'), // creates a new distributed table
|
||||
sessionTables[1].replaceAll('sessions_new_20251123', 'sessions'), // creates a new distributed table
|
||||
);
|
||||
} else {
|
||||
sqls.push(
|
||||
...renameTable({
|
||||
from: 'events_new_20251123',
|
||||
to: 'events',
|
||||
isClustered,
|
||||
}),
|
||||
);
|
||||
sqls.push(
|
||||
...renameTable({
|
||||
from: 'sessions_new_20251123',
|
||||
to: 'sessions',
|
||||
isClustered,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
fs.writeFileSync(
|
||||
path.join(__filename.replace('.ts', '.sql')),
|
||||
sqls
|
||||
.map((sql) =>
|
||||
sql
|
||||
.trim()
|
||||
.replace(/;$/, '')
|
||||
.replace(/\n{2,}/g, '\n')
|
||||
.concat(';'),
|
||||
)
|
||||
.join('\n\n---\n\n'),
|
||||
);
|
||||
|
||||
if (!process.argv.includes('--dry')) {
|
||||
await runClickhouseMigrationCommands(sqls);
|
||||
}
|
||||
}
|
||||
@@ -30,7 +30,3 @@ export function getIsSelfHosting() {
|
||||
export function getIsDry() {
|
||||
return process.argv.includes('--dry');
|
||||
}
|
||||
|
||||
export function getShouldIgnoreRecord() {
|
||||
return process.argv.includes('--no-record');
|
||||
}
|
||||
|
||||
@@ -10,7 +10,6 @@ import {
|
||||
getIsCluster,
|
||||
getIsDry,
|
||||
getIsSelfHosting,
|
||||
getShouldIgnoreRecord,
|
||||
printBoxMessage,
|
||||
} from './helpers';
|
||||
|
||||
@@ -56,12 +55,8 @@ async function migrate() {
|
||||
]);
|
||||
|
||||
if (!getIsSelfHosting()) {
|
||||
if (!getIsDry()) {
|
||||
printBoxMessage('🕒 Migrations starts in 10 seconds', []);
|
||||
await new Promise((resolve) => setTimeout(resolve, 10000));
|
||||
} else {
|
||||
printBoxMessage('🕒 Migrations starts now (dry run)', []);
|
||||
}
|
||||
printBoxMessage('🕒 Migrations starts in 10 seconds', []);
|
||||
await new Promise((resolve) => setTimeout(resolve, 10000));
|
||||
}
|
||||
|
||||
if (migration) {
|
||||
@@ -86,7 +81,7 @@ async function runMigration(migrationsDir: string, file: string) {
|
||||
try {
|
||||
const migration = await import(path.join(migrationsDir, file));
|
||||
await migration.up();
|
||||
if (!getIsDry() && !getShouldIgnoreRecord()) {
|
||||
if (!getIsDry()) {
|
||||
await db.codeMigration.upsert({
|
||||
where: {
|
||||
name: file,
|
||||
|
||||
@@ -28,5 +28,4 @@ export * from './src/types';
|
||||
export * from './src/clickhouse/query-builder';
|
||||
export * from './src/services/import.service';
|
||||
export * from './src/services/overview.service';
|
||||
export * from './src/services/insights';
|
||||
export * from './src/session-context';
|
||||
|
||||
@@ -1,57 +0,0 @@
|
||||
-- CreateEnum
|
||||
CREATE TYPE "public"."InsightState" AS ENUM ('active', 'suppressed', 'closed');
|
||||
|
||||
-- CreateTable
|
||||
CREATE TABLE "public"."project_insights" (
|
||||
"id" UUID NOT NULL DEFAULT gen_random_uuid(),
|
||||
"projectId" TEXT NOT NULL,
|
||||
"moduleKey" TEXT NOT NULL,
|
||||
"dimensionKey" TEXT NOT NULL,
|
||||
"windowKind" TEXT NOT NULL,
|
||||
"state" "public"."InsightState" NOT NULL DEFAULT 'active',
|
||||
"title" TEXT NOT NULL,
|
||||
"summary" TEXT,
|
||||
"payload" JSONB,
|
||||
"currentValue" DOUBLE PRECISION,
|
||||
"compareValue" DOUBLE PRECISION,
|
||||
"changePct" DOUBLE PRECISION,
|
||||
"direction" TEXT,
|
||||
"impactScore" DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
"severityBand" TEXT,
|
||||
"version" INTEGER NOT NULL DEFAULT 1,
|
||||
"threadId" UUID NOT NULL DEFAULT gen_random_uuid(),
|
||||
"windowStart" TIMESTAMP(3),
|
||||
"windowEnd" TIMESTAMP(3),
|
||||
"firstDetectedAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"lastUpdatedAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
"lastSeenAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
|
||||
CONSTRAINT "project_insights_pkey" PRIMARY KEY ("id")
|
||||
);
|
||||
|
||||
-- CreateTable
|
||||
CREATE TABLE "public"."insight_events" (
|
||||
"id" UUID NOT NULL DEFAULT gen_random_uuid(),
|
||||
"insightId" UUID NOT NULL,
|
||||
"eventKind" TEXT NOT NULL,
|
||||
"changeFrom" JSONB,
|
||||
"changeTo" JSONB,
|
||||
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
|
||||
CONSTRAINT "insight_events_pkey" PRIMARY KEY ("id")
|
||||
);
|
||||
|
||||
-- CreateIndex
|
||||
CREATE INDEX "project_insights_projectId_impactScore_idx" ON "public"."project_insights"("projectId", "impactScore" DESC);
|
||||
|
||||
-- CreateIndex
|
||||
CREATE INDEX "project_insights_projectId_moduleKey_windowKind_state_idx" ON "public"."project_insights"("projectId", "moduleKey", "windowKind", "state");
|
||||
|
||||
-- CreateIndex
|
||||
CREATE UNIQUE INDEX "project_insights_projectId_moduleKey_dimensionKey_windowKin_key" ON "public"."project_insights"("projectId", "moduleKey", "dimensionKey", "windowKind", "state");
|
||||
|
||||
-- CreateIndex
|
||||
CREATE INDEX "insight_events_insightId_createdAt_idx" ON "public"."insight_events"("insightId", "createdAt");
|
||||
|
||||
-- AddForeignKey
|
||||
ALTER TABLE "public"."insight_events" ADD CONSTRAINT "insight_events_insightId_fkey" FOREIGN KEY ("insightId") REFERENCES "public"."project_insights"("id") ON DELETE CASCADE ON UPDATE CASCADE;
|
||||
@@ -1,9 +0,0 @@
|
||||
/*
|
||||
Warnings:
|
||||
|
||||
- Made the column `payload` on table `project_insights` required. This step will fail if there are existing NULL values in that column.
|
||||
|
||||
*/
|
||||
-- AlterTable
|
||||
ALTER TABLE "public"."project_insights" ALTER COLUMN "payload" SET NOT NULL,
|
||||
ALTER COLUMN "payload" SET DEFAULT '{}';
|
||||
@@ -1,13 +0,0 @@
|
||||
/*
|
||||
Warnings:
|
||||
|
||||
- You are about to drop the column `changePct` on the `project_insights` table. All the data in the column will be lost.
|
||||
- You are about to drop the column `compareValue` on the `project_insights` table. All the data in the column will be lost.
|
||||
- You are about to drop the column `currentValue` on the `project_insights` table. All the data in the column will be lost.
|
||||
|
||||
*/
|
||||
-- AlterTable
|
||||
ALTER TABLE "public"."project_insights" DROP COLUMN "changePct",
|
||||
DROP COLUMN "compareValue",
|
||||
DROP COLUMN "currentValue",
|
||||
ADD COLUMN "displayName" TEXT NOT NULL DEFAULT '';
|
||||
@@ -497,58 +497,3 @@ model Import {
|
||||
|
||||
@@map("imports")
|
||||
}
|
||||
|
||||
enum InsightState {
|
||||
active
|
||||
suppressed
|
||||
closed
|
||||
}
|
||||
|
||||
model ProjectInsight {
|
||||
id String @id @default(dbgenerated("gen_random_uuid()")) @db.Uuid
|
||||
projectId String
|
||||
moduleKey String // e.g. "referrers", "entry-pages"
|
||||
dimensionKey String // e.g. "referrer:instagram", "page:/pricing"
|
||||
windowKind String // "yesterday" | "rolling_7d" | "rolling_30d"
|
||||
state InsightState @default(active)
|
||||
|
||||
title String
|
||||
summary String?
|
||||
displayName String @default("")
|
||||
/// [IPrismaProjectInsightPayload]
|
||||
payload Json @default("{}") // Rendered insight payload (typed)
|
||||
|
||||
direction String? // "up" | "down" | "flat"
|
||||
impactScore Float @default(0)
|
||||
severityBand String? // "low" | "moderate" | "severe"
|
||||
|
||||
version Int @default(1)
|
||||
threadId String @default(dbgenerated("gen_random_uuid()")) @db.Uuid
|
||||
|
||||
windowStart DateTime?
|
||||
windowEnd DateTime?
|
||||
|
||||
firstDetectedAt DateTime @default(now())
|
||||
lastUpdatedAt DateTime @default(now()) @updatedAt
|
||||
lastSeenAt DateTime @default(now())
|
||||
|
||||
events InsightEvent[]
|
||||
|
||||
@@unique([projectId, moduleKey, dimensionKey, windowKind, state])
|
||||
@@index([projectId, impactScore(sort: Desc)])
|
||||
@@index([projectId, moduleKey, windowKind, state])
|
||||
@@map("project_insights")
|
||||
}
|
||||
|
||||
model InsightEvent {
|
||||
id String @id @default(dbgenerated("gen_random_uuid()")) @db.Uuid
|
||||
insightId String @db.Uuid
|
||||
insight ProjectInsight @relation(fields: [insightId], references: [id], onDelete: Cascade)
|
||||
eventKind String // "created" | "updated" | "severity_up" | "direction_flip" | "closed" | etc
|
||||
changeFrom Json?
|
||||
changeTo Json?
|
||||
createdAt DateTime @default(now())
|
||||
|
||||
@@index([insightId, createdAt])
|
||||
@@map("insight_events")
|
||||
}
|
||||
|
||||
@@ -68,30 +68,6 @@ export class ProfileBuffer extends BaseBuffer {
|
||||
|
||||
const existingProfile = await this.fetchProfile(profile, logger);
|
||||
|
||||
// Delete any properties that are not server related if we have a non-server profile
|
||||
if (
|
||||
existingProfile?.properties.device !== 'server' &&
|
||||
profile.properties.device === 'server'
|
||||
) {
|
||||
profile.properties = omit(
|
||||
[
|
||||
'city',
|
||||
'country',
|
||||
'region',
|
||||
'longitude',
|
||||
'latitude',
|
||||
'os',
|
||||
'osVersion',
|
||||
'browser',
|
||||
'device',
|
||||
'isServer',
|
||||
'os_version',
|
||||
'browser_version',
|
||||
],
|
||||
profile.properties,
|
||||
);
|
||||
}
|
||||
|
||||
const mergedProfile: IClickhouseProfile = existingProfile
|
||||
? deepMergeObjects(existingProfile, omit(['created_at'], profile))
|
||||
: profile;
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { type Redis, getRedisCache } from '@openpanel/redis';
|
||||
|
||||
import { toDots } from '@openpanel/common';
|
||||
import { getSafeJson } from '@openpanel/json';
|
||||
import { assocPath, clone } from 'ramda';
|
||||
import { TABLE_NAMES, ch } from '../clickhouse/client';
|
||||
@@ -90,6 +91,10 @@ export class SessionBuffer extends BaseBuffer {
|
||||
session: newSession,
|
||||
});
|
||||
}
|
||||
newSession.properties = toDots({
|
||||
...(event.properties || {}),
|
||||
...(newSession.properties || {}),
|
||||
});
|
||||
|
||||
const addedRevenue = event.name === 'revenue' ? (event.revenue ?? 0) : 0;
|
||||
newSession.revenue = (newSession.revenue ?? 0) + addedRevenue;
|
||||
@@ -163,6 +168,7 @@ export class SessionBuffer extends BaseBuffer {
|
||||
: '',
|
||||
sign: 1,
|
||||
version: 1,
|
||||
properties: toDots(event.properties || {}),
|
||||
},
|
||||
];
|
||||
}
|
||||
|
||||
@@ -217,7 +217,6 @@ export function moveDataBetweenTables({
|
||||
from,
|
||||
to,
|
||||
batch,
|
||||
columns,
|
||||
}: {
|
||||
from: string;
|
||||
to: string;
|
||||
@@ -228,15 +227,11 @@ export function moveDataBetweenTables({
|
||||
endDate?: Date;
|
||||
startDate?: Date;
|
||||
};
|
||||
columns?: string[];
|
||||
}): string[] {
|
||||
const sqls: string[] = [];
|
||||
|
||||
// Build the SELECT clause
|
||||
const selectClause = columns && columns.length > 0 ? columns.join(', ') : '*';
|
||||
|
||||
if (!batch) {
|
||||
return [`INSERT INTO ${to} SELECT ${selectClause} FROM ${from}`];
|
||||
return [`INSERT INTO ${to} SELECT * FROM ${from}`];
|
||||
}
|
||||
|
||||
// Start from today and go back 3 years
|
||||
@@ -252,109 +247,32 @@ export function moveDataBetweenTables({
|
||||
let currentDate = endDate;
|
||||
const interval = batch.interval || 'day';
|
||||
|
||||
// Helper function to get the start of the week (Monday) for a given date
|
||||
const getWeekStart = (date: Date): Date => {
|
||||
const d = new Date(date);
|
||||
const day = d.getDay();
|
||||
const diff = d.getDate() - day + (day === 0 ? -6 : 1); // Adjust to Monday
|
||||
d.setDate(diff);
|
||||
d.setHours(0, 0, 0, 0); // Normalize to start of day
|
||||
return d;
|
||||
};
|
||||
|
||||
// Helper function to compare dates based on interval
|
||||
const shouldContinue = (
|
||||
current: Date,
|
||||
start: Date,
|
||||
intervalType: string,
|
||||
): boolean => {
|
||||
if (intervalType === 'month') {
|
||||
// For months, compare by year and month
|
||||
// Continue if current month is >= start month
|
||||
const currentYear = current.getFullYear();
|
||||
const currentMonth = current.getMonth();
|
||||
const startYear = start.getFullYear();
|
||||
const startMonth = start.getMonth();
|
||||
return (
|
||||
currentYear > startYear ||
|
||||
(currentYear === startYear && currentMonth >= startMonth)
|
||||
);
|
||||
}
|
||||
if (intervalType === 'week') {
|
||||
// For weeks, compare by week start dates
|
||||
const currentWeekStart = getWeekStart(current);
|
||||
const startWeekStart = getWeekStart(start);
|
||||
return currentWeekStart >= startWeekStart;
|
||||
}
|
||||
return current > start;
|
||||
};
|
||||
|
||||
while (shouldContinue(currentDate, startDate, interval)) {
|
||||
while (currentDate > startDate) {
|
||||
const previousDate = new Date(currentDate);
|
||||
|
||||
switch (interval) {
|
||||
case 'month':
|
||||
previousDate.setMonth(previousDate.getMonth() - 1);
|
||||
// If we've gone below startDate's month, adjust to start of startDate's month
|
||||
// This ensures we generate SQL for the month containing startDate
|
||||
if (
|
||||
previousDate.getFullYear() < startDate.getFullYear() ||
|
||||
(previousDate.getFullYear() === startDate.getFullYear() &&
|
||||
previousDate.getMonth() < startDate.getMonth())
|
||||
) {
|
||||
previousDate.setFullYear(startDate.getFullYear());
|
||||
previousDate.setMonth(startDate.getMonth());
|
||||
previousDate.setDate(1);
|
||||
}
|
||||
break;
|
||||
case 'week': {
|
||||
case 'week':
|
||||
previousDate.setDate(previousDate.getDate() - 7);
|
||||
// If we've gone below startDate's week, adjust to start of startDate's week
|
||||
const startWeekStart = getWeekStart(startDate);
|
||||
const prevWeekStart = getWeekStart(previousDate);
|
||||
if (prevWeekStart < startWeekStart) {
|
||||
previousDate.setTime(startWeekStart.getTime());
|
||||
// Ensure we don't go below startDate
|
||||
if (previousDate < startDate) {
|
||||
previousDate.setTime(startDate.getTime());
|
||||
}
|
||||
break;
|
||||
}
|
||||
// day
|
||||
default:
|
||||
previousDate.setDate(previousDate.getDate() - 1);
|
||||
break;
|
||||
}
|
||||
|
||||
// For monthly/weekly intervals with transform, upperBoundDate should be currentDate
|
||||
// because currentDate already represents the start of the period we're processing
|
||||
// The WHERE clause uses > previousDate AND <= currentDate to get exactly one period
|
||||
let upperBoundDate = currentDate;
|
||||
// Don't exceed the endDate
|
||||
if (upperBoundDate > endDate) {
|
||||
upperBoundDate = endDate;
|
||||
}
|
||||
|
||||
const sql = `INSERT INTO ${to}
|
||||
SELECT ${selectClause} FROM ${from}
|
||||
SELECT * FROM ${from}
|
||||
WHERE ${batch.column} > '${batch.transform ? batch.transform(previousDate) : formatClickhouseDate(previousDate, true)}'
|
||||
AND ${batch.column} <= '${batch.transform ? batch.transform(upperBoundDate) : formatClickhouseDate(upperBoundDate, true)}'`;
|
||||
AND ${batch.column} <= '${batch.transform ? batch.transform(currentDate) : formatClickhouseDate(currentDate, true)}'`;
|
||||
sqls.push(sql);
|
||||
|
||||
// For monthly/weekly intervals, stop if we've reached the start period
|
||||
if (interval === 'month') {
|
||||
const prevYear = previousDate.getFullYear();
|
||||
const prevMonth = previousDate.getMonth();
|
||||
const startYear = startDate.getFullYear();
|
||||
const startMonth = startDate.getMonth();
|
||||
if (prevYear === startYear && prevMonth === startMonth) {
|
||||
break;
|
||||
}
|
||||
} else if (interval === 'week') {
|
||||
const prevWeekStart = getWeekStart(previousDate);
|
||||
const startWeekStart = getWeekStart(startDate);
|
||||
if (prevWeekStart.getTime() === startWeekStart.getTime()) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
currentDate = previousDate;
|
||||
}
|
||||
|
||||
|
||||
@@ -43,7 +43,7 @@ class Expression {
|
||||
}
|
||||
|
||||
export class Query<T = any> {
|
||||
private _select: (string | Expression)[] = [];
|
||||
private _select: string[] = [];
|
||||
private _except: string[] = [];
|
||||
private _from?: string | Expression;
|
||||
private _where: WhereCondition[] = [];
|
||||
@@ -81,19 +81,17 @@ export class Query<T = any> {
|
||||
|
||||
// Select methods
|
||||
select<U>(
|
||||
columns: (string | Expression | null | undefined | false)[],
|
||||
columns: (string | null | undefined | false)[],
|
||||
type: 'merge' | 'replace' = 'replace',
|
||||
): Query<U> {
|
||||
if (this._skipNext) return this as unknown as Query<U>;
|
||||
if (type === 'merge') {
|
||||
this._select = [
|
||||
...this._select,
|
||||
...columns.filter((col): col is string | Expression => Boolean(col)),
|
||||
...columns.filter((col): col is string => Boolean(col)),
|
||||
];
|
||||
} else {
|
||||
this._select = columns.filter((col): col is string | Expression =>
|
||||
Boolean(col),
|
||||
);
|
||||
this._select = columns.filter((col): col is string => Boolean(col));
|
||||
}
|
||||
return this as unknown as Query<U>;
|
||||
}
|
||||
@@ -374,14 +372,7 @@ export class Query<T = any> {
|
||||
if (this._select.length > 0) {
|
||||
parts.push(
|
||||
'SELECT',
|
||||
this._select
|
||||
// Important: Expressions are treated as raw SQL; do not run escapeDate()
|
||||
// on them, otherwise any embedded date strings get double-escaped
|
||||
// (e.g. ''2025-12-16 23:59:59'') which ClickHouse rejects.
|
||||
.map((col) =>
|
||||
col instanceof Expression ? col.toString() : this.escapeDate(col),
|
||||
)
|
||||
.join(', '),
|
||||
this._select.map((col) => this.escapeDate(col)).join(', '),
|
||||
);
|
||||
} else {
|
||||
parts.push('SELECT *');
|
||||
|
||||
@@ -42,11 +42,11 @@ const getPrismaClient = () => {
|
||||
operation === 'update' ||
|
||||
operation === 'delete'
|
||||
) {
|
||||
// logger.info('Prisma operation', {
|
||||
// operation,
|
||||
// args,
|
||||
// model,
|
||||
// });
|
||||
logger.info('Prisma operation', {
|
||||
operation,
|
||||
args,
|
||||
model,
|
||||
});
|
||||
}
|
||||
return query(args);
|
||||
},
|
||||
|
||||
@@ -267,22 +267,43 @@ export function getChartSql({
|
||||
'COUNT(*)::float / COUNT(DISTINCT profile_id)::float as count';
|
||||
}
|
||||
|
||||
const mathFunction = {
|
||||
property_sum: 'sum',
|
||||
property_average: 'avg',
|
||||
property_max: 'max',
|
||||
property_min: 'min',
|
||||
}[event.segment as string];
|
||||
|
||||
if (mathFunction && event.property) {
|
||||
const propertyKey = getSelectPropertyKey(event.property);
|
||||
|
||||
if (isNumericColumn(event.property)) {
|
||||
sb.select.count = `${mathFunction}(${propertyKey}) as count`;
|
||||
sb.where.property = `${propertyKey} IS NOT NULL`;
|
||||
if (event.segment === 'property_sum' && event.property) {
|
||||
if (event.property === 'revenue') {
|
||||
sb.select.count = 'sum(revenue) as count';
|
||||
sb.where.property = 'revenue > 0';
|
||||
} else {
|
||||
sb.select.count = `${mathFunction}(toFloat64OrNull(${propertyKey})) as count`;
|
||||
sb.where.property = `${propertyKey} IS NOT NULL AND notEmpty(${propertyKey})`;
|
||||
sb.select.count = `sum(toFloat64(${getSelectPropertyKey(event.property)})) as count`;
|
||||
sb.where.property = `${getSelectPropertyKey(event.property)} IS NOT NULL AND notEmpty(${getSelectPropertyKey(event.property)})`;
|
||||
}
|
||||
}
|
||||
|
||||
if (event.segment === 'property_average' && event.property) {
|
||||
if (event.property === 'revenue') {
|
||||
sb.select.count = 'avg(revenue) as count';
|
||||
sb.where.property = 'revenue > 0';
|
||||
} else {
|
||||
sb.select.count = `avg(toFloat64(${getSelectPropertyKey(event.property)})) as count`;
|
||||
sb.where.property = `${getSelectPropertyKey(event.property)} IS NOT NULL AND notEmpty(${getSelectPropertyKey(event.property)})`;
|
||||
}
|
||||
}
|
||||
|
||||
if (event.segment === 'property_max' && event.property) {
|
||||
if (event.property === 'revenue') {
|
||||
sb.select.count = 'max(revenue) as count';
|
||||
sb.where.property = 'revenue > 0';
|
||||
} else {
|
||||
sb.select.count = `max(toFloat64(${getSelectPropertyKey(event.property)})) as count`;
|
||||
sb.where.property = `${getSelectPropertyKey(event.property)} IS NOT NULL AND notEmpty(${getSelectPropertyKey(event.property)})`;
|
||||
}
|
||||
}
|
||||
|
||||
if (event.segment === 'property_min' && event.property) {
|
||||
if (event.property === 'revenue') {
|
||||
sb.select.count = 'min(revenue) as count';
|
||||
sb.where.property = 'revenue > 0';
|
||||
} else {
|
||||
sb.select.count = `min(toFloat64(${getSelectPropertyKey(event.property)})) as count`;
|
||||
sb.where.property = `${getSelectPropertyKey(event.property)} IS NOT NULL AND notEmpty(${getSelectPropertyKey(event.property)})`;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -131,6 +131,7 @@ export function transformSessionToEvent(
|
||||
duration: 0,
|
||||
revenue: session.revenue,
|
||||
properties: {
|
||||
...session.properties,
|
||||
is_bounce: session.is_bounce,
|
||||
__query: {
|
||||
utm_medium: session.utm_medium,
|
||||
@@ -591,9 +592,6 @@ export async function getEventList(options: GetEventListOptions) {
|
||||
if (select.sdkVersion) {
|
||||
sb.select.sdkVersion = 'sdk_version';
|
||||
}
|
||||
if (select.revenue) {
|
||||
sb.select.revenue = 'revenue';
|
||||
}
|
||||
|
||||
if (profileId) {
|
||||
sb.where.deviceId = `(device_id IN (SELECT device_id as did FROM ${TABLE_NAMES.events} WHERE project_id = ${sqlstring.escape(projectId)} AND device_id != '' AND profile_id = ${sqlstring.escape(profileId)} group by did) OR profile_id = ${sqlstring.escape(profileId)})`;
|
||||
@@ -630,7 +628,8 @@ export async function getEventList(options: GetEventListOptions) {
|
||||
}
|
||||
}
|
||||
|
||||
sb.orderBy.created_at = 'created_at DESC';
|
||||
sb.orderBy.created_at =
|
||||
'toDate(created_at) DESC, created_at DESC, profile_id DESC, name DESC';
|
||||
|
||||
if (custom) {
|
||||
custom(sb);
|
||||
|
||||
@@ -1,68 +0,0 @@
|
||||
import crypto from 'node:crypto';
|
||||
import type { ClickHouseClient } from '@clickhouse/client';
|
||||
import {
|
||||
type Query,
|
||||
clix as originalClix,
|
||||
} from '../../clickhouse/query-builder';
|
||||
|
||||
/**
|
||||
* Creates a cached wrapper around clix that automatically caches query results
|
||||
* based on query hash. This eliminates duplicate queries within the same module/window context.
|
||||
*
|
||||
* @param client - ClickHouse client
|
||||
* @param cache - Optional cache Map to store query results
|
||||
* @param timezone - Timezone for queries (defaults to UTC)
|
||||
* @returns A function that creates cached Query instances (compatible with clix API)
|
||||
*/
|
||||
export function createCachedClix(
|
||||
client: ClickHouseClient,
|
||||
cache?: Map<string, any>,
|
||||
timezone?: string,
|
||||
) {
|
||||
function clixCached(): Query {
|
||||
const query = originalClix(client, timezone);
|
||||
const queryTimezone = timezone ?? 'UTC';
|
||||
|
||||
// Override execute() method to add caching
|
||||
const originalExecute = query.execute.bind(query);
|
||||
query.execute = async () => {
|
||||
// Build the query SQL string
|
||||
const querySQL = query.toSQL();
|
||||
|
||||
// Create cache key from query SQL + timezone
|
||||
const cacheKey = crypto
|
||||
.createHash('sha256')
|
||||
.update(`${querySQL}|${queryTimezone}`)
|
||||
.digest('hex');
|
||||
|
||||
// Check cache first
|
||||
if (cache?.has(cacheKey)) {
|
||||
return cache.get(cacheKey);
|
||||
}
|
||||
|
||||
// Execute query
|
||||
const result = await originalExecute();
|
||||
|
||||
// Cache the result
|
||||
if (cache) {
|
||||
cache.set(cacheKey, result);
|
||||
}
|
||||
|
||||
return result;
|
||||
};
|
||||
|
||||
return query;
|
||||
}
|
||||
|
||||
// Copy static methods from original clix
|
||||
clixCached.exp = originalClix.exp;
|
||||
clixCached.date = originalClix.date;
|
||||
clixCached.datetime = originalClix.datetime;
|
||||
clixCached.dynamicDatetime = originalClix.dynamicDatetime;
|
||||
clixCached.toStartOf = originalClix.toStartOf;
|
||||
clixCached.toStartOfInterval = originalClix.toStartOfInterval;
|
||||
clixCached.toInterval = originalClix.toInterval;
|
||||
clixCached.toDate = originalClix.toDate;
|
||||
|
||||
return clixCached;
|
||||
}
|
||||
@@ -1,303 +0,0 @@
|
||||
import { createCachedClix } from './cached-clix';
|
||||
import { materialDecision } from './material';
|
||||
import { defaultImpactScore, severityBand } from './scoring';
|
||||
import type {
|
||||
Cadence,
|
||||
ComputeContext,
|
||||
ComputeResult,
|
||||
InsightModule,
|
||||
InsightStore,
|
||||
WindowKind,
|
||||
} from './types';
|
||||
import { resolveWindow } from './windows';
|
||||
|
||||
const DEFAULT_WINDOWS: WindowKind[] = [
|
||||
'yesterday',
|
||||
'rolling_7d',
|
||||
'rolling_30d',
|
||||
];
|
||||
|
||||
export interface EngineConfig {
|
||||
keepTopNPerModuleWindow: number; // e.g. 5
|
||||
closeStaleAfterDays: number; // e.g. 7
|
||||
dimensionBatchSize: number; // e.g. 50
|
||||
globalThresholds: {
|
||||
minTotal: number; // e.g. 200
|
||||
minAbsDelta: number; // e.g. 80
|
||||
minPct: number; // e.g. 0.15
|
||||
};
|
||||
}
|
||||
|
||||
/** Simple gating to cut noise; modules can override via thresholds. */
|
||||
function passesThresholds(
|
||||
r: ComputeResult,
|
||||
mod: InsightModule,
|
||||
cfg: EngineConfig,
|
||||
): boolean {
|
||||
const t = mod.thresholds ?? {};
|
||||
const minTotal = t.minTotal ?? cfg.globalThresholds.minTotal;
|
||||
const minAbsDelta = t.minAbsDelta ?? cfg.globalThresholds.minAbsDelta;
|
||||
const minPct = t.minPct ?? cfg.globalThresholds.minPct;
|
||||
const cur = r.currentValue ?? 0;
|
||||
const cmp = r.compareValue ?? 0;
|
||||
const total = cur + cmp;
|
||||
const absDelta = Math.abs(cur - cmp);
|
||||
const pct = Math.abs(r.changePct ?? 0);
|
||||
if (total < minTotal) return false;
|
||||
if (absDelta < minAbsDelta) return false;
|
||||
if (pct < minPct) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
function chunk<T>(arr: T[], size: number): T[][] {
|
||||
if (size <= 0) return [arr];
|
||||
const out: T[][] = [];
|
||||
for (let i = 0; i < arr.length; i += size) out.push(arr.slice(i, i + size));
|
||||
return out;
|
||||
}
|
||||
|
||||
export function createEngine(args: {
|
||||
store: InsightStore;
|
||||
modules: InsightModule[];
|
||||
db: any;
|
||||
logger?: Pick<Console, 'info' | 'warn' | 'error'>;
|
||||
config: EngineConfig;
|
||||
}) {
|
||||
const { store, modules, db, config } = args;
|
||||
const logger = args.logger ?? console;
|
||||
|
||||
function isProjectOldEnoughForWindow(
|
||||
projectCreatedAt: Date | null | undefined,
|
||||
baselineStart: Date,
|
||||
): boolean {
|
||||
if (!projectCreatedAt) return true; // best-effort; don't block if unknown
|
||||
return projectCreatedAt.getTime() <= baselineStart.getTime();
|
||||
}
|
||||
|
||||
async function runProject(opts: {
|
||||
projectId: string;
|
||||
cadence: Cadence;
|
||||
now: Date;
|
||||
projectCreatedAt?: Date | null;
|
||||
}): Promise<void> {
|
||||
const { projectId, cadence, now, projectCreatedAt } = opts;
|
||||
const projLogger = logger;
|
||||
const eligible = modules.filter((m) => m.cadence.includes(cadence));
|
||||
|
||||
for (const mod of eligible) {
|
||||
const windows = mod.windows ?? DEFAULT_WINDOWS;
|
||||
for (const windowKind of windows) {
|
||||
let window: ReturnType<typeof resolveWindow>;
|
||||
let ctx: ComputeContext;
|
||||
try {
|
||||
window = resolveWindow(windowKind, now);
|
||||
if (
|
||||
!isProjectOldEnoughForWindow(projectCreatedAt, window.baselineStart)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
// Initialize cache for this module+window combination.
|
||||
// Cache is automatically garbage collected when context goes out of scope.
|
||||
const cache = new Map<string, any>();
|
||||
ctx = {
|
||||
projectId,
|
||||
window,
|
||||
db,
|
||||
now,
|
||||
logger: projLogger,
|
||||
clix: createCachedClix(db, cache),
|
||||
};
|
||||
} catch (e) {
|
||||
projLogger.error('[insights] failed to create compute context', {
|
||||
projectId,
|
||||
module: mod.key,
|
||||
windowKind,
|
||||
err: e,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
// 1) enumerate dimensions
|
||||
let dims: string[] = [];
|
||||
try {
|
||||
dims = mod.enumerateDimensions
|
||||
? await mod.enumerateDimensions(ctx)
|
||||
: [];
|
||||
} catch (e) {
|
||||
// Important: enumeration failures should not abort the whole project run.
|
||||
// Also avoid lifecycle close/suppression when we didn't actually evaluate dims.
|
||||
projLogger.error('[insights] module enumerateDimensions failed', {
|
||||
projectId,
|
||||
module: mod.key,
|
||||
windowKind,
|
||||
err: e,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const maxDims = mod.thresholds?.maxDims ?? 25;
|
||||
if (dims.length > maxDims) dims = dims.slice(0, maxDims);
|
||||
|
||||
if (dims.length === 0) {
|
||||
// Still do lifecycle close / suppression based on "nothing emitted"
|
||||
await store.closeMissingActiveInsights({
|
||||
projectId,
|
||||
moduleKey: mod.key,
|
||||
windowKind,
|
||||
seenDimensionKeys: [],
|
||||
now,
|
||||
staleDays: config.closeStaleAfterDays,
|
||||
});
|
||||
|
||||
await store.applySuppression({
|
||||
projectId,
|
||||
moduleKey: mod.key,
|
||||
windowKind,
|
||||
keepTopN: config.keepTopNPerModuleWindow,
|
||||
now,
|
||||
});
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
// 2) compute in batches
|
||||
const seen: string[] = [];
|
||||
const dimBatches = chunk(dims, config.dimensionBatchSize);
|
||||
for (const batch of dimBatches) {
|
||||
let results: ComputeResult[] = [];
|
||||
try {
|
||||
results = await mod.computeMany(ctx, batch);
|
||||
} catch (e) {
|
||||
projLogger.error('[insights] module computeMany failed', {
|
||||
projectId,
|
||||
module: mod.key,
|
||||
windowKind,
|
||||
err: e,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
for (const r of results) {
|
||||
if (!r?.ok) continue;
|
||||
if (!r.dimensionKey) continue;
|
||||
|
||||
// 3) gate noise
|
||||
if (!passesThresholds(r, mod, config)) continue;
|
||||
|
||||
// 4) score
|
||||
const impact = mod.score
|
||||
? mod.score(r, ctx)
|
||||
: defaultImpactScore(r);
|
||||
const sev = severityBand(r.changePct);
|
||||
|
||||
// 5) dedupe/material change requires loading prev identity
|
||||
const prev = await store.getActiveInsightByIdentity({
|
||||
projectId,
|
||||
moduleKey: mod.key,
|
||||
dimensionKey: r.dimensionKey,
|
||||
windowKind,
|
||||
});
|
||||
|
||||
const decision = materialDecision(prev, {
|
||||
changePct: r.changePct,
|
||||
direction: r.direction,
|
||||
});
|
||||
|
||||
// 6) render
|
||||
const card = mod.render(r, ctx);
|
||||
|
||||
// 7) upsert
|
||||
const persisted = await store.upsertInsight({
|
||||
projectId,
|
||||
moduleKey: mod.key,
|
||||
dimensionKey: r.dimensionKey,
|
||||
window,
|
||||
card,
|
||||
metrics: {
|
||||
direction: r.direction,
|
||||
impactScore: impact,
|
||||
severityBand: sev,
|
||||
},
|
||||
now,
|
||||
decision,
|
||||
prev,
|
||||
});
|
||||
|
||||
seen.push(r.dimensionKey);
|
||||
|
||||
// 8) events only when material
|
||||
if (!prev) {
|
||||
await store.insertEvent({
|
||||
projectId,
|
||||
insightId: persisted.id,
|
||||
moduleKey: mod.key,
|
||||
dimensionKey: r.dimensionKey,
|
||||
windowKind,
|
||||
eventKind: 'created',
|
||||
changeFrom: null,
|
||||
changeTo: {
|
||||
title: card.title,
|
||||
changePct: r.changePct,
|
||||
direction: r.direction,
|
||||
impact,
|
||||
severityBand: sev,
|
||||
},
|
||||
now,
|
||||
});
|
||||
} else if (decision.material) {
|
||||
const eventKind =
|
||||
decision.reason === 'direction_flip'
|
||||
? 'direction_flip'
|
||||
: decision.reason === 'severity_change'
|
||||
? sev && prev.severityBand && sev > prev.severityBand
|
||||
? 'severity_up'
|
||||
: 'severity_down'
|
||||
: 'updated';
|
||||
|
||||
await store.insertEvent({
|
||||
projectId,
|
||||
insightId: persisted.id,
|
||||
moduleKey: mod.key,
|
||||
dimensionKey: r.dimensionKey,
|
||||
windowKind,
|
||||
eventKind,
|
||||
changeFrom: {
|
||||
direction: prev.direction,
|
||||
impactScore: prev.impactScore,
|
||||
severityBand: prev.severityBand,
|
||||
},
|
||||
changeTo: {
|
||||
changePct: r.changePct,
|
||||
direction: r.direction,
|
||||
impactScore: impact,
|
||||
severityBand: sev,
|
||||
},
|
||||
now,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 10) lifecycle: close missing insights for this module/window
|
||||
await store.closeMissingActiveInsights({
|
||||
projectId,
|
||||
moduleKey: mod.key,
|
||||
windowKind,
|
||||
seenDimensionKeys: seen,
|
||||
now,
|
||||
staleDays: config.closeStaleAfterDays,
|
||||
});
|
||||
|
||||
// 11) suppression: keep top N
|
||||
await store.applySuppression({
|
||||
projectId,
|
||||
moduleKey: mod.key,
|
||||
windowKind,
|
||||
keepTopN: config.keepTopNPerModuleWindow,
|
||||
now,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return { runProject };
|
||||
}
|
||||
@@ -1,8 +0,0 @@
|
||||
export * from './types';
|
||||
export * from './windows';
|
||||
export * from './scoring';
|
||||
export * from './material';
|
||||
export * from './engine';
|
||||
export * from './store';
|
||||
export * from './utils';
|
||||
export * from './modules';
|
||||
@@ -1,43 +0,0 @@
|
||||
import { severityBand as band } from './scoring';
|
||||
import type { MaterialDecision, PersistedInsight } from './types';
|
||||
|
||||
export function materialDecision(
|
||||
prev: PersistedInsight | null,
|
||||
next: {
|
||||
changePct?: number;
|
||||
direction?: 'up' | 'down' | 'flat';
|
||||
},
|
||||
): MaterialDecision {
|
||||
const nextBand = band(next.changePct);
|
||||
if (!prev) {
|
||||
return { material: true, reason: 'created', newSeverityBand: nextBand };
|
||||
}
|
||||
|
||||
// direction flip is always meaningful
|
||||
const prevDir = (prev.direction ?? 'flat') as any;
|
||||
const nextDir = next.direction ?? 'flat';
|
||||
if (prevDir !== nextDir && (nextDir === 'up' || nextDir === 'down')) {
|
||||
return {
|
||||
material: true,
|
||||
reason: 'direction_flip',
|
||||
newSeverityBand: nextBand,
|
||||
};
|
||||
}
|
||||
|
||||
// severity band change
|
||||
const prevBand = (prev.severityBand ?? null) as any;
|
||||
if (prevBand !== nextBand && nextBand !== null) {
|
||||
return {
|
||||
material: true,
|
||||
reason: 'severity_change',
|
||||
newSeverityBand: nextBand,
|
||||
};
|
||||
}
|
||||
|
||||
// Otherwise: treat as non-material (silent refresh). You can add deadband crossing here if you store prior changePct.
|
||||
return {
|
||||
material: false,
|
||||
reason: 'none',
|
||||
newSeverityBand: prevBand ?? nextBand,
|
||||
};
|
||||
}
|
||||
@@ -1,275 +0,0 @@
|
||||
import { TABLE_NAMES, formatClickhouseDate } from '../../../clickhouse/client';
|
||||
import type {
|
||||
ComputeContext,
|
||||
ComputeResult,
|
||||
InsightModule,
|
||||
RenderedCard,
|
||||
} from '../types';
|
||||
import {
|
||||
buildLookupMap,
|
||||
computeChangePct,
|
||||
computeDirection,
|
||||
computeMedian,
|
||||
getEndOfDay,
|
||||
getWeekday,
|
||||
selectTopDimensions,
|
||||
} from '../utils';
|
||||
|
||||
async function fetchDeviceAggregates(ctx: ComputeContext): Promise<{
|
||||
currentMap: Map<string, number>;
|
||||
baselineMap: Map<string, number>;
|
||||
totalCurrent: number;
|
||||
totalBaseline: number;
|
||||
}> {
|
||||
if (ctx.window.kind === 'yesterday') {
|
||||
const [currentResults, baselineResults, totals] = await Promise.all([
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ device: string; cnt: number }>(['device', 'count(*) as cnt'])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.start,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.groupBy(['device'])
|
||||
.execute(),
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ date: string; device: string; cnt: number }>([
|
||||
'toDate(created_at) as date',
|
||||
'device',
|
||||
'count(*) as cnt',
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.baselineEnd),
|
||||
])
|
||||
.groupBy(['date', 'device'])
|
||||
.execute(),
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ cur_total: number }>([
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${formatClickhouseDate(ctx.window.start)}' AND '${formatClickhouseDate(getEndOfDay(ctx.window.end))}') as cur_total`,
|
||||
),
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.execute(),
|
||||
]);
|
||||
|
||||
const currentMap = buildLookupMap(currentResults, (r) => r.device);
|
||||
|
||||
const targetWeekday = getWeekday(ctx.window.start);
|
||||
const aggregated = new Map<string, { date: string; cnt: number }[]>();
|
||||
for (const r of baselineResults) {
|
||||
if (!aggregated.has(r.device)) {
|
||||
aggregated.set(r.device, []);
|
||||
}
|
||||
const entries = aggregated.get(r.device)!;
|
||||
const existing = entries.find((e) => e.date === r.date);
|
||||
if (existing) {
|
||||
existing.cnt += Number(r.cnt ?? 0);
|
||||
} else {
|
||||
entries.push({ date: r.date, cnt: Number(r.cnt ?? 0) });
|
||||
}
|
||||
}
|
||||
|
||||
const baselineMap = new Map<string, number>();
|
||||
for (const [deviceType, entries] of aggregated) {
|
||||
const sameWeekdayValues = entries
|
||||
.filter((e) => getWeekday(new Date(e.date)) === targetWeekday)
|
||||
.map((e) => e.cnt)
|
||||
.sort((a, b) => a - b);
|
||||
|
||||
if (sameWeekdayValues.length > 0) {
|
||||
baselineMap.set(deviceType, computeMedian(sameWeekdayValues));
|
||||
}
|
||||
}
|
||||
|
||||
const totalCurrent = totals[0]?.cur_total ?? 0;
|
||||
const totalBaseline =
|
||||
baselineMap.size > 0
|
||||
? Array.from(baselineMap.values()).reduce((sum, val) => sum + val, 0)
|
||||
: 0;
|
||||
|
||||
return { currentMap, baselineMap, totalCurrent, totalBaseline };
|
||||
}
|
||||
|
||||
const curStart = formatClickhouseDate(ctx.window.start);
|
||||
const curEnd = formatClickhouseDate(getEndOfDay(ctx.window.end));
|
||||
const baseStart = formatClickhouseDate(ctx.window.baselineStart);
|
||||
const baseEnd = formatClickhouseDate(getEndOfDay(ctx.window.baselineEnd));
|
||||
|
||||
const [results, totals] = await Promise.all([
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ device: string; cur: number; base: number }>([
|
||||
'device',
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur`,
|
||||
),
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base`,
|
||||
),
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.groupBy(['device'])
|
||||
.execute(),
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ cur_total: number; base_total: number }>([
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur_total`,
|
||||
),
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base_total`,
|
||||
),
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.execute(),
|
||||
]);
|
||||
|
||||
const currentMap = buildLookupMap(
|
||||
results,
|
||||
(r) => r.device,
|
||||
(r) => Number(r.cur ?? 0),
|
||||
);
|
||||
|
||||
const baselineMap = buildLookupMap(
|
||||
results,
|
||||
(r) => r.device,
|
||||
(r) => Number(r.base ?? 0),
|
||||
);
|
||||
|
||||
const totalCurrent = totals[0]?.cur_total ?? 0;
|
||||
const totalBaseline = totals[0]?.base_total ?? 0;
|
||||
|
||||
return { currentMap, baselineMap, totalCurrent, totalBaseline };
|
||||
}
|
||||
|
||||
export const devicesModule: InsightModule = {
|
||||
key: 'devices',
|
||||
cadence: ['daily'],
|
||||
thresholds: { minTotal: 100, minAbsDelta: 0, minPct: 0.08, maxDims: 5 },
|
||||
|
||||
async enumerateDimensions(ctx) {
|
||||
const { currentMap, baselineMap } = await fetchDeviceAggregates(ctx);
|
||||
const topDims = selectTopDimensions(
|
||||
currentMap,
|
||||
baselineMap,
|
||||
this.thresholds?.maxDims ?? 5,
|
||||
);
|
||||
return topDims.map((dim) => `device:${dim}`);
|
||||
},
|
||||
|
||||
async computeMany(ctx, dimensionKeys): Promise<ComputeResult[]> {
|
||||
const { currentMap, baselineMap, totalCurrent, totalBaseline } =
|
||||
await fetchDeviceAggregates(ctx);
|
||||
const results: ComputeResult[] = [];
|
||||
|
||||
for (const dimKey of dimensionKeys) {
|
||||
if (!dimKey.startsWith('device:')) continue;
|
||||
const deviceType = dimKey.replace('device:', '');
|
||||
|
||||
const currentValue = currentMap.get(deviceType) ?? 0;
|
||||
const compareValue = baselineMap.get(deviceType) ?? 0;
|
||||
|
||||
const currentShare = totalCurrent > 0 ? currentValue / totalCurrent : 0;
|
||||
const compareShare = totalBaseline > 0 ? compareValue / totalBaseline : 0;
|
||||
|
||||
const shareShiftPp = (currentShare - compareShare) * 100;
|
||||
const changePct = computeChangePct(currentValue, compareValue);
|
||||
const direction = computeDirection(changePct);
|
||||
|
||||
results.push({
|
||||
ok: true,
|
||||
dimensionKey: dimKey,
|
||||
currentValue,
|
||||
compareValue,
|
||||
changePct,
|
||||
direction,
|
||||
extra: {
|
||||
shareShiftPp,
|
||||
currentShare,
|
||||
compareShare,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
return results;
|
||||
},
|
||||
|
||||
render(result, ctx): RenderedCard {
|
||||
const device = result.dimensionKey.replace('device:', '');
|
||||
const changePct = result.changePct ?? 0;
|
||||
const isIncrease = changePct >= 0;
|
||||
|
||||
const sessionsCurrent = result.currentValue ?? 0;
|
||||
const sessionsCompare = result.compareValue ?? 0;
|
||||
const shareCurrent = Number(result.extra?.currentShare ?? 0);
|
||||
const shareCompare = Number(result.extra?.compareShare ?? 0);
|
||||
|
||||
return {
|
||||
title: `${device} ${isIncrease ? '↑' : '↓'} ${Math.abs(changePct * 100).toFixed(0)}%`,
|
||||
summary: `${ctx.window.label}. Device traffic change.`,
|
||||
displayName: device,
|
||||
payload: {
|
||||
kind: 'insight_v1',
|
||||
dimensions: [{ key: 'device', value: device, displayName: device }],
|
||||
primaryMetric: 'sessions',
|
||||
metrics: {
|
||||
sessions: {
|
||||
current: sessionsCurrent,
|
||||
compare: sessionsCompare,
|
||||
delta: sessionsCurrent - sessionsCompare,
|
||||
changePct: sessionsCompare > 0 ? (result.changePct ?? 0) : null,
|
||||
direction: result.direction ?? 'flat',
|
||||
unit: 'count',
|
||||
},
|
||||
share: {
|
||||
current: shareCurrent,
|
||||
compare: shareCompare,
|
||||
delta: shareCurrent - shareCompare,
|
||||
changePct:
|
||||
shareCompare > 0
|
||||
? (shareCurrent - shareCompare) / shareCompare
|
||||
: null,
|
||||
direction:
|
||||
shareCurrent - shareCompare > 0.0005
|
||||
? 'up'
|
||||
: shareCurrent - shareCompare < -0.0005
|
||||
? 'down'
|
||||
: 'flat',
|
||||
unit: 'ratio',
|
||||
},
|
||||
},
|
||||
extra: {
|
||||
// keep module-specific flags/fields if needed later
|
||||
},
|
||||
},
|
||||
};
|
||||
},
|
||||
};
|
||||
@@ -1,287 +0,0 @@
|
||||
import { TABLE_NAMES, formatClickhouseDate } from '../../../clickhouse/client';
|
||||
import type {
|
||||
ComputeContext,
|
||||
ComputeResult,
|
||||
InsightModule,
|
||||
RenderedCard,
|
||||
} from '../types';
|
||||
import {
|
||||
buildLookupMap,
|
||||
computeChangePct,
|
||||
computeDirection,
|
||||
computeWeekdayMedians,
|
||||
getEndOfDay,
|
||||
getWeekday,
|
||||
selectTopDimensions,
|
||||
} from '../utils';
|
||||
|
||||
const DELIMITER = '|||';
|
||||
|
||||
async function fetchEntryPageAggregates(ctx: ComputeContext): Promise<{
|
||||
currentMap: Map<string, number>;
|
||||
baselineMap: Map<string, number>;
|
||||
totalCurrent: number;
|
||||
totalBaseline: number;
|
||||
}> {
|
||||
if (ctx.window.kind === 'yesterday') {
|
||||
const [currentResults, baselineResults, totals] = await Promise.all([
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ entry_origin: string; entry_path: string; cnt: number }>([
|
||||
'entry_origin',
|
||||
'entry_path',
|
||||
'count(*) as cnt',
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.start,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.groupBy(['entry_origin', 'entry_path'])
|
||||
.execute(),
|
||||
ctx
|
||||
.clix()
|
||||
.select<{
|
||||
date: string;
|
||||
entry_origin: string;
|
||||
entry_path: string;
|
||||
cnt: number;
|
||||
}>([
|
||||
'toDate(created_at) as date',
|
||||
'entry_origin',
|
||||
'entry_path',
|
||||
'count(*) as cnt',
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.baselineEnd),
|
||||
])
|
||||
.groupBy(['date', 'entry_origin', 'entry_path'])
|
||||
.execute(),
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ cur_total: number }>([
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${formatClickhouseDate(ctx.window.start)}' AND '${formatClickhouseDate(getEndOfDay(ctx.window.end))}') as cur_total`,
|
||||
),
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.execute(),
|
||||
]);
|
||||
|
||||
const currentMap = buildLookupMap(
|
||||
currentResults,
|
||||
(r) => `${r.entry_origin || ''}${DELIMITER}${r.entry_path || '/'}`,
|
||||
);
|
||||
|
||||
const targetWeekday = getWeekday(ctx.window.start);
|
||||
const baselineMap = computeWeekdayMedians(
|
||||
baselineResults,
|
||||
targetWeekday,
|
||||
(r) => `${r.entry_origin || ''}${DELIMITER}${r.entry_path || '/'}`,
|
||||
);
|
||||
|
||||
const totalCurrent = totals[0]?.cur_total ?? 0;
|
||||
const totalBaseline = Array.from(baselineMap.values()).reduce(
|
||||
(sum, val) => sum + val,
|
||||
0,
|
||||
);
|
||||
|
||||
return { currentMap, baselineMap, totalCurrent, totalBaseline };
|
||||
}
|
||||
|
||||
const curStart = formatClickhouseDate(ctx.window.start);
|
||||
const curEnd = formatClickhouseDate(getEndOfDay(ctx.window.end));
|
||||
const baseStart = formatClickhouseDate(ctx.window.baselineStart);
|
||||
const baseEnd = formatClickhouseDate(getEndOfDay(ctx.window.baselineEnd));
|
||||
|
||||
const [results, totals] = await Promise.all([
|
||||
ctx
|
||||
.clix()
|
||||
.select<{
|
||||
entry_origin: string;
|
||||
entry_path: string;
|
||||
cur: number;
|
||||
base: number;
|
||||
}>([
|
||||
'entry_origin',
|
||||
'entry_path',
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur`,
|
||||
),
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base`,
|
||||
),
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.groupBy(['entry_origin', 'entry_path'])
|
||||
.execute(),
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ cur_total: number; base_total: number }>([
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur_total`,
|
||||
),
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base_total`,
|
||||
),
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.execute(),
|
||||
]);
|
||||
|
||||
const currentMap = buildLookupMap(
|
||||
results,
|
||||
(r) => `${r.entry_origin || ''}${DELIMITER}${r.entry_path || '/'}`,
|
||||
(r) => Number(r.cur ?? 0),
|
||||
);
|
||||
|
||||
const baselineMap = buildLookupMap(
|
||||
results,
|
||||
(r) => `${r.entry_origin || ''}${DELIMITER}${r.entry_path || '/'}`,
|
||||
(r) => Number(r.base ?? 0),
|
||||
);
|
||||
|
||||
const totalCurrent = totals[0]?.cur_total ?? 0;
|
||||
const totalBaseline = totals[0]?.base_total ?? 0;
|
||||
|
||||
return { currentMap, baselineMap, totalCurrent, totalBaseline };
|
||||
}
|
||||
|
||||
export const entryPagesModule: InsightModule = {
|
||||
key: 'entry-pages',
|
||||
cadence: ['daily'],
|
||||
thresholds: { minTotal: 100, minAbsDelta: 30, minPct: 0.2, maxDims: 100 },
|
||||
|
||||
async enumerateDimensions(ctx) {
|
||||
const { currentMap, baselineMap } = await fetchEntryPageAggregates(ctx);
|
||||
const topDims = selectTopDimensions(
|
||||
currentMap,
|
||||
baselineMap,
|
||||
this.thresholds?.maxDims ?? 100,
|
||||
);
|
||||
return topDims.map((dim) => `entry:${dim}`);
|
||||
},
|
||||
|
||||
async computeMany(ctx, dimensionKeys): Promise<ComputeResult[]> {
|
||||
const { currentMap, baselineMap, totalCurrent, totalBaseline } =
|
||||
await fetchEntryPageAggregates(ctx);
|
||||
const results: ComputeResult[] = [];
|
||||
|
||||
for (const dimKey of dimensionKeys) {
|
||||
if (!dimKey.startsWith('entry:')) continue;
|
||||
const originPath = dimKey.replace('entry:', '');
|
||||
|
||||
const currentValue = currentMap.get(originPath) ?? 0;
|
||||
const compareValue = baselineMap.get(originPath) ?? 0;
|
||||
|
||||
const currentShare = totalCurrent > 0 ? currentValue / totalCurrent : 0;
|
||||
const compareShare = totalBaseline > 0 ? compareValue / totalBaseline : 0;
|
||||
|
||||
const shareShiftPp = (currentShare - compareShare) * 100;
|
||||
const changePct = computeChangePct(currentValue, compareValue);
|
||||
const direction = computeDirection(changePct);
|
||||
|
||||
results.push({
|
||||
ok: true,
|
||||
dimensionKey: dimKey,
|
||||
currentValue,
|
||||
compareValue,
|
||||
changePct,
|
||||
direction,
|
||||
extra: {
|
||||
shareShiftPp,
|
||||
currentShare,
|
||||
compareShare,
|
||||
isNew: compareValue === 0 && currentValue > 0,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
return results;
|
||||
},
|
||||
|
||||
render(result, ctx): RenderedCard {
|
||||
const originPath = result.dimensionKey.replace('entry:', '');
|
||||
const [origin, path] = originPath.split(DELIMITER);
|
||||
const displayValue = origin ? `${origin}${path}` : path || '/';
|
||||
const pct = ((result.changePct ?? 0) * 100).toFixed(1);
|
||||
const isIncrease = (result.changePct ?? 0) >= 0;
|
||||
const isNew = result.extra?.isNew as boolean | undefined;
|
||||
|
||||
const title = isNew
|
||||
? `New entry page: ${displayValue}`
|
||||
: `Entry page ${displayValue} ${isIncrease ? '↑' : '↓'} ${Math.abs(Number(pct))}%`;
|
||||
|
||||
const sessionsCurrent = result.currentValue ?? 0;
|
||||
const sessionsCompare = result.compareValue ?? 0;
|
||||
const shareCurrent = Number(result.extra?.currentShare ?? 0);
|
||||
const shareCompare = Number(result.extra?.compareShare ?? 0);
|
||||
|
||||
return {
|
||||
title,
|
||||
summary: `${ctx.window.label}. Sessions ${sessionsCurrent} vs ${sessionsCompare}.`,
|
||||
displayName: displayValue,
|
||||
payload: {
|
||||
kind: 'insight_v1',
|
||||
dimensions: [
|
||||
{ key: 'origin', value: origin ?? '', displayName: origin ?? '' },
|
||||
{ key: 'path', value: path ?? '', displayName: path ?? '' },
|
||||
],
|
||||
primaryMetric: 'sessions',
|
||||
metrics: {
|
||||
sessions: {
|
||||
current: sessionsCurrent,
|
||||
compare: sessionsCompare,
|
||||
delta: sessionsCurrent - sessionsCompare,
|
||||
changePct: sessionsCompare > 0 ? (result.changePct ?? 0) : null,
|
||||
direction: result.direction ?? 'flat',
|
||||
unit: 'count',
|
||||
},
|
||||
share: {
|
||||
current: shareCurrent,
|
||||
compare: shareCompare,
|
||||
delta: shareCurrent - shareCompare,
|
||||
changePct:
|
||||
shareCompare > 0
|
||||
? (shareCurrent - shareCompare) / shareCompare
|
||||
: null,
|
||||
direction:
|
||||
shareCurrent - shareCompare > 0.0005
|
||||
? 'up'
|
||||
: shareCurrent - shareCompare < -0.0005
|
||||
? 'down'
|
||||
: 'flat',
|
||||
unit: 'ratio',
|
||||
},
|
||||
},
|
||||
extra: {
|
||||
isNew: result.extra?.isNew,
|
||||
},
|
||||
},
|
||||
};
|
||||
},
|
||||
};
|
||||
@@ -1,271 +0,0 @@
|
||||
import { getCountry } from '@openpanel/constants';
|
||||
import { TABLE_NAMES, formatClickhouseDate } from '../../../clickhouse/client';
|
||||
import type {
|
||||
ComputeContext,
|
||||
ComputeResult,
|
||||
InsightModule,
|
||||
RenderedCard,
|
||||
} from '../types';
|
||||
import {
|
||||
buildLookupMap,
|
||||
computeChangePct,
|
||||
computeDirection,
|
||||
computeWeekdayMedians,
|
||||
getEndOfDay,
|
||||
getWeekday,
|
||||
selectTopDimensions,
|
||||
} from '../utils';
|
||||
|
||||
async function fetchGeoAggregates(ctx: ComputeContext): Promise<{
|
||||
currentMap: Map<string, number>;
|
||||
baselineMap: Map<string, number>;
|
||||
totalCurrent: number;
|
||||
totalBaseline: number;
|
||||
}> {
|
||||
if (ctx.window.kind === 'yesterday') {
|
||||
const [currentResults, baselineResults, totals] = await Promise.all([
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ country: string; cnt: number }>([
|
||||
'country',
|
||||
'count(*) as cnt',
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.start,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.groupBy(['country'])
|
||||
.execute(),
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ date: string; country: string; cnt: number }>([
|
||||
'toDate(created_at) as date',
|
||||
'country',
|
||||
'count(*) as cnt',
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.baselineEnd),
|
||||
])
|
||||
.groupBy(['date', 'country'])
|
||||
.execute(),
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ cur_total: number }>([
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${formatClickhouseDate(ctx.window.start)}' AND '${formatClickhouseDate(getEndOfDay(ctx.window.end))}') as cur_total`,
|
||||
),
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.execute(),
|
||||
]);
|
||||
|
||||
const currentMap = buildLookupMap(
|
||||
currentResults,
|
||||
(r) => r.country || 'unknown',
|
||||
);
|
||||
|
||||
const targetWeekday = getWeekday(ctx.window.start);
|
||||
const baselineMap = computeWeekdayMedians(
|
||||
baselineResults,
|
||||
targetWeekday,
|
||||
(r) => r.country || 'unknown',
|
||||
);
|
||||
|
||||
const totalCurrent = totals[0]?.cur_total ?? 0;
|
||||
const totalBaseline = Array.from(baselineMap.values()).reduce(
|
||||
(sum, val) => sum + val,
|
||||
0,
|
||||
);
|
||||
|
||||
return { currentMap, baselineMap, totalCurrent, totalBaseline };
|
||||
}
|
||||
|
||||
const curStart = formatClickhouseDate(ctx.window.start);
|
||||
const curEnd = formatClickhouseDate(getEndOfDay(ctx.window.end));
|
||||
const baseStart = formatClickhouseDate(ctx.window.baselineStart);
|
||||
const baseEnd = formatClickhouseDate(getEndOfDay(ctx.window.baselineEnd));
|
||||
|
||||
const [results, totals] = await Promise.all([
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ country: string; cur: number; base: number }>([
|
||||
'country',
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur`,
|
||||
),
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base`,
|
||||
),
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.groupBy(['country'])
|
||||
.execute(),
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ cur_total: number; base_total: number }>([
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur_total`,
|
||||
),
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base_total`,
|
||||
),
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.execute(),
|
||||
]);
|
||||
|
||||
const currentMap = buildLookupMap(
|
||||
results,
|
||||
(r) => r.country || 'unknown',
|
||||
(r) => Number(r.cur ?? 0),
|
||||
);
|
||||
|
||||
const baselineMap = buildLookupMap(
|
||||
results,
|
||||
(r) => r.country || 'unknown',
|
||||
(r) => Number(r.base ?? 0),
|
||||
);
|
||||
|
||||
const totalCurrent = totals[0]?.cur_total ?? 0;
|
||||
const totalBaseline = totals[0]?.base_total ?? 0;
|
||||
|
||||
return { currentMap, baselineMap, totalCurrent, totalBaseline };
|
||||
}
|
||||
|
||||
export const geoModule: InsightModule = {
|
||||
key: 'geo',
|
||||
cadence: ['daily'],
|
||||
thresholds: { minTotal: 100, minAbsDelta: 0, minPct: 0.08, maxDims: 30 },
|
||||
|
||||
async enumerateDimensions(ctx) {
|
||||
const { currentMap, baselineMap } = await fetchGeoAggregates(ctx);
|
||||
const topDims = selectTopDimensions(
|
||||
currentMap,
|
||||
baselineMap,
|
||||
this.thresholds?.maxDims ?? 30,
|
||||
);
|
||||
return topDims.map((dim) => `country:${dim}`);
|
||||
},
|
||||
|
||||
async computeMany(ctx, dimensionKeys): Promise<ComputeResult[]> {
|
||||
const { currentMap, baselineMap, totalCurrent, totalBaseline } =
|
||||
await fetchGeoAggregates(ctx);
|
||||
const results: ComputeResult[] = [];
|
||||
|
||||
for (const dimKey of dimensionKeys) {
|
||||
if (!dimKey.startsWith('country:')) continue;
|
||||
const country = dimKey.replace('country:', '');
|
||||
|
||||
const currentValue = currentMap.get(country) ?? 0;
|
||||
const compareValue = baselineMap.get(country) ?? 0;
|
||||
|
||||
const currentShare = totalCurrent > 0 ? currentValue / totalCurrent : 0;
|
||||
const compareShare = totalBaseline > 0 ? compareValue / totalBaseline : 0;
|
||||
|
||||
const shareShiftPp = (currentShare - compareShare) * 100;
|
||||
const changePct = computeChangePct(currentValue, compareValue);
|
||||
const direction = computeDirection(changePct);
|
||||
|
||||
results.push({
|
||||
ok: true,
|
||||
dimensionKey: dimKey,
|
||||
currentValue,
|
||||
compareValue,
|
||||
changePct,
|
||||
direction,
|
||||
extra: {
|
||||
shareShiftPp,
|
||||
currentShare,
|
||||
compareShare,
|
||||
isNew: compareValue === 0 && currentValue > 0,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
return results;
|
||||
},
|
||||
|
||||
render(result, ctx): RenderedCard {
|
||||
const country = result.dimensionKey.replace('country:', '');
|
||||
const changePct = result.changePct ?? 0;
|
||||
const isIncrease = changePct >= 0;
|
||||
const isNew = result.extra?.isNew as boolean | undefined;
|
||||
const displayName = getCountry(country);
|
||||
|
||||
const title = isNew
|
||||
? `New traffic from: ${displayName}`
|
||||
: `${displayName} ${isIncrease ? '↑' : '↓'} ${Math.abs(changePct * 100).toFixed(0)}%`;
|
||||
|
||||
const sessionsCurrent = result.currentValue ?? 0;
|
||||
const sessionsCompare = result.compareValue ?? 0;
|
||||
const shareCurrent = Number(result.extra?.currentShare ?? 0);
|
||||
const shareCompare = Number(result.extra?.compareShare ?? 0);
|
||||
|
||||
return {
|
||||
title,
|
||||
summary: `${ctx.window.label}. Traffic change from ${displayName}.`,
|
||||
displayName,
|
||||
payload: {
|
||||
kind: 'insight_v1',
|
||||
dimensions: [
|
||||
{ key: 'country', value: country, displayName: displayName },
|
||||
],
|
||||
primaryMetric: 'sessions',
|
||||
metrics: {
|
||||
sessions: {
|
||||
current: sessionsCurrent,
|
||||
compare: sessionsCompare,
|
||||
delta: sessionsCurrent - sessionsCompare,
|
||||
changePct: sessionsCompare > 0 ? (result.changePct ?? 0) : null,
|
||||
direction: result.direction ?? 'flat',
|
||||
unit: 'count',
|
||||
},
|
||||
share: {
|
||||
current: shareCurrent,
|
||||
compare: shareCompare,
|
||||
delta: shareCurrent - shareCompare,
|
||||
changePct:
|
||||
shareCompare > 0
|
||||
? (shareCurrent - shareCompare) / shareCompare
|
||||
: null,
|
||||
direction:
|
||||
shareCurrent - shareCompare > 0.0005
|
||||
? 'up'
|
||||
: shareCurrent - shareCompare < -0.0005
|
||||
? 'down'
|
||||
: 'flat',
|
||||
unit: 'ratio',
|
||||
},
|
||||
},
|
||||
extra: {
|
||||
isNew: result.extra?.isNew,
|
||||
},
|
||||
},
|
||||
};
|
||||
},
|
||||
};
|
||||
@@ -1,5 +0,0 @@
|
||||
export { referrersModule } from './referrers.module';
|
||||
export { entryPagesModule } from './entry-pages.module';
|
||||
export { pageTrendsModule } from './page-trends.module';
|
||||
export { geoModule } from './geo.module';
|
||||
export { devicesModule } from './devices.module';
|
||||
@@ -1,298 +0,0 @@
|
||||
import { TABLE_NAMES, formatClickhouseDate } from '../../../clickhouse/client';
|
||||
import type {
|
||||
ComputeContext,
|
||||
ComputeResult,
|
||||
InsightModule,
|
||||
RenderedCard,
|
||||
} from '../types';
|
||||
import {
|
||||
buildLookupMap,
|
||||
computeChangePct,
|
||||
computeDirection,
|
||||
computeWeekdayMedians,
|
||||
getEndOfDay,
|
||||
getWeekday,
|
||||
selectTopDimensions,
|
||||
} from '../utils';
|
||||
|
||||
const DELIMITER = '|||';
|
||||
|
||||
async function fetchPageTrendAggregates(ctx: ComputeContext): Promise<{
|
||||
currentMap: Map<string, number>;
|
||||
baselineMap: Map<string, number>;
|
||||
totalCurrent: number;
|
||||
totalBaseline: number;
|
||||
}> {
|
||||
if (ctx.window.kind === 'yesterday') {
|
||||
const [currentResults, baselineResults, totals] = await Promise.all([
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ origin: string; path: string; cnt: number }>([
|
||||
'origin',
|
||||
'path',
|
||||
'count(*) as cnt',
|
||||
])
|
||||
.from(TABLE_NAMES.events)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('name', '=', 'screen_view')
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.start,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.groupBy(['origin', 'path'])
|
||||
.execute(),
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ date: string; origin: string; path: string; cnt: number }>([
|
||||
'toDate(created_at) as date',
|
||||
'origin',
|
||||
'path',
|
||||
'count(*) as cnt',
|
||||
])
|
||||
.from(TABLE_NAMES.events)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('name', '=', 'screen_view')
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.baselineEnd),
|
||||
])
|
||||
.groupBy(['date', 'origin', 'path'])
|
||||
.execute(),
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ cur_total: number }>([
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${formatClickhouseDate(ctx.window.start)}' AND '${formatClickhouseDate(getEndOfDay(ctx.window.end))}') as cur_total`,
|
||||
),
|
||||
])
|
||||
.from(TABLE_NAMES.events)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('name', '=', 'screen_view')
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.execute(),
|
||||
]);
|
||||
|
||||
const currentMap = buildLookupMap(
|
||||
currentResults,
|
||||
(r) => `${r.origin || ''}${DELIMITER}${r.path || '/'}`,
|
||||
);
|
||||
|
||||
const targetWeekday = getWeekday(ctx.window.start);
|
||||
const baselineMap = computeWeekdayMedians(
|
||||
baselineResults,
|
||||
targetWeekday,
|
||||
(r) => `${r.origin || ''}${DELIMITER}${r.path || '/'}`,
|
||||
);
|
||||
|
||||
const totalCurrent = totals[0]?.cur_total ?? 0;
|
||||
const totalBaseline = Array.from(baselineMap.values()).reduce(
|
||||
(sum, val) => sum + val,
|
||||
0,
|
||||
);
|
||||
|
||||
return { currentMap, baselineMap, totalCurrent, totalBaseline };
|
||||
}
|
||||
|
||||
const curStart = formatClickhouseDate(ctx.window.start);
|
||||
const curEnd = formatClickhouseDate(getEndOfDay(ctx.window.end));
|
||||
const baseStart = formatClickhouseDate(ctx.window.baselineStart);
|
||||
const baseEnd = formatClickhouseDate(getEndOfDay(ctx.window.baselineEnd));
|
||||
|
||||
const [results, totals] = await Promise.all([
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ origin: string; path: string; cur: number; base: number }>([
|
||||
'origin',
|
||||
'path',
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur`,
|
||||
),
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base`,
|
||||
),
|
||||
])
|
||||
.from(TABLE_NAMES.events)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('name', '=', 'screen_view')
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.groupBy(['origin', 'path'])
|
||||
.execute(),
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ cur_total: number; base_total: number }>([
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur_total`,
|
||||
),
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base_total`,
|
||||
),
|
||||
])
|
||||
.from(TABLE_NAMES.events)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('name', '=', 'screen_view')
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.execute(),
|
||||
]);
|
||||
|
||||
const currentMap = buildLookupMap(
|
||||
results,
|
||||
(r) => `${r.origin || ''}${DELIMITER}${r.path || '/'}`,
|
||||
(r) => Number(r.cur ?? 0),
|
||||
);
|
||||
|
||||
const baselineMap = buildLookupMap(
|
||||
results,
|
||||
(r) => `${r.origin || ''}${DELIMITER}${r.path || '/'}`,
|
||||
(r) => Number(r.base ?? 0),
|
||||
);
|
||||
|
||||
const totalCurrent = totals[0]?.cur_total ?? 0;
|
||||
const totalBaseline = totals[0]?.base_total ?? 0;
|
||||
|
||||
return { currentMap, baselineMap, totalCurrent, totalBaseline };
|
||||
}
|
||||
|
||||
export const pageTrendsModule: InsightModule = {
|
||||
key: 'page-trends',
|
||||
cadence: ['daily'],
|
||||
// Share-based thresholds (values in basis points: 100 = 1%)
|
||||
// minTotal: require at least 0.5% combined share (current + baseline)
|
||||
// minAbsDelta: require at least 0.5 percentage point shift
|
||||
// minPct: require at least 25% relative change in share
|
||||
thresholds: { minTotal: 50, minAbsDelta: 50, minPct: 0.25, maxDims: 100 },
|
||||
|
||||
async enumerateDimensions(ctx) {
|
||||
const { currentMap, baselineMap } = await fetchPageTrendAggregates(ctx);
|
||||
const topDims = selectTopDimensions(
|
||||
currentMap,
|
||||
baselineMap,
|
||||
this.thresholds?.maxDims ?? 100,
|
||||
);
|
||||
return topDims.map((dim) => `page:${dim}`);
|
||||
},
|
||||
|
||||
async computeMany(ctx, dimensionKeys): Promise<ComputeResult[]> {
|
||||
const { currentMap, baselineMap, totalCurrent, totalBaseline } =
|
||||
await fetchPageTrendAggregates(ctx);
|
||||
const results: ComputeResult[] = [];
|
||||
|
||||
for (const dimKey of dimensionKeys) {
|
||||
if (!dimKey.startsWith('page:')) continue;
|
||||
const originPath = dimKey.replace('page:', '');
|
||||
|
||||
const pageviewsCurrent = currentMap.get(originPath) ?? 0;
|
||||
const pageviewsCompare = baselineMap.get(originPath) ?? 0;
|
||||
|
||||
const currentShare =
|
||||
totalCurrent > 0 ? pageviewsCurrent / totalCurrent : 0;
|
||||
const compareShare =
|
||||
totalBaseline > 0 ? pageviewsCompare / totalBaseline : 0;
|
||||
|
||||
// Use share values in basis points (100 = 1%) for thresholding
|
||||
// This makes thresholds intuitive: minAbsDelta=50 means 0.5pp shift
|
||||
const currentShareBp = currentShare * 10000;
|
||||
const compareShareBp = compareShare * 10000;
|
||||
|
||||
const shareShiftPp = (currentShare - compareShare) * 100;
|
||||
// changePct is relative change in share, not absolute pageviews
|
||||
const shareChangePct = computeChangePct(currentShare, compareShare);
|
||||
const direction = computeDirection(shareChangePct);
|
||||
|
||||
results.push({
|
||||
ok: true,
|
||||
dimensionKey: dimKey,
|
||||
// Use share in basis points for threshold checks
|
||||
currentValue: currentShareBp,
|
||||
compareValue: compareShareBp,
|
||||
changePct: shareChangePct,
|
||||
direction,
|
||||
extra: {
|
||||
// Keep absolute values for display
|
||||
pageviewsCurrent,
|
||||
pageviewsCompare,
|
||||
shareShiftPp,
|
||||
currentShare,
|
||||
compareShare,
|
||||
isNew: pageviewsCompare === 0 && pageviewsCurrent > 0,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
return results;
|
||||
},
|
||||
|
||||
render(result, ctx): RenderedCard {
|
||||
const originPath = result.dimensionKey.replace('page:', '');
|
||||
const [origin, path] = originPath.split(DELIMITER);
|
||||
const displayValue = origin ? `${origin}${path}` : path || '/';
|
||||
|
||||
// Get absolute pageviews from extra (currentValue/compareValue are now share-based)
|
||||
const pageviewsCurrent = Number(result.extra?.pageviewsCurrent ?? 0);
|
||||
const pageviewsCompare = Number(result.extra?.pageviewsCompare ?? 0);
|
||||
const shareCurrent = Number(result.extra?.currentShare ?? 0);
|
||||
const shareCompare = Number(result.extra?.compareShare ?? 0);
|
||||
const shareShiftPp = Number(result.extra?.shareShiftPp ?? 0);
|
||||
const isNew = result.extra?.isNew as boolean | undefined;
|
||||
|
||||
// Display share shift in percentage points
|
||||
const isIncrease = shareShiftPp >= 0;
|
||||
const shareShiftDisplay = Math.abs(shareShiftPp).toFixed(1);
|
||||
|
||||
const title = isNew
|
||||
? `New page getting views: ${displayValue}`
|
||||
: `Page ${displayValue} share ${isIncrease ? '↑' : '↓'} ${shareShiftDisplay}pp`;
|
||||
|
||||
return {
|
||||
title,
|
||||
summary: `${ctx.window.label}. Share ${(shareCurrent * 100).toFixed(1)}% vs ${(shareCompare * 100).toFixed(1)}%.`,
|
||||
displayName: displayValue,
|
||||
payload: {
|
||||
kind: 'insight_v1',
|
||||
dimensions: [
|
||||
{ key: 'origin', value: origin ?? '', displayName: origin ?? '' },
|
||||
{ key: 'path', value: path ?? '', displayName: path ?? '' },
|
||||
],
|
||||
primaryMetric: 'share',
|
||||
metrics: {
|
||||
pageviews: {
|
||||
current: pageviewsCurrent,
|
||||
compare: pageviewsCompare,
|
||||
delta: pageviewsCurrent - pageviewsCompare,
|
||||
changePct:
|
||||
pageviewsCompare > 0
|
||||
? (pageviewsCurrent - pageviewsCompare) / pageviewsCompare
|
||||
: null,
|
||||
direction:
|
||||
pageviewsCurrent > pageviewsCompare
|
||||
? 'up'
|
||||
: pageviewsCurrent < pageviewsCompare
|
||||
? 'down'
|
||||
: 'flat',
|
||||
unit: 'count',
|
||||
},
|
||||
share: {
|
||||
current: shareCurrent,
|
||||
compare: shareCompare,
|
||||
delta: shareCurrent - shareCompare,
|
||||
changePct: result.changePct ?? null, // This is now share-based
|
||||
direction: result.direction ?? 'flat',
|
||||
unit: 'ratio',
|
||||
},
|
||||
},
|
||||
extra: {
|
||||
isNew: result.extra?.isNew,
|
||||
shareShiftPp,
|
||||
},
|
||||
},
|
||||
};
|
||||
},
|
||||
};
|
||||
@@ -1,275 +0,0 @@
|
||||
import { TABLE_NAMES, formatClickhouseDate } from '../../../clickhouse/client';
|
||||
import type {
|
||||
ComputeContext,
|
||||
ComputeResult,
|
||||
InsightModule,
|
||||
RenderedCard,
|
||||
} from '../types';
|
||||
import {
|
||||
buildLookupMap,
|
||||
computeChangePct,
|
||||
computeDirection,
|
||||
computeWeekdayMedians,
|
||||
getEndOfDay,
|
||||
getWeekday,
|
||||
selectTopDimensions,
|
||||
} from '../utils';
|
||||
|
||||
async function fetchReferrerAggregates(ctx: ComputeContext): Promise<{
|
||||
currentMap: Map<string, number>;
|
||||
baselineMap: Map<string, number>;
|
||||
totalCurrent: number;
|
||||
totalBaseline: number;
|
||||
}> {
|
||||
if (ctx.window.kind === 'yesterday') {
|
||||
const [currentResults, baselineResults, totals] = await Promise.all([
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ referrer_name: string; cnt: number }>([
|
||||
'referrer_name',
|
||||
'count(*) as cnt',
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.start,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.groupBy(['referrer_name'])
|
||||
.execute(),
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ date: string; referrer_name: string; cnt: number }>([
|
||||
'toDate(created_at) as date',
|
||||
'referrer_name',
|
||||
'count(*) as cnt',
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.baselineEnd),
|
||||
])
|
||||
.groupBy(['date', 'referrer_name'])
|
||||
.execute(),
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ cur_total: number }>([
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${formatClickhouseDate(ctx.window.start)}' AND '${formatClickhouseDate(getEndOfDay(ctx.window.end))}') as cur_total`,
|
||||
),
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.execute(),
|
||||
]);
|
||||
|
||||
const currentMap = buildLookupMap(
|
||||
currentResults,
|
||||
(r) => r.referrer_name || 'direct',
|
||||
);
|
||||
|
||||
const targetWeekday = getWeekday(ctx.window.start);
|
||||
const baselineMap = computeWeekdayMedians(
|
||||
baselineResults,
|
||||
targetWeekday,
|
||||
(r) => r.referrer_name || 'direct',
|
||||
);
|
||||
|
||||
const totalCurrent = totals[0]?.cur_total ?? 0;
|
||||
const totalBaseline = Array.from(baselineMap.values()).reduce(
|
||||
(sum, val) => sum + val,
|
||||
0,
|
||||
);
|
||||
|
||||
return { currentMap, baselineMap, totalCurrent, totalBaseline };
|
||||
}
|
||||
|
||||
const curStart = formatClickhouseDate(ctx.window.start);
|
||||
const curEnd = formatClickhouseDate(getEndOfDay(ctx.window.end));
|
||||
const baseStart = formatClickhouseDate(ctx.window.baselineStart);
|
||||
const baseEnd = formatClickhouseDate(getEndOfDay(ctx.window.baselineEnd));
|
||||
|
||||
const [results, totals] = await Promise.all([
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ referrer_name: string; cur: number; base: number }>([
|
||||
'referrer_name',
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur`,
|
||||
),
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base`,
|
||||
),
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.groupBy(['referrer_name'])
|
||||
.execute(),
|
||||
ctx
|
||||
.clix()
|
||||
.select<{ cur_total: number; base_total: number }>([
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${curStart}' AND '${curEnd}') as cur_total`,
|
||||
),
|
||||
ctx.clix.exp(
|
||||
`countIf(created_at BETWEEN '${baseStart}' AND '${baseEnd}') as base_total`,
|
||||
),
|
||||
])
|
||||
.from(TABLE_NAMES.sessions)
|
||||
.where('project_id', '=', ctx.projectId)
|
||||
.where('sign', '=', 1)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
ctx.window.baselineStart,
|
||||
getEndOfDay(ctx.window.end),
|
||||
])
|
||||
.execute(),
|
||||
]);
|
||||
|
||||
const currentMap = buildLookupMap(
|
||||
results,
|
||||
(r) => r.referrer_name || 'direct',
|
||||
(r) => Number(r.cur ?? 0),
|
||||
);
|
||||
|
||||
const baselineMap = buildLookupMap(
|
||||
results,
|
||||
(r) => r.referrer_name || 'direct',
|
||||
(r) => Number(r.base ?? 0),
|
||||
);
|
||||
|
||||
const totalCurrent = totals[0]?.cur_total ?? 0;
|
||||
const totalBaseline = totals[0]?.base_total ?? 0;
|
||||
|
||||
return { currentMap, baselineMap, totalCurrent, totalBaseline };
|
||||
}
|
||||
|
||||
export const referrersModule: InsightModule = {
|
||||
key: 'referrers',
|
||||
cadence: ['daily'],
|
||||
thresholds: { minTotal: 100, minAbsDelta: 20, minPct: 0.15, maxDims: 50 },
|
||||
|
||||
async enumerateDimensions(ctx) {
|
||||
const { currentMap, baselineMap } = await fetchReferrerAggregates(ctx);
|
||||
const topDims = selectTopDimensions(
|
||||
currentMap,
|
||||
baselineMap,
|
||||
this.thresholds?.maxDims ?? 50,
|
||||
);
|
||||
return topDims.map((dim) => `referrer:${dim}`);
|
||||
},
|
||||
|
||||
async computeMany(ctx, dimensionKeys): Promise<ComputeResult[]> {
|
||||
const { currentMap, baselineMap, totalCurrent, totalBaseline } =
|
||||
await fetchReferrerAggregates(ctx);
|
||||
const results: ComputeResult[] = [];
|
||||
|
||||
for (const dimKey of dimensionKeys) {
|
||||
if (!dimKey.startsWith('referrer:')) continue;
|
||||
const referrerName = dimKey.replace('referrer:', '');
|
||||
|
||||
const currentValue = currentMap.get(referrerName) ?? 0;
|
||||
const compareValue = baselineMap.get(referrerName) ?? 0;
|
||||
|
||||
const currentShare = totalCurrent > 0 ? currentValue / totalCurrent : 0;
|
||||
const compareShare = totalBaseline > 0 ? compareValue / totalBaseline : 0;
|
||||
|
||||
const shareShiftPp = (currentShare - compareShare) * 100;
|
||||
const changePct = computeChangePct(currentValue, compareValue);
|
||||
const direction = computeDirection(changePct);
|
||||
|
||||
results.push({
|
||||
ok: true,
|
||||
dimensionKey: dimKey,
|
||||
currentValue,
|
||||
compareValue,
|
||||
changePct,
|
||||
direction,
|
||||
extra: {
|
||||
shareShiftPp,
|
||||
currentShare,
|
||||
compareShare,
|
||||
isNew: compareValue === 0 && currentValue > 0,
|
||||
isGone: currentValue === 0 && compareValue > 0,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
return results;
|
||||
},
|
||||
|
||||
render(result, ctx): RenderedCard {
|
||||
const referrer = result.dimensionKey.replace('referrer:', '');
|
||||
const pct = ((result.changePct ?? 0) * 100).toFixed(1);
|
||||
const isIncrease = (result.changePct ?? 0) >= 0;
|
||||
const isNew = result.extra?.isNew as boolean | undefined;
|
||||
|
||||
const title = isNew
|
||||
? `New traffic source: ${referrer}`
|
||||
: `Traffic from ${referrer} ${isIncrease ? '↑' : '↓'} ${Math.abs(Number(pct))}%`;
|
||||
|
||||
const sessionsCurrent = result.currentValue ?? 0;
|
||||
const sessionsCompare = result.compareValue ?? 0;
|
||||
const shareCurrent = Number(result.extra?.currentShare ?? 0);
|
||||
const shareCompare = Number(result.extra?.compareShare ?? 0);
|
||||
|
||||
return {
|
||||
title,
|
||||
summary: `${ctx.window.label}. Sessions ${sessionsCurrent} vs ${sessionsCompare}.`,
|
||||
displayName: referrer,
|
||||
payload: {
|
||||
kind: 'insight_v1',
|
||||
dimensions: [
|
||||
{
|
||||
key: 'referrer_name',
|
||||
value: referrer,
|
||||
displayName: referrer,
|
||||
},
|
||||
],
|
||||
primaryMetric: 'sessions',
|
||||
metrics: {
|
||||
sessions: {
|
||||
current: sessionsCurrent,
|
||||
compare: sessionsCompare,
|
||||
delta: sessionsCurrent - sessionsCompare,
|
||||
changePct: sessionsCompare > 0 ? (result.changePct ?? 0) : null,
|
||||
direction: result.direction ?? 'flat',
|
||||
unit: 'count',
|
||||
},
|
||||
share: {
|
||||
current: shareCurrent,
|
||||
compare: shareCompare,
|
||||
delta: shareCurrent - shareCompare,
|
||||
changePct:
|
||||
shareCompare > 0
|
||||
? (shareCurrent - shareCompare) / shareCompare
|
||||
: null,
|
||||
direction:
|
||||
shareCurrent - shareCompare > 0.0005
|
||||
? 'up'
|
||||
: shareCurrent - shareCompare < -0.0005
|
||||
? 'down'
|
||||
: 'flat',
|
||||
unit: 'ratio',
|
||||
},
|
||||
},
|
||||
extra: {
|
||||
isNew: result.extra?.isNew,
|
||||
isGone: result.extra?.isGone,
|
||||
},
|
||||
},
|
||||
};
|
||||
},
|
||||
};
|
||||
@@ -1,18 +0,0 @@
|
||||
import type { ComputeResult } from './types';
|
||||
|
||||
export function defaultImpactScore(r: ComputeResult): number {
|
||||
const vol = (r.currentValue ?? 0) + (r.compareValue ?? 0);
|
||||
const pct = Math.abs(r.changePct ?? 0);
|
||||
// stable-ish: bigger change + bigger volume => higher impact
|
||||
return Math.log1p(vol) * (pct * 100);
|
||||
}
|
||||
|
||||
export function severityBand(
|
||||
changePct?: number | null,
|
||||
): 'low' | 'moderate' | 'severe' | null {
|
||||
const p = Math.abs(changePct ?? 0);
|
||||
if (p < 0.1) return null;
|
||||
if (p < 0.5) return 'low';
|
||||
if (p < 1) return 'moderate';
|
||||
return 'severe';
|
||||
}
|
||||
@@ -1,343 +0,0 @@
|
||||
import { Prisma, db } from '../../prisma-client';
|
||||
import type {
|
||||
Cadence,
|
||||
InsightStore,
|
||||
PersistedInsight,
|
||||
RenderedCard,
|
||||
WindowKind,
|
||||
WindowRange,
|
||||
} from './types';
|
||||
|
||||
export const insightStore: InsightStore = {
|
||||
async listProjectIdsForCadence(cadence: Cadence): Promise<string[]> {
|
||||
const projects = await db.project.findMany({
|
||||
where: {
|
||||
deleteAt: null,
|
||||
eventsCount: { gt: 10_000 },
|
||||
updatedAt: { gt: new Date(Date.now() - 1000 * 60 * 60 * 24) },
|
||||
organization: {
|
||||
subscriptionStatus: 'active',
|
||||
},
|
||||
},
|
||||
select: { id: true },
|
||||
});
|
||||
return projects.map((p) => p.id);
|
||||
},
|
||||
|
||||
async getProjectCreatedAt(projectId: string): Promise<Date | null> {
|
||||
const project = await db.project.findFirst({
|
||||
where: { id: projectId, deleteAt: null },
|
||||
select: { createdAt: true },
|
||||
});
|
||||
return project?.createdAt ?? null;
|
||||
},
|
||||
|
||||
async getActiveInsightByIdentity({
|
||||
projectId,
|
||||
moduleKey,
|
||||
dimensionKey,
|
||||
windowKind,
|
||||
}): Promise<PersistedInsight | null> {
|
||||
const insight = await db.projectInsight.findFirst({
|
||||
where: {
|
||||
projectId,
|
||||
moduleKey,
|
||||
dimensionKey,
|
||||
windowKind,
|
||||
state: 'active',
|
||||
},
|
||||
});
|
||||
|
||||
if (!insight) return null;
|
||||
|
||||
return {
|
||||
id: insight.id,
|
||||
projectId: insight.projectId,
|
||||
moduleKey: insight.moduleKey,
|
||||
dimensionKey: insight.dimensionKey,
|
||||
windowKind: insight.windowKind as WindowKind,
|
||||
state: insight.state as 'active' | 'suppressed' | 'closed',
|
||||
version: insight.version,
|
||||
impactScore: insight.impactScore,
|
||||
lastSeenAt: insight.lastSeenAt,
|
||||
lastUpdatedAt: insight.lastUpdatedAt,
|
||||
direction: insight.direction,
|
||||
severityBand: insight.severityBand,
|
||||
};
|
||||
},
|
||||
|
||||
async upsertInsight({
|
||||
projectId,
|
||||
moduleKey,
|
||||
dimensionKey,
|
||||
window,
|
||||
card,
|
||||
metrics,
|
||||
now,
|
||||
decision,
|
||||
prev,
|
||||
}): Promise<PersistedInsight> {
|
||||
const baseData = {
|
||||
projectId,
|
||||
moduleKey,
|
||||
dimensionKey,
|
||||
windowKind: window.kind,
|
||||
state: prev?.state === 'closed' ? 'active' : (prev?.state ?? 'active'),
|
||||
title: card.title,
|
||||
summary: card.summary ?? null,
|
||||
displayName: card.displayName,
|
||||
payload: card.payload,
|
||||
direction: metrics.direction ?? null,
|
||||
impactScore: metrics.impactScore,
|
||||
severityBand: metrics.severityBand ?? null,
|
||||
version: prev ? (decision.material ? prev.version + 1 : prev.version) : 1,
|
||||
windowStart: window.start,
|
||||
windowEnd: window.end,
|
||||
lastSeenAt: now,
|
||||
lastUpdatedAt: now,
|
||||
};
|
||||
|
||||
// Try to find existing insight first
|
||||
const existing = prev
|
||||
? await db.projectInsight.findFirst({
|
||||
where: {
|
||||
projectId,
|
||||
moduleKey,
|
||||
dimensionKey,
|
||||
windowKind: window.kind,
|
||||
state: prev.state,
|
||||
},
|
||||
})
|
||||
: null;
|
||||
|
||||
let insight: any;
|
||||
if (existing) {
|
||||
// Update existing
|
||||
insight = await db.projectInsight.update({
|
||||
where: { id: existing.id },
|
||||
data: {
|
||||
...baseData,
|
||||
threadId: existing.threadId, // Preserve threadId
|
||||
},
|
||||
});
|
||||
} else {
|
||||
// Create new - need to check if there's a closed/suppressed one to reopen
|
||||
const closed = await db.projectInsight.findFirst({
|
||||
where: {
|
||||
projectId,
|
||||
moduleKey,
|
||||
dimensionKey,
|
||||
windowKind: window.kind,
|
||||
state: { in: ['closed', 'suppressed'] },
|
||||
},
|
||||
orderBy: { lastUpdatedAt: 'desc' },
|
||||
});
|
||||
|
||||
if (closed) {
|
||||
// Reopen and update
|
||||
insight = await db.projectInsight.update({
|
||||
where: { id: closed.id },
|
||||
data: {
|
||||
...baseData,
|
||||
state: 'active',
|
||||
threadId: closed.threadId, // Preserve threadId
|
||||
},
|
||||
});
|
||||
} else {
|
||||
// Create new
|
||||
insight = await db.projectInsight.create({
|
||||
data: {
|
||||
...baseData,
|
||||
firstDetectedAt: now,
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
id: insight.id,
|
||||
projectId: insight.projectId,
|
||||
moduleKey: insight.moduleKey,
|
||||
dimensionKey: insight.dimensionKey,
|
||||
windowKind: insight.windowKind as WindowKind,
|
||||
state: insight.state as 'active' | 'suppressed' | 'closed',
|
||||
version: insight.version,
|
||||
impactScore: insight.impactScore,
|
||||
lastSeenAt: insight.lastSeenAt,
|
||||
lastUpdatedAt: insight.lastUpdatedAt,
|
||||
direction: insight.direction,
|
||||
severityBand: insight.severityBand,
|
||||
};
|
||||
},
|
||||
|
||||
async insertEvent({
|
||||
projectId,
|
||||
insightId,
|
||||
moduleKey,
|
||||
dimensionKey,
|
||||
windowKind,
|
||||
eventKind,
|
||||
changeFrom,
|
||||
changeTo,
|
||||
now,
|
||||
}): Promise<void> {
|
||||
await db.insightEvent.create({
|
||||
data: {
|
||||
insightId,
|
||||
eventKind,
|
||||
changeFrom: changeFrom
|
||||
? (changeFrom as Prisma.InputJsonValue)
|
||||
: Prisma.DbNull,
|
||||
changeTo: changeTo
|
||||
? (changeTo as Prisma.InputJsonValue)
|
||||
: Prisma.DbNull,
|
||||
createdAt: now,
|
||||
},
|
||||
});
|
||||
},
|
||||
|
||||
async closeMissingActiveInsights({
|
||||
projectId,
|
||||
moduleKey,
|
||||
windowKind,
|
||||
seenDimensionKeys,
|
||||
now,
|
||||
staleDays,
|
||||
}): Promise<number> {
|
||||
const staleDate = new Date(now);
|
||||
staleDate.setDate(staleDate.getDate() - staleDays);
|
||||
|
||||
const result = await db.projectInsight.updateMany({
|
||||
where: {
|
||||
projectId,
|
||||
moduleKey,
|
||||
windowKind,
|
||||
state: 'active',
|
||||
lastSeenAt: { lt: staleDate },
|
||||
dimensionKey: { notIn: seenDimensionKeys },
|
||||
},
|
||||
data: {
|
||||
state: 'closed',
|
||||
lastUpdatedAt: now,
|
||||
},
|
||||
});
|
||||
|
||||
return result.count;
|
||||
},
|
||||
|
||||
async applySuppression({
|
||||
projectId,
|
||||
moduleKey,
|
||||
windowKind,
|
||||
keepTopN,
|
||||
now,
|
||||
}): Promise<{ suppressed: number; unsuppressed: number }> {
|
||||
// Get all active insights for this module/window, ordered by impactScore desc
|
||||
const insights = await db.projectInsight.findMany({
|
||||
where: {
|
||||
projectId,
|
||||
moduleKey,
|
||||
windowKind,
|
||||
state: { in: ['active', 'suppressed'] },
|
||||
},
|
||||
orderBy: { impactScore: 'desc' },
|
||||
});
|
||||
|
||||
if (insights.length === 0) {
|
||||
return { suppressed: 0, unsuppressed: 0 };
|
||||
}
|
||||
|
||||
let suppressed = 0;
|
||||
let unsuppressed = 0;
|
||||
|
||||
// For "yesterday" insights, suppress any that are stale (windowEnd is not actually yesterday)
|
||||
// This prevents showing confusing insights like "Yesterday traffic dropped" when it's from 2+ days ago
|
||||
if (windowKind === 'yesterday') {
|
||||
const yesterday = new Date(now);
|
||||
yesterday.setUTCHours(0, 0, 0, 0);
|
||||
yesterday.setUTCDate(yesterday.getUTCDate() - 1);
|
||||
const yesterdayTime = yesterday.getTime();
|
||||
|
||||
for (const insight of insights) {
|
||||
// If windowEnd is null, consider it stale
|
||||
const isStale = insight.windowEnd
|
||||
? new Date(insight.windowEnd).setUTCHours(0, 0, 0, 0) !==
|
||||
yesterdayTime
|
||||
: true;
|
||||
|
||||
if (isStale && insight.state === 'active') {
|
||||
await db.projectInsight.update({
|
||||
where: { id: insight.id },
|
||||
data: { state: 'suppressed', lastUpdatedAt: now },
|
||||
});
|
||||
suppressed++;
|
||||
}
|
||||
}
|
||||
|
||||
// Filter to only non-stale insights for top-N logic
|
||||
const freshInsights = insights.filter((insight) => {
|
||||
if (!insight.windowEnd) return false;
|
||||
const windowEndTime = new Date(insight.windowEnd).setUTCHours(
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
);
|
||||
return windowEndTime === yesterdayTime;
|
||||
});
|
||||
|
||||
const topN = freshInsights.slice(0, keepTopN);
|
||||
const belowN = freshInsights.slice(keepTopN);
|
||||
|
||||
for (const insight of belowN) {
|
||||
if (insight.state === 'active') {
|
||||
await db.projectInsight.update({
|
||||
where: { id: insight.id },
|
||||
data: { state: 'suppressed', lastUpdatedAt: now },
|
||||
});
|
||||
suppressed++;
|
||||
}
|
||||
}
|
||||
|
||||
for (const insight of topN) {
|
||||
if (insight.state === 'suppressed') {
|
||||
await db.projectInsight.update({
|
||||
where: { id: insight.id },
|
||||
data: { state: 'active', lastUpdatedAt: now },
|
||||
});
|
||||
unsuppressed++;
|
||||
}
|
||||
}
|
||||
|
||||
return { suppressed, unsuppressed };
|
||||
}
|
||||
|
||||
// For non-yesterday windows, apply standard top-N suppression
|
||||
const topN = insights.slice(0, keepTopN);
|
||||
const belowN = insights.slice(keepTopN);
|
||||
|
||||
// Suppress those below top N
|
||||
for (const insight of belowN) {
|
||||
if (insight.state === 'active') {
|
||||
await db.projectInsight.update({
|
||||
where: { id: insight.id },
|
||||
data: { state: 'suppressed', lastUpdatedAt: now },
|
||||
});
|
||||
suppressed++;
|
||||
}
|
||||
}
|
||||
|
||||
// Unsuppress those in top N
|
||||
for (const insight of topN) {
|
||||
if (insight.state === 'suppressed') {
|
||||
await db.projectInsight.update({
|
||||
where: { id: insight.id },
|
||||
data: { state: 'active', lastUpdatedAt: now },
|
||||
});
|
||||
unsuppressed++;
|
||||
}
|
||||
}
|
||||
|
||||
return { suppressed, unsuppressed };
|
||||
},
|
||||
};
|
||||
@@ -1,191 +0,0 @@
|
||||
import type {
|
||||
InsightDimension,
|
||||
InsightMetricEntry,
|
||||
InsightMetricKey,
|
||||
InsightPayload,
|
||||
} from '@openpanel/validation';
|
||||
|
||||
export type Cadence = 'daily';
|
||||
|
||||
export type WindowKind = 'yesterday' | 'rolling_7d' | 'rolling_30d';
|
||||
|
||||
export interface WindowRange {
|
||||
kind: WindowKind;
|
||||
start: Date; // inclusive
|
||||
end: Date; // inclusive (or exclusive, but be consistent)
|
||||
baselineStart: Date;
|
||||
baselineEnd: Date;
|
||||
label: string; // e.g. "Yesterday" / "Last 7 days"
|
||||
}
|
||||
|
||||
export interface ComputeContext {
|
||||
projectId: string;
|
||||
window: WindowRange;
|
||||
db: any; // your DB client
|
||||
now: Date;
|
||||
logger: Pick<Console, 'info' | 'warn' | 'error'>;
|
||||
/**
|
||||
* Cached clix function that automatically caches query results based on query hash.
|
||||
* This eliminates duplicate queries within the same module+window context.
|
||||
* Use this instead of importing clix directly to benefit from automatic caching.
|
||||
*/
|
||||
clix: ReturnType<typeof import('./cached-clix').createCachedClix>;
|
||||
}
|
||||
|
||||
export interface ComputeResult {
|
||||
ok: boolean;
|
||||
dimensionKey: string; // e.g. "referrer:instagram" / "page:/pricing"
|
||||
currentValue?: number;
|
||||
compareValue?: number;
|
||||
changePct?: number; // -0.15 = -15%
|
||||
direction?: 'up' | 'down' | 'flat';
|
||||
extra?: Record<string, unknown>; // share delta pp, rank, sparkline, etc.
|
||||
}
|
||||
|
||||
// Types imported from @openpanel/validation:
|
||||
// - InsightMetricKey
|
||||
// - InsightMetricEntry
|
||||
// - InsightDimension
|
||||
// - InsightPayload
|
||||
|
||||
/**
|
||||
* Render should be deterministic and safe to call multiple times.
|
||||
* Returns the shape that matches ProjectInsight create input.
|
||||
* The payload contains all metric data and display metadata.
|
||||
*/
|
||||
export interface RenderedCard {
|
||||
title: string;
|
||||
summary?: string;
|
||||
displayName: string;
|
||||
payload: InsightPayload; // Contains dimensions, primaryMetric, metrics, extra
|
||||
}
|
||||
|
||||
/** Optional per-module thresholds (the engine can still apply global defaults) */
|
||||
export interface ModuleThresholds {
|
||||
minTotal?: number; // min current+baseline
|
||||
minAbsDelta?: number; // min abs(current-compare)
|
||||
minPct?: number; // min abs(changePct)
|
||||
maxDims?: number; // cap enumerateDimensions
|
||||
}
|
||||
|
||||
export interface InsightModule {
|
||||
key: string;
|
||||
cadence: Cadence[];
|
||||
/** Optional per-module override; engine applies a default if omitted. */
|
||||
windows?: WindowKind[];
|
||||
thresholds?: ModuleThresholds;
|
||||
enumerateDimensions?(ctx: ComputeContext): Promise<string[]>;
|
||||
/** Preferred path: batch compute many dimensions in one go. */
|
||||
computeMany(
|
||||
ctx: ComputeContext,
|
||||
dimensionKeys: string[],
|
||||
): Promise<ComputeResult[]>;
|
||||
/** Must not do DB reads; just format output. */
|
||||
render(result: ComputeResult, ctx: ComputeContext): RenderedCard;
|
||||
/** Score decides what to show (top-N). */
|
||||
score?(result: ComputeResult, ctx: ComputeContext): number;
|
||||
/** Optional: compute "drivers" for AI explain step */
|
||||
drivers?(
|
||||
result: ComputeResult,
|
||||
ctx: ComputeContext,
|
||||
): Promise<Record<string, unknown>>;
|
||||
}
|
||||
|
||||
/** Insight row shape returned from persistence (minimal fields engine needs). */
|
||||
export interface PersistedInsight {
|
||||
id: string;
|
||||
projectId: string;
|
||||
moduleKey: string;
|
||||
dimensionKey: string;
|
||||
windowKind: WindowKind;
|
||||
state: 'active' | 'suppressed' | 'closed';
|
||||
version: number;
|
||||
impactScore: number;
|
||||
lastSeenAt: Date;
|
||||
lastUpdatedAt: Date;
|
||||
direction?: string | null;
|
||||
severityBand?: string | null;
|
||||
}
|
||||
|
||||
/** Material change decision used for events/notifications. */
|
||||
export type MaterialReason =
|
||||
| 'created'
|
||||
| 'direction_flip'
|
||||
| 'severity_change'
|
||||
| 'cross_deadband'
|
||||
| 'reopened'
|
||||
| 'none';
|
||||
|
||||
export interface MaterialDecision {
|
||||
material: boolean;
|
||||
reason: MaterialReason;
|
||||
newSeverityBand?: 'low' | 'moderate' | 'severe' | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Persistence interface: implement with Postgres.
|
||||
* Keep engine independent of query builder choice.
|
||||
*/
|
||||
export interface InsightStore {
|
||||
listProjectIdsForCadence(cadence: Cadence): Promise<string[]>;
|
||||
/** Used by the engine/worker to decide if a window has enough baseline history. */
|
||||
getProjectCreatedAt(projectId: string): Promise<Date | null>;
|
||||
getActiveInsightByIdentity(args: {
|
||||
projectId: string;
|
||||
moduleKey: string;
|
||||
dimensionKey: string;
|
||||
windowKind: WindowKind;
|
||||
}): Promise<PersistedInsight | null>;
|
||||
upsertInsight(args: {
|
||||
projectId: string;
|
||||
moduleKey: string;
|
||||
dimensionKey: string;
|
||||
window: WindowRange;
|
||||
card: RenderedCard;
|
||||
metrics: {
|
||||
direction?: 'up' | 'down' | 'flat';
|
||||
impactScore: number;
|
||||
severityBand?: string | null;
|
||||
};
|
||||
now: Date;
|
||||
decision: MaterialDecision;
|
||||
prev: PersistedInsight | null;
|
||||
}): Promise<PersistedInsight>;
|
||||
insertEvent(args: {
|
||||
projectId: string;
|
||||
insightId: string;
|
||||
moduleKey: string;
|
||||
dimensionKey: string;
|
||||
windowKind: WindowKind;
|
||||
eventKind:
|
||||
| 'created'
|
||||
| 'updated'
|
||||
| 'severity_up'
|
||||
| 'severity_down'
|
||||
| 'direction_flip'
|
||||
| 'closed'
|
||||
| 'reopened'
|
||||
| 'suppressed'
|
||||
| 'unsuppressed';
|
||||
changeFrom?: Record<string, unknown> | null;
|
||||
changeTo?: Record<string, unknown> | null;
|
||||
now: Date;
|
||||
}): Promise<void>;
|
||||
/** Mark insights as not seen this run if you prefer lifecycle via closeMissing() */
|
||||
closeMissingActiveInsights(args: {
|
||||
projectId: string;
|
||||
moduleKey: string;
|
||||
windowKind: WindowKind;
|
||||
seenDimensionKeys: string[];
|
||||
now: Date;
|
||||
staleDays: number; // close if not seen for X days
|
||||
}): Promise<number>; // count closed
|
||||
/** Enforce top-N display by suppressing below-threshold insights. */
|
||||
applySuppression(args: {
|
||||
projectId: string;
|
||||
moduleKey: string;
|
||||
windowKind: WindowKind;
|
||||
keepTopN: number;
|
||||
now: Date;
|
||||
}): Promise<{ suppressed: number; unsuppressed: number }>;
|
||||
}
|
||||
@@ -1,151 +0,0 @@
|
||||
/**
|
||||
* Shared utilities for insight modules
|
||||
*/
|
||||
|
||||
/**
|
||||
* Get UTC weekday (0 = Sunday, 6 = Saturday)
|
||||
*/
|
||||
export function getWeekday(date: Date): number {
|
||||
return date.getUTCDay();
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute median of a sorted array of numbers
|
||||
*/
|
||||
export function computeMedian(sortedValues: number[]): number {
|
||||
if (sortedValues.length === 0) return 0;
|
||||
const mid = Math.floor(sortedValues.length / 2);
|
||||
return sortedValues.length % 2 === 0
|
||||
? ((sortedValues[mid - 1] ?? 0) + (sortedValues[mid] ?? 0)) / 2
|
||||
: (sortedValues[mid] ?? 0);
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute weekday medians from daily breakdown data.
|
||||
* Groups by dimension, filters to matching weekday, computes median per dimension.
|
||||
*
|
||||
* @param data - Array of { date, dimension, cnt } rows
|
||||
* @param targetWeekday - Weekday to filter to (0-6)
|
||||
* @param getDimension - Function to extract normalized dimension from row
|
||||
* @returns Map of dimension -> median value
|
||||
*/
|
||||
export function computeWeekdayMedians<T>(
|
||||
data: T[],
|
||||
targetWeekday: number,
|
||||
getDimension: (row: T) => string,
|
||||
): Map<string, number> {
|
||||
// Group by dimension, filtered to target weekday
|
||||
const byDimension = new Map<string, number[]>();
|
||||
|
||||
for (const row of data) {
|
||||
const rowWeekday = getWeekday(new Date((row as any).date));
|
||||
if (rowWeekday !== targetWeekday) continue;
|
||||
|
||||
const dim = getDimension(row);
|
||||
const values = byDimension.get(dim) ?? [];
|
||||
values.push(Number((row as any).cnt ?? 0));
|
||||
byDimension.set(dim, values);
|
||||
}
|
||||
|
||||
// Compute median per dimension
|
||||
const result = new Map<string, number>();
|
||||
for (const [dim, values] of byDimension) {
|
||||
values.sort((a, b) => a - b);
|
||||
result.set(dim, computeMedian(values));
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute change percentage between current and compare values
|
||||
*/
|
||||
export function computeChangePct(
|
||||
currentValue: number,
|
||||
compareValue: number,
|
||||
): number {
|
||||
return compareValue > 0
|
||||
? (currentValue - compareValue) / compareValue
|
||||
: currentValue > 0
|
||||
? 1
|
||||
: 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Determine direction based on change percentage
|
||||
*/
|
||||
export function computeDirection(
|
||||
changePct: number,
|
||||
threshold = 0.05,
|
||||
): 'up' | 'down' | 'flat' {
|
||||
return changePct > threshold
|
||||
? 'up'
|
||||
: changePct < -threshold
|
||||
? 'down'
|
||||
: 'flat';
|
||||
}
|
||||
|
||||
/**
|
||||
* Get end of day timestamp (23:59:59.999) for a given date.
|
||||
* Used to ensure BETWEEN queries include the full day.
|
||||
*/
|
||||
export function getEndOfDay(date: Date): Date {
|
||||
const end = new Date(date);
|
||||
end.setUTCHours(23, 59, 59, 999);
|
||||
return end;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a lookup map from query results.
|
||||
* Aggregates counts by key, handling duplicate keys by summing values.
|
||||
*
|
||||
* @param results - Array of result rows
|
||||
* @param getKey - Function to extract the key from each row
|
||||
* @param getCount - Function to extract the count from each row (defaults to 'cnt' field)
|
||||
* @returns Map of key -> aggregated count
|
||||
*/
|
||||
export function buildLookupMap<T>(
|
||||
results: T[],
|
||||
getKey: (row: T) => string,
|
||||
getCount: (row: T) => number = (row) => Number((row as any).cnt ?? 0),
|
||||
): Map<string, number> {
|
||||
const map = new Map<string, number>();
|
||||
for (const row of results) {
|
||||
const key = getKey(row);
|
||||
const cnt = getCount(row);
|
||||
map.set(key, (map.get(key) ?? 0) + cnt);
|
||||
}
|
||||
return map;
|
||||
}
|
||||
|
||||
/**
|
||||
* Select top-N dimensions by ranking on greatest(current, baseline).
|
||||
* This preserves union behavior: dimensions with high values in either period are included.
|
||||
*
|
||||
* @param currentMap - Map of dimension -> current value
|
||||
* @param baselineMap - Map of dimension -> baseline value
|
||||
* @param maxDims - Maximum number of dimensions to return
|
||||
* @returns Array of dimension keys, ranked by greatest(current, baseline)
|
||||
*/
|
||||
export function selectTopDimensions(
|
||||
currentMap: Map<string, number>,
|
||||
baselineMap: Map<string, number>,
|
||||
maxDims: number,
|
||||
): string[] {
|
||||
// Merge all dimensions from both maps
|
||||
const allDims = new Set<string>();
|
||||
for (const dim of currentMap.keys()) allDims.add(dim);
|
||||
for (const dim of baselineMap.keys()) allDims.add(dim);
|
||||
|
||||
// Rank by greatest(current, baseline)
|
||||
const ranked = Array.from(allDims)
|
||||
.map((dim) => ({
|
||||
dim,
|
||||
maxValue: Math.max(currentMap.get(dim) ?? 0, baselineMap.get(dim) ?? 0),
|
||||
}))
|
||||
.sort((a, b) => b.maxValue - a.maxValue)
|
||||
.slice(0, maxDims)
|
||||
.map((x) => x.dim);
|
||||
|
||||
return ranked;
|
||||
}
|
||||
@@ -1,59 +0,0 @@
|
||||
import type { WindowKind, WindowRange } from './types';
|
||||
|
||||
function atUtcMidnight(d: Date) {
|
||||
const x = new Date(d);
|
||||
x.setUTCHours(0, 0, 0, 0);
|
||||
return x;
|
||||
}
|
||||
|
||||
function addDays(d: Date, days: number) {
|
||||
const x = new Date(d);
|
||||
x.setUTCDate(x.getUTCDate() + days);
|
||||
return x;
|
||||
}
|
||||
|
||||
/**
|
||||
* Convention: end is inclusive (end of day). If you prefer exclusive, adapt consistently.
|
||||
*/
|
||||
export function resolveWindow(kind: WindowKind, now: Date): WindowRange {
|
||||
const today0 = atUtcMidnight(now);
|
||||
const yesterday0 = addDays(today0, -1);
|
||||
if (kind === 'yesterday') {
|
||||
const start = yesterday0;
|
||||
const end = yesterday0;
|
||||
// Baseline: median of last 4 same weekdays -> engine/module implements the median.
|
||||
// Here we just define the candidate range; module queries last 28 days and filters weekday.
|
||||
const baselineStart = addDays(yesterday0, -28);
|
||||
const baselineEnd = addDays(yesterday0, -1);
|
||||
return { kind, start, end, baselineStart, baselineEnd, label: 'Yesterday' };
|
||||
}
|
||||
if (kind === 'rolling_7d') {
|
||||
const end = yesterday0;
|
||||
const start = addDays(end, -6); // 7 days inclusive
|
||||
const baselineEnd = addDays(start, -1);
|
||||
const baselineStart = addDays(baselineEnd, -6);
|
||||
return {
|
||||
kind,
|
||||
start,
|
||||
end,
|
||||
baselineStart,
|
||||
baselineEnd,
|
||||
label: 'Last 7 days',
|
||||
};
|
||||
}
|
||||
// rolling_30d
|
||||
{
|
||||
const end = yesterday0;
|
||||
const start = addDays(end, -29);
|
||||
const baselineEnd = addDays(start, -1);
|
||||
const baselineStart = addDays(baselineEnd, -29);
|
||||
return {
|
||||
kind,
|
||||
start,
|
||||
end,
|
||||
baselineStart,
|
||||
baselineEnd,
|
||||
label: 'Last 30 days',
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -7,28 +7,6 @@ import { TABLE_NAMES, ch } from '../clickhouse/client';
|
||||
import { clix } from '../clickhouse/query-builder';
|
||||
import { getEventFiltersWhereClause } from './chart.service';
|
||||
|
||||
// Constants
|
||||
const ROLLUP_DATE_PREFIX = '1970-01-01';
|
||||
|
||||
const COLUMN_PREFIX_MAP: Record<string, string> = {
|
||||
region: 'country',
|
||||
city: 'country',
|
||||
browser_version: 'browser',
|
||||
os_version: 'os',
|
||||
};
|
||||
|
||||
// Types
|
||||
type MetricsRow = {
|
||||
bounce_rate: number;
|
||||
unique_visitors: number;
|
||||
total_sessions: number;
|
||||
avg_session_duration: number;
|
||||
total_screen_views: number;
|
||||
views_per_session: number;
|
||||
};
|
||||
|
||||
type MetricsSeriesRow = MetricsRow & { date: string; total_revenue: number };
|
||||
|
||||
export const zGetMetricsInput = z.object({
|
||||
projectId: z.string(),
|
||||
filters: z.array(z.any()),
|
||||
@@ -107,112 +85,11 @@ export type IGetTopGenericInput = z.infer<typeof zGetTopGenericInput> & {
|
||||
export class OverviewService {
|
||||
constructor(private client: typeof ch) {}
|
||||
|
||||
// Helper methods
|
||||
private isRollupRow(date: string): boolean {
|
||||
return date.startsWith(ROLLUP_DATE_PREFIX);
|
||||
}
|
||||
|
||||
private getFillConfig(interval: string, startDate: string, endDate: string) {
|
||||
const useDateOnly = ['month', 'week'].includes(interval);
|
||||
return {
|
||||
from: clix.toStartOf(
|
||||
clix.datetime(startDate, useDateOnly ? 'toDate' : 'toDateTime'),
|
||||
interval as any,
|
||||
),
|
||||
to: clix.datetime(endDate, useDateOnly ? 'toDate' : 'toDateTime'),
|
||||
step: clix.toInterval('1', interval as any),
|
||||
};
|
||||
}
|
||||
|
||||
private createRevenueQuery({
|
||||
projectId,
|
||||
startDate,
|
||||
endDate,
|
||||
interval,
|
||||
timezone,
|
||||
filters,
|
||||
}: {
|
||||
projectId: string;
|
||||
startDate: string;
|
||||
endDate: string;
|
||||
interval: string;
|
||||
timezone: string;
|
||||
filters: IChartEventFilter[];
|
||||
}) {
|
||||
return clix(this.client, timezone)
|
||||
.select<{ date: string; total_revenue: number }>([
|
||||
`${clix.toStartOf('created_at', interval as any, timezone)} AS date`,
|
||||
'sum(revenue) AS total_revenue',
|
||||
])
|
||||
.from(TABLE_NAMES.events)
|
||||
.where('project_id', '=', projectId)
|
||||
.where('name', '=', 'revenue')
|
||||
.where('revenue', '>', 0)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
clix.datetime(startDate, 'toDateTime'),
|
||||
clix.datetime(endDate, 'toDateTime'),
|
||||
])
|
||||
.rawWhere(this.getRawWhereClause('events', filters))
|
||||
.groupBy(['date'])
|
||||
.rollup()
|
||||
.transform({
|
||||
date: (item) => new Date(item.date).toISOString(),
|
||||
});
|
||||
}
|
||||
|
||||
private mergeRevenueIntoSeries<T extends { date: string }>(
|
||||
series: T[],
|
||||
revenueData: { date: string; total_revenue: number }[],
|
||||
): (T & { total_revenue: number })[] {
|
||||
const revenueByDate = new Map(
|
||||
revenueData
|
||||
.filter((r) => !this.isRollupRow(r.date))
|
||||
.map((r) => [r.date, r.total_revenue]),
|
||||
);
|
||||
return series.map((row) => ({
|
||||
...row,
|
||||
total_revenue: revenueByDate.get(row.date) ?? 0,
|
||||
}));
|
||||
}
|
||||
|
||||
private getOverallRevenue(
|
||||
revenueData: { date: string; total_revenue: number }[],
|
||||
): number {
|
||||
return (
|
||||
revenueData.find((r) => this.isRollupRow(r.date))?.total_revenue ?? 0
|
||||
);
|
||||
}
|
||||
|
||||
private withDistinctSessionsIfNeeded<T>(
|
||||
query: ReturnType<typeof clix>,
|
||||
params: {
|
||||
filters: IChartEventFilter[];
|
||||
projectId: string;
|
||||
startDate: string;
|
||||
endDate: string;
|
||||
timezone: string;
|
||||
},
|
||||
): ReturnType<typeof clix> {
|
||||
if (!this.isPageFilter(params.filters)) {
|
||||
query.rawWhere(this.getRawWhereClause('sessions', params.filters));
|
||||
return query;
|
||||
}
|
||||
|
||||
return clix(this.client, params.timezone)
|
||||
.with('distinct_sessions', this.getDistinctSessions(params))
|
||||
.merge(query)
|
||||
.where(
|
||||
'id',
|
||||
'IN',
|
||||
clix.exp('(SELECT session_id FROM distinct_sessions)'),
|
||||
);
|
||||
}
|
||||
|
||||
isPageFilter(filters: IChartEventFilter[]) {
|
||||
return filters.some((filter) => filter.name === 'path' && filter.value);
|
||||
}
|
||||
|
||||
async getMetrics({
|
||||
getMetrics({
|
||||
projectId,
|
||||
filters,
|
||||
startDate,
|
||||
@@ -239,42 +116,170 @@ export class OverviewService {
|
||||
views_per_session: number;
|
||||
total_revenue: number;
|
||||
}[];
|
||||
}> {
|
||||
return this.isPageFilter(filters)
|
||||
? this.getMetricsWithPageFilter({
|
||||
projectId,
|
||||
filters,
|
||||
startDate,
|
||||
endDate,
|
||||
interval,
|
||||
timezone,
|
||||
})
|
||||
: this.getMetricsFromSessions({
|
||||
projectId,
|
||||
filters,
|
||||
startDate,
|
||||
endDate,
|
||||
interval,
|
||||
timezone,
|
||||
});
|
||||
}
|
||||
|
||||
private async getMetricsFromSessions({
|
||||
projectId,
|
||||
filters,
|
||||
startDate,
|
||||
endDate,
|
||||
interval,
|
||||
timezone,
|
||||
}: IGetMetricsInput): Promise<{
|
||||
metrics: MetricsRow & { total_revenue: number };
|
||||
series: MetricsSeriesRow[];
|
||||
}> {
|
||||
const where = this.getRawWhereClause('sessions', filters);
|
||||
const fillConfig = this.getFillConfig(interval, startDate, endDate);
|
||||
if (this.isPageFilter(filters)) {
|
||||
// Session aggregation with bounce rates
|
||||
const sessionAggQuery = clix(this.client, timezone)
|
||||
.select([
|
||||
`${clix.toStartOf('created_at', interval, timezone)} AS date`,
|
||||
'round((countIf(is_bounce = 1 AND sign = 1) * 100.) / countIf(sign = 1), 2) AS bounce_rate',
|
||||
'sum(revenue * sign) AS total_revenue',
|
||||
])
|
||||
.from(TABLE_NAMES.sessions, true)
|
||||
.where('sign', '=', 1)
|
||||
.where('project_id', '=', projectId)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
clix.datetime(startDate, 'toDateTime'),
|
||||
clix.datetime(endDate, 'toDateTime'),
|
||||
])
|
||||
.rawWhere(where)
|
||||
.groupBy(['date'])
|
||||
.rollup()
|
||||
.orderBy('date', 'ASC');
|
||||
|
||||
// Session metrics query
|
||||
const sessionQuery = clix(this.client, timezone)
|
||||
// Overall unique visitors
|
||||
const overallUniqueVisitorsQuery = clix(this.client, timezone)
|
||||
.select([
|
||||
'uniq(profile_id) AS unique_visitors',
|
||||
'uniq(session_id) AS total_sessions',
|
||||
])
|
||||
.from(TABLE_NAMES.events)
|
||||
.where('project_id', '=', projectId)
|
||||
.where('name', '=', 'screen_view')
|
||||
.where('created_at', 'BETWEEN', [
|
||||
clix.datetime(startDate, 'toDateTime'),
|
||||
clix.datetime(endDate, 'toDateTime'),
|
||||
])
|
||||
.rawWhere(this.getRawWhereClause('events', filters));
|
||||
|
||||
// Use toDate for month/week intervals, toDateTime for others
|
||||
const rollupDate =
|
||||
interval === 'month' || interval === 'week'
|
||||
? clix.date('1970-01-01')
|
||||
: clix.datetime('1970-01-01 00:00:00');
|
||||
|
||||
return clix(this.client, timezone)
|
||||
.with('session_agg', sessionAggQuery)
|
||||
.with(
|
||||
'overall_bounce_rate',
|
||||
clix(this.client, timezone)
|
||||
.select(['bounce_rate'])
|
||||
.from('session_agg')
|
||||
.where('date', '=', rollupDate),
|
||||
)
|
||||
.with(
|
||||
'overall_total_revenue',
|
||||
clix(this.client, timezone)
|
||||
.select(['total_revenue'])
|
||||
.from('session_agg')
|
||||
.where('date', '=', rollupDate),
|
||||
)
|
||||
.with(
|
||||
'daily_stats',
|
||||
clix(this.client, timezone)
|
||||
.select(['date', 'bounce_rate', 'total_revenue'])
|
||||
.from('session_agg')
|
||||
.where('date', '!=', rollupDate),
|
||||
)
|
||||
.with('overall_unique_visitors', overallUniqueVisitorsQuery)
|
||||
.select<{
|
||||
date: string;
|
||||
bounce_rate: number;
|
||||
unique_visitors: number;
|
||||
total_sessions: number;
|
||||
avg_session_duration: number;
|
||||
total_screen_views: number;
|
||||
views_per_session: number;
|
||||
total_revenue: number;
|
||||
overall_unique_visitors: number;
|
||||
overall_total_sessions: number;
|
||||
overall_bounce_rate: number;
|
||||
overall_total_revenue: number;
|
||||
}>([
|
||||
`${clix.toStartOf('e.created_at', interval)} AS date`,
|
||||
'ds.bounce_rate as bounce_rate',
|
||||
'uniq(e.profile_id) AS unique_visitors',
|
||||
'uniq(e.session_id) AS total_sessions',
|
||||
'round(avgIf(duration, duration > 0), 2) / 1000 AS _avg_session_duration',
|
||||
'if(isNaN(_avg_session_duration), 0, _avg_session_duration) AS avg_session_duration',
|
||||
'count(*) AS total_screen_views',
|
||||
'round((count(*) * 1.) / uniq(e.session_id), 2) AS views_per_session',
|
||||
'ds.total_revenue AS total_revenue',
|
||||
'(SELECT unique_visitors FROM overall_unique_visitors) AS overall_unique_visitors',
|
||||
'(SELECT total_sessions FROM overall_unique_visitors) AS overall_total_sessions',
|
||||
'(SELECT bounce_rate FROM overall_bounce_rate) AS overall_bounce_rate',
|
||||
'(SELECT total_revenue FROM overall_total_revenue) AS overall_total_revenue',
|
||||
])
|
||||
.from(`${TABLE_NAMES.events} AS e`)
|
||||
.leftJoin(
|
||||
'daily_stats AS ds',
|
||||
`${clix.toStartOf('e.created_at', interval)} = ds.date`,
|
||||
)
|
||||
.where('e.project_id', '=', projectId)
|
||||
.where('e.name', '=', 'screen_view')
|
||||
.where('e.created_at', 'BETWEEN', [
|
||||
clix.datetime(startDate, 'toDateTime'),
|
||||
clix.datetime(endDate, 'toDateTime'),
|
||||
])
|
||||
.rawWhere(this.getRawWhereClause('events', filters))
|
||||
.groupBy(['date', 'ds.bounce_rate', 'ds.total_revenue'])
|
||||
.orderBy('date', 'ASC')
|
||||
.fill(
|
||||
clix.toStartOf(
|
||||
clix.datetime(
|
||||
startDate,
|
||||
['month', 'week'].includes(interval) ? 'toDate' : 'toDateTime',
|
||||
),
|
||||
interval,
|
||||
),
|
||||
clix.datetime(
|
||||
endDate,
|
||||
['month', 'week'].includes(interval) ? 'toDate' : 'toDateTime',
|
||||
),
|
||||
clix.toInterval('1', interval),
|
||||
)
|
||||
.transform({
|
||||
date: (item) => new Date(item.date).toISOString(),
|
||||
})
|
||||
.execute()
|
||||
.then((res) => {
|
||||
const anyRowWithData = res.find(
|
||||
(item) =>
|
||||
item.overall_bounce_rate !== null ||
|
||||
item.overall_total_sessions !== null ||
|
||||
item.overall_unique_visitors !== null ||
|
||||
item.overall_total_revenue !== null,
|
||||
);
|
||||
return {
|
||||
metrics: {
|
||||
bounce_rate: anyRowWithData?.overall_bounce_rate ?? 0,
|
||||
unique_visitors: anyRowWithData?.overall_unique_visitors ?? 0,
|
||||
total_sessions: anyRowWithData?.overall_total_sessions ?? 0,
|
||||
avg_session_duration: average(
|
||||
res.map((item) => item.avg_session_duration),
|
||||
),
|
||||
total_screen_views: sum(
|
||||
res.map((item) => item.total_screen_views),
|
||||
),
|
||||
views_per_session: average(
|
||||
res.map((item) => item.views_per_session),
|
||||
),
|
||||
total_revenue: anyRowWithData?.overall_total_revenue ?? 0,
|
||||
},
|
||||
series: res.map(
|
||||
omit([
|
||||
'overall_bounce_rate',
|
||||
'overall_unique_visitors',
|
||||
'overall_total_sessions',
|
||||
'overall_total_revenue',
|
||||
]),
|
||||
),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
const query = clix(this.client, timezone)
|
||||
.select<{
|
||||
date: string;
|
||||
bounce_rate: number;
|
||||
@@ -283,8 +288,9 @@ export class OverviewService {
|
||||
avg_session_duration: number;
|
||||
total_screen_views: number;
|
||||
views_per_session: number;
|
||||
total_revenue: number;
|
||||
}>([
|
||||
`${clix.toStartOf('created_at', interval as any, timezone)} AS date`,
|
||||
`${clix.toStartOf('created_at', interval, timezone)} AS date`,
|
||||
'round(sum(sign * is_bounce) * 100.0 / sum(sign), 2) as bounce_rate',
|
||||
'uniqIf(profile_id, sign > 0) AS unique_visitors',
|
||||
'sum(sign) AS total_sessions',
|
||||
@@ -292,6 +298,7 @@ export class OverviewService {
|
||||
'if(isNaN(_avg_session_duration), 0, _avg_session_duration) AS avg_session_duration',
|
||||
'sum(sign * screen_view_count) AS total_screen_views',
|
||||
'round(sum(sign * screen_view_count) * 1.0 / sum(sign), 2) AS views_per_session',
|
||||
'sum(revenue * sign) AS total_revenue',
|
||||
])
|
||||
.from('sessions')
|
||||
.where('created_at', 'BETWEEN', [
|
||||
@@ -304,200 +311,41 @@ export class OverviewService {
|
||||
.having('sum(sign)', '>', 0)
|
||||
.rollup()
|
||||
.orderBy('date', 'ASC')
|
||||
.fill(fillConfig.from, fillConfig.to, fillConfig.step)
|
||||
.fill(
|
||||
clix.toStartOf(
|
||||
clix.datetime(
|
||||
startDate,
|
||||
['month', 'week'].includes(interval) ? 'toDate' : 'toDateTime',
|
||||
),
|
||||
interval,
|
||||
),
|
||||
clix.datetime(
|
||||
endDate,
|
||||
['month', 'week'].includes(interval) ? 'toDate' : 'toDateTime',
|
||||
),
|
||||
clix.toInterval('1', interval),
|
||||
)
|
||||
.transform({
|
||||
date: (item) => new Date(item.date).toISOString(),
|
||||
});
|
||||
|
||||
// Revenue query
|
||||
const revenueQuery = this.createRevenueQuery({
|
||||
projectId,
|
||||
startDate,
|
||||
endDate,
|
||||
interval,
|
||||
timezone,
|
||||
filters,
|
||||
return query.execute().then((res) => {
|
||||
// First row is the rollup row containing the total values
|
||||
return {
|
||||
metrics: {
|
||||
bounce_rate: res[0]?.bounce_rate ?? 0,
|
||||
unique_visitors: res[0]?.unique_visitors ?? 0,
|
||||
total_sessions: res[0]?.total_sessions ?? 0,
|
||||
avg_session_duration: res[0]?.avg_session_duration ?? 0,
|
||||
total_screen_views: res[0]?.total_screen_views ?? 0,
|
||||
views_per_session: res[0]?.views_per_session ?? 0,
|
||||
total_revenue: res[0]?.total_revenue ?? 0,
|
||||
},
|
||||
series: res
|
||||
.slice(1)
|
||||
.map(omit(['overall_bounce_rate', 'overall_unique_visitors'])),
|
||||
};
|
||||
});
|
||||
|
||||
// Execute both queries in parallel and merge results
|
||||
const [sessionRes, revenueRes] = await Promise.all([
|
||||
sessionQuery.execute(),
|
||||
revenueQuery.execute(),
|
||||
]);
|
||||
|
||||
const overallRevenue = this.getOverallRevenue(revenueRes);
|
||||
const series = this.mergeRevenueIntoSeries(sessionRes.slice(1), revenueRes);
|
||||
|
||||
return {
|
||||
metrics: {
|
||||
bounce_rate: sessionRes[0]?.bounce_rate ?? 0,
|
||||
unique_visitors: sessionRes[0]?.unique_visitors ?? 0,
|
||||
total_sessions: sessionRes[0]?.total_sessions ?? 0,
|
||||
avg_session_duration: sessionRes[0]?.avg_session_duration ?? 0,
|
||||
total_screen_views: sessionRes[0]?.total_screen_views ?? 0,
|
||||
views_per_session: sessionRes[0]?.views_per_session ?? 0,
|
||||
total_revenue: overallRevenue,
|
||||
},
|
||||
series,
|
||||
};
|
||||
}
|
||||
|
||||
private async getMetricsWithPageFilter({
|
||||
projectId,
|
||||
filters,
|
||||
startDate,
|
||||
endDate,
|
||||
interval,
|
||||
timezone,
|
||||
}: IGetMetricsInput): Promise<{
|
||||
metrics: MetricsRow & { total_revenue: number };
|
||||
series: MetricsSeriesRow[];
|
||||
}> {
|
||||
const where = this.getRawWhereClause('sessions', filters);
|
||||
const fillConfig = this.getFillConfig(interval, startDate, endDate);
|
||||
|
||||
// Session aggregation with bounce rates
|
||||
const sessionAggQuery = clix(this.client, timezone)
|
||||
.select([
|
||||
`${clix.toStartOf('created_at', interval as any, timezone)} AS date`,
|
||||
'round((countIf(is_bounce = 1 AND sign = 1) * 100.) / countIf(sign = 1), 2) AS bounce_rate',
|
||||
])
|
||||
.from(TABLE_NAMES.sessions, true)
|
||||
.where('sign', '=', 1)
|
||||
.where('project_id', '=', projectId)
|
||||
.where('created_at', 'BETWEEN', [
|
||||
clix.datetime(startDate, 'toDateTime'),
|
||||
clix.datetime(endDate, 'toDateTime'),
|
||||
])
|
||||
.rawWhere(where)
|
||||
.groupBy(['date'])
|
||||
.rollup()
|
||||
.orderBy('date', 'ASC');
|
||||
|
||||
// Overall unique visitors
|
||||
const overallUniqueVisitorsQuery = clix(this.client, timezone)
|
||||
.select([
|
||||
'uniq(profile_id) AS unique_visitors',
|
||||
'uniq(session_id) AS total_sessions',
|
||||
])
|
||||
.from(TABLE_NAMES.events)
|
||||
.where('project_id', '=', projectId)
|
||||
.where('name', '=', 'screen_view')
|
||||
.where('created_at', 'BETWEEN', [
|
||||
clix.datetime(startDate, 'toDateTime'),
|
||||
clix.datetime(endDate, 'toDateTime'),
|
||||
])
|
||||
.rawWhere(this.getRawWhereClause('events', filters));
|
||||
|
||||
// Use toDate for month/week intervals, toDateTime for others
|
||||
const rollupDate =
|
||||
interval === 'month' || interval === 'week'
|
||||
? clix.date(ROLLUP_DATE_PREFIX)
|
||||
: clix.datetime(`${ROLLUP_DATE_PREFIX} 00:00:00`);
|
||||
|
||||
// Main metrics query (without revenue)
|
||||
const mainQuery = clix(this.client, timezone)
|
||||
.with('session_agg', sessionAggQuery)
|
||||
.with(
|
||||
'overall_bounce_rate',
|
||||
clix(this.client, timezone)
|
||||
.select(['bounce_rate'])
|
||||
.from('session_agg')
|
||||
.where('date', '=', rollupDate),
|
||||
)
|
||||
.with(
|
||||
'daily_session_stats',
|
||||
clix(this.client, timezone)
|
||||
.select(['date', 'bounce_rate'])
|
||||
.from('session_agg')
|
||||
.where('date', '!=', rollupDate),
|
||||
)
|
||||
.with('overall_unique_visitors', overallUniqueVisitorsQuery)
|
||||
.select<{
|
||||
date: string;
|
||||
bounce_rate: number;
|
||||
unique_visitors: number;
|
||||
total_sessions: number;
|
||||
avg_session_duration: number;
|
||||
total_screen_views: number;
|
||||
views_per_session: number;
|
||||
overall_unique_visitors: number;
|
||||
overall_total_sessions: number;
|
||||
overall_bounce_rate: number;
|
||||
}>([
|
||||
`${clix.toStartOf('e.created_at', interval as any)} AS date`,
|
||||
'dss.bounce_rate as bounce_rate',
|
||||
'uniq(e.profile_id) AS unique_visitors',
|
||||
'uniq(e.session_id) AS total_sessions',
|
||||
'round(avgIf(duration, duration > 0), 2) / 1000 AS _avg_session_duration',
|
||||
'if(isNaN(_avg_session_duration), 0, _avg_session_duration) AS avg_session_duration',
|
||||
'count(*) AS total_screen_views',
|
||||
'round((count(*) * 1.) / uniq(e.session_id), 2) AS views_per_session',
|
||||
'(SELECT unique_visitors FROM overall_unique_visitors) AS overall_unique_visitors',
|
||||
'(SELECT total_sessions FROM overall_unique_visitors) AS overall_total_sessions',
|
||||
'(SELECT bounce_rate FROM overall_bounce_rate) AS overall_bounce_rate',
|
||||
])
|
||||
.from(`${TABLE_NAMES.events} AS e`)
|
||||
.leftJoin(
|
||||
'daily_session_stats AS dss',
|
||||
`${clix.toStartOf('e.created_at', interval as any)} = dss.date`,
|
||||
)
|
||||
.where('e.project_id', '=', projectId)
|
||||
.where('e.name', '=', 'screen_view')
|
||||
.where('e.created_at', 'BETWEEN', [
|
||||
clix.datetime(startDate, 'toDateTime'),
|
||||
clix.datetime(endDate, 'toDateTime'),
|
||||
])
|
||||
.rawWhere(this.getRawWhereClause('events', filters))
|
||||
.groupBy(['date', 'dss.bounce_rate'])
|
||||
.orderBy('date', 'ASC')
|
||||
.fill(fillConfig.from, fillConfig.to, fillConfig.step)
|
||||
.transform({
|
||||
date: (item) => new Date(item.date).toISOString(),
|
||||
});
|
||||
|
||||
// Revenue query
|
||||
const revenueQuery = this.createRevenueQuery({
|
||||
projectId,
|
||||
startDate,
|
||||
endDate,
|
||||
interval,
|
||||
timezone,
|
||||
filters,
|
||||
});
|
||||
|
||||
// Execute both queries in parallel and merge results
|
||||
const [mainRes, revenueRes] = await Promise.all([
|
||||
mainQuery.execute(),
|
||||
revenueQuery.execute(),
|
||||
]);
|
||||
|
||||
const overallRevenue = this.getOverallRevenue(revenueRes);
|
||||
const series = this.mergeRevenueIntoSeries(mainRes, revenueRes);
|
||||
|
||||
const anyRowWithData = mainRes.find(
|
||||
(item) =>
|
||||
item.overall_bounce_rate !== null ||
|
||||
item.overall_total_sessions !== null ||
|
||||
item.overall_unique_visitors !== null,
|
||||
);
|
||||
|
||||
return {
|
||||
metrics: {
|
||||
bounce_rate: anyRowWithData?.overall_bounce_rate ?? 0,
|
||||
unique_visitors: anyRowWithData?.overall_unique_visitors ?? 0,
|
||||
total_sessions: anyRowWithData?.overall_total_sessions ?? 0,
|
||||
avg_session_duration: average(
|
||||
mainRes.map((item) => item.avg_session_duration),
|
||||
),
|
||||
total_screen_views: sum(mainRes.map((item) => item.total_screen_views)),
|
||||
views_per_session: average(
|
||||
mainRes.map((item) => item.views_per_session),
|
||||
),
|
||||
total_revenue: overallRevenue,
|
||||
},
|
||||
series,
|
||||
};
|
||||
}
|
||||
|
||||
getRawWhereClause(type: 'events' | 'sessions', filters: IChartEventFilter[]) {
|
||||
@@ -520,6 +368,12 @@ export class OverviewService {
|
||||
}
|
||||
return item;
|
||||
}),
|
||||
// .filter((item) => {
|
||||
// if (this.isPageFilter(filters) && type === 'sessions') {
|
||||
// return item.name !== 'entry_path' && item.name !== 'entry_origin';
|
||||
// }
|
||||
// return true;
|
||||
// }),
|
||||
);
|
||||
|
||||
return Object.values(where).join(' AND ');
|
||||
@@ -560,6 +414,7 @@ export class OverviewService {
|
||||
'entry_path',
|
||||
'entry_origin',
|
||||
'coalesce(round(countIf(is_bounce = 1 AND sign = 1) * 100.0 / countIf(sign = 1), 2), 0) as bounce_rate',
|
||||
'sum(revenue * sign) as revenue',
|
||||
])
|
||||
.from(TABLE_NAMES.sessions, true)
|
||||
.where('sign', '=', 1)
|
||||
@@ -591,6 +446,7 @@ export class OverviewService {
|
||||
'p.avg_duration',
|
||||
'p.count as sessions',
|
||||
'b.bounce_rate',
|
||||
'coalesce(b.revenue, 0) as revenue',
|
||||
])
|
||||
.from('page_stats p', false)
|
||||
.leftJoin(
|
||||
@@ -613,6 +469,16 @@ export class OverviewService {
|
||||
limit = 10,
|
||||
timezone,
|
||||
}: IGetTopEntryExitInput) {
|
||||
const where = this.getRawWhereClause('sessions', filters);
|
||||
|
||||
const distinctSessionQuery = this.getDistinctSessions({
|
||||
projectId,
|
||||
filters,
|
||||
startDate,
|
||||
endDate,
|
||||
timezone,
|
||||
});
|
||||
|
||||
const offset = (cursor - 1) * limit;
|
||||
|
||||
const query = clix(this.client, timezone)
|
||||
@@ -637,19 +503,25 @@ export class OverviewService {
|
||||
clix.datetime(startDate, 'toDateTime'),
|
||||
clix.datetime(endDate, 'toDateTime'),
|
||||
])
|
||||
.rawWhere(where)
|
||||
.groupBy([`${mode}_origin`, `${mode}_path`])
|
||||
.having('sum(sign)', '>', 0)
|
||||
.orderBy('sessions', 'DESC')
|
||||
.limit(limit)
|
||||
.offset(offset);
|
||||
|
||||
const mainQuery = this.withDistinctSessionsIfNeeded(query, {
|
||||
projectId,
|
||||
filters,
|
||||
startDate,
|
||||
endDate,
|
||||
timezone,
|
||||
});
|
||||
let mainQuery = query;
|
||||
|
||||
if (this.isPageFilter(filters)) {
|
||||
mainQuery = clix(this.client, timezone)
|
||||
.with('distinct_sessions', distinctSessionQuery)
|
||||
.merge(query)
|
||||
.where(
|
||||
'id',
|
||||
'IN',
|
||||
clix.exp('(SELECT session_id FROM distinct_sessions)'),
|
||||
);
|
||||
}
|
||||
|
||||
return mainQuery.execute();
|
||||
}
|
||||
@@ -688,7 +560,28 @@ export class OverviewService {
|
||||
limit = 10,
|
||||
timezone,
|
||||
}: IGetTopGenericInput) {
|
||||
const prefixColumn = COLUMN_PREFIX_MAP[column] ?? null;
|
||||
const distinctSessionQuery = this.getDistinctSessions({
|
||||
projectId,
|
||||
filters,
|
||||
startDate,
|
||||
endDate,
|
||||
timezone,
|
||||
});
|
||||
|
||||
const prefixColumn = (() => {
|
||||
switch (column) {
|
||||
case 'region':
|
||||
return 'country';
|
||||
case 'city':
|
||||
return 'country';
|
||||
case 'browser_version':
|
||||
return 'browser';
|
||||
case 'os_version':
|
||||
return 'os';
|
||||
}
|
||||
return null;
|
||||
})();
|
||||
|
||||
const offset = (cursor - 1) * limit;
|
||||
|
||||
const query = clix(this.client, timezone)
|
||||
@@ -719,15 +612,24 @@ export class OverviewService {
|
||||
.limit(limit)
|
||||
.offset(offset);
|
||||
|
||||
const mainQuery = this.withDistinctSessionsIfNeeded(query, {
|
||||
projectId,
|
||||
filters,
|
||||
startDate,
|
||||
endDate,
|
||||
timezone,
|
||||
});
|
||||
let mainQuery = query;
|
||||
|
||||
return mainQuery.execute();
|
||||
if (this.isPageFilter(filters)) {
|
||||
mainQuery = clix(this.client, timezone)
|
||||
.with('distinct_sessions', distinctSessionQuery)
|
||||
.merge(query)
|
||||
.where(
|
||||
'id',
|
||||
'IN',
|
||||
clix.exp('(SELECT session_id FROM distinct_sessions)'),
|
||||
);
|
||||
} else {
|
||||
mainQuery.rawWhere(this.getRawWhereClause('sessions', filters));
|
||||
}
|
||||
|
||||
const res = await mainQuery.execute();
|
||||
|
||||
return res;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { cacheable } from '@openpanel/redis';
|
||||
import type { IChartEventFilter } from '@openpanel/validation';
|
||||
import { uniq } from 'ramda';
|
||||
import sqlstring from 'sqlstring';
|
||||
import {
|
||||
TABLE_NAMES,
|
||||
@@ -52,6 +53,7 @@ export type IClickhouseSession = {
|
||||
revenue: number;
|
||||
sign: 1 | 0;
|
||||
version: number;
|
||||
properties: Record<string, string>;
|
||||
};
|
||||
|
||||
export interface IServiceSession {
|
||||
@@ -90,6 +92,7 @@ export interface IServiceSession {
|
||||
utmContent: string;
|
||||
utmTerm: string;
|
||||
revenue: number;
|
||||
properties: Record<string, string>;
|
||||
profile?: IServiceProfile;
|
||||
}
|
||||
|
||||
@@ -141,6 +144,7 @@ export function transformSession(session: IClickhouseSession): IServiceSession {
|
||||
utmContent: session.utm_content,
|
||||
utmTerm: session.utm_term,
|
||||
revenue: session.revenue,
|
||||
properties: session.properties,
|
||||
profile: undefined,
|
||||
};
|
||||
}
|
||||
@@ -196,13 +200,12 @@ export async function getSessionList({
|
||||
|
||||
if (cursor) {
|
||||
const cAt = sqlstring.escape(cursor.createdAt);
|
||||
// TODO: remove id from cursor
|
||||
const cId = sqlstring.escape(cursor.id);
|
||||
sb.where.cursor = `created_at < toDateTime64(${cAt}, 3)`;
|
||||
sb.where.cursor = `(created_at < toDateTime64(${cAt}, 3) OR (created_at = toDateTime64(${cAt}, 3) AND id < ${cId}))`;
|
||||
sb.where.cursorWindow = `created_at >= toDateTime64(${cAt}, 3) - INTERVAL ${dateIntervalInDays} DAY`;
|
||||
sb.orderBy.created_at = 'created_at DESC';
|
||||
sb.orderBy.created_at = 'toDate(created_at) DESC, created_at DESC, id DESC';
|
||||
} else {
|
||||
sb.orderBy.created_at = 'created_at DESC';
|
||||
sb.orderBy.created_at = 'toDate(created_at) DESC, created_at DESC, id DESC';
|
||||
sb.where.created_at = `created_at > now() - INTERVAL ${dateIntervalInDays} DAY`;
|
||||
}
|
||||
|
||||
|
||||
@@ -180,11 +180,11 @@ export function sessionConsistency() {
|
||||
|
||||
// For write operations with session: cache WAL LSN after write
|
||||
if (isWriteOperation(operation)) {
|
||||
// logger.info('Prisma operation', {
|
||||
// operation,
|
||||
// args,
|
||||
// model,
|
||||
// });
|
||||
logger.info('Prisma operation', {
|
||||
operation,
|
||||
args,
|
||||
model,
|
||||
});
|
||||
|
||||
const result = await query(args);
|
||||
|
||||
|
||||
@@ -3,7 +3,6 @@ import type {
|
||||
IIntegrationConfig,
|
||||
INotificationRuleConfig,
|
||||
IProjectFilters,
|
||||
InsightPayload,
|
||||
} from '@openpanel/validation';
|
||||
import type {
|
||||
IClickhouseBotEvent,
|
||||
@@ -19,7 +18,6 @@ declare global {
|
||||
type IPrismaIntegrationConfig = IIntegrationConfig;
|
||||
type IPrismaNotificationPayload = INotificationPayload;
|
||||
type IPrismaProjectFilters = IProjectFilters[];
|
||||
type IPrismaProjectInsightPayload = InsightPayload;
|
||||
type IPrismaClickhouseEvent = IClickhouseEvent;
|
||||
type IPrismaClickhouseProfile = IClickhouseProfile;
|
||||
type IPrismaClickhouseBotEvent = IClickhouseBotEvent;
|
||||
|
||||
@@ -111,18 +111,13 @@ export type CronQueuePayloadProject = {
|
||||
type: 'deleteProjects';
|
||||
payload: undefined;
|
||||
};
|
||||
export type CronQueuePayloadInsightsDaily = {
|
||||
type: 'insightsDaily';
|
||||
payload: undefined;
|
||||
};
|
||||
export type CronQueuePayload =
|
||||
| CronQueuePayloadSalt
|
||||
| CronQueuePayloadFlushEvents
|
||||
| CronQueuePayloadFlushSessions
|
||||
| CronQueuePayloadFlushProfiles
|
||||
| CronQueuePayloadPing
|
||||
| CronQueuePayloadProject
|
||||
| CronQueuePayloadInsightsDaily;
|
||||
| CronQueuePayloadProject;
|
||||
|
||||
export type MiscQueuePayloadTrialEndingSoon = {
|
||||
type: 'trialEndingSoon';
|
||||
@@ -151,7 +146,7 @@ export const eventsGroupQueues = Array.from({
|
||||
}).map(
|
||||
(_, index, list) =>
|
||||
new GroupQueue<EventsQueuePayloadIncomingEvent['payload']>({
|
||||
logger: process.env.NODE_ENV === 'production' ? queueLogger : undefined,
|
||||
logger: queueLogger,
|
||||
namespace: getQueueName(
|
||||
list.length === 1 ? 'group_events' : `group_events_${index}`,
|
||||
),
|
||||
@@ -240,21 +235,6 @@ export const importQueue = new Queue<ImportQueuePayload>(
|
||||
},
|
||||
);
|
||||
|
||||
export type InsightsQueuePayloadProject = {
|
||||
type: 'insightsProject';
|
||||
payload: { projectId: string; date: string };
|
||||
};
|
||||
|
||||
export const insightsQueue = new Queue<InsightsQueuePayloadProject>(
|
||||
getQueueName('insights'),
|
||||
{
|
||||
connection: getRedisQueue(),
|
||||
defaultJobOptions: {
|
||||
removeOnComplete: 100,
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
export function addTrialEndingSoonJob(organizationId: string, delay: number) {
|
||||
return miscQueue.add(
|
||||
'misc',
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@openpanel/astro",
|
||||
"version": "1.0.6-local",
|
||||
"version": "1.0.5-local",
|
||||
"config": {
|
||||
"transformPackageJson": false,
|
||||
"transformEnvs": true
|
||||
@@ -14,7 +14,7 @@
|
||||
"files": ["src", "index.ts"],
|
||||
"keywords": ["astro-component"],
|
||||
"dependencies": {
|
||||
"@openpanel/web": "workspace:1.0.6-local"
|
||||
"@openpanel/web": "workspace:1.0.5-local"
|
||||
},
|
||||
"devDependencies": {
|
||||
"astro": "^5.7.7"
|
||||
|
||||
@@ -33,7 +33,7 @@ const methods: { name: OpenPanelMethodNames; value: unknown }[] = [
|
||||
value: {
|
||||
...options,
|
||||
sdk: 'astro',
|
||||
sdkVersion: '1.0.6',
|
||||
sdkVersion: '1.0.5',
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
{
|
||||
"name": "@openpanel/express",
|
||||
"version": "1.0.4-local",
|
||||
"version": "1.0.3-local",
|
||||
"module": "index.ts",
|
||||
"scripts": {
|
||||
"build": "rm -rf dist && tsup",
|
||||
"typecheck": "tsc --noEmit"
|
||||
},
|
||||
"dependencies": {
|
||||
"@openpanel/sdk": "workspace:1.0.3-local",
|
||||
"@openpanel/sdk": "workspace:1.0.2-local",
|
||||
"@openpanel/common": "workspace:*"
|
||||
},
|
||||
"peerDependencies": {
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
{
|
||||
"name": "@openpanel/nextjs",
|
||||
"version": "1.1.2-local",
|
||||
"version": "1.1.1-local",
|
||||
"module": "index.ts",
|
||||
"scripts": {
|
||||
"build": "rm -rf dist && tsup",
|
||||
"typecheck": "tsc --noEmit"
|
||||
},
|
||||
"dependencies": {
|
||||
"@openpanel/web": "workspace:1.0.6-local"
|
||||
"@openpanel/web": "workspace:1.0.5-local"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"next": "^12.0.0 || ^13.0.0 || ^14.0.0 || ^15.0.0 || ^16.0.0",
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
{
|
||||
"name": "@openpanel/react-native",
|
||||
"version": "1.0.4-local",
|
||||
"version": "1.0.3-local",
|
||||
"module": "index.ts",
|
||||
"scripts": {
|
||||
"build": "rm -rf dist && tsup",
|
||||
"typecheck": "tsc --noEmit"
|
||||
},
|
||||
"dependencies": {
|
||||
"@openpanel/sdk": "workspace:1.0.3-local"
|
||||
"@openpanel/sdk": "workspace:1.0.2-local"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@openpanel/tsconfig": "workspace:*",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@openpanel/sdk",
|
||||
"version": "1.0.3-local",
|
||||
"version": "1.0.2-local",
|
||||
"module": "index.ts",
|
||||
"scripts": {
|
||||
"build": "rm -rf dist && tsup",
|
||||
|
||||
@@ -68,7 +68,6 @@ export type OpenPanelOptions = {
|
||||
waitForProfile?: boolean;
|
||||
filter?: (payload: TrackHandlerPayload) => boolean;
|
||||
disabled?: boolean;
|
||||
debug?: boolean;
|
||||
};
|
||||
|
||||
export class OpenPanel {
|
||||
@@ -130,7 +129,6 @@ export class OpenPanel {
|
||||
}
|
||||
|
||||
async track(name: string, properties?: TrackProperties) {
|
||||
this.log('track event', name, properties);
|
||||
return this.send({
|
||||
type: 'track',
|
||||
payload: {
|
||||
@@ -145,7 +143,6 @@ export class OpenPanel {
|
||||
}
|
||||
|
||||
async identify(payload: IdentifyPayload) {
|
||||
this.log('identify user', payload);
|
||||
if (payload.profileId) {
|
||||
this.profileId = payload.profileId;
|
||||
this.flush();
|
||||
@@ -165,10 +162,12 @@ export class OpenPanel {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @deprecated This method is deprecated and will be removed in a future version.
|
||||
*/
|
||||
async alias(payload: AliasPayload) {}
|
||||
async alias(payload: AliasPayload) {
|
||||
return this.send({
|
||||
type: 'alias',
|
||||
payload,
|
||||
});
|
||||
}
|
||||
|
||||
async increment(payload: IncrementPayload) {
|
||||
return this.send({
|
||||
@@ -225,10 +224,4 @@ export class OpenPanel {
|
||||
});
|
||||
this.queue = [];
|
||||
}
|
||||
|
||||
log(...args: any[]) {
|
||||
if (this.options.debug) {
|
||||
console.log('[OpenPanel.dev]', ...args);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
{
|
||||
"name": "@openpanel/web",
|
||||
"version": "1.0.6-local",
|
||||
"version": "1.0.5-local",
|
||||
"module": "index.ts",
|
||||
"scripts": {
|
||||
"build": "rm -rf dist && tsup",
|
||||
"typecheck": "tsc --noEmit"
|
||||
},
|
||||
"dependencies": {
|
||||
"@openpanel/sdk": "workspace:1.0.3-local"
|
||||
"@openpanel/sdk": "workspace:1.0.2-local"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@openpanel/tsconfig": "workspace:*",
|
||||
|
||||
@@ -38,6 +38,8 @@ export class OpenPanel extends OpenPanelBase {
|
||||
});
|
||||
|
||||
if (!this.isServer()) {
|
||||
console.log('OpenPanel.dev - Initialized', this.options);
|
||||
|
||||
try {
|
||||
const pending = sessionStorage.getItem('openpanel-pending-revenues');
|
||||
if (pending) {
|
||||
@@ -200,6 +202,7 @@ export class OpenPanel extends OpenPanelBase {
|
||||
}
|
||||
|
||||
this.lastPath = path;
|
||||
console.log('OpenPanel.dev - Track page view');
|
||||
super.track('screen_view', {
|
||||
...(properties ?? {}),
|
||||
__path: path,
|
||||
|
||||
@@ -5,7 +5,6 @@ import { clientRouter } from './routers/client';
|
||||
import { dashboardRouter } from './routers/dashboard';
|
||||
import { eventRouter } from './routers/event';
|
||||
import { importRouter } from './routers/import';
|
||||
import { insightRouter } from './routers/insight';
|
||||
import { integrationRouter } from './routers/integration';
|
||||
import { notificationRouter } from './routers/notification';
|
||||
import { onboardingRouter } from './routers/onboarding';
|
||||
@@ -48,7 +47,6 @@ export const appRouter = createTRPCRouter({
|
||||
overview: overviewRouter,
|
||||
realtime: realtimeRouter,
|
||||
chat: chatRouter,
|
||||
insight: insightRouter,
|
||||
});
|
||||
|
||||
// export type definition of API
|
||||
|
||||
@@ -10,10 +10,12 @@ import {
|
||||
convertClickhouseDateToJs,
|
||||
db,
|
||||
eventService,
|
||||
formatClickhouseDate,
|
||||
getChartStartEndDate,
|
||||
getConversionEventNames,
|
||||
getEventList,
|
||||
getEventMetasCached,
|
||||
getEvents,
|
||||
getSettingsForProject,
|
||||
overviewService,
|
||||
sessionService,
|
||||
@@ -141,7 +143,6 @@ export const eventRouter = createTRPCRouter({
|
||||
path: columnVisibility?.name ?? true,
|
||||
duration: columnVisibility?.name ?? true,
|
||||
projectId: false,
|
||||
revenue: true,
|
||||
},
|
||||
});
|
||||
|
||||
@@ -220,7 +221,6 @@ export const eventRouter = createTRPCRouter({
|
||||
path: columnVisibility?.name ?? true,
|
||||
duration: columnVisibility?.name ?? true,
|
||||
projectId: false,
|
||||
revenue: true,
|
||||
},
|
||||
custom: (sb) => {
|
||||
sb.where.name = `name IN (${filteredConversions.map((event) => sqlstring.escape(event.name)).join(',')})`;
|
||||
|
||||
@@ -1,102 +0,0 @@
|
||||
import { db } from '@openpanel/db';
|
||||
import { z } from 'zod';
|
||||
import { getProjectAccess } from '../access';
|
||||
import { TRPCAccessError } from '../errors';
|
||||
import { createTRPCRouter, protectedProcedure } from '../trpc';
|
||||
|
||||
export const insightRouter = createTRPCRouter({
|
||||
list: protectedProcedure
|
||||
.input(
|
||||
z.object({
|
||||
projectId: z.string(),
|
||||
limit: z.number().min(1).max(100).optional().default(50),
|
||||
}),
|
||||
)
|
||||
.query(async ({ input: { projectId, limit }, ctx }) => {
|
||||
const access = await getProjectAccess({
|
||||
userId: ctx.session.userId,
|
||||
projectId,
|
||||
});
|
||||
|
||||
if (!access) {
|
||||
throw TRPCAccessError('You do not have access to this project');
|
||||
}
|
||||
|
||||
// Fetch more insights than needed to account for deduplication
|
||||
const allInsights = await db.projectInsight.findMany({
|
||||
where: {
|
||||
projectId,
|
||||
state: 'active',
|
||||
moduleKey: {
|
||||
notIn: ['page-trends', 'entry-pages'],
|
||||
},
|
||||
},
|
||||
orderBy: {
|
||||
impactScore: 'desc',
|
||||
},
|
||||
take: limit * 3, // Fetch 3x to account for deduplication
|
||||
});
|
||||
|
||||
// WindowKind priority: yesterday (1) > rolling_7d (2) > rolling_30d (3)
|
||||
const windowKindPriority: Record<string, number> = {
|
||||
yesterday: 1,
|
||||
rolling_7d: 2,
|
||||
rolling_30d: 3,
|
||||
};
|
||||
|
||||
// Group by moduleKey + dimensionKey, keep only highest priority windowKind
|
||||
const deduplicated = new Map<string, (typeof allInsights)[0]>();
|
||||
for (const insight of allInsights) {
|
||||
const key = `${insight.moduleKey}:${insight.dimensionKey}`;
|
||||
const existing = deduplicated.get(key);
|
||||
const currentPriority = windowKindPriority[insight.windowKind] ?? 999;
|
||||
const existingPriority = existing
|
||||
? (windowKindPriority[existing.windowKind] ?? 999)
|
||||
: 999;
|
||||
|
||||
// Keep if no existing, or if current has higher priority (lower number)
|
||||
if (!existing || currentPriority < existingPriority) {
|
||||
deduplicated.set(key, insight);
|
||||
}
|
||||
}
|
||||
|
||||
// Convert back to array, sort by impactScore, and limit
|
||||
const insights = Array.from(deduplicated.values())
|
||||
.sort((a, b) => (b.impactScore ?? 0) - (a.impactScore ?? 0))
|
||||
.slice(0, limit)
|
||||
.map(({ impactScore, ...rest }) => rest); // Remove impactScore from response
|
||||
|
||||
return insights;
|
||||
}),
|
||||
|
||||
listAll: protectedProcedure
|
||||
.input(
|
||||
z.object({
|
||||
projectId: z.string(),
|
||||
limit: z.number().min(1).max(500).optional().default(200),
|
||||
}),
|
||||
)
|
||||
.query(async ({ input: { projectId, limit }, ctx }) => {
|
||||
const access = await getProjectAccess({
|
||||
userId: ctx.session.userId,
|
||||
projectId,
|
||||
});
|
||||
|
||||
if (!access) {
|
||||
throw TRPCAccessError('You do not have access to this project');
|
||||
}
|
||||
|
||||
const insights = await db.projectInsight.findMany({
|
||||
where: {
|
||||
projectId,
|
||||
state: 'active',
|
||||
},
|
||||
orderBy: {
|
||||
impactScore: 'desc',
|
||||
},
|
||||
take: limit,
|
||||
});
|
||||
|
||||
return insights;
|
||||
}),
|
||||
});
|
||||
@@ -1,3 +1,2 @@
|
||||
export * from './src/index';
|
||||
export * from './src/types.validation';
|
||||
export * from './src/types.insights';
|
||||
|
||||
@@ -553,5 +553,3 @@ export const zCreateImport = z.object({
|
||||
});
|
||||
|
||||
export type ICreateImport = z.infer<typeof zCreateImport>;
|
||||
|
||||
export * from './types.insights';
|
||||
|
||||
@@ -1,43 +0,0 @@
|
||||
export type InsightMetricKey = 'sessions' | 'pageviews' | 'share';
|
||||
|
||||
export type InsightMetricUnit = 'count' | 'ratio';
|
||||
|
||||
export interface InsightMetricEntry {
|
||||
current: number;
|
||||
compare: number;
|
||||
delta: number;
|
||||
changePct: number | null;
|
||||
direction: 'up' | 'down' | 'flat';
|
||||
unit: InsightMetricUnit;
|
||||
}
|
||||
|
||||
export interface InsightDimension {
|
||||
key: string;
|
||||
value: string;
|
||||
displayName?: string;
|
||||
}
|
||||
|
||||
export interface InsightExtra {
|
||||
[key: string]: unknown;
|
||||
currentShare?: number;
|
||||
compareShare?: number;
|
||||
shareShiftPp?: number;
|
||||
isNew?: boolean;
|
||||
isGone?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Shared payload shape for insights cards. This is embedded in DB rows and
|
||||
* shipped to the frontend, so it must remain backwards compatible.
|
||||
*/
|
||||
export interface InsightPayload {
|
||||
kind?: 'insight_v1';
|
||||
dimensions: InsightDimension[];
|
||||
primaryMetric: InsightMetricKey;
|
||||
metrics: Partial<Record<InsightMetricKey, InsightMetricEntry>>;
|
||||
|
||||
/**
|
||||
* Module-specific extra data.
|
||||
*/
|
||||
extra?: Record<string, unknown>;
|
||||
}
|
||||
2635
pnpm-lock.yaml
generated
2635
pnpm-lock.yaml
generated
File diff suppressed because it is too large
Load Diff
@@ -7,10 +7,10 @@ packages:
|
||||
# Define a catalog of version ranges.
|
||||
catalog:
|
||||
zod: ^3.24.2
|
||||
react: ^19.2.3
|
||||
"@types/react": ^19.2.3
|
||||
"react-dom": ^19.2.3
|
||||
"@types/react-dom": ^19.2.3
|
||||
react: ^19.0.0
|
||||
"@types/react": ^19.0.0
|
||||
"react-dom": ^19.0.0
|
||||
"@types/react-dom": ^19.0.0
|
||||
"@types/node": ^24.7.1
|
||||
typescript: ^5.9.3
|
||||
groupmq: 1.1.1-next.2
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
version: "3"
|
||||
version: '3'
|
||||
|
||||
services:
|
||||
op-proxy:
|
||||
image: caddy:2-alpine
|
||||
restart: always
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
- '80:80'
|
||||
- '443:443'
|
||||
volumes:
|
||||
- op-proxy-data:/data
|
||||
- op-proxy-config:/config
|
||||
@@ -28,7 +28,7 @@ services:
|
||||
volumes:
|
||||
- op-db-data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U postgres"]
|
||||
test: [ 'CMD-SHELL', 'pg_isready -U postgres' ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
@@ -49,9 +49,9 @@ services:
|
||||
restart: always
|
||||
volumes:
|
||||
- op-kv-data:/data
|
||||
command: ["redis-server", "--maxmemory-policy", "noeviction"]
|
||||
command: [ 'redis-server', '--maxmemory-policy', 'noeviction' ]
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "redis-cli ping"]
|
||||
test: [ 'CMD-SHELL', 'redis-cli ping' ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
@@ -65,7 +65,7 @@ services:
|
||||
# - 6379:6379
|
||||
|
||||
op-ch:
|
||||
image: clickhouse/clickhouse-server:25.10.2.65
|
||||
image: clickhouse/clickhouse-server:24.3.2-alpine
|
||||
restart: always
|
||||
volumes:
|
||||
- op-ch-data:/var/lib/clickhouse
|
||||
@@ -73,10 +73,8 @@ services:
|
||||
- ./clickhouse/clickhouse-config.xml:/etc/clickhouse-server/config.d/op-config.xml:ro
|
||||
- ./clickhouse/clickhouse-user-config.xml:/etc/clickhouse-server/users.d/op-user-config.xml:ro
|
||||
- ./clickhouse/init-db.sh:/docker-entrypoint-initdb.d/init-db.sh:ro
|
||||
environment:
|
||||
- CLICKHOUSE_SKIP_USER_SETUP=1
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", 'clickhouse-client --query "SELECT 1"']
|
||||
test: [ 'CMD-SHELL', 'clickhouse-client --query "SELECT 1"' ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
@@ -91,7 +89,7 @@ services:
|
||||
max-file: "3"
|
||||
|
||||
op-api:
|
||||
image: lindesvard/openpanel-api:2.0.0
|
||||
image: lindesvard/openpanel-api:latest
|
||||
restart: always
|
||||
command: >
|
||||
sh -c "
|
||||
@@ -101,7 +99,7 @@ services:
|
||||
pnpm start
|
||||
"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -f http://localhost:3000/healthcheck || exit 1"]
|
||||
test: [ 'CMD-SHELL', 'curl -f http://localhost:3000/healthcheck || exit 1' ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
@@ -121,7 +119,7 @@ services:
|
||||
max-file: "3"
|
||||
|
||||
op-dashboard:
|
||||
image: lindesvard/openpanel-dashboard:2.0.0
|
||||
image: lindesvard/openpanel-dashboard:latest
|
||||
restart: always
|
||||
depends_on:
|
||||
op-api:
|
||||
@@ -129,8 +127,7 @@ services:
|
||||
env_file:
|
||||
- .env
|
||||
healthcheck:
|
||||
test:
|
||||
["CMD-SHELL", "curl -f http://localhost:3000/api/healthcheck || exit 1"]
|
||||
test: [ 'CMD-SHELL', 'curl -f http://localhost:3000/api/healthcheck || exit 1' ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
@@ -141,7 +138,7 @@ services:
|
||||
max-file: "3"
|
||||
|
||||
op-worker:
|
||||
image: lindesvard/openpanel-worker:2.0.0
|
||||
image: lindesvard/openpanel-worker:latest
|
||||
restart: always
|
||||
depends_on:
|
||||
op-api:
|
||||
@@ -149,7 +146,7 @@ services:
|
||||
env_file:
|
||||
- .env
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -f http://localhost:3000/healthcheck || exit 1"]
|
||||
test: [ 'CMD-SHELL', 'curl -f http://localhost:3000/healthcheck || exit 1' ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
@@ -174,4 +171,4 @@ volumes:
|
||||
op-proxy-data:
|
||||
driver: local
|
||||
op-proxy-config:
|
||||
driver: local
|
||||
driver: local
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user