Stats

Looks after the data of the instances a host serves. Today it measures facts about each instance (disk used, last request served) and reports them. Later it will also carry out decisions about that data, such as archiving.

Status. Working, first step: facts and the measure decision. Archiving, restoring, offloading backups and purging come next, as further decisions. The rules engine is ready as a Go library but isn't read from pocketone.yaml yet.

Install · Configure · Use · Integrations · Example

Install

Lifecycle runs on a host that serves many instances (ext/multisite, with pocker or docker), not on a single app.

The stock pocketone binary turns it on by itself in host mode: when pocketone.yaml names a provider (pocker or docker), every instance under that provider's root folder is measured. There's no switch to flip.

provider: pocker   # or docker
instances: { file: ./instances.yaml }

In your own host build, add it as a gate (to see requests) and a background job:

lc := &stats.Lifecycle{Directory: h.Directory, Folder: func(i multisite.Instance) string { return filepath.Join(root, i.ID) }}
h.Gates = append(h.Gates, lc)                // records each local instance's last request
h.Background = append(h.Background, lc.Run) // measures and reports on the policy's schedule

Folder says where an instance's data lives. The stock binary uses the instance's own dir when it has one, else <root>/<id>.

Configure

The lifecycle section in pocketone.yaml sets the schedule. Everything is optional.

lifecycle:
  measure_every: 24h   # how old a disk measurement may get
  report_every: 1h     # how often changed last-request times are reported
  concurrency: 2       # instances measured at once

The policy

What to do comes from a policy. The built-in one (YAMLPolicy) reads this section and decides one thing: measure an instance when it was never measured, or when its measurement is older than measure_every. Zero or missing values fall back to the defaults.

Rules with criteria, all/any/not, vetoes and budgets exist in the rules engine, but the lifecycle section doesn't take them yet.

pocketone.yaml reference

lifecycle:

Instance data on this host: how often disk use is measured and last-request times are reported.

Use

What it measures

For each instance this host serves itself (not the ones it forwards to another edge):

• disk_bytes: the size of the regular files in the instance's folder (symlinks aren't followed; 0 when there's no folder), and measured_at, when that was measured.
• last_active: the last request this host served for it, recorded at most once a minute per instance.

When

• As soon as the instance list loads, every instance is measured, because none has been measured yet.
• After that it checks once a minute and measures the instances whose measurement is older than measure_every.
• Changed last-request times are reported every report_every.

Facts are kept in memory. With the default log sink, a restart measures everything again.

What you'll see

Log lines from the default sink and the measuring pass:

INFO lifecycle: facts instances=12 disk_bytes=48213504
INFO lifecycle: measured instances=12 disk_bytes=48213504 ms=37

lifecycle: facts is each report the sink receives (after a measuring pass, and for last-request times). lifecycle: measured follows a measuring pass, with how long it took. Problems show as warnings: lifecycle: measuring (a folder couldn't be read), lifecycle: report (the sink failed; last-request times are retried next time) and lifecycle: policy.

Try it

example/main.go measures the instance folders under a root once and prints each instance's size:

go run ./example --root ./pb_root --instances instances.yaml

Integrations

Works with

It knows nothing about plans, payments or tenants: that belongs to whichever policy provider is configured.

Your own policy

Set Lifecycle.Policy to anything that implements PolicyProvider:

type PolicyProvider interface {
	Decide(ctx context.Context, facts []Facts) ([]Decision, error) // what to do now, and why
	ReportEvery() time.Duration                                    // how often last-request times are reported
	Concurrency() int                                              // instances worked on at once
}

A control plane can provide its own, for example one that delegates the decisions to a mothership, and providers can wrap each other. Only the measure action is carried out today; any other action is logged as lifecycle: action not supported yet.

Your own sink

Set Lifecycle.Sink to anything that implements Sink (Report(ctx, facts []Facts) error) to send facts somewhere other than the log. A sink that also implements Loader (Load(ctx) ([]Facts, error)) gives the facts back at start, so a restart doesn't measure everything again or forget last-request times.

Rules: criteria an app can replace

rules (package github.com/pockethost/pocketone/ext/multisite-stats/rules, standard library only) is the engine policies are built from, on an edge or a mothership. A subject is whatever the caller evaluates: an instance's facts on an edge, an account with its instances on a mothership.

• Criteria answer one question about a subject (stale? orphaned? over its plan?) and say why. They live in a Registry by name: an app registers its own or replaces a built-in one (Register("stale", ...)), and every rule that refers to the name (Ref("stale")) follows, so what "stale" means is the app's to decide.
• Where they come from is pluggable too: code, or definitions from config (Load) built from kinds the app registered (RegisterKind("idle", ...)) and combined with all, any and not. A YAML section and a mothership are both just sources of definitions. Load registers nothing when a definition is wrong.
• Rules lead from a criterion to an action, with named vetoes that always win and a budget per run.
• Reports list each decision with its reasons, what was vetoed and by what, what the budget held back and what failed. Summary gives one line per rule (archive-stale: 3 decided, 1 vetoed). Evaluating does nothing by itself: acting on the decisions, or only logging them in a dry run, is the caller's.

A definition, as config would supply it:

stale:      { kind: idle, params: { after: 8760h } }
archivable: { kind: all, of: [stale, free_plan] }
keep:       { kind: not, of: [archivable] }

And in Go:

r := rules.NewRegistry[Facts]()
r.RegisterKind("idle", idleFor)                       // params: { after: 8760h }
_ = r.Load(map[string]rules.Def{
	"stale":      {Kind: "idle", Params: map[string]any{"after": "8760h"}},
	"archivable": {Kind: "all", Of: []string{"stale", "free_plan"}},
})
e := rules.Engine[Facts]{Key: id, Rules: []rules.Rule[Facts]{{
	Name: "archive-stale", Action: "archive", When: r.Ref("archivable"),
	Unless: []rules.Veto[Facts]{{Name: "pinned", If: r.Ref("pinned")}}, Budget: 200,
}}}
report := e.Evaluate(ctx, facts)

Every rule sees every subject, so one subject can get several decisions. Make rules exclusive by building them from exclusive criteria.

Example

example/main.go measures the instance folders under a root once and prints their facts, which is what a host's lifecycle does daily.

// Measure the instance folders under a root once and print their facts (what a host's lifecycle does daily).
//
//	go run ./example --root ./pb_root --instances instances.yaml
package main

import (
	"context"
	"flag"
	"fmt"
	"log"
	"log/slog"
	"path/filepath"

	"github.com/pockethost/pocketone/ext/multisite"
	"github.com/pockethost/pocketone/ext/multisite-stats"
)

func main() {
	root := flag.String("root", "./pb_root", "one folder per instance: <root>/<id>")
	instances := flag.String("instances", "instances.yaml", "instances file")
	flag.Parse()

	src := &multisite.YAMLSource{Path: *instances, Required: true}
	list, err := src.Snapshot(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	dir := multisite.NewDirectory(slog.Default())
	dir.Load(list)

	lc := &stats.Lifecycle{
		Directory: dir,
		Folder: func(i multisite.Instance) string {
			if i.Dir != "" {
				return i.Dir
			}
			return filepath.Join(*root, i.ID)
		},
		Sink: stats.LogSink{},
	}
	lc.Step(context.Background()) // never measured: every instance is measured now
	for _, f := range lc.Local() {
		fmt.Printf("%-24s %12d bytes\n", f.Instance, f.DiskBytes)
	}
}

How it works

• Go module github.com/pockethost/pocketone/ext/multisite-stats. The stock binary wires it in examples/pocketone/multi.go.
• Run loads recorded facts from a Loader sink, then waits for the host's instance list to load (checking every second). From then on, once a minute, it asks the policy for decisions on the current facts, measures the instances it was asked to (concurrency at a time) and reports what it measured to the sink.
• Measuring walks the folder and adds up regular file sizes. An instance's pending last-request time goes out with its measurement, so it isn't reported twice.
• Admit records last_active only for instances this host serves itself (Remote empty), and never refuses a request.

License

MIT

Edit this page on GitHub