Recipes

CLI recipes and data loaders

Recipes are declarative pipelines — a fixed sequence of ingest, feature, and (optionally) classify steps. CLI recipes are executable Python scripts installed as console scripts and are available now; they train on the per-event aggregates in the manifest. Agent recipes are MCP tool sequences whose scatter and reduce steps run on the open deterministic scattering core, which is in development.

vikshep-recipe calibrate

Fits a ridge-regularised linear regression from aggregate features to a target scalar. Useful for detector calibration and response correction.

vikshep-recipe calibrate \
  --features <manifest.json>   # required: path to the ingest manifest
  --target   <column_name>     # required: aggregate scalar to predict
  --out      <dir>             # optional: default = manifest directory

Flag	Required	Description
--features	Yes	Path to manifest.json from vikshep-ingest
--target	Yes	Aggregate column name to regress against
--out	No	Output directory for calibrate_report.json (default: manifest directory)

vikshep-recipe tag

Trains a logistic classifier on aggregate features with a weighted DisCo penalty on the dependence between the score and the --protect variable on background events, to suppress mass sculpting. Training uses a Pearson-correlation proxy for the penalty gradient; the dCorr² in the report is the exact weighted value, evaluated once after training. The report records both as training_gradient: "pearson_proxy" and reported_dcorr2: "exact_weighted".

vikshep-recipe tag \
  --features <manifest.json>   # required
  --label    <column_name>     # required: binary target
  --protect  <column_name>     # required: variable to decorrelate from
  --weights  <column_name>     # optional: per-event weights
  --lambda   <float>           # optional: DisCo penalty (default 1.0)
  --out      <dir>             # optional: default = manifest directory

Flag	Required	Description
--features	Yes	Path to manifest.json
--label	Yes	Binary label column (0/1 or bool)
--protect	Yes	Column whose dependence on the score is penalised
--weights	No	Per-event weight column (default: uniform)
--lambda	No	DisCo penalty weight (0 = off, default 1.0, >1 = stricter)
--out	No	Output directory for tag_report.json (default: manifest directory)

Data loaders

Loaders are discovered via Python entry points (vikshep.loaders). Each loader writes POSIX shared memory and returns a 28-hex SHA3-256 OID. All loaders are in backend/ingest and installed with pip install -e backend/ingest.

Loader	Format	CLI entry point
g4	Geant4 CSV (event_id, layer, phi, theta, momentum[, energy])	vikshep-ingest g4
root-uproot	.root (Geant4, CMS Open Data)	Python API (vikshep.loaders entry point)
hdf5	.h5, generic HDF5	Python API (vikshep.loaders entry point)
well	The Well HDF5 (15 TB physics simulations dataset)	Python API (vikshep_ingest.well)

All loaders are CPU-only and require no account. The well loader is available in the Free tier; hosted well-scale ingest (> 1 TB) requires Lab or Enterprise.

Agent recipes

Agent recipes are declarative MCP tool sequences defined in agent/src/recipes/. They run through the TypeScript/Bun MCP orchestrator (async streaming, provenance logging) and call the scatter and reduce tools, so they need the open deterministic scattering core, which is in development.

Recipe	Pipeline
hep-tagging-disco	ingest → scatter → r₂ → DisCo classifier
bsm-anomaly	ingest → scatter → log-mean → HNSW → detect
general-feature	ingest → scatter (Dim, Group from request) → reduce

Manifest anatomy

The manifest is the control-plane payload produced by any loader. It contains OIDs and scalar metadata — no raw tensors.

{
  "n_events": 10,
  "schema": "komal_v1",
  "aggregate_names": ["layer1_n_hits", "layer1_e_mean", ...],  // 32 scalars/event
  "grid_oids": ["a3f1...", ...],  // 28-char SHA3-256 hex, one per event
  "pad_policy_per_axis": {
    "phi": "Circular",     // correct for azimuthal angle
    "theta": "ZeroPad"     // correct for polar angle
  }
}

The 28-char OIDs are the only handles that cross the AGPL boundary into the engine. Raw grid tensors live in POSIX shared memory and never reach TypeScript or the browser. The schema is frozen; changing it is a breaking change requiring a major version bump in both control plane and data plane.

Benchmark harness

Run the harness on a real analysis pipeline to sweep the DisCo lambda and record AUC-vs-dCorr² trade-offs:

python -m bench.run \
  --manifest manifest.json \
  --label    <signal_col> \
  --protect  <mass_col> \
  --lambdas  0,0.1,1,10

The harness outputs a sweep table and computes the Asimov proxy and Jensen-Shannon divergence for each lambda value. All free, all local.