Add or update a framework

Adding an Academic Standards framework should normally be a data and configuration change, not a curriculum-specific Python change.

The backend is designed to apply the same package validation, catalog, search, graph, resource, prompt, and MCP behavior to each accepted package.

Before you start

This repository consumes already-created curriculum graph artifacts. It does not include the upstream PDF/document extraction or knowledge-graph construction pipeline.

You need accepted source artifacts that satisfy the delivery and detailed artifact contracts before the package builder can create a graph package.

Read these references first:

Decide what is changing

Use a new framework ID when the conceptual framework family is different.

Use a new snapshot when the source curriculum/version changes within the same framework family. Snapshot IDs are immutable and namespaced by framework ID.

Do not edit an already passed package in place. The current package builder supports initial package revision 1; the current loader/validator also supports revision 1. An update therefore should be represented as a new immutable snapshot rather than an invented later package revision.

End-to-end workflow

%%{init: {"themeVariables": {"fontSize": "18px"}}}%%
flowchart TB
    SOURCE[Accepted curriculum graph artifacts] --> PROFILE[Create/select interpretation profile]
    PROFILE --> PROMPT[Optional prompt configuration]
    PROMPT --> DRY[Build package --dry-run]
    DRY --> BUILD[Create pending graph package]
    BUILD --> READONLY[Validate read-only]
    READONLY -->|valid| PERSIST[Persist validation]
    PERSIST --> PASSED[Terminal passed package]
    PASSED --> SMOKE[Repository STDIO smoke]
    SMOKE --> DOCS[Update framework catalog/docs]
    DOCS --> BUNDLE{Release bundle?}
    BUNDLE -- Yes --> MCPB[Build + stage-smoke MCPB]
    BUNDLE -- No --> DONE[Review]
    MCPB --> DONE

Hold "Alt" / "Option" to enable pan & zoom

1. Create or version the interpretation profile

Profiles live at:

config/profiles/<profile-id>/<profile-version>/profile.json

The profile is the semantic contract between generic Python and the curriculum. It should express source-local behavior such as:

framework and jurisdiction terminology;
local grade labels and normalized discovery facets;
source statement types and normalized classifications;
hierarchy levels and allowed parent relationships;
tree versus multi-parent DAG behavior;
root/grouping semantics;
code format, normalization, and prefix-search policy;
language/terminology preservation;
rights and derivative-use policy;
known source anomalies; and
required disclosures.

Do not solve a profile problem by adding if framework_id == ... branches to generic runtime code.

If profile semantics change for an existing immutable snapshot/package, create a new appropriate version/snapshot relationship rather than silently changing the bytes that a passed package checksum already binds.

2. Add optional prompt guidance

Prompt configuration lives at:

config/prompts/<profile-id>/<profile-version>/prompts.json

Prompt overlays are optional. Use them for framework-local terminology, warnings, context, or output guidance—not for retrieval logic that belongs in ordinary services or profile semantics.

The prompt configuration must resolve against the same profile identity/version used by the package.

3. Prepare delivery artifacts

The package builder requires canonical node and relationship JSONL inputs:

as_lc_lp_nodes_*.jsonl
as_lc_lp_relationships_*.jsonl

Recognized detailed artifacts may also be supplied, including the standards framework/items, provenance, KG bundle, hierarchy relationships, unresolved items, and validation report described in Graph package format.

Put every input in a new input set, data/input_artifacts/<set>/, with the same delivery/, detailed/ and additional/ layout as the existing sets. The builder records each input's repository-relative path in the package's evidence archive record, so inputs outside the repository are rejected. The existing ignore rules then keep the archive-only files out of Git. See Input sets.

Treat source wording, identifiers, hierarchy, unresolved relationships, rights, and provenance as evidence to preserve rather than values to normalize away.

Learning progression inputs and updates

The seven mixed AS/LC/LP packages reuse one graph/runtime per curriculum. Nodes and the AS/LC relationship prefix remain byte-identical; normalized LP wire records are appended in (relationship type, identifier) order. Source/target CASE UUIDs resolve to the existing outer standard IDs. Combined upstream exports are reconciliation inputs, not interchangeable delivery files: five use a flat format; CBSE and CAPS use envelopes. Preparation normalizes the split LP records and preserves exact IDs, endpoint orientation, origin, attribution and the original provenance. It does not rerun the producer/checker pipeline or accept final claims as edges.

Input sets

Input set	Framework	Original source document key
`data/input_artifacts/ghana_english`	`ghana-nacca-primary-english-language-basic-1-3`	`e49a792637011b32ed2ed906d58f9992e5208d2b7c15425ccc971f4f51de43c8`
`data/input_artifacts/ghana_math`	`ghana-nacca-primary-mathematics-basic-4-6`	`8d59d76cb439110ad9409e9c12561023df60d12f234ced7285796f7e320fd9fa`
`data/input_artifacts/pratham_science`	`india-cbse-science-learning-framework-classes-9-10`	`3f8c25c19ed8395bf6625b7958092b8d219cae743b9ea36fcfd56efd59b6d2fc`
`data/input_artifacts/madhi_math`	`india-tamil-nadu-tnscert-mathematics-classes-1-5`	`33c5da78839a7611308dd30f342b72f931a661235e094cc641832ec2a5548444`
`data/input_artifacts/nigeria_math`	`nigeria-nerdc-mathematics-primary-1-3`	`09d6b52b54b2f6b00d5279a58650d0ef10ab5a6a5d3da02d10b24c42edb1058a`
`data/input_artifacts/rwanda_math`	`rwanda-reb-mathematics-lower-primary-1-3`	`7b9629e6bd5ad5566e6420892908a307687543ede8cc0c676ddf5b0321ba5874`
`data/input_artifacts/south_africa_english`	`south-africa-dbe-caps-english-home-language-grades-r-3`	`c939a3a9dcce92ed61d970f4599ced6f2ddc425d1220eeeab9828472b878d24d`

Each input set holds the full evidence for one package, and its package_build.json is the build spec. Only small files are committed:

data/input_artifacts/<set>/
├── package_build.json                     committed
├── delivery/                              git-ignored (same bytes as the package's delivery files)
├── additional/                            git-ignored (64 original LP provenance shards)
└── detailed/
    ├── as_lc_validation_report.json       committed
    ├── as_standards_framework.json        committed
    ├── as_unresolved_items.json           committed
    ├── lc_dedup_groups.json               committed
    ├── lc_generation_summary.json         committed
    ├── lp_generation_summary.json         committed
    ├── lp_normalization_receipt.json      committed
    ├── lp_unresolved_items.json           committed
    ├── lp_validation_report.json          committed
    └── 11 archive-only files              git-ignored

The 11 git-ignored detailed/ files are as_entity_provenance.json, as_kg_bundle.json, as_lc_kg_bundle.json, as_relationships_has_child.jsonl, as_standards_framework_items.jsonl, lc_entity_provenance.json, lp_final_claims.json, lp_relationship_provenance.json, lp_relationship_provenance_index.json, lp_relationships_builds_towards.jsonl and lp_relationships_relates_to.jsonl. The ignore rules in .gitignore match any folder under data/input_artifacts/, so a new input set with the same layout is covered too.

The git-ignored files are the evidence archive. A copy of the input-set folders is kept in IDinsight Google Drive. Uploading to Drive and managing access are done by hand, outside this repository; no command here contacts Drive. The server, the image and the bundle never read data/input_artifacts/.

A fresh clone has only the committed files. That is enough to run the server, the test suite and kgfegmcp-validate-packages --runtime-only. Rebuilding or accepting a package needs the archive restored first.

Restore and verify archived evidence

Download the input-set folders you need from the IDinsight Google Drive archive.
Place them under data/input_artifacts/<set>/, so each file sits at the localPath listed in the package's evidence/evidence_archive_record.json. The committed files in the Drive copy are the same bytes, so overwriting them changes nothing.
Check the restored files against every package's record:
```
uv --directory backend run --locked --no-dev kgfegmcp-verify-evidence-archive all
```
Use one --framework-id <framework-id> --snapshot-id <snapshot-id> to check a single package.

The command reads only the records and the local files; it never downloads, uploads or changes anything. It prints JSON with the archive location and, per package, a status of ok, incomplete or record_unavailable, ok/missing/mismatch counts, and one entry per file:

{"localPath": "data/input_artifacts/ghana_english/detailed/as_kg_bundle.json", "logicalName": "academicStandardsBundle", "status": "ok", "expectedSha256": "sha256:6447…", "actualSha256": "sha256:6447…", "expectedSizeBytes": 9330374, "actualSizeBytes": 9330374}

A missing entry has a reason of absent or unreadable; mismatch means the size or SHA-256 differs. With the full archive in place, each of the seven packages reports 86 ok entries. In a fresh clone each reports 9 ok (the committed small files) and 77 missing.

Exit	Meaning
`0`	Every entry is `ok`
`1`	An entry is missing or mismatched, or a package has no readable record
`2`	Invocation or configuration error

Rebuild and accept separately

Restore and verify the archive first. The builder stops if a file the spec names directly (a delivery file or an original shard) is missing, but it skips any expected file that is absent from the detailed/ directory, so a partly restored input set can build a package that leaves out evidence. The verify command catches this before you build. For the current maintained Nigeria inputs, make a local specification with a fresh output root. The maintained spec writes to data/source_artifacts/learning_progressions/rebuilt_packages, which is git-ignored and may already hold terminal packages from an earlier rebuild; the builder refuses those even in dry-run mode. This example preserves existing files and requires unused paths:

python3 - <<'PYTHON'
import json
from pathlib import Path

root = Path.cwd()
local = root / "data/source_artifacts/learning_progressions/operator-rebuild"
output = local / "packages"
spec_path = local / "nigeria-build.json"
if spec_path.exists() or output.exists():
    raise FileExistsError("Choose new local specification/output paths.")
spec = json.loads((root / "data/input_artifacts/nigeria_math/package_build.json").read_text())
spec["outputRoot"] = str(output)
local.mkdir(parents=True, exist_ok=True)
spec_path.write_text(json.dumps(spec, indent=2) + "\n")
PYTHON
project_root="$PWD"
uv --directory backend run --locked kgfegmcp-build-manifest \
  --spec "$project_root/data/source_artifacts/learning_progressions/operator-rebuild/nigeria-build.json" \
  --dry-run

Each maintained spec selects profile version 2.0, the delivery files, the detailed/ directory and the 64 original shards. Input paths in the spec resolve from the project root and must stay inside the repository, because the evidence archive record stores each one as a repository-relative localPath. --spec itself is an absolute path because uv changes cwd to backend. Omit --dry-run only to create the proposed pending package in this fresh root. For other curricula, copy the corresponding maintained specification and review its output root before each build. Never patch a sealed accepted package; changed artifacts/profile bytes require new immutable identities.

The builder writes only the runtime package: it copies the delivery and small detailed/ files, derives the slim/ files and writes the evidence archive record and a pending 1.2 manifest. Rebuilding the same spec from the same inputs gives the same snapshot ID and the same files; only the manifest's createdAt (and its validation state, until accepted) differs.

Validate the separate package root before switching active data:

project_root="$PWD"
KGFEGMCP_GRAPH_PACKAGES_ROOT="$project_root/data/source_artifacts/learning_progressions/operator-rebuild/packages" \
  uv --directory backend run --locked kgfegmcp-validate-packages one \
  --framework-id <framework-id> --snapshot-id <built-snapshot-id> --read-only

This runs both validation tiers, so it reads the restored archive. Inspect findings, then omit --read-only for an intentional pending-to-terminal acceptance; terminal revalidation never persists changes. If the build reproduces a snapshot ID that is already in data/graph_packages/, it has confirmed that package: keep the committed package unchanged and discard the rebuild, because a sealed package is never replaced. Otherwise, move the accepted package into data/graph_packages/<framework-id>/ and keep one unambiguous current snapshot per framework. Commit the package directory and any changed small input files. The ignore rules keep archive-only files out of Git; add new or changed ones to the Drive archive yourself. Local preparation/activation and distribution verification do not publish or deploy; deployment remains user-owned.

Prepare new source exports locally

Each source document's export has this 23-file copy set:

as_entity_provenance.json
as_kg_bundle.json
as_lc_kg_bundle.json
as_lc_lp_kg_bundle.json
as_lc_lp_nodes.jsonl
as_lc_lp_relationships.jsonl
as_lc_nodes.jsonl
as_lc_relationships.jsonl
as_lc_validation_report.json
as_relationships_has_child.jsonl
as_standards_framework.json
as_standards_framework_items.jsonl
as_unresolved_items.json
lc_dedup_groups.json
lc_entity_provenance.json
lc_generation_summary.json
lp_final_claims.json
lp_generation_summary.json
lp_relationship_provenance.json
lp_relationships_builds_towards.jsonl
lp_relationships_relates_to.jsonl
lp_unresolved_items.json
lp_validation_report.json

For a future export, copy only the listed files from the mapped <doc-key>/kgs/ directory of the external project into a repository-local source copy at data/source_artifacts/learning_progressions/<doc-key>/kgs/; do not edit the external source. That folder is git-ignored local preparation evidence and is not in a fresh clone. Its copy_receipt.json records source paths, framework/CASE mapping, size, SHA-256 before and after copying, destination hash and package/profile identities. Verify CASE/framework mapping and AS/LC equality, and reconcile per-type counts against split/combined/provenance evidence before relying on it. A source change needs a new verified receipt, not reuse of an old hash. The strict CopyReceipt, CopiedFile and CopiedFramework models in packages/normalization_models.py define the preparation input contract: receiptVersion: 1.0, status: verified, framework mappings, selected-file count and verified per-file fields. The local receipt can retain private source paths; package receipts expose only relative paths/logical names and hashes.

From repository root, normalize existing verified local copies into the separate default preparation root:

uv --directory backend run --locked kgfegmcp-prepare-learning-progressions

Optional --framework-id selects one mapped framework; --receipt selects a verified local receipt and --output-root a separate safe preparation directory. Run --help for those options. The default output is data/source_artifacts/learning_progressions/prepared/<framework-id>/. Identical output is reusable; conflicting output is preserved and rejected. Unsafe/symlinked paths and overlap with source/config/runtime roots are rejected. Preparation verifies local bytes and never reads external receipt source paths. It returns hashes/counts and created or existing_identical; it does not build or accept a package.

Provenance and independent validation

Current contracts are manifest 1.2, delivery 1.2, source 1.0, package revision 1, profile schema 1.1/version 2.0, prompt-config schema 1.1/version 2.0.0, public prompt library 1.4.0 (server/MCPB 1.0.0).

CBSE's original provenance map is about 40.9 MB and CAPS's about 119.2 MB, both above the default 32 MiB resource source-read limit, and all seven maps together are about 221 MB. Packages therefore carry 64 slim shards instead: each holds the slim records for the edges whose SHA256(UTF8(relationship_id))[0] modulo 64 matches its number, under logical names learningProgressionProvenanceShard00 through 63. The slim index binds their exact hashes and placement, and their union must equal the package's LP delivery edges. A client reads one edge's evidence by opening one small shard, within rights and byte limits. The original map, its index and its 64 original shards are archived and bound by hash.

Packages declare learningProgressionNormalization, learningProgressionProvenanceIndex, learningProgressionSummary, learningProgressionUnresolved, learningProgressionValidation and the 64 shards. learningProgressionBuildsTowards, learningProgressionRelatesTo, learningProgressionProvenance and learningProgressionFinalClaims are archive-only and listed in the evidence archive record. The normalization receipt binds the hash of every other full-evidence file; it also binds unchanged nodes, AS/LC prefix/separator, sorted normalized suffix, input/output hashes and copy-receipt identity.

Both validation tiers check IDs/types, standard endpoints, exact CASE correspondence, canonical relates ordering, duplicates/pair exclusivity, counts and builds cycles separately from hasChild. Acceptance's evidence tier also reconciles final claims/provenance/report/unresolved evidence while excluding rejected, no_relation and needs_review claims from exported edges, and checks that every slim file matches what the full evidence yields. AS/LC reports are compared with their subgraph, LP reports with LP counts. Acceptance is structural/process integrity, not semantic/pedagogical certification. Preserve source truncation, warning pairs and eligibility notices, including CBSE's one needs-review claim. See Validation and lifecycle.

Use local tests and acceptance commands for package/service/static/STDIO/loopback HTTP/MCPB stage checks, and build docs strictly. Tests must mock live-model/paid boundaries, including failures; a costs-money marker cannot authorize calls. Build a fresh retained stage for changed runtime inputs, verify that stage using its own roots, and leave hosted deployment/publication to the user.

4. Dry-run the package build

The safest operator interface is a JSON build specification:

uv --directory backend run --locked --no-dev \
  kgfegmcp-build-manifest \
  --spec ./package-build-spec.json \
  --dry-run

A minimal specification has this shape:

{
  "jurisdictionType": "<operator-approved type>",
  "nodes": "/absolute/path/to/as_lc_lp_nodes_XXX.jsonl",
  "outputRoot": "/absolute/path/to/data/graph_packages",
  "profileId": "<profile-id>",
  "profileVersion": "<profile-version>",
  "relationships": "/absolute/path/to/as_lc_lp_relationships_XXX.jsonl",
  "versionToken": "<immutable source version token>"
}

Optional fields cover detailed artifacts, explicitly named additional artifacts, snapshot relations, a source document, and authoritative source publication date.

The builder creates/proposes a pending runtime-only package: delivery files, small detailed/ files, derived slim/ provenance and the evidence archive record. It does not replace complete package validation.

5. Create the pending package

After reviewing the dry run:

uv --directory backend run --locked --no-dev \
  kgfegmcp-build-manifest --spec ./package-build-spec.json

The package is written beneath:

data/graph_packages/<framework-id>/<snapshot-id>/

The builder is deterministic: an already-existing byte-identical pending target can be reported as existing_identical; it will not overwrite a different or terminal package as though it were the same build.

6. Validate read-only first

Resolve the generated framework and snapshot IDs from the builder result, then run:

uv --directory backend run --locked --no-dev \
  kgfegmcp-validate-packages one \
  --framework-id <framework-id> \
  --snapshot-id <snapshot-id> \
  --read-only

Review every finding. Validation covers package closure and checksums, profile binding, delivery decoding, graph semantics, topology, counts, capabilities, rights metadata, and other acceptance invariants. It also runs the evidence tier against the full inputs, so they must still be present locally; see Validation and lifecycle.

Do not weaken validation merely to admit one framework. If the source genuinely needs a new generic capability, model that capability explicitly and test it across appropriate packages.

7. Persist the terminal state

When read-only validation is clean, persist the pending-to-passed transition:

uv --directory backend run --locked --no-dev \
  kgfegmcp-validate-packages one \
  --framework-id <framework-id> \
  --snapshot-id <snapshot-id>

Only a terminal passed package that still passes catalog read-only revalidation is eligible to be served.

A failed or quarantined package is terminal and excluded. A valid but still pending package is also excluded.

Commit the package directory, the input set's package_build.json and its small committed files. Add the archive-only files to the IDinsight Google Drive archive so others can restore them; Git does not hold them.

8. Verify discovery and retrieval

Run the end-to-end smoke test:

uv --directory backend run --locked --no-dev kgfegmcp-stdio-smoke

Then verify the new framework through an MCP client or focused service/tool tests:

list_frameworks returns the framework/snapshot;
get_framework resolves it exactly;
get_capabilities reports package-appropriate search/resource behavior;
representative text search returns source-grounded hits;
code search behaves exactly as the profile permits;
hierarchy context preserves all valid parent edges; and
declared resource access follows rights policy.

For a multi-parent package, include a representative node whose complete root paths prove that all valid parent edges survive loading and traversal.

9. Update documentation

Update the generated or maintained framework catalog and any framework-specific caveats:

Available frameworks
examples that intentionally name current framework/snapshot IDs;
search-code support matrices;
rights/disclosure notes; and
totals derived from accepted manifests.

Do not hand-copy source facts into many pages when they can instead be generated from the manifest/profile contracts.

10. Validate packaged distribution when applicable

If the new package is part of the MCPB release, build and retain a stage:

rm -rf ./dist/kgfegmcp-stage

uv --directory backend run --locked --no-dev kgfegmcp-build-mcpb \
  --stage-output ./dist/kgfegmcp-stage

uv --directory backend run --locked --no-dev kgfegmcp-stdio-smoke \
  --bundle-root ./dist/kgfegmcp-stage

The stage smoke proves that configuration, profile, data, and backend files were all included independently of the source checkout.

Updating only prompt guidance

If source/package semantics do not change but prompt guidance does, version the prompt configuration according to the repository's profile/prompt policy and verify prompt rendering without altering accepted source evidence.

Prompt changes never turn model-generated text into source assertions.

When Python changes are justified

A new framework should trigger generic Python changes only when it exposes a genuinely new reusable requirement—for example a new framework-independent topology capability, resource class, or search policy that the current typed contracts cannot represent.

In that case:

extend the generic model/service first;
keep framework-specific values in profiles/manifests;
add positive and negative tests across representative packages;
update capability reporting honestly; and
update MCP/reference contracts only if the public surface changes.

Next: Testing and acceptance