Testing Plan¶

Tests should be written against SPEC.md.

See test-verification-report.md for the latest verification pass, confidence rating, and known gaps.

How to add a test¶

Choose a fixture — add a YAML or JSON file under tests/fixtures/ (or reuse an existing one).
Register expectations — add an entry to tests/fixture_expectations.json for parse/validation outcomes, or to phase-specific manifests (plan_expectations.json, optimize_expectations.json, compile_expectations.json, capability_expectations.json).
Add integration coverage — extend the appropriate phase test file (tests/phase_0_*.rs) or parametrize in python/tests/test_dtcs.py.
Golden files — for plan, optimize, or compile output, add or regenerate JSON under tests/fixtures/plans/, plans_optimized/, or execution_plans/. Regenerate optimize goldens with cargo run --example write_optimize_goldens.

Run locally:

cargo test --locked
pytest python/tests -v

Concern	Example fixtures
Semantic actions	`stdlib_action_normalize_whitespace_valid.yaml`, `invalid_semantic_action.yaml`
Functions	`stdlib_function_concat_valid.yaml`, `stdlib_function_coalesce_valid.yaml`
Rules	`stdlib_rule_range_valid.yaml`, `stdlib_rule_regex_match_valid.yaml`
Signature errors	`stdlib_function_concat_signature_invalid.yaml`, `stdlib_function_substr_param_order_invalid.yaml`

Covered by tests/fixture_expectations.json and validation integration tests.

Concern	Example fixtures
Registry loading	`tests/fixtures/registry/vendor_catalog.yaml`, `vendor_mandatory_extension.yaml`
Namespace safety	`invalid_http_rule.yaml`, `invalid_http_action.yaml`, `invalid_http_type.yaml`

Concern	Example fixtures
Expression analysis	`expression_with_type.yaml`, `expression_precedence_*.yaml`, `analysis_constant_expr.yaml`
Semantic analysis	`analysis_dtcs_call_valid.yaml`, `analysis_logical_ops.yaml`
Invalid semantics	`analysis_duplicate_action_target.yaml`, expression/semantics error fixtures

Concern	Example fixtures
Compatibility (5 levels)	`tests/fixtures/compatibility/` — `identical_`, `backward_`, `forward_`, `conditional_`, `incompatible_*`
Evolution	`tests/fixtures/compatibility/evolution/rev1.yaml`, `rev2.yaml`, `deprecated.yaml`
Versioning	`invalid_version.yaml`, `version_conflict.yaml`
Lineage analysis	`lineage_multi.yaml`

Concern	Example fixtures
Plan lowering	`valid_customer.yaml`, `valid_minimal.json`, `plan_explicit_ordering.yaml`
Dependency graph	`lineage_multi.yaml`
Ambiguous action order	`analysis_duplicate_action_target.yaml` (plan invalid)
Golden plans	`tests/fixtures/plans/*.plan.json` via `tests/plan_expectations.json`

Concern	Example fixtures
Constant folding	`optimize_constant_fold.yaml`
Algebraic simplification	`optimize_algebraic.yaml`
Action fusion	`optimize_action_fusion.yaml`
Function inlining	`optimize_function_inline.yaml`
Rule deduplication	`optimize_rule_dedup.yaml`, `optimize_rule_dedup_params.yaml`
Dead-expression elimination	`optimize_dead_expr.yaml`, `optimize_dead_after_fold.yaml`
Golden optimized plans	`tests/fixtures/plans_optimized/*.plan.json` via `tests/optimize_expectations.json`

Integration tests: tests/phase_0_8.rs. Python tests parametrize over tests/optimize_expectations.json.

Concern	Example fixtures
Capability matching	`valid_customer.yaml`, `plan_field_write_chain.yaml`, vendor action rejection
Compilation	`tests/compile_expectations.json`, `tests/fixtures/execution_plans/*.exec.json`
Runtime	`tests/fixtures/runtime/customer_normalize_input.json`, `plan_field_write_chain_input.json`
End-to-end	`examples/customer_normalize.dtcs.yaml`

Integration tests: tests/phase_0_9.rs. Manifest-driven compile goldens use tests/compile_expectations.json; capability and runtime I/O expectations use tests/capability_expectations.json.

Concern	Example fixtures / artifacts
Conformance manifest	`tests/conformance/manifest.json`, embedded `src/conformance/manifest.json`
Profile coverage	`valid_customer.yaml`, `missing_lineage.yaml`, `optimize_constant_fold.yaml`
Security vectors	`registry/evil_dtcs_injection.yaml`, `invalid_rule_duplicate_params.json`
Integrated Platform E2E	`examples/customer_normalize.dtcs.yaml`, runtime I/O fixtures

Integration tests: tests/phase_0_10.rs. CI runs cargo test --test phase_0_10 and dtcs conformance run --profile all.

Concern	Example fixtures / artifacts
Flagship pipeline	`examples/customer_pipeline.dtcs.yaml`, `tests/fixtures/runtime/customer_pipeline_input.json`
Portable differential	`tests/fixtures/portable/*.json`
Portable integration	`tests/portable_relational.rs`
Capability accuracy	`validate_capability_accuracy` / `reference_portable_manifest` unit tests

Add tests/fixtures/portable/<id>.json with actions, input, expected (or expectError).
Use the fixture token dialect ($missing / $invalid) — not {"$dtcs":…} — see portable-conformance.md.
Register a portableDifferential case in both src/conformance/manifest.json and tests/conformance/manifest.json.

Run:

cargo test --test portable_relational --locked
cargo test --test phase_0_10 --locked
cargo run --bin dtcs -- conformance run --profile all

Concern	Example fixtures
Metadata	`valid_metadata.yaml`, `invalid_metadata_timestamp.yaml`, `invalid_metadata_impossible_date.yaml`
Types	`map_type_valid.yaml`, `nested_collection_valid.yaml`, `valid_conversion_lossy.yaml`, `invalid_type_trailing_garbage.yaml`
Expressions	`expression_with_type.yaml`, `expression_missing_type.yaml`, `expression_type_mismatch.yaml`
Interfaces	`optional_input.yaml`, `input_precondition.yaml`, `invalid_io_extension.yaml`, `streaming_unbounded.yaml`
Registry	`tests/fixtures/registry/`, `tests/phase_0_4.rs`
Namespace safety	`invalid_http_rule.yaml`, `invalid_http_action.yaml`, `invalid_http_type.yaml`

Add snapshot tests for diagnostics where useful.