## Long-Term Memory

- 2026-04-08: Main-session workspace context initialized for `chinext50_regime_project`.
- 2026-04-08: User asked for a thorough reading and logic analysis of the project.
- 2026-04-08: Workspace did not contain `SOUL.md`, `USER.md`, `MEMORY.md`, or daily memory files before this session.
- 2026-04-08: User specifically asked to reference `chinext50_regime_build_handoff_2026-04-08.md` when analyzing the project.
- 2026-04-09: Created OpenSpec change `add-real-data-regime-input` and completed proposal/design/specs/tasks artifacts focused on real-data contract, point-in-time breadth merge, and pipeline quality gate.
- 2026-04-09: Implemented `add-real-data-regime-input` tasks end-to-end (9/9), including data contract validation, sidecar merge, strict/non-strict quality gate integration for both pipelines, tests, and README updates.
- 2026-04-09: Installed missing `pytest` and `PyYAML`; verification passed with 13 tests and both pipeline smoke runs producing expected quality artifacts.
- 2026-04-09: Archived `add-real-data-regime-input` and synced three new main specs (`pipeline-data-quality-gate`, `point-in-time-breadth-merge`, `regime-market-data-contract`).
- 2026-04-09: Completed and archived `strengthen-frozen-hypothesis-walkforward`, adding train-select/test-freeze walk-forward flow, audit board outputs, and corresponding tests.
- 2026-04-09: Frozen validation pipeline now supports candidate-set configuration (`config` or `--candidates-json`) and min-train/min-test row controls.
- 2026-04-09: Completed and archived `add-event-anchored-diagnostics`, adding event taxonomy diagnostics (`crash_onset`, `false_rebound`, `true_repair`, `crowded_unwind`) and `event_log.csv` output in demo pipeline.
- 2026-04-09: Full regression after this phase passed (`21` tests), and demo/frozen smoke pipelines both succeeded.
- 2026-04-09: Completed and archived `enhance-execution-layer-constraints`, adding extreme-day/gap execution costs and tracking gap/error diagnostics in ledger + metrics.
- 2026-04-09: Completed and archived `extend-breadth-concentration-features`, adding top1/top10/sector concentration features and integrating them into crowding score.
- 2026-04-09: Latest full regression passed (`25` tests), and demo/frozen smoke pipelines remained successful.
- 2026-04-09: User reaffirmed autonomous OpenSpec-driven execution (no extra confirmations) and requested to continue by change flow.
- 2026-04-09: Completed and archived `build-real-data-pit-dataset`, adding PIT builder (`data/pit_builder.py`), PIT CLI (`pipelines/build_pit_dataset.py`), summary metadata fields (`sources`, `pit_columns`), dedicated PIT tests, and README PIT usage docs.
- 2026-04-09: Post-change verification passed with `29` tests and PIT smoke outputs (`chinext50_pit.csv`, `pit_quality_summary.json`).
- 2026-04-09: Synced two new main specs from this change: `pit-dataset-construction` and `pit-build-artifacts`.
- 2026-04-09: Completed and archived `require-real-pit-input`, enforcing full PIT input contract in runtime pipelines and removing synthetic fallback from `run_demo`/`frozen_hypothesis_validation`.
- 2026-04-09: Completed and archived `tiered-data-quality-gate`, adding breach severity (`error`/`warning`) and `blocking_columns`-driven strict blocking semantics across all data-quality entrypoints.
- 2026-04-09: Completed and archived `add-real-walkforward-report`, adding `pipelines/real_walkforward_report.py` with summary JSON + markdown report outputs for strategy-vs-baseline evaluation.
- 2026-04-09: Completed and archived `add-execution-constraint-calibration`, adding `pipelines/calibrate_execution_constraints.py` for execution-constraint grid calibration outputs.
- 2026-04-09: Latest verification after these changes passed (`38` tests, 18 specs strict validation, no active OpenSpec changes).
- 2026-04-09: User approved immediate implementation of data-fetch/ingestion plan.
- 2026-04-09: Completed and archived `add-real-data-ingestion-pipeline`, adding provider-based ingestion (`csv` + optional `akshare`), layered artifacts (`raw`/`staging`/`pit`), and incremental date-safe merge behavior.
- 2026-04-09: Added `pipelines/ingest_real_data.py` as unified entrypoint from source ingestion to PIT publication.
- 2026-04-09: Added ingestion regression tests and updated README ingestion command docs; latest verification passed (`41` tests, 19 specs strict validation, no active OpenSpec changes).
- 2026-04-09: User requested Akshare + Mairui hybrid ingestion; completed and archived `add-mairui-fallback-ingestion`.
- 2026-04-09: Ingestion now supports `mairui` provider and Akshare-to-Mairui fallback in `akshare` mode, including configurable Mairui breadth endpoint mapping.
- 2026-04-09: Verified with full regression (`42` tests), specs validation (`20` strict passes), and live Mairui smoke output generation.
- 2026-04-09: Ran full strict end-to-end system chain (ingestion -> demo -> frozen -> real report -> execution calibration) on unified PIT dataset and produced complete artifacts under `outputs/system_e2e_20260409_strict/`.
- 2026-04-09: Validated GPT Pro review (`chinext50_regime_review_2026-04-09.md`) against current code/output; the immediate bottleneck is system integrity rather than threshold tuning: NaN propagation breaks `breadth_score` and `crowding_score`, hazards are neutralized to `0.5`, the state space collapses, exposure quantization hides candidate differences, and the current walk-forward window set is statistically too weak.
- 2026-04-09: User supplied GPT Pro blocker checklist `chinext50_blocker_checklist_for_codex.md`; executed a new OpenSpec change `fix-regime-integrity-blockers` strictly in `B1 -> B6` order.
- 2026-04-09: Integrity blocker outcomes:
  - B1 repaired score/hazard integrity (`breadth_score` non-null `0.9611`, `crowding_score` non-null `0.9216`, hazards no longer constant `0.5`)
  - B2 added explicit `warmup` and invalid-after-warmup fail-fast
  - B3 added post-feature low-information gate; current real PIT now correctly fails strict mode on constant `concentration_spread_5`
  - B4 replaced coarse `0.25` exposure quantization with configurable `0.10` ladder
  - B5 replaced hard-coded walk-forward windows with PIT-driven expanding windows (`5/5` valid windows on current real PIT)
  - B6 redesigned execution calibration score with tracking tolerance bands
- 2026-04-09: Post-blocker regression passed (`59` tests, `20` strict OpenSpec spec validations), and the change `fix-regime-integrity-blockers` reached `all_done`.
- 2026-04-09: Completed and archived `derive-real-breadth-sidecar`, adding constituent-derived breadth builder, breadth-source integrity gate, ingestion integration for `--derive-breadth`, and new artifacts (`breadth_derivation_summary.json`, `breadth_integrity_summary.json`).
- 2026-04-09: Added Akshare Tencent-line fallback for stock history in breadth derivation, plus Mairui retry/backoff handling for 429/5xx.
- 2026-04-09: Post-change verification passed (`63` tests, `24` strict OpenSpec spec validations).
- 2026-04-09: Derived-breadth smoke: strict mode now blocks on low-information breadth diagnostics (as intended), and non-strict run completed at `outputs/ingestion_derived_smoke_20260409_v9/` with spread variability restored (`spread_unique_count=22` vs placeholder baseline `1`).
- 2026-04-09: During cache rollout verification on `outputs/ingestion_derived_smoke_20260409_v12`, found cache miss bug: repeated runs still reported `hit_count=0`, `miss_count=12`.
- 2026-04-09: Root cause was non-trading-day boundary mismatch in `_history_covers_range` (requested `2020-01-01`, cached series starts `2020-01-02`), causing false cache invalidation.
- 2026-04-09: Fixed in `data/breadth_builder.py` by adding boundary tolerance (`CACHE_BOUNDARY_TOLERANCE_DAYS=10`) and updated coverage checks to accept small boundary gaps while rejecting true out-of-range panels.
- 2026-04-09: Added regression test `test_derive_breadth_sidecar_cache_handles_non_trading_boundaries` in `tests/test_breadth_builder.py`.
- 2026-04-09: Verification after fix passed (`66` tests, `26` strict spec validations), and cache metrics improved on rerun: `hit_count 0 -> 12`, `miss_count 12 -> 0`, provider mix switched to `cache:12`.
- 2026-04-09: Re-ran end-to-end downstream chain on cache-fixed PIT at `outputs/system_e2e_derived_v12_cachefix/`; pipeline execution succeeded for demo/frozen/report/calibration with updated artifacts.
- 2026-04-09: Started and completed implementation for new OpenSpec change `harden-derived-breadth-production` (artifacts + tasks all done, awaiting archive decision).
- 2026-04-09: Added resilient metadata sourcing and diagnostics in `data/breadth_builder.py`:
  - Akshare metadata retries
  - optional Mairui metadata fallback
  - metadata provider/error diagnostics in derivation summary
  - metadata cache file `_meta_cache.json` under constituent-history cache
- 2026-04-09: Added backward-compatible comparison aliases in `real_walkforward_summary.json` (`annual_return_delta_vs_baseline`, `max_drawdown_delta_vs_baseline`) and updated report tests.
- 2026-04-09: Full regression after this round passed (`68` tests, `26` strict spec validations).
- 2026-04-09: Full50 strict derived-breadth ingestion now publishes successfully at `outputs/ingestion_derived_full50_20260409_v2/` with `1517` PIT rows and passing breadth integrity.
- 2026-04-09: Performance improvement via metadata cache confirmed: first full50 run ~`1199.9s` vs second rerun ~`26.9s`; second run shows `meta_cache.hit_count=50`.
- 2026-04-09: Remaining unresolved blocker documented for GPT Pro in `deliverables/gpt_pro_blockers_harden_derived_breadth_2026-04-09.md`: upstream metadata instability persists (`industry_unknown_ratio=1.0`, Akshare `RemoteDisconnected`, Mairui `HTTP 429`), causing fallback to `weight_hhi_proxy` sector concentration mode.
- 2026-04-09: Archived OpenSpec change `harden-derived-breadth-production` as `openspec/changes/archive/2026-04-09-harden-derived-breadth-production/`; synced modified specs (`constituent-derived-breadth`, `real-walkforward-report`) and added new spec (`implementation-issue-log`).
- 2026-04-09: Post-archive spec validation passed (`27` strict spec checks), and there are no active OpenSpec changes.
- 2026-04-09: Prepared GPT Pro review package `deliverables/gpt_pro_bundle_harden_derived_2026-04-09.zip` with blocker log, focused code/spec diffs, archived OpenSpec change artifacts, and key full50 output summaries.
- 2026-04-09: Completed OpenSpec change uild-snapshot-first-index-metadata-and-weight-layer per latest hardening handoff. Added snapshot-weight forward-fill behavior in breadth builder and expanded coverage for semantic gate + metadata cache freshness + membership semantics. Final verification: 75 tests passed and OpenSpec strict validation passed (27 specs), change state ll_done.
- 2026-04-09: Archived uild-snapshot-first-index-metadata-and-weight-layer. Main specs now include readth-semantic-gate and index-metadata-snapshot-layer; strict spec validation is 29 passed with no active changes.
- 2026-04-09: Archived 
ecalibrate-walkforward-economics-after-semantic-hardening and synced new/updated specs (walkforward-candidate-ranking, rozen-train-window-selection, 
eal-walkforward-report). Current recalibrated report (outputs/real_walkforward_recalibrated_20260409_v2/) now shows diversified frozen selection and higher upside capture with improved utility gap vs baseline, though utility remains negative and additional economics iteration is still needed.
- 2026-04-10: Applied GPT Pro recalibrate guidance sequence for report semantics and validation tests. `real_walkforward_report.py` now separates `default` vs `stitched_frozen_oos` metrics/comparisons, emits `stitched_frozen_oos_ledger.csv`, and adds primary-vs-partial window success diagnostics (`primary_window_success_ratio`, etc.). Added/updated tests in `test_frozen_walkforward.py` and `test_real_walkforward_report_pipeline.py`; verification passed (`81` tests). New semantic report run output: `outputs/real_walkforward_recalibrated_20260410_semantic_v1/`.
- 2026-04-10: Began OpenSpec change `execute-fullcode-guidance-20260410` to execute strict sequence `B1->B2->B3->H1->H2->H3` with per-block guardrails and rollback. Completed B1 (stitched same-period baseline semantics) and B2 (utility core/net split + frozen stability de-dup turnover). B3 attempt (state precedence reorder) was rolled back because guardrail failed (`risk_off` state share stayed high at ~0.3686 > 0.32). Execution halted before H1/H2/H3 by rule.
- 2026-04-10: Continued with post-B3 sequence guidance (`B3 -> B4 -> H1a -> ...`). B3 and B4 completed; H1a executed without triggering stop conditions but failed acceptance floor on stitched annual return, so progression to H1b/H2/H3 is currently blocked per guidance rule.
- 2026-04-10: Under `chinext50_post_b3_detailed_guidance_for_codex_2026-04-10.md`, completed R0/R1 and adopted one-shot H1a microprobe thresholds (`risk_off_down_hazard=0.67`, `risk_off_stress=0.89`, `risk_off_trend_floor=-0.14`, `crash_override_down_hazard=0.77`). Attempted H1b.1 repair-cleanup pass but rolled it back due stop condition breach (`drawdown_ratio_vs_baseline > 0.64`).
- 2026-04-10: Executed post-B3 feedback ladder strictly (D0 -> prep -> H1b.1 L1/L2/L3 -> H1b.2). D0 and prep landed with zero semantic drift vs R1; all H1b.1 ladder variants hit stop conditions and were rolled back per block rule; H1b.2-direct-from-R1 did not trigger stop but failed acceptance (offense carry and trend+euphoric target not met). Consolidated artifacts at `deliverables/chinext50_post_b3_feedback_execution_summary_2026-04-10.{md,json}`.
- 2026-04-10: Added and archived `build-regime-lite-mvp`, introducing `regime-lite-runtime` spec and lightweight operational pipeline (`pipelines/regime_lite_run.py`) with focused tests and real PIT smoke outputs. This establishes a practical small-team path while keeping large research modules intact.
- 2026-04-11: Implemented and archived `experiment-timing-and-entry-exit`, adding `regime_lite_experiment_board` capability with HTML/CSV/JSON outputs and focused tests. Initial PIT results show large improvement for `combo_fast_hold3`, especially trend re-entry speed and return/drawdown profile; requires out-of-sample confirmation before promoting to runtime default.
- 2026-04-24: User set a new operating preference for this workspace: I should act as the main supervising thread, keep pushing until done or truly blocked, oversee subtasks, make local decisions when issues appear, and enforce convergence so work does not sprawl indefinitely. Subtasks should prefer OpenSpec-style progression.
- 2026-04-24: User clarified the supervisor objective: the main thread should drive the entire project toward a final converged end state, not merely complete isolated changes. Intermediate routing, prioritization, and local tradeoff decisions are delegated to me unless a true direction-level decision is needed from the user.
- 2026-04-24: Created and completed OpenSpec change `validate-regime-lite-fast-entry-promotion`, adding a deterministic lite promotion gate, explicit runtime execution profiles, promotion decision artifacts, and updated lite experiment/report contracts.
- 2026-04-24: Targeted verification passed (`7` focused lite tests). Real PIT promotion run at `outputs/regime_lite_promotion_20260424/` returned `promote` for `combo_fast_hold3` -> runtime profile `promoted_fast_entry_hold3`.
- 2026-04-24: Validation window (`2024-09-10` to `2026-04-09`) delta vs baseline for the promoted lite profile: annual_return `+0.053994`, max_drawdown `-0.036674`, annual_turnover `-0.866667`, trend re-entry speed `-6.291667` days. Supervisor baseline now prefers `promoted_fast_entry_hold3` while retaining `baseline` as rollback/reference.
- 2026-04-24: Synced the new lite-promotion requirements into main OpenSpec specs (`regime-lite-runtime`, `regime-lite-experiment-board`, `regime-lite-promotion-gate`) and archived the completed change at `openspec/changes/archive/2026-04-24-validate-regime-lite-fast-entry-promotion/`.
- 2026-04-24: Added `pytest.ini` to constrain default test collection to the main `tests/` directory so bundled deliverables/backups no longer break full-project verification. Full regression now runs cleanly with `py -m pytest -q` and currently passes at `101` tests.
- 2026-04-24: Closed and archived the stale full-framework change `execute-fullcode-guidance-20260410` at `openspec/changes/archive/2026-04-24-execute-fullcode-guidance-20260410/` with an explicit convergence note that H2/H3 were intentionally not pursued further.
- 2026-04-24: Updated `README.md` to reflect the converged operational baseline: preferred lite runtime profile `promoted_fast_entry_hold3`, promotion artifact location, and the default project health-check command.