cyb50-quant/research/chinext50_regime_project/MEMORY.md

Long-Term Memory

• 2026-04-08: Main-session workspace context initialized for chinext50_regime_project.
• 2026-04-08: User asked for a thorough reading and logic analysis of the project.
• 2026-04-08: Workspace did not contain SOUL.md, USER.md, MEMORY.md, or daily memory files before this session.
• 2026-04-08: User specifically asked to reference chinext50_regime_build_handoff_2026-04-08.md when analyzing the project.
• 2026-04-09: Created OpenSpec change add-real-data-regime-input and completed proposal/design/specs/tasks artifacts focused on real-data contract, point-in-time breadth merge, and pipeline quality gate.
• 2026-04-09: Implemented add-real-data-regime-input tasks end-to-end (9/9), including data contract validation, sidecar merge, strict/non-strict quality gate integration for both pipelines, tests, and README updates.
• 2026-04-09: Installed missing pytest and PyYAML; verification passed with 13 tests and both pipeline smoke runs producing expected quality artifacts.
• 2026-04-09: Archived add-real-data-regime-input and synced three new main specs (pipeline-data-quality-gate, point-in-time-breadth-merge, regime-market-data-contract).
• 2026-04-09: Completed and archived strengthen-frozen-hypothesis-walkforward, adding train-select/test-freeze walk-forward flow, audit board outputs, and corresponding tests.
• 2026-04-09: Frozen validation pipeline now supports candidate-set configuration (config or --candidates-json) and min-train/min-test row controls.
• 2026-04-09: Completed and archived add-event-anchored-diagnostics, adding event taxonomy diagnostics (crash_onset, false_rebound, true_repair, crowded_unwind) and event_log.csv output in demo pipeline.
• 2026-04-09: Full regression after this phase passed (21 tests), and demo/frozen smoke pipelines both succeeded.
• 2026-04-09: Completed and archived enhance-execution-layer-constraints, adding extreme-day/gap execution costs and tracking gap/error diagnostics in ledger + metrics.
• 2026-04-09: Completed and archived extend-breadth-concentration-features, adding top1/top10/sector concentration features and integrating them into crowding score.
• 2026-04-09: Latest full regression passed (25 tests), and demo/frozen smoke pipelines remained successful.
• 2026-04-09: User reaffirmed autonomous OpenSpec-driven execution (no extra confirmations) and requested to continue by change flow.
• 2026-04-09: Completed and archived build-real-data-pit-dataset, adding PIT builder (data/pit_builder.py), PIT CLI (pipelines/build_pit_dataset.py), summary metadata fields (sources, pit_columns), dedicated PIT tests, and README PIT usage docs.
• 2026-04-09: Post-change verification passed with 29 tests and PIT smoke outputs (chinext50_pit.csv, pit_quality_summary.json).
• 2026-04-09: Synced two new main specs from this change: pit-dataset-construction and pit-build-artifacts.
• 2026-04-09: Completed and archived require-real-pit-input, enforcing full PIT input contract in runtime pipelines and removing synthetic fallback from run_demo/frozen_hypothesis_validation.
• 2026-04-09: Completed and archived tiered-data-quality-gate, adding breach severity (error/warning) and blocking_columns-driven strict blocking semantics across all data-quality entrypoints.
• 2026-04-09: Completed and archived add-real-walkforward-report, adding pipelines/real_walkforward_report.py with summary JSON + markdown report outputs for strategy-vs-baseline evaluation.
• 2026-04-09: Completed and archived add-execution-constraint-calibration, adding pipelines/calibrate_execution_constraints.py for execution-constraint grid calibration outputs.
• 2026-04-09: Latest verification after these changes passed (38 tests, 18 specs strict validation, no active OpenSpec changes).
• 2026-04-09: User approved immediate implementation of data-fetch/ingestion plan.
• 2026-04-09: Completed and archived add-real-data-ingestion-pipeline, adding provider-based ingestion (csv + optional akshare), layered artifacts (raw/staging/pit), and incremental date-safe merge behavior.
• 2026-04-09: Added pipelines/ingest_real_data.py as unified entrypoint from source ingestion to PIT publication.
• 2026-04-09: Added ingestion regression tests and updated README ingestion command docs; latest verification passed (41 tests, 19 specs strict validation, no active OpenSpec changes).
• 2026-04-09: User requested Akshare + Mairui hybrid ingestion; completed and archived add-mairui-fallback-ingestion.
• 2026-04-09: Ingestion now supports mairui provider and Akshare-to-Mairui fallback in akshare mode, including configurable Mairui breadth endpoint mapping.
• 2026-04-09: Verified with full regression (42 tests), specs validation (20 strict passes), and live Mairui smoke output generation.
• 2026-04-09: Ran full strict end-to-end system chain (ingestion -> demo -> frozen -> real report -> execution calibration) on unified PIT dataset and produced complete artifacts under outputs/system_e2e_20260409_strict/.
• 2026-04-09: Validated GPT Pro review (chinext50_regime_review_2026-04-09.md) against current code/output; the immediate bottleneck is system integrity rather than threshold tuning: NaN propagation breaks breadth_score and crowding_score, hazards are neutralized to 0.5, the state space collapses, exposure quantization hides candidate differences, and the current walk-forward window set is statistically too weak.
• 2026-04-09: User supplied GPT Pro blocker checklist chinext50_blocker_checklist_for_codex.md; executed a new OpenSpec change fix-regime-integrity-blockers strictly in B1 -> B6 order.
• 2026-04-09: Integrity blocker outcomes:
  • B1 repaired score/hazard integrity (breadth_score non-null 0.9611, crowding_score non-null 0.9216, hazards no longer constant 0.5)
  • B2 added explicit warmup and invalid-after-warmup fail-fast
  • B3 added post-feature low-information gate; current real PIT now correctly fails strict mode on constant concentration_spread_5
  • B4 replaced coarse 0.25 exposure quantization with configurable 0.10 ladder
  • B5 replaced hard-coded walk-forward windows with PIT-driven expanding windows (5/5 valid windows on current real PIT)
  • B6 redesigned execution calibration score with tracking tolerance bands
• 2026-04-09: Post-blocker regression passed (59 tests, 20 strict OpenSpec spec validations), and the change fix-regime-integrity-blockers reached all_done.
• 2026-04-09: Completed and archived derive-real-breadth-sidecar, adding constituent-derived breadth builder, breadth-source integrity gate, ingestion integration for --derive-breadth, and new artifacts (breadth_derivation_summary.json, breadth_integrity_summary.json).
• 2026-04-09: Added Akshare Tencent-line fallback for stock history in breadth derivation, plus Mairui retry/backoff handling for 429/5xx.
• 2026-04-09: Post-change verification passed (63 tests, 24 strict OpenSpec spec validations).
• 2026-04-09: Derived-breadth smoke: strict mode now blocks on low-information breadth diagnostics (as intended), and non-strict run completed at outputs/ingestion_derived_smoke_20260409_v9/ with spread variability restored (spread_unique_count=22 vs placeholder baseline 1).
• 2026-04-09: During cache rollout verification on outputs/ingestion_derived_smoke_20260409_v12, found cache miss bug: repeated runs still reported hit_count=0, miss_count=12.
• 2026-04-09: Root cause was non-trading-day boundary mismatch in _history_covers_range (requested 2020-01-01, cached series starts 2020-01-02), causing false cache invalidation.
• 2026-04-09: Fixed in data/breadth_builder.py by adding boundary tolerance (CACHE_BOUNDARY_TOLERANCE_DAYS=10) and updated coverage checks to accept small boundary gaps while rejecting true out-of-range panels.
• 2026-04-09: Added regression test test_derive_breadth_sidecar_cache_handles_non_trading_boundaries in tests/test_breadth_builder.py.
• 2026-04-09: Verification after fix passed (66 tests, 26 strict spec validations), and cache metrics improved on rerun: hit_count 0 -> 12, miss_count 12 -> 0, provider mix switched to cache:12.
• 2026-04-09: Re-ran end-to-end downstream chain on cache-fixed PIT at outputs/system_e2e_derived_v12_cachefix/; pipeline execution succeeded for demo/frozen/report/calibration with updated artifacts.
• 2026-04-09: Started and completed implementation for new OpenSpec change harden-derived-breadth-production (artifacts + tasks all done, awaiting archive decision).
• 2026-04-09: Added resilient metadata sourcing and diagnostics in data/breadth_builder.py:
  • Akshare metadata retries
  • optional Mairui metadata fallback
  • metadata provider/error diagnostics in derivation summary
  • metadata cache file _meta_cache.json under constituent-history cache
• 2026-04-09: Added backward-compatible comparison aliases in real_walkforward_summary.json (annual_return_delta_vs_baseline, max_drawdown_delta_vs_baseline) and updated report tests.
• 2026-04-09: Full regression after this round passed (68 tests, 26 strict spec validations).
• 2026-04-09: Full50 strict derived-breadth ingestion now publishes successfully at outputs/ingestion_derived_full50_20260409_v2/ with 1517 PIT rows and passing breadth integrity.
• 2026-04-09: Performance improvement via metadata cache confirmed: first full50 run ~1199.9s vs second rerun ~26.9s; second run shows meta_cache.hit_count=50.
• 2026-04-09: Remaining unresolved blocker documented for GPT Pro in deliverables/gpt_pro_blockers_harden_derived_breadth_2026-04-09.md: upstream metadata instability persists (industry_unknown_ratio=1.0, Akshare RemoteDisconnected, Mairui HTTP 429), causing fallback to weight_hhi_proxy sector concentration mode.
• 2026-04-09: Archived OpenSpec change harden-derived-breadth-production as openspec/changes/archive/2026-04-09-harden-derived-breadth-production/; synced modified specs (constituent-derived-breadth, real-walkforward-report) and added new spec (implementation-issue-log).
• 2026-04-09: Post-archive spec validation passed (27 strict spec checks), and there are no active OpenSpec changes.
• 2026-04-09: Prepared GPT Pro review package deliverables/gpt_pro_bundle_harden_derived_2026-04-09.zip with blocker log, focused code/spec diffs, archived OpenSpec change artifacts, and key full50 output summaries.
• 2026-04-09: Completed OpenSpec change uild-snapshot-first-index-metadata-and-weight-layer per latest hardening handoff. Added snapshot-weight forward-fill behavior in breadth builder and expanded coverage for semantic gate + metadata cache freshness + membership semantics. Final verification: 75 tests passed and OpenSpec strict validation passed (27 specs), change state ll_done.
• 2026-04-09: Archived uild-snapshot-first-index-metadata-and-weight-layer. Main specs now include readth-semantic-gate and index-metadata-snapshot-layer; strict spec validation is 29 passed with no active changes.
• 2026-04-09: Archived ecalibrate-walkforward-economics-after-semantic-hardening and synced new/updated specs (walkforward-candidate-ranking, rozen-train-window-selection, eal-walkforward-report). Current recalibrated report (outputs/real_walkforward_recalibrated_20260409_v2/) now shows diversified frozen selection and higher upside capture with improved utility gap vs baseline, though utility remains negative and additional economics iteration is still needed.
• 2026-04-10: Applied GPT Pro recalibrate guidance sequence for report semantics and validation tests. real_walkforward_report.py now separates default vs stitched_frozen_oos metrics/comparisons, emits stitched_frozen_oos_ledger.csv, and adds primary-vs-partial window success diagnostics (primary_window_success_ratio, etc.). Added/updated tests in test_frozen_walkforward.py and test_real_walkforward_report_pipeline.py; verification passed (81 tests). New semantic report run output: outputs/real_walkforward_recalibrated_20260410_semantic_v1/.
• 2026-04-10: Began OpenSpec change execute-fullcode-guidance-20260410 to execute strict sequence B1->B2->B3->H1->H2->H3 with per-block guardrails and rollback. Completed B1 (stitched same-period baseline semantics) and B2 (utility core/net split + frozen stability de-dup turnover). B3 attempt (state precedence reorder) was rolled back because guardrail failed (risk_off state share stayed high at ~0.3686 > 0.32). Execution halted before H1/H2/H3 by rule.
• 2026-04-10: Continued with post-B3 sequence guidance (B3 -> B4 -> H1a -> ...). B3 and B4 completed; H1a executed without triggering stop conditions but failed acceptance floor on stitched annual return, so progression to H1b/H2/H3 is currently blocked per guidance rule.
• 2026-04-10: Under chinext50_post_b3_detailed_guidance_for_codex_2026-04-10.md, completed R0/R1 and adopted one-shot H1a microprobe thresholds (risk_off_down_hazard=0.67, risk_off_stress=0.89, risk_off_trend_floor=-0.14, crash_override_down_hazard=0.77). Attempted H1b.1 repair-cleanup pass but rolled it back due stop condition breach (drawdown_ratio_vs_baseline > 0.64).
• 2026-04-10: Executed post-B3 feedback ladder strictly (D0 -> prep -> H1b.1 L1/L2/L3 -> H1b.2). D0 and prep landed with zero semantic drift vs R1; all H1b.1 ladder variants hit stop conditions and were rolled back per block rule; H1b.2-direct-from-R1 did not trigger stop but failed acceptance (offense carry and trend+euphoric target not met). Consolidated artifacts at deliverables/chinext50_post_b3_feedback_execution_summary_2026-04-10.{md,json}.
• 2026-04-10: Added and archived build-regime-lite-mvp, introducing regime-lite-runtime spec and lightweight operational pipeline (pipelines/regime_lite_run.py) with focused tests and real PIT smoke outputs. This establishes a practical small-team path while keeping large research modules intact.
• 2026-04-11: Implemented and archived experiment-timing-and-entry-exit, adding regime_lite_experiment_board capability with HTML/CSV/JSON outputs and focused tests. Initial PIT results show large improvement for combo_fast_hold3, especially trend re-entry speed and return/drawdown profile; requires out-of-sample confirmation before promoting to runtime default.
• 2026-04-24: User set a new operating preference for this workspace: I should act as the main supervising thread, keep pushing until done or truly blocked, oversee subtasks, make local decisions when issues appear, and enforce convergence so work does not sprawl indefinitely. Subtasks should prefer OpenSpec-style progression.
• 2026-04-24: User clarified the supervisor objective: the main thread should drive the entire project toward a final converged end state, not merely complete isolated changes. Intermediate routing, prioritization, and local tradeoff decisions are delegated to me unless a true direction-level decision is needed from the user.
• 2026-04-24: Created and completed OpenSpec change validate-regime-lite-fast-entry-promotion, adding a deterministic lite promotion gate, explicit runtime execution profiles, promotion decision artifacts, and updated lite experiment/report contracts.
• 2026-04-24: Targeted verification passed (7 focused lite tests). Real PIT promotion run at outputs/regime_lite_promotion_20260424/ returned promote for combo_fast_hold3 -> runtime profile promoted_fast_entry_hold3.
• 2026-04-24: Validation window (2024-09-10 to 2026-04-09) delta vs baseline for the promoted lite profile: annual_return +0.053994, max_drawdown -0.036674, annual_turnover -0.866667, trend re-entry speed -6.291667 days. Supervisor baseline now prefers promoted_fast_entry_hold3 while retaining baseline as rollback/reference.
• 2026-04-24: Synced the new lite-promotion requirements into main OpenSpec specs (regime-lite-runtime, regime-lite-experiment-board, regime-lite-promotion-gate) and archived the completed change at openspec/changes/archive/2026-04-24-validate-regime-lite-fast-entry-promotion/.
• 2026-04-24: Added pytest.ini to constrain default test collection to the main tests/ directory so bundled deliverables/backups no longer break full-project verification. Full regression now runs cleanly with py -m pytest -q and currently passes at 101 tests.
• 2026-04-24: Closed and archived the stale full-framework change execute-fullcode-guidance-20260410 at openspec/changes/archive/2026-04-24-execute-fullcode-guidance-20260410/ with an explicit convergence note that H2/H3 were intentionally not pursued further.
• 2026-04-24: Updated README.md to reflect the converged operational baseline: preferred lite runtime profile promoted_fast_entry_hold3, promotion artifact location, and the default project health-check command.