chore: consolidate platform services, dedupe helpers, de-monolith CLIs #93
Merged
wasabi opened 1 month ago

Summary

Consolidation, deduplication, and de-monolithization pass across the platform to improve stability, scalability, and maintainability.

Bugs fixed

  • services/predictions/cli.py: removed def get_db(): return get_db() infinite-recursion override that crashed every predictions subcommand.

Deduplication

  • services/agents/base.py (new): BaseSynthesisAgent + location_filter + query_rows_cursor shared scaffolding. All 10 synthesis agents now inherit the identical get_connection/store_summary/run_synthesis/main boilerplate (~60 lines each) instead of re-implementing it.
  • services/common/utils.py (new): serialize_value/serialize_rows extracted from export/report_generator.py and export/exporter.py (removed duplicate isoformat/bytes-hash serialization).
  • services/ingestion/base.py: added batch_ranges() helper; baal_indexer now uses it instead of a hand-rolled while loop.

CLI de-monolithization

  • services/analytics/cli.py: collapsed 23 near-identical 5-line dispatch blocks into a 13-entry table + shared get_connection/print_json.
  • 19 analytics cli_*.py + 13 service CLIs (crisp, data_module, events, attestation, certificates, finance, fortune500, geostatistics, graph_projection, iri, metadata_api, office, revenue_multiplier): replaced local _out/_connection/json.dumps boilerplate with the shared services.common.cli helpers.
  • 23 __main__.py files standardized to the canonical from .cli import main + guard pattern (was 6 variants).

Test consolidation

  • Replaced 48 copy-pasted test_cli_*.py template files with one parametrized tests/test_cli_registry.py covering all 47 CLI modules plus their subcommand --help targets.

Governance

  • services/migration/cli.py: documented that seed version-prefix sharing (9 pairs) is intentional and enforced by unique seed:<filename> IDs; schema versions remain strictly unique.

Verification

  • 202 focused tests pass (CLI registry, migration, all agent suites)
  • ruff clean on all changed files

Net: 121 files changed, +1,424 / −3,666

Commits were merged into target branch
  • chore: consolidate platform services, dedupe helpers, de-monolith CLIs
    Consolidation, deduplication, and de-monolithization pass across the
    platform to improve stability, scalability, and maintainability.
    
    Bugs fixed:
    - services/predictions/cli.py: removed `def get_db(): return get_db()`
      infinite-recursion override that crashed every predictions subcommand.
    
    Deduplication:
    - services/agents/base.py (new): BaseSynthesisAgent + location_filter +
      query_rows_cursor shared scaffolding. All 10 synthesis agents now
      inherit the identical get_connection/store_summary/run_synthesis/main
      boilerplate (~60 lines each) instead of re-implementing it.
    - services/common/utils.py (new): serialize_value/serialize_rows
      extracted from export/report_generator.py and export/exporter.py
      (removed duplicate isoformat/bytes-hash serialization).
    - services/ingestion/base.py: added batch_ranges() helper; baal_indexer
      now uses it instead of a hand-rolled while loop.
    
    CLI de-monolithization:
    - services/analytics/cli.py: collapsed 23 near-identical 5-line dispatch
      blocks into a 13-entry table + shared get_connection/print_json.
    - 19 analytics cli_*.py + 13 service CLIs (crisp, data_module, events,
      attestation, certificates, finance, fortune500, geostatistics,
      graph_projection, iri, metadata_api, office, revenue_multiplier):
      replaced local _out/_connection/json.dumps boilerplate with the shared
      services.common.cli helpers.
    - 23 __main__.py files standardized to the canonical
      `from .cli import main` + guard pattern (was 6 variants).
    
    Test consolidation:
    - Replaced 48 copy-pasted test_cli_*.py template files with one
      parametrized tests/test_cli_registry.py covering all 47 CLI modules
      plus their subcommand --help targets.
    
    Governance:
    - services/migration/cli.py: documented that seed version-prefix sharing
      (9 pairs) is intentional and enforced by unique seed:<filename> IDs;
      schema versions remain strictly unique.
    
    Verification: 202 focused tests pass (CLI registry, migration, all agent
    suites); ruff clean on all changed files.
    
    Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
    1 month ago
  • chore: consolidate platform services, dedupe helpers, de-monolith CLIs
    Consolidation, deduplication, and de-monolithization pass across the
    platform to improve stability, scalability, and maintainability.
    
    Bugs fixed:
    - services/predictions/cli.py: removed `def get_db(): return get_db()`
      infinite-recursion override that crashed every predictions subcommand.
    
    Deduplication:
    - services/agents/base.py (new): BaseSynthesisAgent + location_filter +
      query_rows_cursor shared scaffolding. All 10 synthesis agents now
      inherit the identical get_connection/store_summary/run_synthesis/main
      boilerplate (~60 lines each) instead of re-implementing it.
    - services/common/utils.py (new): serialize_value/serialize_rows
      extracted from export/report_generator.py and export/exporter.py
      (removed duplicate isoformat/bytes-hash serialization).
    - services/ingestion/base.py: added batch_ranges() helper; baal_indexer
      now uses it instead of a hand-rolled while loop.
    
    CLI de-monolithization:
    - services/analytics/cli.py: collapsed 23 near-identical 5-line dispatch
      blocks into a 13-entry table + shared get_connection/print_json.
    - 19 analytics cli_*.py + 13 service CLIs (crisp, data_module, events,
      attestation, certificates, finance, fortune500, geostatistics,
      graph_projection, iri, metadata_api, office, revenue_multiplier):
      replaced local _out/_connection/json.dumps boilerplate with the shared
      services.common.cli helpers.
    - 23 __main__.py files standardized to the canonical
      `from .cli import main` + guard pattern (was 6 variants).
    
    Test consolidation:
    - Replaced 48 copy-pasted test_cli_*.py template files with one
      parametrized tests/test_cli_registry.py covering all 47 CLI modules
      plus their subcommand --help targets.
    
    Governance:
    - services/migration/cli.py: documented that seed version-prefix sharing
      (9 pairs) is intentional and enforced by unique seed:<filename> IDs;
      schema versions remain strictly unique.
    
    Verification: 202 focused tests pass (CLI registry, migration, all agent
    suites); ruff clean on all changed files.
    
    Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
    1 month ago
  • refactor: shared query helper, dispatch table, factory + serializer consolidation
    - services/common/database.py: add query()/query_one() helpers that work on
      both raw psycopg2 connections and DatabaseConnection; migrate clean
      single-query cursor blocks in ecology, carbon_balance, capacity_utilization,
      token_integration, and export/reports/commons to use them.
    - services/analytics/cli.py: collapse 24 sequential if/elif dispatch blocks
      into a _DB_LOCATION_COMMANDS table using shared get_connection/print_json;
      normalize inconsistent get_db import paths.
    - Delegate get_pg()/get_connection() copies in finance/cli, dataset_refresh,
      exporter, and reports/common to the canonical common.database.get_db.
    - reports/common: alias _serialize_value/_serialize_rows to the canonical
      services.common.utils implementations (removes the duplicated copy).
    - ingestion: gnosis_indexer uses shared batch_ranges(); update_indexer_status
      gains an optional metadata param so eas_indexer's duplicate status upsert
      is removed and routed through the shared function.
    
    Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
    1 month ago
    chore(scripts): extract shared bootstrap into scripts/lib/common.sh
    Collapse the copy-pasted secrets-source blocks, compose() helper, and
    wait_for_postgres() retry loop across 9 scripts into one sourced library:
    
    - scripts/lib/common.sh (new): source_secrets [strict|warn], compose,
      wait_for_postgres, psql_exec, seed_apply.
    - seed.sh: replace 30 hand-written psql seed blocks with an array-driven
      loop using psql_exec (291 -> 131 lines).
    - seed-pilot.sh: collapse 4 seed loops onto seed_apply with the pilot
      context (147 -> 126 lines).
    - backup/restore/rollback/upgrade/health-check/schema-snapshot/
      seed-metabase/verify-upgrade: source the lib, drop local copies.
    
    Verified: bash -n passes on all scripts.
    
    Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
    1 month ago
    refactor: canonical extensions, governance boundary doc, shared http adoption
    - schemas/postgres/000_extensions.sql: single canonical extension list
      (uuid-ossp, pgcrypto, postgis, postgis_topology, pg_trgm,
      pg_stat_statements); config/postgres/init.sql aligned to the same set;
      extensions.sql kept as the legacy bootstrap pointer.
    - services/governance/README.md: document the boundary between the
      off-chain governance CLIs (analytics/cli_governance_*, argparse, DB) and
      the on-chain DAO read client (services/governance, typer, read-first);
      add the same note at the services/cli.py mount points.
    - services/common/http.py: re-export Timeout/ConnectionError exception
      types so callers don't import requests directly.
    - Migrate raw urllib.request/requests call sites to the shared http client:
      gateway/router.py (Directus proxies), core/health.py (Directus ping),
      events/handlers.py (webhook + Directus notification), and remove the dead
      requests import in ingestion/climate_data.py; copernicus_remote_sensing
      now uses the re-exported exception types.
    
    Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
    1 month ago
    feat(cli): mount all analytics cli_*.py modules into the unified meta-CLI
    Wire the 18 previously-unreachable analytics CLI modules (stakeholders,
    consent, coordination, capability/strategy maps, governance-*, etc.) into
    services/cli.py so the unified `kokonut` tree exposes them consistently
    with the standalone python3 -m invocation. Also fix the pre-existing
    undefined `Any` annotation in _legacy.
    
    Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
    1 month ago
    refactor(analytics): extract pest_management CLI into cli_pest_management.py
    Split the 3,066-line pest_management.py monolith: domain functions
    (scouting, thresholds, interventions, traps, MoA rotation, IPM compliance,
    organic scoring) stay in the domain module; the ~550 lines of _cmd_*
    handlers, CommandLine subcommand definitions, and main() move to
    services/analytics/cli_pest_management.py. Register the new CLI in the
    test_cli_registry parametrized suite.
    
    pest_management.py: 3066 -> 2509 lines; cli_pest_management.py: 606 lines.
    
    Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
    1 month ago
    refactor(analytics): extract cooperative CLI into cli_cooperative.py
    Split the 1,358-line cooperative.py monolith: domain functions (coops,
    members, assets, collective purchasing, market orders, member dashboard)
    stay in the domain module; the _format_* renderers, _cmd_* handlers,
    CommandLine definitions, and main() move to
    services/analytics/cli_cooperative.py. Register in test_cli_registry.
    
    cooperative.py: 1358 -> 981 lines; cli_cooperative.py: 404 lines.
    
    Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
    1 month ago
    refactor(analytics): extract inline CLI handlers from 7 more monoliths
    Split the remaining analytics domain monoliths that mix domain queries with
    CommandLine subcommand definitions, _cmd_* handlers, and main():
    
    - digital_finance.py: 1396 -> 1221 lines, cli_digital_finance.py: 194
    - farmer_identity.py: 1181 -> 882 lines, cli_farmer_identity.py: 319
    - data_governance.py: 1174 -> 966 lines, cli_data_governance.py: 226
    - traceability.py: 1012 -> 817 lines, cli_traceability.py: 220
    - crop_rotation.py: 1013 -> 791 lines, cli_crop_rotation.py: 237
    - energy_monitoring.py: 867 -> 647 lines, cli_energy_monitoring.py: 235
    - pollinator_health.py: 806 -> 625 lines, cli_pollinator_health.py: 198
    
    Domain modules keep business logic and drop the now-unused CommandLine
    import; each new cli_<name>.py owns the CommandLine definitions, handlers,
    and main(). All 7 new CLIs registered in tests/test_cli_registry.py.
    
    Verification: 250 domain-module tests + 133 CLI registry tests pass.
    
    Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
    1 month ago
    refactor(analytics): extract argparse-style CLI sections from 3 monoliths
    Split precision_irrigation (1230 -> 1008 lines), marketplace (1224 -> 877),
    and extension (1161 -> 960) by moving their inline argparse main() +
    dispatcher into cli_<name>.py, matching the cli_*.py convention. Domain
    modules keep business logic and drop the unused argparse import; new CLI
    modules import the domain functions and any render helpers that moved with
    them. All 3 registered in tests/test_cli_registry.py.
    
    Verification: 139 CLI registry + 76 domain-module tests pass.
    
    Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
    1 month ago
    refactor(reports): split commons.py into a themed commons/ package
    The 1,114-line reports/commons.py is split by theme into
    services/export/reports/commons/:
    
    - bio_factory.py (205), liberation.py (150), wellbeing.py (171),
      stewardship.py (151), scaling.py (171), kokonut.py (157), ecological.py (147)
    - commons.py becomes a re-export facade over the package so existing
      importers (reports/__init__.py, reports/state.py) keep working unchanged.
    
    Also fix the shared query() helper to honor mock cursors without a
    ``description`` attribute (surfaced by the split test suite).
    
    Verification: 47 report tests + export/smoke tests pass; ruff clean.
    
    Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
    1 month ago
    test: shared location_id fixture + assert_sql_contains helper
    Add ADELPHI_LOCATION_ID / location_id fixture and assert_sql_contains to
    conftest.py for the analytics + schema-integrity test genres. Migrate
    test_consent_append_only and test_evidence_lineage_integrity to the shared
    helper; update test_relational_integrity to reference the renamed
    seed_apply() from the scripts/lib/common.sh refactor.
    
    Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
    1 month ago
    fix(analytics): resolve missing imports in extracted CLI modules
    The auto-extraction of CLI sections dropped several stdlib imports (date,
    datetime, json) and missed domain functions called without a ``return``
    prefix (get_plan, check_consent, record_transaction, etc.). Add the missing
    imports and domain references so every cli_<name>.py imports cleanly and
    all subcommands resolve their handlers.
    
    Verified: all 31 analytics cli_*.py modules import; 139 CLI registry + 461
    domain-module tests pass.
    
    Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
    1 month ago
  • test: fix gateway audit unit tests + bootstrap local test env
    The 3 gateway audit tests were implicitly hitting the real database:
    they patched only AuditLogger, but GatewayAudit.log() opens a connection
    via _get_conn() before the mocked logger is reached, so they failed with
    a DB auth error instead of testing the normalization logic. Mock
    _get_conn in all three so they are pure unit tests.
    
    The 9 agent_safety trigger tests skipped because the test process could
    not reach the Kokonut database from the host: services.common.db falls
    back to a placeholder password when secrets aren't loaded, and the
    compose DB port was not published (a separate project held host 5432).
    Bootstrap KOKONUT_ALLOW_PLAINTEXT_ENV and the compose POSTGRES_PASSWORD
    in tests/conftest.py so local `pytest` runs behave like the documented
    plaintext fallback, and expose the DB port via docker-compose.ci.yml.
    
    Verified: 3 audit + 9 trigger tests now pass; 68 gateway/agent and 262
    platform-integrity tests pass with no skips.
    
    Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
    1 month ago
  • test: do not hardcode POSTGRES_PASSWORD in conftest
    The previous conftest bootstrap set POSTGRES_PASSWORD=dev-kokonut-postgres-2026
    via setdefault, which is correct locally but broke CI: the buildspec writes
    POSTGRES_PASSWORD=ci-placeholder only to the .env *file* (not the shell env),
    so conftest's setdefault wonlocked the local dev password before
    services.common.db.load_dotenv() could read ci-placeholder from .env — every
    db-auth test then connected with the wrong credential and 316 DB-dependent
    tests skipped with "password authentication failed".
    
    Keep the KOKONUT_ALLOW_PLAINTEXT_ENV=true opt-in (needed for local dev to
    load the plaintext .env fallback); drop the hardcoded password so .env
    always wins via load_dotenv(). Verified locally (24 passed) and simulated
    the CI path (resolves to ci-placeholder).
    
    Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
    1 month ago
  • fix(ci): isolate Compose project namespace from host stacks
    CI's Compose project name defaulted to the workspace directory name,
    which collided with the long-running 'work' stack on the CI host
    (/onedev-build/work). Consequences observed in KI-388..390:
    
    - CI 'docker compose down -v' operated on the shared project and
      recreated the stack's data volumes (2026-08-14 05:12)
    - verify-clean-bootstrap created/dropped databases in the shared
      Postgres; a cancelled build (#389) left an orphaned
      kokonut_bootstrap_5367 database whose cleanup trap never fired
    - build #390 then failed on CREATE DATABASE collision
    
    Fix: pin COMPOSE_PROJECT_NAME=ki-ci at the top of the job so every
    docker compose call (buildspec, ci-check.sh, verify-clean-bootstrap.sh)
    targets an isolated, disposable project regardless of checkout path.
    
    Add a pre-flight guard that verifies the namespace is pinned and fails
    fast if it is not, preventing regression to host-shared projects.
    
    Co-authored-by: Syntropic Agent <syntropic-agent@kokonut.network>
    3 weeks ago
    fix(ci): start CI database before ci-check bootstrap gate
    With the isolated ki-ci namespace (previous commit), no PostgreSQL
    container exists when ci-check.sh runs its Clean PostgreSQL bootstrap
    check — previously it accidentally reached the shared host stack's
    database. Start the database service (ki-ci project) before the
    fast-fail gate and wait for pg_isready. The infra step later runs
    'down -v' + fresh 'up', so this early container is disposable.
    
    Co-authored-by: Syntropic Agent <syntropic-agent@kokonut.network>
    3 weeks ago
    fix(ci): keep early bootstrap database off host port 5432
    KI-392: the early database (started for the bootstrap check) published
    127.0.0.1:5432 via docker-compose.ci.yml, colliding with another stack
    holding that port on the CI host. The bootstrap check is exec-based
    ('docker compose exec database psql'), so the port publish is
    unnecessary at this stage — use the base compose file only (ports stay
    private). The infra step later recreates the project with the ci.yml
    override for the host-run test suite.
    
    Co-authored-by: Syntropic Agent <syntropic-agent@kokonut.network>
    3 weeks ago
    fix(ci): make ci.yml host ports configurable to avoid shared-host conflicts
    KI-393 got through all 65 fast-fail checks, then the infra step hit
    'Bind for 127.0.0.1:8123 failed: port is already allocated' — the dev
    'work' stack on the CI host holds 5432/8123.
    
    Make docker-compose.ci.yml host ports configurable via CI_PG_HOST_PORT /
    CI_CH_HOST_PORT (defaults unchanged: 127.0.0.1:5432/8123, so local dev
    behavior is identical). The CI buildspec pins high ports 15432/18123.
    
    Tests reach the databases via the Compose network (PG_HOST=database /
    CH_HOST=clickhouse), so the published ports are for host-side debugging
    only; nothing in CI depends on the specific host port values.
    
    Co-authored-by: Syntropic Agent <syntropic-agent@kokonut.network>
    3 weeks ago
    fix(ci): make directus/gateway/grpc host ports configurable
    KI-394 cleared the database/clickhouse conflicts but hit the next tier:
    the base compose file publishes directus 8055, gateway 8099, and grpc
    50051 — all held by the dev 'work' stack on the shared CI host.
    
    Make these ports env-driven with current defaults (DIRECTUS_HOST_PORT,
    GATEWAY_HOST_PORT, GRPC_HOST_PORT — local dev behavior unchanged) and
    pin high ports in the CI buildspec: 18055 / 18099 / 50052.
    
    The grpc container healthcheck targets 127.0.0.1:50051 inside the
    container (unaffected by host mapping); the one test referencing 50051
    mocks the channel entirely.
    
    Co-authored-by: Syntropic Agent <syntropic-agent@kokonut.network>
    3 weeks ago
    test: fix statistically unsound triangular sampling assertion
    test_scenario_param_sample_triangular asserted the sample mean of 100
    triangular draws ≈ the mode (0.05), but the distribution's true mean is
    (a+b+c)/3 ≈ 0.0733. With n=100 the sample mean has SE ≈ 0.003, so the
    0.03 tolerance fails ~1.5% of runs — a flake that hit KI-395.
    
    Assert against the true distribution mean with n=2000 (SE ≈ 0.0006,
    tolerance 0.01 ≈ 16 SE): zero failures across 500 seeded trials.
    Bounds assertion retained unchanged.
    
    Co-authored-by: Syntropic Agent <syntropic-agent@kokonut.network>
    3 weeks ago
    Merge PR #94: fix(ci) — isolate Compose project namespace from host stacks
    Pins COMPOSE_PROJECT_NAME=ki-ci with pre-flight guard, starts an early
    bootstrap database, and makes all CI host ports configurable
    (15432/18123/18055/18099/50052) so CI never touches other Compose
    stacks on shared hosts. Also fixes the flaky triangular-sampling test
    (asserted mode instead of true distribution mean).
    
    CI KI-396: all gates green — 4098 tests, static analysis, Solidity, hooks.
    3 weeks ago
  • syntropicagent commented 3 weeks ago

    Merging on Wasabi's instruction ("do the merging"). CI KI-397 green on the merge ref — first successful build since the compose-namespace collision (KI-387/389/390 failures). Namespace isolation from PR #94 resolved the blocker. 4,098 tests + static analysis + Solidity + hooks all pass.

  • syntropicagent merged 3 weeks ago (merge commit: 86fa8c39 )
1/1
Submitter wasabi
Target main
Source chore/platform-consolidation-dedup
Jobs
Merge Strategy
Create Merge Commit
Watchers (2)
Reference
pr KI-93
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover