Commits were merged into target branch
-
chore: consolidate platform services, dedupe helpers, de-monolith CLIs
Consolidation, deduplication, and de-monolithization pass across the platform to improve stability, scalability, and maintainability. Bugs fixed: - services/predictions/cli.py: removed `def get_db(): return get_db()` infinite-recursion override that crashed every predictions subcommand. Deduplication: - services/agents/base.py (new): BaseSynthesisAgent + location_filter + query_rows_cursor shared scaffolding. All 10 synthesis agents now inherit the identical get_connection/store_summary/run_synthesis/main boilerplate (~60 lines each) instead of re-implementing it. - services/common/utils.py (new): serialize_value/serialize_rows extracted from export/report_generator.py and export/exporter.py (removed duplicate isoformat/bytes-hash serialization). - services/ingestion/base.py: added batch_ranges() helper; baal_indexer now uses it instead of a hand-rolled while loop. CLI de-monolithization: - services/analytics/cli.py: collapsed 23 near-identical 5-line dispatch blocks into a 13-entry table + shared get_connection/print_json. - 19 analytics cli_*.py + 13 service CLIs (crisp, data_module, events, attestation, certificates, finance, fortune500, geostatistics, graph_projection, iri, metadata_api, office, revenue_multiplier): replaced local _out/_connection/json.dumps boilerplate with the shared services.common.cli helpers. - 23 __main__.py files standardized to the canonical `from .cli import main` + guard pattern (was 6 variants). Test consolidation: - Replaced 48 copy-pasted test_cli_*.py template files with one parametrized tests/test_cli_registry.py covering all 47 CLI modules plus their subcommand --help targets. Governance: - services/migration/cli.py: documented that seed version-prefix sharing (9 pairs) is intentional and enforced by unique seed:<filename> IDs; schema versions remain strictly unique. Verification: 202 focused tests pass (CLI registry, migration, all agent suites); ruff clean on all changed files. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 month ago -
chore: consolidate platform services, dedupe helpers, de-monolith CLIs
Consolidation, deduplication, and de-monolithization pass across the platform to improve stability, scalability, and maintainability. Bugs fixed: - services/predictions/cli.py: removed `def get_db(): return get_db()` infinite-recursion override that crashed every predictions subcommand. Deduplication: - services/agents/base.py (new): BaseSynthesisAgent + location_filter + query_rows_cursor shared scaffolding. All 10 synthesis agents now inherit the identical get_connection/store_summary/run_synthesis/main boilerplate (~60 lines each) instead of re-implementing it. - services/common/utils.py (new): serialize_value/serialize_rows extracted from export/report_generator.py and export/exporter.py (removed duplicate isoformat/bytes-hash serialization). - services/ingestion/base.py: added batch_ranges() helper; baal_indexer now uses it instead of a hand-rolled while loop. CLI de-monolithization: - services/analytics/cli.py: collapsed 23 near-identical 5-line dispatch blocks into a 13-entry table + shared get_connection/print_json. - 19 analytics cli_*.py + 13 service CLIs (crisp, data_module, events, attestation, certificates, finance, fortune500, geostatistics, graph_projection, iri, metadata_api, office, revenue_multiplier): replaced local _out/_connection/json.dumps boilerplate with the shared services.common.cli helpers. - 23 __main__.py files standardized to the canonical `from .cli import main` + guard pattern (was 6 variants). Test consolidation: - Replaced 48 copy-pasted test_cli_*.py template files with one parametrized tests/test_cli_registry.py covering all 47 CLI modules plus their subcommand --help targets. Governance: - services/migration/cli.py: documented that seed version-prefix sharing (9 pairs) is intentional and enforced by unique seed:<filename> IDs; schema versions remain strictly unique. Verification: 202 focused tests pass (CLI registry, migration, all agent suites); ruff clean on all changed files. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 month ago -
refactor: shared query helper, dispatch table, factory + serializer consolidation
- services/common/database.py: add query()/query_one() helpers that work on both raw psycopg2 connections and DatabaseConnection; migrate clean single-query cursor blocks in ecology, carbon_balance, capacity_utilization, token_integration, and export/reports/commons to use them. - services/analytics/cli.py: collapse 24 sequential if/elif dispatch blocks into a _DB_LOCATION_COMMANDS table using shared get_connection/print_json; normalize inconsistent get_db import paths. - Delegate get_pg()/get_connection() copies in finance/cli, dataset_refresh, exporter, and reports/common to the canonical common.database.get_db. - reports/common: alias _serialize_value/_serialize_rows to the canonical services.common.utils implementations (removes the duplicated copy). - ingestion: gnosis_indexer uses shared batch_ranges(); update_indexer_status gains an optional metadata param so eas_indexer's duplicate status upsert is removed and routed through the shared function. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 month agochore(scripts): extract shared bootstrap into scripts/lib/common.shCollapse the copy-pasted secrets-source blocks, compose() helper, and wait_for_postgres() retry loop across 9 scripts into one sourced library: - scripts/lib/common.sh (new): source_secrets [strict|warn], compose, wait_for_postgres, psql_exec, seed_apply. - seed.sh: replace 30 hand-written psql seed blocks with an array-driven loop using psql_exec (291 -> 131 lines). - seed-pilot.sh: collapse 4 seed loops onto seed_apply with the pilot context (147 -> 126 lines). - backup/restore/rollback/upgrade/health-check/schema-snapshot/ seed-metabase/verify-upgrade: source the lib, drop local copies. Verified: bash -n passes on all scripts. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 month agorefactor: canonical extensions, governance boundary doc, shared http adoption- schemas/postgres/000_extensions.sql: single canonical extension list (uuid-ossp, pgcrypto, postgis, postgis_topology, pg_trgm, pg_stat_statements); config/postgres/init.sql aligned to the same set; extensions.sql kept as the legacy bootstrap pointer. - services/governance/README.md: document the boundary between the off-chain governance CLIs (analytics/cli_governance_*, argparse, DB) and the on-chain DAO read client (services/governance, typer, read-first); add the same note at the services/cli.py mount points. - services/common/http.py: re-export Timeout/ConnectionError exception types so callers don't import requests directly. - Migrate raw urllib.request/requests call sites to the shared http client: gateway/router.py (Directus proxies), core/health.py (Directus ping), events/handlers.py (webhook + Directus notification), and remove the dead requests import in ingestion/climate_data.py; copernicus_remote_sensing now uses the re-exported exception types. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 month agofeat(cli): mount all analytics cli_*.py modules into the unified meta-CLIWire the 18 previously-unreachable analytics CLI modules (stakeholders, consent, coordination, capability/strategy maps, governance-*, etc.) into services/cli.py so the unified `kokonut` tree exposes them consistently with the standalone python3 -m invocation. Also fix the pre-existing undefined `Any` annotation in _legacy. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 month agorefactor(analytics): extract pest_management CLI into cli_pest_management.pySplit the 3,066-line pest_management.py monolith: domain functions (scouting, thresholds, interventions, traps, MoA rotation, IPM compliance, organic scoring) stay in the domain module; the ~550 lines of _cmd_* handlers, CommandLine subcommand definitions, and main() move to services/analytics/cli_pest_management.py. Register the new CLI in the test_cli_registry parametrized suite. pest_management.py: 3066 -> 2509 lines; cli_pest_management.py: 606 lines. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 month agorefactor(analytics): extract cooperative CLI into cli_cooperative.pySplit the 1,358-line cooperative.py monolith: domain functions (coops, members, assets, collective purchasing, market orders, member dashboard) stay in the domain module; the _format_* renderers, _cmd_* handlers, CommandLine definitions, and main() move to services/analytics/cli_cooperative.py. Register in test_cli_registry. cooperative.py: 1358 -> 981 lines; cli_cooperative.py: 404 lines. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 month agorefactor(analytics): extract inline CLI handlers from 7 more monolithsSplit the remaining analytics domain monoliths that mix domain queries with CommandLine subcommand definitions, _cmd_* handlers, and main(): - digital_finance.py: 1396 -> 1221 lines, cli_digital_finance.py: 194 - farmer_identity.py: 1181 -> 882 lines, cli_farmer_identity.py: 319 - data_governance.py: 1174 -> 966 lines, cli_data_governance.py: 226 - traceability.py: 1012 -> 817 lines, cli_traceability.py: 220 - crop_rotation.py: 1013 -> 791 lines, cli_crop_rotation.py: 237 - energy_monitoring.py: 867 -> 647 lines, cli_energy_monitoring.py: 235 - pollinator_health.py: 806 -> 625 lines, cli_pollinator_health.py: 198 Domain modules keep business logic and drop the now-unused CommandLine import; each new cli_<name>.py owns the CommandLine definitions, handlers, and main(). All 7 new CLIs registered in tests/test_cli_registry.py. Verification: 250 domain-module tests + 133 CLI registry tests pass. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 month agorefactor(analytics): extract argparse-style CLI sections from 3 monolithsSplit precision_irrigation (1230 -> 1008 lines), marketplace (1224 -> 877), and extension (1161 -> 960) by moving their inline argparse main() + dispatcher into cli_<name>.py, matching the cli_*.py convention. Domain modules keep business logic and drop the unused argparse import; new CLI modules import the domain functions and any render helpers that moved with them. All 3 registered in tests/test_cli_registry.py. Verification: 139 CLI registry + 76 domain-module tests pass. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 month agorefactor(reports): split commons.py into a themed commons/ packageThe 1,114-line reports/commons.py is split by theme into services/export/reports/commons/: - bio_factory.py (205), liberation.py (150), wellbeing.py (171), stewardship.py (151), scaling.py (171), kokonut.py (157), ecological.py (147) - commons.py becomes a re-export facade over the package so existing importers (reports/__init__.py, reports/state.py) keep working unchanged. Also fix the shared query() helper to honor mock cursors without a ``description`` attribute (surfaced by the split test suite). Verification: 47 report tests + export/smoke tests pass; ruff clean. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 month agotest: shared location_id fixture + assert_sql_contains helperAdd ADELPHI_LOCATION_ID / location_id fixture and assert_sql_contains to conftest.py for the analytics + schema-integrity test genres. Migrate test_consent_append_only and test_evidence_lineage_integrity to the shared helper; update test_relational_integrity to reference the renamed seed_apply() from the scripts/lib/common.sh refactor. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 month agofix(analytics): resolve missing imports in extracted CLI modulesThe auto-extraction of CLI sections dropped several stdlib imports (date, datetime, json) and missed domain functions called without a ``return`` prefix (get_plan, check_consent, record_transaction, etc.). Add the missing imports and domain references so every cli_<name>.py imports cleanly and all subcommands resolve their handlers. Verified: all 31 analytics cli_*.py modules import; 139 CLI registry + 461 domain-module tests pass. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 month ago -
test: fix gateway audit unit tests + bootstrap local test env
The 3 gateway audit tests were implicitly hitting the real database: they patched only AuditLogger, but GatewayAudit.log() opens a connection via _get_conn() before the mocked logger is reached, so they failed with a DB auth error instead of testing the normalization logic. Mock _get_conn in all three so they are pure unit tests. The 9 agent_safety trigger tests skipped because the test process could not reach the Kokonut database from the host: services.common.db falls back to a placeholder password when secrets aren't loaded, and the compose DB port was not published (a separate project held host 5432). Bootstrap KOKONUT_ALLOW_PLAINTEXT_ENV and the compose POSTGRES_PASSWORD in tests/conftest.py so local `pytest` runs behave like the documented plaintext fallback, and expose the DB port via docker-compose.ci.yml. Verified: 3 audit + 9 trigger tests now pass; 68 gateway/agent and 262 platform-integrity tests pass with no skips. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 month ago -
test: do not hardcode POSTGRES_PASSWORD in conftest
The previous conftest bootstrap set POSTGRES_PASSWORD=dev-kokonut-postgres-2026 via setdefault, which is correct locally but broke CI: the buildspec writes POSTGRES_PASSWORD=ci-placeholder only to the .env *file* (not the shell env), so conftest's setdefault wonlocked the local dev password before services.common.db.load_dotenv() could read ci-placeholder from .env — every db-auth test then connected with the wrong credential and 316 DB-dependent tests skipped with "password authentication failed". Keep the KOKONUT_ALLOW_PLAINTEXT_ENV=true opt-in (needed for local dev to load the plaintext .env fallback); drop the hardcoded password so .env always wins via load_dotenv(). Verified locally (24 passed) and simulated the CI path (resolves to ci-placeholder). Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
1 month ago -
fix(ci): isolate Compose project namespace from host stacks
CI's Compose project name defaulted to the workspace directory name, which collided with the long-running 'work' stack on the CI host (/onedev-build/work). Consequences observed in KI-388..390: - CI 'docker compose down -v' operated on the shared project and recreated the stack's data volumes (2026-08-14 05:12) - verify-clean-bootstrap created/dropped databases in the shared Postgres; a cancelled build (#389) left an orphaned kokonut_bootstrap_5367 database whose cleanup trap never fired - build #390 then failed on CREATE DATABASE collision Fix: pin COMPOSE_PROJECT_NAME=ki-ci at the top of the job so every docker compose call (buildspec, ci-check.sh, verify-clean-bootstrap.sh) targets an isolated, disposable project regardless of checkout path. Add a pre-flight guard that verifies the namespace is pinned and fails fast if it is not, preventing regression to host-shared projects. Co-authored-by: Syntropic Agent <syntropic-agent@kokonut.network>
3 weeks agofix(ci): start CI database before ci-check bootstrap gateWith the isolated ki-ci namespace (previous commit), no PostgreSQL container exists when ci-check.sh runs its Clean PostgreSQL bootstrap check — previously it accidentally reached the shared host stack's database. Start the database service (ki-ci project) before the fast-fail gate and wait for pg_isready. The infra step later runs 'down -v' + fresh 'up', so this early container is disposable. Co-authored-by: Syntropic Agent <syntropic-agent@kokonut.network>
3 weeks agofix(ci): keep early bootstrap database off host port 5432KI-392: the early database (started for the bootstrap check) published 127.0.0.1:5432 via docker-compose.ci.yml, colliding with another stack holding that port on the CI host. The bootstrap check is exec-based ('docker compose exec database psql'), so the port publish is unnecessary at this stage — use the base compose file only (ports stay private). The infra step later recreates the project with the ci.yml override for the host-run test suite. Co-authored-by: Syntropic Agent <syntropic-agent@kokonut.network>
3 weeks agofix(ci): make ci.yml host ports configurable to avoid shared-host conflictsKI-393 got through all 65 fast-fail checks, then the infra step hit 'Bind for 127.0.0.1:8123 failed: port is already allocated' — the dev 'work' stack on the CI host holds 5432/8123. Make docker-compose.ci.yml host ports configurable via CI_PG_HOST_PORT / CI_CH_HOST_PORT (defaults unchanged: 127.0.0.1:5432/8123, so local dev behavior is identical). The CI buildspec pins high ports 15432/18123. Tests reach the databases via the Compose network (PG_HOST=database / CH_HOST=clickhouse), so the published ports are for host-side debugging only; nothing in CI depends on the specific host port values. Co-authored-by: Syntropic Agent <syntropic-agent@kokonut.network>
3 weeks agofix(ci): make directus/gateway/grpc host ports configurableKI-394 cleared the database/clickhouse conflicts but hit the next tier: the base compose file publishes directus 8055, gateway 8099, and grpc 50051 — all held by the dev 'work' stack on the shared CI host. Make these ports env-driven with current defaults (DIRECTUS_HOST_PORT, GATEWAY_HOST_PORT, GRPC_HOST_PORT — local dev behavior unchanged) and pin high ports in the CI buildspec: 18055 / 18099 / 50052. The grpc container healthcheck targets 127.0.0.1:50051 inside the container (unaffected by host mapping); the one test referencing 50051 mocks the channel entirely. Co-authored-by: Syntropic Agent <syntropic-agent@kokonut.network>
3 weeks agotest: fix statistically unsound triangular sampling assertiontest_scenario_param_sample_triangular asserted the sample mean of 100 triangular draws ≈ the mode (0.05), but the distribution's true mean is (a+b+c)/3 ≈ 0.0733. With n=100 the sample mean has SE ≈ 0.003, so the 0.03 tolerance fails ~1.5% of runs — a flake that hit KI-395. Assert against the true distribution mean with n=2000 (SE ≈ 0.0006, tolerance 0.01 ≈ 16 SE): zero failures across 500 seeded trials. Bounds assertion retained unchanged. Co-authored-by: Syntropic Agent <syntropic-agent@kokonut.network>
3 weeks agoMerge PR #94: fix(ci) — isolate Compose project namespace from host stacksPins COMPOSE_PROJECT_NAME=ki-ci with pre-flight guard, starts an early bootstrap database, and makes all CI host ports configurable (15432/18123/18055/18099/50052) so CI never touches other Compose stacks on shared hosts. Also fixes the flaky triangular-sampling test (asserted mode instead of true distribution mean). CI KI-396: all gates green — 4098 tests, static analysis, Solidity, hooks.
3 weeks ago -
-
Summary
Consolidation, deduplication, and de-monolithization pass across the platform to improve stability, scalability, and maintainability.
Bugs fixed
services/predictions/cli.py: removeddef get_db(): return get_db()infinite-recursion override that crashed every predictions subcommand.Deduplication
services/agents/base.py(new):BaseSynthesisAgent+location_filter+query_rows_cursorshared scaffolding. All 10 synthesis agents now inherit the identicalget_connection/store_summary/run_synthesis/mainboilerplate (~60 lines each) instead of re-implementing it.services/common/utils.py(new):serialize_value/serialize_rowsextracted fromexport/report_generator.pyandexport/exporter.py(removed duplicate isoformat/bytes-hash serialization).services/ingestion/base.py: addedbatch_ranges()helper;baal_indexernow uses it instead of a hand-rolled while loop.CLI de-monolithization
services/analytics/cli.py: collapsed 23 near-identical 5-line dispatch blocks into a 13-entry table + sharedget_connection/print_json.cli_*.py+ 13 service CLIs (crisp, data_module, events, attestation, certificates, finance, fortune500, geostatistics, graph_projection, iri, metadata_api, office, revenue_multiplier): replaced local_out/_connection/json.dumpsboilerplate with the sharedservices.common.clihelpers.__main__.pyfiles standardized to the canonicalfrom .cli import main+ guard pattern (was 6 variants).Test consolidation
test_cli_*.pytemplate files with one parametrizedtests/test_cli_registry.pycovering all 47 CLI modules plus their subcommand--helptargets.Governance
services/migration/cli.py: documented that seed version-prefix sharing (9 pairs) is intentional and enforced by uniqueseed:<filename>IDs; schema versions remain strictly unique.Verification
ruffclean on all changed filesNet: 121 files changed, +1,424 / −3,666