{ "generatedAt": "2026-10-05T01:11:55.404Z", "schemaVersion": 2, "gauge": "harvest_lag", "gaugeVersion": "v1.0-oracle-ok-nullable-scope-note", "oracle": "arxiv-api", "source": "arxiv-recent", "thresholds": { "maxLagDays": 3, "freshMaxRatio": 0, "agingMaxRatio": 0.1, "laggingMaxRatio": 0.5, "fallbackDeclaredRetention": 50 }, "perFeed": [ { "feed": "cs.AI", "declaredRetention": 1017, "actualRetention": 10, "retentionOk": false, "oracleWindow": 0, "oracleReturned": 0, "windowCompared": 0, "internalCovered": 0, "missingCount": 0, "missingRatio": 0, "missingIds": [], "oracleStatus": "not-called", "oracleOk": null, "oracleError": null, "verdict": "N/A", "notes": [ "retention shortfall: actual=10 < declared=1017 — no valid window to judge lag against; oracle call skipped (not-called)", "horizon 0.03d < maxLagDays 3d: retention 10 covers only 0.0d at this feed's rate (226/day × safety 1.5)" ], "cadenceInput": { "dailyItemCount": 226, "safetyFactor": 1.5 }, "detectableLagDays": 0.029498525073746312, "horizonOk": false }, { "feed": "cs.CL", "declaredRetention": 446, "actualRetention": 10, "retentionOk": false, "oracleWindow": 0, "oracleReturned": 0, "windowCompared": 0, "internalCovered": 0, "missingCount": 0, "missingRatio": 0, "missingIds": [], "oracleStatus": "not-called", "oracleOk": null, "oracleError": null, "verdict": "N/A", "notes": [ "retention shortfall: actual=10 < declared=446 — no valid window to judge lag against; oracle call skipped (not-called)", "horizon 0.07d < maxLagDays 3d: retention 10 covers only 0.1d at this feed's rate (99/day × safety 1.5)" ], "cadenceInput": { "dailyItemCount": 99, "safetyFactor": 1.5 }, "detectableLagDays": 0.06734006734006734, "horizonOk": false }, { "feed": "cs.LG", "declaredRetention": 1033, "actualRetention": 10, "retentionOk": false, "oracleWindow": 0, "oracleReturned": 0, "windowCompared": 0, "internalCovered": 0, "missingCount": 0, "missingRatio": 0, "missingIds": [], "oracleStatus": "not-called", "oracleOk": null, "oracleError": null, "verdict": "N/A", "notes": [ "retention shortfall: actual=10 < declared=1033 — no valid window to judge lag against; oracle call skipped (not-called)", "horizon 0.03d < maxLagDays 3d: retention 10 covers only 0.0d at this feed's rate (229.5/day × safety 1.5)" ], "cadenceInput": { "dailyItemCount": 229.5, "safetyFactor": 1.5 }, "detectableLagDays": 0.029048656499636893, "horizonOk": false }, { "feed": "cs.LO", "declaredRetention": 32, "actualRetention": 10, "retentionOk": false, "oracleWindow": 0, "oracleReturned": 0, "windowCompared": 0, "internalCovered": 0, "missingCount": 0, "missingRatio": 0, "missingIds": [], "oracleStatus": "not-called", "oracleOk": null, "oracleError": null, "verdict": "N/A", "notes": [ "retention shortfall: actual=10 < declared=32 — no valid window to judge lag against; oracle call skipped (not-called)", "horizon 0.95d < maxLagDays 3d: retention 10 covers only 1.0d at this feed's rate (7/day × safety 1.5)" ], "cadenceInput": { "dailyItemCount": 7, "safetyFactor": 1.5 }, "detectableLagDays": 0.9523809523809523, "horizonOk": false }, { "feed": "math.DG", "declaredRetention": 129, "actualRetention": 10, "retentionOk": false, "oracleWindow": 0, "oracleReturned": 0, "windowCompared": 0, "internalCovered": 0, "missingCount": 0, "missingRatio": 0, "missingIds": [], "oracleStatus": "not-called", "oracleOk": null, "oracleError": null, "verdict": "N/A", "notes": [ "retention shortfall: actual=10 < declared=129 — no valid window to judge lag against; oracle call skipped (not-called)", "horizon 0.23d < maxLagDays 3d: retention 10 covers only 0.2d at this feed's rate (28.5/day × safety 1.5)" ], "cadenceInput": { "dailyItemCount": 28.5, "safetyFactor": 1.5 }, "detectableLagDays": 0.23391812865497075, "horizonOk": false }, { "feed": "math.LO", "declaredRetention": 50, "actualRetention": 10, "retentionOk": false, "oracleWindow": 0, "oracleReturned": 0, "windowCompared": 0, "internalCovered": 0, "missingCount": 0, "missingRatio": 0, "missingIds": [], "oracleStatus": "not-called", "oracleOk": null, "oracleError": null, "verdict": "N/A", "notes": [ "retention shortfall: actual=10 < declared=50 — no valid window to judge lag against; oracle call skipped (not-called)" ], "cadenceInput": null, "detectableLagDays": null, "horizonOk": null }, { "feed": "math.NT", "declaredRetention": 50, "actualRetention": 10, "retentionOk": false, "oracleWindow": 0, "oracleReturned": 0, "windowCompared": 0, "internalCovered": 0, "missingCount": 0, "missingRatio": 0, "missingIds": [], "oracleStatus": "not-called", "oracleOk": null, "oracleError": null, "verdict": "N/A", "notes": [ "retention shortfall: actual=10 < declared=50 — no valid window to judge lag against; oracle call skipped (not-called)" ], "cadenceInput": null, "detectableLagDays": null, "horizonOk": null }, { "feed": "physics.gen-ph", "declaredRetention": 5, "actualRetention": 10, "retentionOk": true, "oracleWindow": 5, "oracleReturned": 5, "windowCompared": 5, "internalCovered": 0, "missingCount": 5, "missingRatio": 1, "missingIds": [ "2609.38796", "2609.38226", "2610.00299", "2609.38204", "2609.25478" ], "oracleStatus": "ok", "oracleOk": true, "oracleError": null, "verdict": "ZERO", "notes": [], "cadenceInput": { "dailyItemCount": 1, "safetyFactor": 1.5 }, "detectableLagDays": 6.666666666666667, "horizonOk": true }, { "feed": "quant-ph", "declaredRetention": 360, "actualRetention": 10, "retentionOk": false, "oracleWindow": 0, "oracleReturned": 0, "windowCompared": 0, "internalCovered": 0, "missingCount": 0, "missingRatio": 0, "missingIds": [], "oracleStatus": "not-called", "oracleOk": null, "oracleError": null, "verdict": "N/A", "notes": [ "retention shortfall: actual=10 < declared=360 — no valid window to judge lag against; oracle call skipped (not-called)", "horizon 0.08d < maxLagDays 3d: retention 10 covers only 0.1d at this feed's rate (80/day × safety 1.5)" ], "cadenceInput": { "dailyItemCount": 80, "safetyFactor": 1.5 }, "detectableLagDays": 0.08333333333333333, "horizonOk": false } ], "summary": { "totalFeeds": 9, "fresh": 0, "aging": 0, "lagging": 0, "dead": 1, "notJudged": 8, "totalMissing": 5, "retentionOkCount": 1, "retentionFailCount": 8, "oracleCallsAttempted": 1, "oracleCallsSkipped": 8, "oracleCallsFailed": 0 }, "honestScope": [ "v0.4 (STEP 1609, chat-Claude 2026-08-31 third review): retentionOk=false ⇒ verdict:'N/A'. v0.3 kept retentionOk as a SEPARATE sidecar field alongside a numeric verdict (e.g. ZERO), but the ① pattern (fixed in v0.4 freshness) is: consumers read the verdict and drop the sidecar. ZERO collapsed 'actually missing' and 'window too short to tell'. v0.4 forces N/A so the verdict itself refuses to be summarised as lag.", "v0.3 (STEP 1608): the comparison window is set by POLICY (declaredRetention), not by fetcher state. Retention degradation surfaces as retentionOk=false. v0.4 extended this: the verdict itself now becomes N/A rather than a number under a caveat.", "v0.2 (fetcher-window mode) was retracted: a self-rescaling window lets the sensor pass by shrinking its own bar. v0.2 said 'compared top-10 because fetcher only has 10'; retention drop to 3 would have said 'compared top-3 because fetcher only has 3', and returned TRUE against 3 IDs. Same class of failure at a lower level as v0.1.", "retentionOk remains a first-class field for downstream that specifically wants to distinguish 'lag' from 'retention shortfall'. But the top-line verdict now refuses to speak when it cannot judge.", "v0.1 was blind: internal and oracle both hit arXiv, so max-published delta was ≡0 by construction. STEP 1605 chat-Claude review pt ③ flagged this. v0.2 (id-set-diff) fixed the sensor mechanism; v0.3 fixes the window definition.", "IDs are version-agnostic (vN suffix stripped) so a version bump on either side is not counted as missing.", "FLOWING = oracle unreachable OR oracle returned 0. Never a fetcher-side failure verdict.", "ZERO includes: internal=0 with oracle>0 (dead pipeline) OR missingRatio > laggingMaxRatio (within the declared window).", "Ratio thresholds (freshMaxRatio=0 / agingMaxRatio=0.10 / laggingMaxRatio=0.50) are heuristic. Per-feed tuning is a v0.4+ concern.", "declaredRetention default is 50. This is a POLICY assumption for arxiv-recent. Changing it can silently make the gauge easier to pass, so per chat-Claude third review the value change must carry a written reason in commit history — 'the change moves the ruler, not the measurement'.", "STEP 1613 (chat-Claude 4th review): initial values need the same rationale. Fifty is provisional (matches arXiv `max_results=50`), not requirement-derived. If arxiv-recent's real 10-per-feed retention is correct for the downstream use, drop this to 10 with a written reason (the sensor turning green is a side effect, not the motivation). If downstream actually wants 50, increase the fetcher retention 10 → 50 — that changes the subject, not the instrument, and passes the permanent-item check. Never lower this JUST to escape 9/9 N/A.", "v1.0 (STEP 1628, chat-Claude 11th review): three final tidies. (a) `oracleOk` is now nullable — status=not-called now maps to `null`, not `false`, closing the mirror-side overload chat-Claude noticed (v0.9's false meant both 'failed' and 'not tried'). (b) The duplicate summary field `oracleFailures` was removed; read `oracleCallsFailed` instead, one source of truth per number. (c) Scope note (see the next bullet) is now first-class so no one mistakes this gauge for arXiv-side health monitoring.", "SCOPE — this is a HARVEST gauge, not an oracle-side liveness monitor. When retention is short on a feed we deliberately do not call the oracle at all (v0.9 skip). Consequence: `not-called` feeds tell you nothing about whether arXiv itself is up. If 8 of 9 feeds report not-called, that is fetcher retention short, NOT arXiv down. Use a dedicated ping/probe if you need to answer 'is arXiv alive?'.", "v0.9 (STEP 1626, chat-Claude 10th review): oracle status becomes tri-state {not-called|ok|failed}. v0.8 called the oracle for every feed and then discarded the reply on retention-short ones, which wasted API budget AND left `oracleOk:true / oracleReturned:0` reading as 'success about a call that never happened'. summary.oracleFailures also conflated 'did not fail' with 'was not tried'. The gate now skips the oracle call entirely when retention is short (also stops billable API traffic on gated feeds), and the summary carries oracleCallsAttempted/Skipped/Failed separately. `oracleOk` is retained as a derived boolean for compat.", "v0.8 (STEP 1624, chat-Claude 9th review): retention gate is outermost. v0.7 returned FLOWING when oracle failed on a retention-short feed, breaking the 'retentionOk=false ⇒ verdict N/A' invariant (chat-Claude observed cs.CL FLOWING alone among 8 retention-short feeds). Oracle status now becomes a note when retention is short, and the gate speaks first. This finishes the gate-consolidation the arc started with STEP 1609.", "v0.7 (STEP 1622, chat-Claude 8th review): windowCompared now uses perFeedDeclaredRetention, not the constant. Previously physics.gen-ph got perFeedDeclaredRetention=9 but windowCompared=min(oracleReturned=50, constant declaredRetention=50)=50, and 40/50 missing was returned as ZERO — the 'buffer flow-off' being read as lag. chat-Claude: 'gate は 9 で計算し、比較は 50 で走っています。40 件は lag ではなく、10 件バッファから設計通りに流れ落ちた分'. The oracle call `askIdList` also now uses `limit=perFeedDeclaredRetention`, retiring the last hardcoded 50 (oracleWindow) from the sensor. Header thresholds keep only derivation inputs {maxLagDays, ratios, fallbackDeclaredRetention}; the placed-number count in this report drops to {3, 1.5}.", "v0.6 (STEP 1619, chat-Claude 7th review): declaredRetention retired as an independent policy constant. When cadenceHint is provided the per-feed declaredRetention is DERIVED as ceil(maxLagDays × dailyItemCount × safetyFactor) — the same inequality as horizonOk. This removes the pathology chat-Claude flagged: physics.gen-ph had horizonOk=true (3.33d ≥ 3d target) yet verdict=N/A because the provisional 50-item policy constant vetoed it. Now the gate is the derivation itself; the arc's 'placed number' count drops from {50, 3, 1.5} to {3, 1.5}. Fallback to the DEFAULT_DECLARED_RETENTION=50 remains only when no cadenceHint is available for a feed.", "STEP 1616 (chat-Claude 6th review): v0.5 adds `detectableLagDays` per feed when a cadenceHint is provided. detectableLagDays = actualRetention / (dailyItemCount × safetyFactor) — the gauge's own resolution declaration for this feed. `horizonOk` flags whether that resolution covers `maxLagDays`. This turns the two-value cliff ('50 met = judge / 50 not met = N/A') into a graded readout so N/A carries information ('this feed can see 1.6h at current retention; you would need X more IDs to reach the 3-day target'). Recommended usage: pass cadenceHint from measure-feed-cadence output so retention policy conversations start from feed-specific reality rather than a single guessed constant.", "STEP 1616: maxLagDays default 3 is REQUIREMENT-DERIVED (weekend-spanning cron gap coverage) and passes chat-Claude Q2. safetyFactor is upstream in cadenceHint; that constant is documented in measure-feed-cadence honestScope as the acknowledged residual arbitrariness after this arc (see chat-Claude 6th review: 'arbitrariness aggregates to a few named places rather than disappearing').", "arXiv ToU ≥ 3s/call. Runner rate-limits at 3.5s. Do not co-run with other arXiv callers.", "arXiv only in v0.3. Zenodo/HF/OpenAlex adapters that implement OracleWithIdList plug into the same gauge." ] }