4 ms·

I think you've really hit the mark on how the harness should be structured: 1. Guardrails - deterministic, social intelligence, team alignment & accountability

• polotics 1mo ago
  Hi. This is very interesting, could you link to the research? There is a dearth of proper research studies that A/B test what approach is best in terms of harness structure based on repeatable benchmark data with relevant sample uses-cases.
  • rush86999 1mo ago
    Reasoning / self-consistency (voter in core/llm/self_consistency_voter.py): - Wang et al. Self-Consistency Improves Chain-of-Thought — ICLR 2023, Google Brain, 4k+ cites — https://arxiv.org/abs/2203.11171 https://arxiv.org/abs/2203.11171 — N-sample majority vote we use verbatim - Chen et al. Universal Self-Consistency — ICML 2024 — https://arxiv.org/abs/2311.17311 https://arxiv.org/abs/2311.17311 — judge fallback when no hash collides - Soft Self-Consistency — ACL 2024 — https://aclanthology.org/2024.acl-short.28.pdf https://aclanthology.org/2024.acl-short.28.pdf - Too Consistent to Detect — EMNLP 2025 — https://aclanthology.org/2025.emnlp-main.238/ https://aclanthology.org/2025.emnlp-main.238/ — why SC doesn't fix systematic bias - Self-Consistency Falls Short — TACL — https://direct.mit.org/tacl/article/doi/10.1162/TACL.a.625/ https://direct.mit.org/tacl/article/doi/10.1162/TACL.a.625/ — position-bias failure mode Multi-agent / org (core/agent_radio/, core/fleet_orchestration/): - Stanford Virtual Biotech — bioRxiv 2026.02.23.707551, Zou Lab — https://www.biorxiv.org/content/10.64898/2026.02.23.707551v1 https://www.biorxiv.org/content/10.64898/2026.02.23.707551v1 — 37k agents, CSO->scientists->reviewer->re-delegation, Merck external validation of B7-H3 design. Basis for VFS + hierarchy. - Debate or Vote (Choi & Li) — NeurIPS 2025 — https://arxiv.org/abs/2508.17536 https://arxiv.org/abs/2508.17536 — MAD gains = majority vote, not debate (why we didn't build debate) Sandbox / eval: - DABstep — arXiv:2506.23719 — https://arxiv.org/abs/2506.23719 https://arxiv.org/abs/2506.23719 — 450 real Adyen tasks, justifies code-interpreter + sandbox isolation - Spotlighting — Microsoft Research — https://arxiv.org/abs/2403.14720 https://arxiv.org/abs/2403.14720 — provenance delimiters cut injection ASR 50% -> <2% Governance: - OWASP Top 10 for Agentic Applications 2026 — globally peer-reviewed by 100+ experts, Dec 2025 — https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/ https://genai.owasp.org/resource/owasp-top-10-for-agentic-ap... — HIGH. Atom maps 1:1 (Goal Hijack -> match-confidence + oracle, Tool Misuse -> sandbox whitelist/caps, Privilege Abuse -> capability bindings, Memory Poisoning -> verified-episode graduation, etc.) docs/marketing/RESEARCH_NOTES.md:130 - NIST AI Agent Standards Initiative — Feb 17 2026, NIST CAISI — https://www.nist.gov/artificial-intelligence/ai-agent-standards-initiative https://www.nist.gov/artificial-intelligence/ai-agent-standa... + RFI summary May 2026 https://www.nist.gov/publications/summary-analysis-responses-request-information-regarding-security-considerations-ai https://www.nist.gov/publications/summary-analysis-responses... — HIGH (US gov standard). Defines the 4 enterprise minimums Atom implements: identification, authorization, access delegation, logging. - Stanford Virtual Biotech — bioRxiv 2026.02.23.707551 — https://www.biorxiv.org/content/10.64898/2026.02.23.707551v1 https://www.biorxiv.org/content/10.64898/2026.02.23.707551v1 — CSO -> 4 divisions -> 8 scientists -> reviewer -> re-delegation, no debate, no SFT — HIGH (Stanford Zou lab + Merck external validation). Basis for Atom's fleet hierarchy core/agent_radio/ and why maturity is routing not security. - Spotlighting — Microsoft Research — https://arxiv.org/abs/2403.14720 https://arxiv.org/abs/2403.14720 — HIGH — provenance delimiters <provenance type="tool_output"> cut indirect injection ASR 50% -> <2%, used in core/provenance.py:10 - IntentGuard — https://arxiv.org/abs/2512.00966 https://arxiv.org/abs/2512.00966 + OpenReview — HIGH — intent tracing ASR 100% -> 8.5% on AgentDojo/Mind2Web, basis for sandbox egress allowlist + core/sandbox_tripwire.py
• stasomatic 1mo ago
  [flagged]