Assay Project

Context

In 2026, AI agents have proliferated, shifting from experimental copilots into delegated software actors that can plan, invoke tools, access data, call APIs, execute code, and complete multi-step workflows on behalf of users, organizations, and, increasingly, other agents. Recent studies and surveys show agentic AI adoption is already widespread and moving quickly from experimentation to production:

• McKinsey's 2025 State of AI survey reported that 23% of organizations are already scaling agentic AI systems somewhere in the enterprise, with another 39% experimenting [1].
• A 2026 study by IBM and Oxford Economics surveyed 2,000 C-level technology executives, and organizational strain is already showing. A majority (77%) report that AI adoption is outpacing their current governance capabilities; nearly 60% cite security and compliance concerns as a top barrier to scaling agents; two-thirds say they are accountable for outcomes in systems they don't fully control; and 70% report teams deploying technology faster than IT can track [2].
• Public open-source agent projects are gaining traction just as fast. TradingAgents — an open-source multi-agent LLM trading system that models a financial trading firm with analyst, researcher, trader, risk-management, and portfolio-manager roles — grew from near-zero GitHub stars in early 2025 to more than 80,000 stars and 16,000 forks by June 2026. GitHub stars are not a proxy for production use or trustworthiness, but this growth illustrates how quickly multi-purpose agent systems can gain attention, be forked, and be adapted by users outside formal enterprise procurement or governance processes [3].

This growth creates a visibility and trust problem. Unlike traditional software components, agents are not defined only by their code dependencies or model selection. Their risk can vary widely depending on what tools they can invoke, what files and systems they can access, what credentials they use, what external destinations they contact, what actions they can perform without approval, and how their behavior is monitored. And as agents ship faster than ever, their public-facing documentation tends to emphasize capabilities and use cases while saying little about safety, guardrails, and risk management. Today, users have no quick, consistent way to evaluate an agent's underlying components and overall potential risk.

Scope

Assay evaluates public AI agent code repositories and produces evidence-based, transparent cards from observable repository artifacts, including source code, configuration files, dependency definitions, documentation, prompts, examples, and deployment files.

AI Agents

For purposes of this methodology, Assay treats an AI agent as a software system that uses an AI model to pursue a goal with a degree of autonomy (perceiving its context, deciding its own next steps, and using tools to act) rather than executing a fixed, predetermined script [4, 5, 6]. This synthesizes the converging industry definitions from Anthropic, Google Cloud, and AWS. In practice, this means Assay focuses on systems that can operate in a self-directed loop: planning, acting, observing results, adjusting, and continuing until the task is complete or human input or approval is required.

Assay does not evaluate the following:

• Frameworks, SDKs, and orchestration libraries such as LangChain, LangGraph, OpenAI Agents SDK, AutoGen, Semantic Kernel, LlamaIndex, or similar tooling when they provide building blocks for creating agents rather than a specific agent implementation.
• Simple model wrappers or chatbots that primarily pass user input to a model and return generated text without meaningful tool use, planning, or external system interaction.
• Deterministic automation scripts that execute fixed workflows without model-directed decisions, tool selection, or adaptive control flow.
• Model providers, model APIs, or model runtimes such as OpenAI, Anthropic, Gemini, Mistral, Ollama, or local inference runtimes when the repository only integrates with the model and does not define agent behavior.

Languages

Source-level analysis (the capability surface and everything derived from it) currently covers Python. This scope is data-driven, not incidental. A June 2026 census of the public agent population on GitHub (full search counts across five agent topics, ~70,000 repositories, cross-checked against a ~213,000-repository keyword frame) found that Python accounts for roughly 39–41% of agent repositories and TypeScript/JavaScript for a further 21–27%. Weighted by stars, a proxy for the agents users actually encounter, the two ecosystems together cover 80–92% of the population, while no other language exceeds roughly 3% [13]. TypeScript/JavaScript support is planned next.

For agents written in other languages, the deterministic sections (identity, maintenance, transparency, dependencies, adoption) are unaffected, because they come from GitHub's APIs and manifest parsing rather than source scanning. Fields the inspector cannot derive from source are recorded as null: unknown, never clean.

Analysis

Currently, Assay performs only static code analysis to determine the values on an Assay Card. It does not sandbox agents or test them at runtime. Runtime evaluation is part of our planned improvements.

Solution

The objective of the Assay project is to provide transparent, evidence-backed, data-driven information about public AI agents before they are run, integrated, or deployed.

Assay's purpose is to be a public, centralized registry of open-source AI agents. Each agent in the registry has an Assay Card: a structured JSON document derived primarily from the agent's public code repository. Publisher cooperation is not required to produce a card, though maintainers may later submit new or updated code, corrections, clarifications, or supplemental evidence.

An Assay Card summarizes what the repository appears capable of supporting, including agent purpose, autonomy, tool use, file access, execution behavior, outbound destinations, credential usage, data movement, persistence, and human approval requirements.

Assay is not a certification program or certifying body. A card and its content do not certify or guarantee that an agent is safe, secure, compliant, non-malicious, performant, or suitable for a specific deployment. It provides information that humans or agents can use to make their own decisions about whether to deploy or further evaluate the agent.

Core principles

Independence. An agent's assessment is determined by its code, not its publisher. No publisher can pay for, request, or opt out of a more favorable card, and Assay has no commercial relationship with the agents it assesses — the assessor is independent of the assessed.

Transparency. The methodology, the scoring criteria, the card schema, the category bank, the threat taxonomy, and the inspector engine are all open source. Anyone can review how a card was produced, challenge a classification, or propose an improvement.

Evidence over claims. Every field on an assay card is supported by observable evidence, deterministic analysis, or an explicit inference from code. When an LLM is used to produce a field, that is stated explicitly. Values are never fabricated or guessed; if the inspector cannot determine a field's value from available evidence, it records null.

Data-driven. From the frameworks we assess against (MITRE ATLAS, a maintained incident corpus, the category bank) to the signals on each card (taken from the strongest available source and applied uniformly to every agent), Assay's assessments follow real, structured data rather than opinion.

How a card is produced

Every part of a card is derived by a specific analysis step, and the steps run in order so later judgments can build on earlier facts. The table below maps each part of a card to how it is produced and what that derivation rests on — a deliberate control, so any value can be traced back to its basis.

Basis indicates how a value was established — and how to weigh it:

• Deterministic — computed from observable facts; the same inputs always produce the same value.
• Code-inferred — derived from static analysis of the source, with the evidence (file and line) recorded on the card.
• LLM-assisted — produced or labeled by a language model, and flagged as such so it can be weighed accordingly. These steps run at a fixed temperature for repeatability, and the exact model is recorded on each card.

Results are cached per repository and commit, so re-running on unchanged code reproduces the same card and avoids repeating language-model calls.

Assay Card Field Details

This section walks through each section of the card and provides additional context on what each field tells you, how it was derived, and how much to trust it.

Overview

The Overview section summarizes the repository's identity, public adoption signals, stated purpose, and assessed autonomy level. Its goal is to allow the user to quickly answer four questions:

• What is this agent?
• Who maintains it?
• What does it claim to do?
• How independently can it act?

Identity and Adoption

This section records the project's basic public metadata: repository name, owner, source URL, license, stars, forks, publisher type, and assessment date.

Name, owner, source URL, stars, and forks come directly from the GitHub repository API at the time of assessment. License is recorded using the Software Package Data Exchange (SPDX) identifier reported by GitHub, with a fallback to GitHub's license endpoint when needed. Assay then maps that SPDX identifier to two derived flags: whether the license is permissive (imposes no copyleft or share-alike obligation) and whether it allows commercial use. Unrecognized licenses mark the values as null.

The one-line project summary is created by an LLM as it reads the repository's README file and summarizes the project so a user can understand what this agent is used for.

Basis: Deterministic (GitHub repository metadata); LLM-assisted (README one-line summary).

Classification

Assay classifies each project into a primary domain and function, with optional secondary classifications when the repository clearly spans multiple use cases.

An LLM uses a combination of the data from the README, the code-detected orchestration framework, and the repository's GitHub topics to map the agent to the best fitting domain and function. Classifications are selected from an internally maintained category bank so cards are comparable and searchable across the ecosystem.

Basis: LLM-assisted (README and code review signals classified against an internal function and domain bank).

Intended use

The intended use field summarizes what the publisher says the project is for.

It is extracted by an LLM from the README and recorded as a publisher claim, not Assay's independent verification that the project does that thing, or does it safely. Stated limitations, exclusions, or target users are captured where the README describes them.

Basis: LLM-assisted (extraction from the README).

Autonomy

Autonomy is Assay's assessment of how much human oversight is required before the system produces a side effect. It is reported on a five-tier scale, with the evidence behind the tier recorded alongside it.

The scale begins with the research. Feng, McDonald & Zhang, Levels of Autonomy for AI Agents (arXiv:2506.12469) [7], characterizes autonomy by the human's control role (operator, collaborator, consultant, approver, observer), and Assay adopts its core axis: more agent autonomy means less human control. Assay does not score Feng's levels verbatim, because each is defined by the human's role in a live deployment, which a public repository cannot reveal (the same code can be run operator-style or observer-style). Assay scores only what the code shows. It therefore treats autonomy as a property of the artifact, derived from code-observable dimensions drawn from the system-level literature: the stage at which automation acts (Parasuraman, Sheridan & Wickens [9]); the dimensions of agency, namely directness of impact, goal-directedness, long-term planning, and underspecification (Chan et al. [8]); and of agenticness, namely autonomy, efficacy, goal complexity, and generality (Shavit et al., OpenAI [10]). Capability and autonomy are kept as separate axes that inform one another, following DeepMind's Levels of AGI [11]: what an agent can do (Capabilities) is distinct from how much it acts without a human (Autonomy).

The boundary that carries the most risk, and is clearest in code, is side-effecting execution rather than planning. A system can produce a sophisticated multi-step plan and still be Tier 2 if it only advises. This is the jump from Parasuraman's decision-selection stage to its action-implementation stage [9], and it anchors the T2/T3 split. Tier 5 (Self-modifying) is a distinct top, separating an agent that rewrites its own code, prompts, or tools from one that merely acts autonomously.

Rather than a single judgment, the tier is a transparent function over code-observable dimensions, each grounded in the literature or, where the research is silent, in an explicit Assay rationale:

Two of these dimensions are Assay's own, because no framework cited here isolates them. Initiation (trigger type) matters because an unattended trigger (cron, webhook, daemon, event) removes the human from the initiation loop entirely, which raises how autonomously an agent operates regardless of its per-action gating. Self-modification is a distinct top-tier cut: an agent that edits its own code, prompts, or tools, or spawns new agents, at runtime changes its own future behavior, a risk the oversight and capability ladders do not separate out.

The dimensions do not all weigh equally. A few gating dimensions place the tier. Whether the agent produces a side effect at all separates the advisory tiers (T1 and T2) from the acting tiers (T3 and up); whether those side effects sit behind a code-enforced approval gate separates acting with approval (T3) from acting autonomously (T4); and runtime self-modification marks the top tier (T5). The remaining dimensions (action reach, planning and looping, persistent memory, trigger type, goal openness, multi-agent structure) do not by themselves move the tier. They corroborate the placement, resolve ambiguous cases, and feed the rationale and integration_risk. Assay keeps this linkage at the level of principle rather than a rigid formula, so it stays durable as detection improves and agent patterns evolve.

The dimensions are read from the code where they are statically detectable, primarily the code-derived Surface (network, filesystem, code execution, credentials), which is trusted over any README claim, and combined into a tier by a transparent, documented rule rather than a black-box prompt. The same dimensions surface as the card booleans human_in_the_loop, acts_on_external_systems, and reversible. A language model is used only where a dimension is not reliably detectable from source, and to author the rationale that cites the specific evidence behind the tier. It never overrides a code-derived signal, and identical code yields an identical tier.

Assay does not assess runtime properties a public repository cannot reveal, such as granted IAM or OAuth scopes and production blast radius (Assay sees requested credentials, not what a deployed identity is granted), or whether decision-to-action logging is enabled.

Basis: Code-inferred (the dimension signals, from the Surface); LLM-assisted (judgment where a dimension is not fully determinable from source, and the rationale).

Integration risk

Integration risk is reported separately from the as-shipped tier: how easily the agent's output could be wrapped into higher-autonomy use. An agent can be Tier 2 (advisory) yet carry high integration risk if its output maps directly onto a real-world action, for example a trading-decision framework whose output can drive brokerage API calls. This mirrors the autonomy-versus-agency distinction [12]: what a system can do (agency) is distinct from how much it acts without a human (autonomy).

Basis: Code-inferred; LLM-assisted (code-derived surface and model judgment).

Threats

The card's structured fields describe what an agent can do. The Threats section names the adversarial techniques that capability exposes it to, using MITRE ATLAS [17] as the controlled vocabulary, and grades the evidence for each. A technique listed here means the agent's own capabilities make it applicable. It does not mean the agent is malicious, compromised, or vulnerable. The section turns the Surface findings into named, comparable risk vocabulary.

Each ATLAS technique maps to a rule over the card's own fields (threat_taxonomy.v0.1.yaml). When an agent's fields satisfy a rule, that technique is listed in threat_assessment.exposed_techniques. Each entry carries its ATLAS id and tactic, the card fields that triggered it (so the exposure is traceable to evidence, never asserted), and a maturity grade from ATLAS:

Assay maps only techniques visible in the agent's own surface (Execution, Exfiltration, Impact, Persistence, supply chain), not attacker-campaign tactics (Reconnaissance, Discovery, Credential Access), which describe a deployment rather than a repository. A rule over a null field never fires: the absence of an exposure means "undetermined," not "safe."

Incident counts per technique are joined live at serve time against the incident corpus, so the corpus grows without re-inspecting cards. Zero matched incidents still shows the ATLAS maturity grade, never "safe."

Basis: deterministic (capability predicates over the card's code-derived fields via the shared rule engine; technique names, tactics, and maturity from MITRE ATLAS [17]; incident counts joined at serve time).

Capabilities

The Capabilities section answers two distinct questions, kept deliberately separate because breadth is not danger: a platform that ships three hundred read-only connectors is not more dangerous than a bare agent with one shell tool. The two facets are never summed.

Tool verbs — the action space. The first facet inventories the primitive actions the agent's model can invoke, drawn from a fixed, versioned vocabulary so they are comparable across cards rather than free text:

The strongest evidence that the model can invoke a verb is a tool definition the model is given — a @tool/@mcp.tool decorator, a Tool(...)/FunctionTool(...) construction, or a framework code-execution tool class. These produce a high-confidence verb. A verb re-projected from an ordinary call site the agent performs (a subprocess.run, an exec) with no registration is recorded at low confidence, and the card distinguishes the two so a registered tool never reads the same as a bare inference. Each verb carries the actual exposed tool names and a {file, line, snippet} citation. A library merely being imported is never enough on its own — import smtplib is not send_message. read_file/write_file are emitted only when the model can request the file operation, never for an application reading its own configuration; ordinary application I/O belongs to Surface & Access.

Bundled extensions — the breadth. The second facet inventories the named, countable units the repository ships in the box: skills (.claude/skills/, .agents/skills/), MCP servers (config entries, FastMCP modules, or known server packages invoked via npx/uvx/pip), and plugins/connectors (e.g. activepieces packages/pieces/*). Names come from a manifest where one exists (a SKILL.md title, an mcp.json server name, a package.json/piece.json name) before falling back to the directory name. Each unit records a binding that preserves repo-shipped ≠ runtime-enabled — configured (declared in a runtime config) and present_on_disk (a matching unit with no wiring evidence) are not the same claim, so a repo carrying three hundred connectors never reads as an agent that invokes three hundred connectors. A large platform reports a count rather than flooding the card with every item. Where an MCP server's tools are statically detectable, they are enumerated on the unit. Each detected unit also carries code-derived identifiers (package, repository, install command) — the deterministic join key an external enrichment layer can match at serve time, without that volatile external data ever being frozen onto the card.

Tool-verb detection is an AST scan of Python source. On a non-Python agent an empty verb list means not assessed, not no tools — a coverage limitation disclosed alongside the finding, never read as a clean result. Bundled-extension detection is file-tree based and language-agnostic.

Basis: deterministic (closed-vocabulary tool verbs from tool-registration sites and code-execution call sites in the Python AST, with confidence and evidence; bundled extensions from the file tree and manifests, with binding and code-derived identifiers). No language model is used for either facet.

Surface & Access

The Surface & Access section describes the external capabilities Assay can observe from the repository: whether the project can call out to the network, read or write files, execute code or shell commands, and request credentials. This section is focused on capability exposure, not on whether the project is malicious or unsafe. A broad surface may be appropriate for some agents, but it should be visible to users, because these are the paths through which an agent can affect systems, move data, or depend on secrets.

Most findings here are inferred from source code, configuration patterns, and imported libraries. Where Assay cannot determine a capability from static analysis, it records the value as unknown rather than assuming the capability is absent.

The source-level scan is Python-AST only, so the section also carries a coverage disclosure (surface.analysis_coverage): how many Python files were scanned, how many failed to parse, and how many source files in other languages (TypeScript, JavaScript, Go, …) the scanner could not read. When that last count is non-zero the coverage is marked incomplete — on a non-Python agent an empty surface means not assessed, never clean.

Network

The Network subsection identifies whether the project appears capable of making outbound calls and, where possible, where those calls go. Outbound network is the channel through which an agent can move data off-device or act on the world. It is the precondition for the Exfiltration techniques in the threat layer (MITRE ATLAS) and the realization path for prompt-injection and sensitive-information-disclosure risk (OWASP Top 10 for LLM Applications, LLM01 and LLM02) [14][15].

Assay establishes outbound capability from code-level signals: imported HTTP and networking libraries (such as requests, httpx, aiohttp), AI-provider SDKs (such as OpenAI, Anthropic, AWS), and orchestration frameworks that wrap HTTP (such as LangChain, CrewAI). Destinations are then read from the strongest evidence available, in order: endpoints declared in configuration and manifest files (an http: or services: section, an MCP server entry, a base_url), URLs and host literals extracted from the source, and the default endpoints of detected SDKs. Each destination carries its provenance and, where it comes from code or config, a {file, line, snippet} citation. A language model is used only to normalize and label those observed destinations (canonical service name, purpose, required vs optional); it never invents a destination that has no code or config behind it. Required destinations are part of the core operating path; optional destinations are enabled through configuration, plugins, integrations, or specific user workflows.

Egress scope describes how constrained, and how attacker-influenceable, the outbound behavior appears. It is derived from the destinations and their provenance rather than guessed. The broader the scope, the larger the exfiltration channel and the greater the server-side request forgery (SSRF) surface; exfiltration through an agent's own egress after prompt injection has been demonstrated in the wild [15].

Assay does not treat the absence of an HTTP import as proof that network access is impossible. A project may still reach the network through shell commands, subprocesses, browser automation, plugins, or dependencies outside the scanned source, in which case makes_outbound_calls is recorded as unknown (null), never "none." A destination list is "what Assay could identify with evidence," not a guarantee of an exhaustive map.

Basis: code-inferred (capability, and destinations from code and config); LLM-assisted (labeling observed destinations).

Filesystem

The Filesystem subsection describes what parts of the local filesystem the agent reads from and writes to, and, as a privacy signal, whether it reads secret-bearing locations.

Assay records the paths the agent writes (generated output, caches, logs, workspace files) and reads, extracted from the source with a {file, line} citation, and groups them by reach.

The privacy-sensitive part is local reads of secret-bearing locations. Assay specifically flags reads of SSH keys, cloud credentials (AWS, GCP), Kubernetes and Docker config, GPG keys, .netrc, private keys and certificates, the macOS keychain, browser profiles and cookies, and system secrets such as /etc/passwd. These categories map to the credential-access techniques catalogued in MITRE ATT&CK, Unsecured Credentials (T1552) and Credentials from Password Stores (T1555) [16], so the taxonomy is grounded rather than ad hoc. Each sensitive read carries {category, file, line, snippet}, so the finding can be verified rather than trusted.

This matters because reads become materially more sensitive when combined with network egress. An agent that can read secrets and reach external destinations has a potential exfiltration path, even if the repository does not appear malicious. This read-then-exfiltrate pattern is the core privacy threat the Surface section exists to make visible (MITRE ATLAS Exfiltration; OWASP LLM02 Sensitive Information Disclosure [14]).

Scope describes how far the agent's filesystem access reaches, its blast radius, derived from the observed paths:

Basis: code-inferred (paths and sensitive reads extracted from source; scope derived from them).

Code execution

The Code execution subsection identifies whether the project runs shell commands or executes arbitrary code, tied to the concrete call site that does it.

Assay ties each execution finding to a call site in the source, never to README claims. It detects shell execution (subprocess, os.system, popen, execv*, and the shell=True argument) and arbitrary-code execution (exec, eval), and reports the surface as two capability signals, executes_shell_commands and executes_arbitrary_code. Each finding carries {kind, file, line, snippet}, so a reader can expand it to exactly what runs and where, distinguishing a narrow hardcoded command from broad execution of generated code.

Sandboxing is reported only on positive source-level evidence: a recognized sandboxing or isolated-runtime library (such as E2B, RestrictedPython, Pyodide, or a WASM/microVM runtime). On a match, sandboxed is true; otherwise it is left unknown (null), never false. Sandboxing is a runtime property, and the absence of a sandbox library does not prove code runs unsandboxed. Assay does not infer safety from a README that describes execution as "safe."

Basis: code-inferred from static analysis (call-site evidence; sandbox from library imports).

Credentials

The Credentials subsection lists the credential-like values the project appears to request: API keys, provider tokens, and other secret-bearing environment variables.

Assay identifies these from environment-variable reads in the source (os.environ[...], os.getenv(...)) whose names match well-known secret patterns (*_API_KEY, *_TOKEN, *SECRET*, *PASSWORD*, *ACCESS_KEY*, BEARER, CREDENTIAL). The result, credentials_required, is the set of those variable names. They represent what the project asks for or expects to use, not what a deployed identity is actually granted.

For example, detecting ANTHROPIC_API_KEY means the project expects access to Anthropic's API. It does not tell Assay what account, billing limits, organization permissions, or operational controls sit behind that key.

Basis: code-inferred (environment-variable reads matched by name pattern).

Supply Chain

The Supply Chain section summarizes what the repository is built with and what it depends on: its primary language, package ecosystem, declared dependencies, model providers, and a basic hygiene signal (whether it publishes an SBOM). It does not judge whether those dependencies are vulnerable or safe. Its purpose is to make the project's software composition visible: the ecosystem it belongs to, the packages it declares, and the third-party model services it appears designed to use.

Language

The primary language and the broader breakdown come from GitHub's language statistics, measured by bytes of code. This reflects the observed mix in the repository rather than a single file extension or a README claim. The primary language is the one with the largest share.

Package manager and dependencies

The package manager and dependency count come from standard manifests such as requirements.txt, pyproject.toml, package.json, go.mod, and Cargo.toml. The count reflects the project's declared direct dependencies, not the full resolved tree, since installing them may pull in many transitive packages. Key dependencies surface the packages that signal what the agent is: recognized orchestration frameworks and LLM-provider SDKs first, then the project's other distinctive dependencies, with generic infrastructure and utility packages filtered out (so backtrader and yfinance surface, not pandas).

Model providers

The model providers the project appears designed to use (such as OpenAI, Anthropic, or a local runtime) are detected from declared dependencies (provider SDKs and the per-provider langchain-* wrappers) and from provider-specific string references in the source (a provider registry, an llm_provider setting, an API base URL). Each is labeled hosted (a third-party API that may send prompts, files, or user data off-device) or local (runs on user-controlled hardware, though actual deployment still depends on configuration). A flexible flag marks agents that route across multiple providers (for example via LiteLLM). This indicates provider support or expected use, not which model runs on every path, what data is sent, or what account-level controls exist.

SBOM

Whether the repository ships a software bill of materials, a presence check by filename (a file named *sbom*, bom.json, or bom.xml). If none is found, it is recorded as not present. This is a presence check only: Assay does not validate the SBOM's completeness, freshness, or accuracy.

Basis: deterministic (GitHub language API; manifest parsing with rule-based dependency and provider classification; repository file checks).

Maintenance

The Maintenance section summarizes how active the repository appears to be: whether the project is actively changing, lightly maintained, dormant, or no longer maintained. These fields come from GitHub repository history, not README claims.

Maintenance is not a security rating. A frequently updated project is not automatically safe, and an older one is not automatically risky. The signal is about activity and upkeep.

Activity

First and last commit dates, total commits, contributor count, release count, latest release, and release cadence are pulled from GitHub's commit, contributor, and release data. Open issues and open pull requests are reported separately: because GitHub represents pull requests through the issues model, Assay subtracts the PR count so the issue count matches how a reader sees the repository's Issues tab. These values reflect the repository state at the time it is queried.

Status

A maintenance status is derived from the time since the last commit:

The status, and the days-since-last-commit it rests on, is time-relative and computed at serve time rather than frozen onto the card. It is evaluated against the current date, so a project that goes quiet moves from actively_maintained toward maintained, dormant, and abandoned as time passes, rather than showing a stale label from whenever it was inspected.

Basis: deterministic (GitHub commit, contributor, issue, pull request, and release history; status and days-since-last-commit recomputed at serve time from the last-commit date).

Transparency

The Transparency section records whether the repository provides common documentation, process, and safety artifacts. These signals help readers gauge how much context the project gives its users, contributors, and security researchers. They are presence checks, not quality judgments: a file being present means the project provides that artifact, not that it is complete, accurate, or strong.

For readability, Assay groups the signals into general project transparency (standard open-source hygiene) and agent transparency (disclosures especially relevant to AI agents). They are individual presence booleans on the card; the grouping is only how they are presented.

Project transparency

Standard open-source documentation and process signals: a README, changelog, contributing guide, code of conduct, CI pipeline, and Dockerfile, plus GitHub's own community_health_percentage (its repo-health score). Each is marked present when Assay finds it via GitHub's community-profile data, the Actions workflow count (for CI), or a repository file-tree check, so a contributing guide is caught whether it sits at the repository root, under docs/, or inside .github/.

Agent transparency

Disclosures that matter most for AI-agent systems: whether the repository ships examples, tests, published evaluation results, documented failure modes or limitations, and a security policy. These help a reader see whether the project explains how the agent behaves, how it has been tested, where it may fail, and how to report security issues. Evaluation results and failure modes are transparency signals, not guarantees: published evals may be narrow or outdated, and a documented-limitations list shows the project acknowledges failure modes without implying the list is exhaustive.

Basis: deterministic (GitHub community-profile and Actions data with a repository file-tree fallback, plus lightweight README content checks for evaluation results and failure modes).

Versioning

This methodology is versioned as a single document. All changes are recorded in the changelog below.

Disputes and corrections

If you believe a card contains an error, open an issue in the Assay repository with:

1. The card ID (card_id field)
2. The field(s) you believe are incorrect
3. Evidence from the repository supporting the correction

Publisher claim flows (where publishers can add attestations and correct factual errors) are planned for v2.

References

Sources cited in this document, in order of appearance:

1. McKinsey & Company. "The State of AI." QuantumBlack, 2025. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai
2. IBM Institute for Business Value. "Redefining the tech leader's mandate: Building the IT foundation for agentic AI at scale." 2026 Tech Leader Study (Global C-suite Series), 2026. https://www.ibm.com/downloads/documents/us-en/16ddce7b4954875d
3. TauricResearch. TradingAgents (GitHub repository). https://github.com/TauricResearch/TradingAgents
4. Anthropic. "Building Effective Agents." Anthropic, December 19, 2024. https://www.anthropic.com/engineering/building-effective-agents
5. Google Cloud. "What are AI agents?" https://cloud.google.com/discover/what-are-ai-agents
6. Amazon Web Services. "What is an AI agent?" https://aws.amazon.com/what-is/ai-agents/
7. K. J. Kevin Feng, David W. McDonald, Amy X. Zhang. "Levels of Autonomy for AI Agents." arXiv:2506.12469 [cs.HC], 2025. https://doi.org/10.48550/arXiv.2506.12469 — Defines autonomy by the human's control role (operator → collaborator → consultant → approver → observer); backs our oversight-based T1–T5 ladder and the principle that autonomy is distinct from capability.
8. Alan Chan, Rebecca Salganik, Alva Markelius, et al. "Harms from Increasingly Agentic Algorithmic Systems." ACM FAccT 2023. arXiv:2302.10329 [cs.AI]. https://doi.org/10.48550/arXiv.2302.10329 — Defines agency by four system properties — directness of impact, goal-directedness, long-term planning, underspecification; backs the code-observable autonomy dimensions and the artifact-level (not deployment-level) framing of the tier.
9. Raja Parasuraman, Thomas B. Sheridan, Christopher D. Wickens. "A Model for Types and Levels of Human Interaction with Automation." IEEE Transactions on Systems, Man, and Cybernetics — Part A, 30(3):286–297, 2000. https://doi.org/10.1109/3468.844354 — Locates automation across four function stages (information acquisition → analysis → decision selection → action implementation); backs our placement of the risk-bearing cut at the action-implementation stage (the T2/T3 side-effect boundary).
10. Yonadav Shavit, Sandhini Agarwal, Miles Brundage, et al. "Practices for Governing Agentic AI Systems." OpenAI, 2023. https://cdn.openai.com/papers/practices-for-governing-agentic-ai-systems.pdf — Characterizes agentic systems by degree of autonomy, efficacy, goal complexity, and generality; backs the action-reach, goal-openness, and orchestration dimensions.
11. Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, et al. "Levels of AGI for Operationalizing Progress on the Path to AGI." ICML 2024. arXiv:2311.02462 [cs.AI]. https://doi.org/10.48550/arXiv.2311.02462 — Treats capability and autonomy as distinct, correlated axes (higher autonomy "unlocked" by capability); backs keeping Capabilities and Autonomy as separate card sections that inform one another.
12. Damir Safin, Dian Balta. "Autonomy and Agency in Agentic AI: Architectural Tactics for Regulated Contexts." arXiv:2605.12105 [cs.AI], 2026. https://doi.org/10.48550/arXiv.2605.12105 — Separates agency (what a system can do) from autonomy (how much it acts without a human); backs our split of autonomy.tier (as-shipped oversight) from integration_risk (agency that is easily wrapped into higher autonomy).
13. Assay project. "Agent Population — Language Measurement." June 2026. docs/agent-population-2026-06.md — Census of agent-repository language distribution via GitHub search counts (topic, keyword, and star-weighted frames); backs the Python + TypeScript/JavaScript analysis-coverage scope. Reproducible via scripts/measure_agent_population.py.
14. OWASP. "OWASP Top 10 for Large Language Model Applications (2025)." OWASP Gen AI Security Project, 2025. https://genai.owasp.org/llm-top-10/ — Names the risks Assay's network surface is a precondition for: LLM01 Prompt Injection (the trigger), LLM02 Sensitive Information Disclosure (the data that leaves), LLM10 Unbounded Consumption (uncontrolled egress); backs surfacing outbound capability and egress scope.
15. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, et al. "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection." ACM AISec 2023. arXiv:2302.12173 [cs.CR]. https://doi.org/10.48550/arXiv.2302.12173 — Demonstrates end-to-end data exfiltration through an application's own outbound network after indirect prompt injection; backs why egress scope (listed → configurable → unrestricted) tracks the size and attacker-influenceability of the exfiltration channel.
16. MITRE. "ATT&CK — Unsecured Credentials (T1552); Credentials from Password Stores (T1555)." MITRE Corporation. https://attack.mitre.org/techniques/T1552/ · https://attack.mitre.org/techniques/T1555/ — Catalogs the credential-access techniques (credentials-in-files, private keys, keychain, browser-stored credentials) that Assay's filesystem sensitive_reads categories map to; grounds the sensitive-read taxonomy in the standard catalog.
17. MITRE. "ATLAS — Adversarial Threat Landscape for Artificial-Intelligence Systems." MITRE Corporation. https://atlas.mitre.org/ — The adversarial-technique catalog for AI systems; Assay uses its Agentic AI platform as the controlled vocabulary (technique ids, tactics, maturity grades) for the Threats section.