A Failure-Mode Taxonomy for LLM-Agent Services: An Empirical Mining Study

Abstract

LLM-agent frameworks increasingly run as service components that call remote model providers and tool servers, yet service-oriented computing (SOC) lacks an empirically grounded vocabulary for how they fail. We present FMTAX , a taxonomy of 22 failure classes in seven groups, built by coding 720 closed GitHub issues from six repositories of four ecosystems (LangChain, LlamaIndex, MCP, CrewAI) and checked for coverage on 240 held-out issues from four further ecosystems. Labels from two rule coders are calibrated against a human-coded anchor (κ = 0.70) and an independent LLM annotator. Up to 40% of the issues are agent-specific; both references indicate that the rule coders over- rather than under-assign such codes. Across all raters, 66–82% of agent-specific failures arise at the agent’s service boundaries — the tool contract and the provider and tool-server links — led by tool-argument malformation and provider-error propagation. Ecosystems differ in their agent-specific share and in two classes that survive a like-for-like sampling check. Weighting a mapping from failure classes to SOC resilience patterns by observed prevalence, three boundary patterns (schema validation, error normalisation, timeout with retry) are candidate mitigations for 70% of agent-specific issues — a testable hypothesis for framework design. Corpus, codebook, and pipeline are released. Keywords: LLM agents, Service-oriented computing, Failure-mode taxonomy, Mining software repositories, Resilience patterns

Cite this paper

@inproceedings{trantruong2026fmtax,
  author    = {Tran-Truong, Phat T. and Le, Xuan-Bach and Ha, Xuan Son},
  title     = {A Failure-Mode Taxonomy for {LLM}-Agent Services: An Empirical Mining Study},
  booktitle = {Proceedings of the 24th International Conference on Service-Oriented Computing (ICSOC 2026)},
  year      = {2026},
  series    = {Lecture Notes in Computer Science},
  publisher = {Springer},
  address   = {Lodz, Poland},
  note      = {Artifact: \url{https://github.com/LeoTTTPhat/FMTAX}}
}