What becomes possible when agents can see, hear, and read beyond text

This blog is part of a series.

Read previous part: Small and Domain-Specific Language Models

Read next part: Scalable Agent Architectures

Most agentic systems are designed to operate in the world of text. They receive goals expressed in language, retrieve information from documents and databases, reason through problems in natural language, and produce outputs that are read by humans or consumed by other systems.

That is already a powerful capability. But organisations do not operate in a text-only world. They operate in a world of images, documents with complex layouts, audio recordings, video feeds, sensor data, structured forms, and visual interfaces. Much of the information that matters exists in these forms, not just in text.

An agent that can only process text is, in many contexts, working with one hand tied behind its back. A multimodal agent is one that can process and reason across multiple forms of input, not just text.

An agent that cannot perceive cannot fully act.

Multimodal agentic systems extend an agent’s perceptual capability. An agent that can see as well as read, interpret a chart as well as a spreadsheet, or process an audio recording as well as a transcript operates in a fundamentally richer context. This expands what is possible. It also introduces a new class of architectural challenge.

The case for multimodal agents

The value of multimodal capability becomes clear when considering the tasks that text-only agents handle poorly or cannot handle at all.

A claims processing agent that relies only on structured data misses the information contained in supporting documents, photographs, and handwritten notes. A compliance agent that can only read policy text cannot assess whether a physical environment or a visual output meets the required standard. A customer service agent that cannot interpret a screenshot cannot assist when the problem is visible but difficult to describe.  In each of these cases, the limitation is not reasoning. It is perception.

Multimodal capability extends the agent’s input to match the richness of the real-world context in which it operates. The resulting use cases span industries. In healthcare, agents that interpret medical imaging alongside clinical notes. In financial services, agents that process complex document layouts alongside structured data. In manufacturing, agents that monitor visual feeds and detect anomalies. In professional services, agents that work with the full range of materials used by human experts.

This is not a marginal extension of existing capability. It is access to a materially different class of tasks.

What changes architecturally

Extending an agentic system to handle multimodal inputs is not simply a matter of using a model that accepts images or audio. The architectural implications are broader.

• Perception and grounding
  Grounding becomes more complex when inputs span modalities. A multimodal agent must reconcile information that may be inconsistent or contradictory. An image and a text description of the same situation may tell different stories. The system must detect and handle these inconsistencies rather than silently favouring one input.
• Cross-modality consistency
  Consistency checks across modalities become essential. A conclusion supported by both visual and textual evidence is more reliable than one supported by only one. A conflict between modalities is not something to resolve automatically. It is a signal that requires investigation.
• Tool chain design
  Multimodal systems depend on a chain of components that acquire and process non-text inputs. Vision models interpret images. Speech systems convert audio. Document pipelines extract structure from complex layouts. Sensor integrations translate physical signals into usable data. Each component feeds the agent’s reasoning and must be designed, validated, and governed accordingly.
• Validation and testing
  Evaluation must extend beyond text quality. Systems must test visual interpretation, cross-modality consistency, and behaviour under ambiguous or degraded inputs. Text-only evaluation frameworks are not sufficient. Multimodal systems require their own testing discipline.

The failure modes specific to multimodal systems

Multimodal agents introduce new failure modes that require explicit design attention.

• Misinterpretation of visual inputs
  Vision models can misread text, misidentify objects, or misunderstand spatial relationships. These errors are often difficult to detect from the output alone. An agent acting on a misinterpreted image may take consequential actions based on incorrect perception.
• Modality imbalance
  An agent may over-rely on one modality. Strong textual context can override contradictory visual evidence. A compelling image can overshadow relevant written information. Without explicit design, the weighting of inputs becomes implicit and unreliable.
• Degraded input quality
  Real-world inputs are often imperfect. Images may be low resolution or obscured. Audio may be noisy. Documents may be distorted. The system must recognise degraded inputs and respond appropriately by seeking clarification, lowering confidence, or declining to act.

What good design looks like

A well-designed multimodal agentic system extends perception and validation together. The agent perceives more, and it verifies more. Cross-modality consistency is treated as a core architectural concern. The tool chain that processes non-text inputs is governed with the same rigour as the systems that support text reasoning. Validation is designed into the system from the start, not added later.

For organisations, the starting point is not capability but use case. Identify where multimodal input materially improves outcomes. Design for those cases specifically. Build validation and testing before expanding scope. Multimodal capability is not a feature to be enabled. It is an architectural extension that requires deliberate design, testing, and operational maturity.

The opportunity is real. Realising it responsibly requires applying the same discipline to perception that is applied to reasoning, action, and control.

Learn more about Bell’s AI Solutions.

Meet the Author:

Alison Hodivala
Director of AI & Data Portfolio Strategy at Bell Integration

Alison Hodivala is the Director of AI & Data Portfolio Strategy at Bell Integration, where she focuses on ensuring the company’s AI and data services reflect market needs and are supported by delivery models that scale reliably. She works across product, delivery and commercial teams to develop offerings that are practical, relevant and capable of delivering business impact.

Her career spans more than 25 years in start-ups and global organisations, covering consultative sales and delivery, enablement and change management. Drawing on a background in organisational psychology, Alison applies an evidence-based approach to AI adoption, aligning people, processes and technology to achieve effective and responsible outcomes.