4.12 Signifiers for Autonomous Agents: Perceiving and Acting in Affordance-Rich Hypermedia Environments

The Web’s immense capacity of interaction possibilities is both what empowers users and what imposes on them the challenge of handling and making the most of such abundance of heterogeneous resources. Since the 1960s, affordance theory [1] has influenced many applied fields in modeling and designing possibilities of action offered to people by virtual and physical resources11. The Web environment itself is instilled with such affordances due to the mechanism that lands at the very depths of its heart: hypermedia. Hypermedia binds together information and controls such that the information becomes the affordance through which people and automated clients (or even autonomous agents) obtain choices and engage in interactions12. However, as the Web expands its uniform interface to accommodate users with varying abilities (i.e. different classes of autonomous agents) and resources whose capabilities are exposed through different mediums (i.e. through physical, virtual or mixed reality interfaces), we need to further investigate the following: a) what type of information is required for describing interactions with heterogeneous resources, and b) when and how this information should be rendered with respect to user abilities and objectives. Specifically, for diminishing the gap between the objectives and actions of software agents, we take inspiration from human-computer interaction towards the design of a new class of information carriers: a new class of signifiers [3] for autonomous agents on the Web. The separation of concerns between affordances and signifiers reduces the coupling between the design of affordances and the design of perceptible information about affordances. This could be beneficial particularly for autonomous agents on the Web. Affordances express which agent abilities and artifact13 capabilities are compatible with each other and when. Affordances can be expressed through relationships that are checked at run-time based on the temporal complementarity of an agent’s abilities, an artifact’s capabilities, and their state within their shared environment. On the other hand, signifiers convey to an agent clear and unambiguous cues on how to exploit an emerging affordance and how relevant this interaction is for the agent (e.g. based on the agent’s intentions). By treating them as separate abstractions, affordances and signifiers can be modified and monitored independently by agents that have different interests and/or access rights (e.g. an agent may have the permission to update a signifier but not the related affordance).
Affordances and signifiers enable agents to pick up from their environment only the minimal information that is most relevant for interaction (i.e. principle of economical perception[1]), thus facilitating them to cope with large-scale environments. Specifically, agents’ percepts could be adjusted quantitavely and qualitatively through limiting the set of perceived signifiers to one that maps to currently exploitable and prioritized affordances. Furthermore, since exploiting an affordance may lead to the perception of information about a new set of exploitable affordances, agents are given the chance to progressively explore their environment based on their intentions and take advantage of newly-discovered action opportunities. This step-wise navigation decouples further the agents from their environment, allowing both to evolve independently at run-time. To this end, it is interesting to investigate the following:

How to model and represent affordances as relationships between autonomous agents and their hypermedia environment.
How to model and represent signifiers for autonomous agents in hypermedia environments.
How to enable autonomous agents, artifact designers and environment designers to publish, share and modify signifiers.
How to enable autonomous agents to perceive signifiers based on the principle of economical perception.
How to design mechanisms for dynamically adjusting the salience of signifiers such as to properly invite autonomous agents to interact.