Integrating Semantic Communication and Human Decision-Making into an End-to-End Sensing-Decision Framework

As early as 1949, Weaver defined communication in a very broad sense to include all procedures by which one mind or technical system can influence another, thus establishing the idea of semantic communication. With the recent success of machine learning in expert assistance systems where sensed information is wirelessly provided to a human to assist task execution, the need to design effective and efficient communications has become increasingly apparent. In particular, semantic communication aims to convey the meaning behind the sensed information relevant for Human Decision-Making (HDM). Regarding the interplay between semantic communication and HDM, many questions remain, such as how to model the entire end-to-end sensing-decision-making process, how to design semantic communication for the HDM and which information should be provided for HDM. To address these questions, we propose to integrate semantic communication and HDM into one probabilistic end-to-end sensing-decision framework that bridges communications and psychology. In our interdisciplinary framework, we model the human through a HDM process, allowing us to explore how feature extraction from semantic communication can best support HDM both in theory and in simulations. In this sense, our study reveals the fundamental design trade-off between maximizing the relevant semantic information and matching the cognitive capabilities of the HDM model. Our initial analysis shows how semantic communication can balance the level of detail with human cognitive capabilities while demanding less bandwidth, power, and latency.

💡 Research Summary

The paper addresses a pressing need in emerging 6G and AI‑native communication systems: how to convey not just raw sensor data but the meaning of that data in a way that supports human decision‑making (HDM) while respecting stringent wireless constraints on bandwidth, latency, and power. To this end, the authors propose a unified probabilistic end‑to‑end (E2E) sensing‑decision framework that explicitly couples semantic communication with a cognitive model of the human decision maker.

Framework Overview
The system is modeled as a Markov chain comprising six stages: (1) a semantic source z that encodes the task‑relevant state (e.g., “tool is worn” vs. “tool is healthy”), (2) a sensor observation s generated from z via a semantic channel p(s|z), (3) a semantic encoder pθ(x|s) that compresses s into a transmit vector x, (4) a physical wireless channel p(y|x) delivering y to the receiver, (5) a semantic decoder qφ(z|y) that reconstructs the latent meaning, and (6) a presentation layer p(ν|y) that formats the recovered meaning into a human‑readable stimulus ν (visual, auditory, or multimodal). Finally, the human applies a decision‑making model p̂(z|ν) to produce a judgment ẑ.

Design Objectives and Trade‑off
Two competing objectives are identified: (i) Information Maximization (InfoMax) – maximize the mutual information Iθ(z; y) between the semantic variable and the received signal, ensuring that as much relevant meaning as possible survives the wireless link; (ii) Cognitive Compatibility – limit the dimensionality, granularity, and presentation style of ν so that it fits the human’s working‑memory capacity, attention budget, and heuristic decision strategies (e.g., “take‑the‑best”). The authors formalize this as a multi‑objective optimization problem, either by weighted sum or Pareto frontier analysis.

Methodology
Building on prior work (e.g., Bao & Basu’s semantic information theory, the SINFONY system, and the Information Bottleneck framework), the encoder and decoder are trained with deep neural networks to solve the InfoMax problem. The presentation layer is separately optimized to align with a parametric HDM model derived from psychological literature (bounded rationality, heuristic use, probabilistic response variability). Alternating training loops allow the communication subsystem and the HDM model to co‑adapt.

Case Study & Results
The authors instantiate the framework with a tool‑wear categorization task. Sensors capture images of a tool from multiple viewpoints and short audio clips describing its condition. Feature extraction is performed at three compression levels (32, 64, 128 latent dimensions). Human decision‑making is simulated with a limited‑capacity “take‑the‑best” heuristic that can attend to at most three cues. Experiments vary (a) the amount of semantic information transmitted, and (b) the cognitive load imposed by the presentation. Key findings:

• When the presentation exceeds the human’s capacity, decision accuracy drops sharply even though the wireless link carries abundant semantic information.
• Matching the presentation detail to the cognitive budget (e.g., 64‑dimensional features presented as three salient cues) yields higher accuracy than transmitting maximal detail with an overloaded human.
• Under adverse conditions—limited prior experience or time pressure—the importance of cognitive compatibility grows, confirming the robustness of the trade‑off.
• Introducing stochastic noise into the HDM model (to mimic real‑world variability) reduces overall system performance, suggesting that end‑to‑end training must also account for human response variability.

Contributions

1. A general probabilistic E2E sensing‑decision framework that bridges communications theory and cognitive psychology.
2. Extension of the InfoMax design criterion to include presentation design and HDM model training, revealing a fundamental trade‑off between “maximizing relevant semantic information” and “matching human cognitive capabilities.”
3. Empirical evidence (via image and audio simulations) that appropriate feature‑level abstraction is more critical for decision accuracy than sheer information volume.
4. Demonstration that joint optimization of semantic communication and HDM can improve accuracy, albeit at the cost of additional alternating training iterations.
5. A roadmap for future work, including visualization‑centric presentation design, game‑theoretic analysis of sender‑receiver conflicts, and real‑time adaptive learning for dynamic environments.

Implications
The study moves semantic communication beyond machine‑to‑machine scenarios, positioning it as a core component of human‑centric assistance systems such as remote surgery, autonomous manufacturing, and disaster response. By quantifying how much and what kind of semantic detail should be delivered to a bounded human mind, the framework offers a principled path to design 6G‑enabled services that are both spectrally efficient and ergonomically effective. Future experimental validation with real users and hardware testbeds will be essential to translate these theoretical insights into practical standards and protocols.