The paper proposes an intelligent assistive system for Deaf and Hard-of-Hearing users designed to transform acoustic information into a visually accessible form. The study is motivated by the growing prevalence of hearing impairment and by the limitations of conventional rehabilitation tools, which do not fully support the perception of complex acoustic environments. The system is built around a mobile architecture that combines ambient sound analysis, automatic speech recognition, and speech emotion analysis. To map sound to color, the paper introduces an auxiliary textual modality and performs semantic matching between sound descriptions and color descriptions. The experiments show that this approach produces stable sound–color pairs in HEX format that can be directly integrated into the system. For audio tagging, EfficientAT models were fine-tuned on the AudioSet-Strong dataset; the best performance was achieved with a 2.5-second analysis window and 1.25-second overlap, while the mn10_as model provided the best trade-off between accuracy and computational efficiency. For speech emotion visualization, the continuous Valence-Arousal-Dominance space proved more suitable than rigid categorical labels because it yields a more balanced and less subjective representation of affective states. The results support the use of multimodal learning, transfer learning, and compact neural architectures for practical assistive technologies for users with hearing loss.