Multimodal learning for mapping genotype–phenotype dynamics
Related Articles
Multimodal insights: enhancing cultural promotion through analysis of Saudi Arabian audiovisual productions
This research explores the application of Dicerto’s (2018) multimodal pragmatic model in analyzing Arabic audiovisual productions for translation purposes, focusing on enhancing cultural promotion. Employing a qualitative descriptive analysis approach, the study examines samples from Saudi productions that promote tourism, mainly focusing on Saudi coffee and its cultural traditions to enlighten foreign visitors about Saudi culture. The analysis reveals that Dicerto’s model provides a clear framework for achieving semantic fidelity in translation, ensuring that the translated text closely resembles its original in interpretative richness. Central to this framework is the principle of optimal relevance, wherein the sender intends the message to be maximally pertinent to the audience, thereby justifying the recipient’s cognitive effort in processing it and facilitating access to the sender’s intentions. This research sheds light on the effectiveness of applying multimodal analysis models in cultural promotion efforts through audiovisual productions, particularly in Saudi Arabian tourism promotion.
Innovating beyond electrophysiology through multimodal neural interfaces
Neural circuits distributed across different brain regions mediate how neural information is processed and integrated, resulting in complex cognitive capabilities and behaviour. To understand dynamics and interactions of neural circuits, it is crucial to capture the complete spectrum of neural activity, ranging from the fast action potentials of individual neurons to the population dynamics driven by slow brain-wide oscillations. In this Review, we discuss how advances in electrical and optical recording technologies, coupled with the emergence of machine learning methodologies, present a unique opportunity to unravel the complex dynamics of the brain. Although great progress has been made in both electrical and optical neural recording technologies, these alone fail to provide a comprehensive picture of the neuronal activity with high spatiotemporal resolution. To address this challenge, multimodal experiments integrating the complementary advantages of different techniques hold great promise. However, they are still hindered by the absence of multimodal data analysis methods capable of providing unified and interpretable explanations of the complex neural dynamics distinctly encoded in these modalities. Combining multimodal studies with advanced data analysis methods will offer novel perspectives to address unresolved questions in basic neuroscience and to develop treatments for various neurological disorders.
Predictive learning as the basis of the testing effect
A prominent learning phenomenon is the testing effect, meaning that testing enhances retention more than studying. Emergent frameworks propose fundamental (Hebbian and predictive) learning principles as its basis. Predictive learning posits that learning occurs based on the contrast (error) between a prediction and the feedback on that prediction (prediction error). Here, we propose that in testing (but not studying) scenarios, participants predict potential answers, and its contrast with the subsequent feedback yields a prediction error, which facilitates testing-based learning. To investigate this, we developed an associative memory network incorporating Hebbian and/or predictive learning, together with an experimental design where human participants studied or tested English-Swahili word pairs followed by recognition. Three behavioral experiments (N = 80, 81, 62) showed robust testing effects when feedback was provided. Model fitting (of 10 different models) suggested that only models incorporating predictive learning can account for the breadth of data associated with the testing effect. Our data and model suggest that predictive learning underlies the testing effect.
A multimodal neural signature of face processing in autism within the fusiform gyrus
Atypical face processing is commonly reported in autism. Its neural correlates have been explored extensively across single neuroimaging modalities within key regions of the face processing network, such as the fusiform gyrus (FFG). Nonetheless, it is poorly understood how variation in brain anatomy and function jointly impacts face processing and social functioning. Here we leveraged a large multimodal sample to study the cross-modal signature of face processing within the FFG across four imaging modalities (structural magnetic resonance imaging (MRI), resting-state functional magnetic resonance imaging, task-functional magnetic resonance imaging and electroencephalography) in 204 autistic and nonautistic individuals aged 7–30 years (case–control design). We combined two methodological innovations—normative modeling and linked independent component analysis—to integrate individual-level deviations across modalities and assessed how multimodal components differentiated groups and informed social functioning in autism. Groups differed significantly in a multimodal component driven by bilateral resting-state functional MRI, bilateral structure, right task-functional MRI and left electroencephalography loadings in face-selective and retinotopic FFG. Multimodal components outperformed unimodal ones in differentiating groups. In autistic individuals, multimodal components were associated with cognitive and clinical features linked to social, but not nonsocial, functioning. These findings underscore the importance of elucidating multimodal neural associations of social functioning in autism, offering potential for the identification of mechanistic and prognostic biomarkers.
Understanding learning through uncertainty and bias
Learning allows humans and other animals to make predictions about the environment that facilitate adaptive behavior. Casting learning as predictive inference can shed light on normative cognitive mechanisms that improve predictions under uncertainty. Drawing on normative learning models, we illustrate how learning should be adjusted to different sources of uncertainty, including perceptual uncertainty, risk, and uncertainty due to environmental changes. Such models explain many hallmarks of human learning in terms of specific statistical considerations that come into play when updating predictions under uncertainty. However, humans also display systematic learning biases that deviate from normative models, as studied in computational psychiatry. Some biases can be explained as normative inference conditioned on inaccurate prior assumptions about the environment, while others reflect approximations to Bayesian inference aimed at reducing cognitive demands. These biases offer insights into cognitive mechanisms underlying learning and how they might go awry in psychiatric illness.
Responses