Innovating beyond electrophysiology through multimodal neural interfaces

Introduction

Neural circuits exhibit dynamics that range from fast electrical and chemical signalling at the subcellular level to brain-wide rhythms that coordinate the activity of various brain regions^1,2,3. To interrogate neural circuits at increasingly larger scales, the neural engineering community has been developing technologies capable of recording and stimulating tens to hundreds of thousands of neurons simultaneously^4,5. Among several key sensing approaches, including functional magnetic resonance imaging (fMRI)⁶, positron emission tomography⁷, functional near-infrared spectroscopy (fNIRS)⁸ and magnetoencephalography (MEG)⁹, electrical¹⁰ and optical^11,12,13 methods have emerged as particularly powerful and versatile tools for brain studies thanks to their ability to record neural activity with high temporal or spatial resolution (Fig. 1). From the first technologies developed in the 1960s, the growth in the number of simultaneously recorded neurons using electrical^{14,15,16,17,18,19,20} and optical^21,22 neural recording technologies (Fig. 2) resembles the exponential trend characterizing Moore’s law, the driving force for semiconductor electronics in the past five decades^23,24. In addition to the ever-increasing number of recording sites in electrical neural interfaces^25,26, advances in optical microscopy and fluorescence imaging techniques have led to even steeper progress in the number of simultaneously recorded neurons in large neuronal populations, enabling studies of complex local and global neural dynamics happening at large scales across the brain⁵.

Innovating beyond electrophysiology through multimodal neural interfaces — **Fig. 1: Spatiotemporal resolution of existing neural recording techniques.**

**Fig. 2: Timeline of neural recording techniques.**

Although optical methods have enabled imaging dynamic activity from thousands of neurons — and even millions at lower sampling rates (~2 Hz)⁵— the need for electrical recordings cannot be eliminated. This necessity arises from the fact that, unlike electrical probes, which can access any point inside the brain, optical methods are fundamentally constrained by light scattering and absorption that limit their access to deep layers of the brain^13,27. Moreover, electrical probes can sample neural activity across multiple brain areas in freely moving animals continuously for many hours²⁵, whereas high-resolution optical recordings captured by scanning multiphoton microscopes are generally confined to head-fixed configurations and, typically, provide a maximum duration of approximately 30 min–2 h due to photobleaching of optical indicators^11,28,29. Miniscope³⁰ and lensless³¹ imaging systems can alleviate the head fixation problem, yet their resolution and depth are limited due to light absorption and scattering. Temporal resolution of new voltage imaging indicators has been progressing, achieving frame acquisition rates up to 1 kHz (ref. ³²); however, electrical recordings still provide superior temporal resolution, exceeding 20 kHz, enabling precise measurements of spiking dynamics. Combining small field of view (FOV), high-resolution imaging (such as two-photon and confocal imaging) with low-resolution wide-field optical microscopy in the same set-up is technically challenging as these two disparate imaging modalities have different requirements of the apparatus set-up (filters, objectives, working distances, scanning modes).

Multimodal experiments — defined as the integration of distinct recording techniques within a single experimental set-up to leverage the complementary strengths and mitigate the limitations of each technology — offer substantial potential to enhance spatiotemporal resolution. The integration of different recording techniques enables precise interrogation and modulation of neuronal activity, facilitating a comprehensive understanding of complex neural dynamics. Conventional electrical neural interface technologies, however, are not compatible with multimodal experiments as they are mostly rigid, opaque and challenging to combine with optical imaging set-ups. A seamless integration of multiple modalities requires transparent, flexible and crosstalk-free electrical neural recording devices compatible with optical imaging techniques. Flexibility facilitates the implantation and integration with various microscopy set-ups, whereas transparency and the elimination of crosstalk are crucial for allowing simultaneous electrical and optical recordings without interference from either modality. With the proper toolbox and techniques offering high spatiotemporal resolution, such multimodal experiments have the potential to enhance our understanding of brain function and to inform practical applications in areas such as neuroprosthetics and the pathophysiology of brain disorders.

In this Review, we explore those electrical neural probes that are compatible with optical imaging and discuss various optical neural recording and stimulation technologies. We also delve into multimodal experiments elucidating their key findings and addressing challenges related to signal processing and data analysis for large and complex multimodal neural datasets. Finally, we discuss the future of multimodal experiments and their impact on the field of neuroscience and medical practice.

Electrophysiology techniques for multimodal set-ups

Comprising billions of cells, the human brain is recognized as the most sophisticated system known to humanity¹. It contains an approximately equal number of neuronal and non-neuronal cells — glia, epithelial cells, pericytes and endothelia cells — which form complex neural networks^33,34,35, essential for processing of sensory information and executing diverse cognitive functions^36,37. These neural circuits exhibit electrical activity at multiple scales across various brain regions^2,3,38. Since the 1930s, electrophysiology has been the backbone of neuroscience in the study of neural dynamics with high temporal resolution, to capture fast action potentials³⁸ as well as slower local field potentials (LFPs) generated by neural populations³⁹.

Opaque and rigid neural interfaces

Conventional rigid probes and microwires have been the workhorse for electrophysiology and are widely used in numerous studies to record action potentials of neurons from different layers of the brain^15,40,41. At the same time, electrical neural interfaces have been evolving constantly to enable recordings from a larger population of neurons, capturing the activity of hundreds of neurons^42,43,44. For example, advanced silicon probes such as Neuropixels^19,26, Neuronexus SiNAPS⁴⁵ — characterized by hundreds of microscale recording sites each measuring less than 400 µm², evenly distributed over single or multiple (up to 4) shanks — have enabled neural recordings from different regions and layers of the brain^26,46. The long shanks of these probes are connected to integrated complementary metal-oxide semiconductor chips that enable on-board signal processing, conditioning and amplification to realize robust recording of action potentials from cortical and subcortical regions in different animal models⁴⁷ and even in humans^48,49. While high-density silicon probes excel in neural recording across various brain layers, neural probes such as Utah⁵⁰ and Argo²⁰ arrays offer a unique perspective on neural activity. These intracortical microelectrode arrays feature tens to hundreds of single-channel microelectrodes distributed on a 2D plane enabling them to record and stimulate neural populations with single-cell resolution across an expansive cortical area, span up to 12 mm × 12 mm. This ability has notably contributed to the field of brain–computer interfaces (BCIs) and has initiated a transformative era in neural prosthetics^51,52. Although these probes have contributed to our understanding of the brain and led to clinical trials for future improvement of life quality for patients⁵³, they are not suitable for integration into multimodal experiments. The main reasons are their rigidity and opacity, which hinder their use with optical imaging set-ups by obstructing the FOV, and their susceptibility to generating light-induced artefacts^54,55,56.

Flexible neural interfaces

Advances in microfabrication technologies and the introduction of new materials (such as flexible polymers) have revolutionized the field of neural engineering, leading to the development of a new generation of neural interfaces⁵⁷. The use of flexible materials in the fabrication of neural interfaces substantially improves mechanical compliance, which is essential not only for reducing potential damage during implantation but also for stable long-term neural recordings^57,58. Flexible polymers such as polyimide (PI)^59,60,61, parylene⁶², polyethylene terephthalate (PET)⁶³, SU-8 (ref. ⁶⁴), silicone⁶⁵ and polydimethylsiloxane (PDMS)⁶⁶ have been tested as substrates and encapsulation layers for flexible probes. These materials offer flexibility, with bending stiffness values that are orders of magnitude lower than those of conventional stiff materials such as silicon, given comparable geometrical dimensions⁴³. This substantial enhancement in mechanical compliance and conformality of neural probes is crucial for reliable long-term recordings and minimizing tissue damage^{67,68,69,70,71}. The incorporation of flexible materials into the fabrication of laminar probes represents a paradigm shift, transitioning from conventional rigid silicon probes to soft, flexible, single-shank and multi-shank intracortical neural interfaces^25,72,73 (Fig. 3Aa–c), enabling chronic experiments with high fidelity⁷⁴. Building on these advancements, active neural probes with integrated microtransistors have been developed to perform local signal amplification and achieve superior signal-to-noise ratio compared with conventional passive electrodes^59,75. These probes are capable of recording ultra-low-frequency oscillations that play an important role in the synchronization of different brain regions and have been crucial for various studies on pathological conditions and functional connectivity^76,77.

**Fig. 3: Electrical neural probes compatible with optical imaging methods.**

Transparent and flexible neural interfaces

Transparency of the neural probes is important to ensure seamless integration of electrophysiological recordings with optical imaging techniques. The diverse choice of transparent and flexible materials for the substrate, encapsulation, interconnects and electrodes has led to a new generation of planar electrodes that can be placed on the brain’s surface (Fig. 3B). Conventional planar probes typically use metal layers for electrodes and interconnects as they do not require optical transparency. However, it is possible to modify the metal structures to improve their transparency while still maintaining high conductivity^78,79. Various semi-transparent neural probes have been developed, including those based on metal nanowires and meshes^{80,81,82,83,84} (Fig. 3Ba). Although these technologies have shown great potential for multimodal experiments, issues with transparency and light-induced artefacts remain to be fully addressed^85,86.

Building on the quest for improved transparency in planar neural interfaces, researchers have shifted their focus to replacing metals with other conductive materials. For example, transparent conductive oxides, typically found in display panels and solar cells⁸⁷, can be employed to develop completely transparent neural devices⁸⁸. Among these materials, indium tin oxide (ITO) has been used to develop micro-electrocorticography arrays that can record neural activity from the surface of the brain^89,90 (Fig. 3Bb). Although these materials offer excellent transparency, their mechanical brittleness poses challenges for long-term, chronic experiments^91,92. Overcoming the flexibility constraints of transparent conductive oxides can enable flexible, biocompatible and transparent neural probes suitable for long-term chronic experiments.

Conductive polymers have also emerged as promising materials in the development of neural probes. A prominent example is poly(3,4-ethylenedioxythiophene) polystyrene sulfonate (PEDOT:PSS), which has been widely employed to realize transparent neural electrodes^88,93 (Fig. 3Bc). Although PEDOT:PSS offers good conductivity and transparency, improvements in its chemical and mechanical stability are necessary for long-term chronic experiments⁹⁴. Carbon-based materials, notably graphene, have been extensively used to develop various types of neural electrodes^59,66,95,96. Monolayer graphene, with its sub-nanometre thickness, offers unique mechanical⁹⁷, electrical⁹⁸ and optical⁹⁹ properties, making it a suitable choice for transparent neural interfaces. These ultrathin graphene layers are grown using chemical vapour deposition and transferred to target substrates via electrochemical delamination methods¹⁰⁰. Microelectrode arrays developed using single or multiple sheets of graphene provide artefact-free recording capability^{55,56,95,101,102,103}, biocompatibility^104,105 and chronic reliability^63,105 (Fig. 3Bd). Carbon nanotubes (CNTs) are rolled up sheets of graphene that offer excellent electrical conductivity, transparency and flexibility, making them ideal for neural electrode fabrication⁶⁶ (Fig. 3Be). However, concerns have been raised regarding the cytotoxicity and biocompatibility of CNTs^106,107.

Miniaturization and scaling challenges

Despite the unique designs and characteristics and the impressive properties of planar and laminar neural recording devices that have been used in multimodal experiments (Supplementary Table 1), there is room for improvement in areas such as transparency, flexibility, crosstalk, conductivity, channel count and electrode size. For example, most transparent planar arrays have limited channel counts (<64) and large electrodes (>300 µm²), which substantially limit their recording capabilities. Continuous efforts in the field of neural engineering are focused on developing new methods to overcome these limitations. For example, multilayer graphene sheets combined with interlayer doping and platinum nanoparticle deposition have shown promise in achieving high-density, large-area transparent graphene arrays with small electrodes¹⁰². Moreover, issues such as biocompatibility and longevity of these technologies require thorough investigation to ensure their effective function in chronic experiments and future BCIs. Addressing these limitations is crucial for achieving electrical recordings with enhanced fidelity and higher spatial resolution, fundamental for studying neural circuits and understanding brain functions.

In addition to miniaturization challenges, scaling neural interfaces to achieve whole-brain coverage presents additional problems related to connectors and packaging. To achieve high-density chronic recordings with high-channel-count neural probes that cover larger areas of the brain, it is essential to reduce the number of wires connecting to the data acquisition system. Developing fine-pitch bonding methods between amplifier chips and flexible substrates is crucial for miniaturizing head stages while providing high-bandwidth connectivity¹⁰⁸. Custom neural interface system-on-chip and application-specific integrated circuits with ultra-low noise and power are necessary to ensure high-precision signal integrity with minimal form factors (size, weight and power). Integrating these low-power, low-noise neural signal conditioning and acquisition chips with high-density arrays would enable high-bandwidth recording within compact form factors, making them suitable for head-mounted operation in chronic recordings.

Electrical signals recorded by laminar and planar neural interfaces

The electrical signals recorded by laminar and planar neural interfaces comprise a broad spectrum of low-frequency and high-frequency components, each of which offers unique insights into neural dynamics³⁹ (Fig. 4). These interfaces record extracellular potentials and currents that are generated by the superposition of different mechanisms including synaptic currents, action potentials, calcium spikes, and other transmembrane currents, oscillations and ionic fluxes in the extracellular fluid³⁹. Neural signals are categorized into the electroencephalogram, electrocorticogram and LFP, depending on the recording method, although they are all generated by the same underlying biophysical processes³⁹. The frequency spectrum and specificity of the recorded signals depend on the size, geometry and location of the electrode array (for example, macroelectrodes versus microelectrodes and planar versus laminar arrays). Conventional clinical recording technologies, such as electroencephalography (EEG) and electrocorticography, utilize millimetre-scale electrodes that are placed on the scalp surface or implanted epidurally or subdurally on the brain surface. These macroelectrodes capture filtered and spatially averaged versions of extracellular potentials after their transmission through the intermediate brain tissue and the skull, in the case of EEG¹⁰⁹. By contrast, laminar probes equipped with microscale recording sites, placed in closer proximity to neural populations, can capture low-frequency and high-frequency activity including the LFP, single-unit activity (SUA) and multi-unit activity (MUA) (Fig. 4a). LFPs include various frequency bands such as delta (1–4 Hz), theta (3–10 Hz), alpha/beta (8–30 Hz), gamma (30–80 Hz) and fast oscillations with dynamics up to 200 Hz, with slight variations in these ranges across different species¹¹⁰. SUA refers to action potentials generated by single neurons and is characterized by a brief and rapid (sub-millisecond) exchange of ions across the neuron membranes. MUA represents spiking activity that stems from multiple neurons. Spike sorting methods are used to categorize neural recordings as SUA, when traceable to individual neurons, or as MUA, when originating from multiple neurons¹¹¹. To accurately detect these high-frequency components in the extracellular potentials, small electrodes with low impedances are essential to minimize spatial averaging, which can otherwise filter out these fast spikes^112,113. For example, planar surface arrays with low-impedance microscale electrodes can record high-frequency LFP and MUA spikes with high fidelity (Fig. 4b), whereas this is not possible with millimetre-scale electrocorticography electrodes^62,102. Additionally, the location of the implanted neural interface plays an important role in capturing specific neural dynamics. For example, probes positioned in the hippocampus can detect sharp wave ripples (SWRs) — the high-frequency activity (100–250 Hz) that typically occurs during rest or sleep (Fig. 4c). These fast oscillations, which are generated by the synchronous activity of excitatory and inhibitory neurons, play an important role in memory formation and retrieval^114,115. Furthermore, both planar and multi-shank laminar arrays are effective in identifying various types of travelling waves (Fig. 4b). These waves are crucial for the coordination of information transfer across different brain regions, highlighting their importance in neural communication and function^63,116,117.

**Fig. 4: Neural signals captured by laminar and planar microelectrode arrays.**

Optical recording and stimulation methods

Although neural communication primarily occurs through electrical signals, optical methods have been introduced to non-electrically capture neural dynamics. These methods rely on a diverse range of direct or indirect measurement of neural activity. For example, calcium-binding proteins respond to changes in intracellular calcium concentrations that occur during an action potential. More direct approaches such as voltage indicators aim to directly capture fast neural dynamics and even subthreshold potentials. Although most of these methods are not fully non-invasive and require procedures such as thinned skull or cranial window installation, they generally cause less tissue disruption compared with electrophysiology techniques. Furthermore, the ability to dynamically select recording sites without direct contact with the neural tissue offers substantial advantages, including reduced impact on the surrounding areas and more precise monitoring of neural activity^118,119. Combining these optical approaches with electrical neural probes has propelled multimodal neurophysiology forward, enhancing both spatial and temporal resolutions^{120,121,122,123}.

Optogenetics

The field of optical neurophysiology experienced a marked breakthrough with the advent of optogenetics^124,125. This technique, which involves transfecting microbial opsin-encoding genes into neuronal cells, enables the expression of light-sensitive opsin proteins, allowing for light-activated neural stimulation. Since its initial demonstration, the tools in the optogenetic arsenal have undergone continuous refinement and expansion. The development of techniques for delivering light beyond the brain’s surface has been a pivotal advancement in transforming optical neuron manipulation into a more practical tool for neurophysiology^126,127. However, unlike electrodes, which can be inserted into deep brain regions with relative ease, deep penetration of light into the brain is hampered by absorption and scattering^13,27 (Fig. 5a). For example, blue light — approximately 480 nm in wavelength and commonly used to excite channelrhodopsin-2 — is limited to penetration depths of only a few hundred micrometres into the brain tissue.

**Fig. 5: Latest developments in optical neurophysiology.**

A potential solution for deep brain light delivery involves inserting an optical fibre or an on-chip waveguide into the brain^125,128. However, this approach limits the spatial flexibility of optogenetic excitation and results in increased invasiveness¹²⁹. Without sacrificing the advantages of optogenetic methods, multiphoton (two-photon and three-photon) excitation achieves deeper tissue penetration by using longer-wavelength near-infrared lasers, which are less prone to scattering and absorption in biological tissues (Fig. 5b). Furthermore, nanoparticles capable of up-converting near-infrared light into blue light on site can be used to achieve neuron stimulation more efficiently than multiphoton excitation methods (Fig. 5c). For example, lanthanide-doped up-converting nanoparticles enabled optogenetic excitation of neurons up to 4.5 mm deep in mouse brains, using a near-infrared light source positioned outside the mouse skull¹³⁰.

Precise spatial and temporal control of excitation light delivered to neurons can be achieved with multiphoton holographic optogenetics (Fig. 5d), which permits temporally accurate simultaneous excitation of target neurons in a specific structured pattern¹³¹. This precise control of excitation patterns is essential for multimodal experiments, as it enables selective and targeted activation of neurons to interrogate neural circuits¹³². When combined with optically transparent electrodes in multimodal experiments, the precise optical control over neuron excitation in both space and time can enable in-depth studies on causal relationships in the brain. Moreover, precise temporal control of neuron excitation can facilitate closed-loop neural experiments, where stimuli (either electrical or optical) are automatically adjusted based on the difference between desired and actual target outputs (in terms of behavioural, electrophysiological or optical neural readouts)¹³³. In closed-loop experiments, the spatiotemporal variation of optical or electrical excitations — determined by real-time monitoring and processing of neural recordings — can be used to induce physiological or behavioural activities in a feedback-controlled manner. In this context, closed-loop optogenetics within multimodal experiments has been particularly promising in exploring oscillatory neural behaviour in the brain. For example, leveraging the closed-loop optogenetics paradigm, the connection between memory and various low-frequency brain oscillations such as theta (4–12 Hz)¹³⁴ and gamma (30–110 Hz) cycles^135,136 have been investigated.

Optical voltage imaging of neural activity

The fundamental challenge in all-optical neural voltage recording is how to efficiently transduce electrical activity into optical signals. The two types of widely adopted indicators for optical neural recording are the calcium fluorescent indicators and the fluorescent voltage-sensitive indicators. Calcium indicators, which can be either synthetic fluorescent molecules or genetically encoded fluorescent proteins, alter their fluorescence properties upon binding to calcium ions¹¹. During neuronal activation, the influx of calcium ions through voltage-dependent channels on the neuron membrane enable the transduction of neuronal electrical activity into an optical signal as a change in fluorescence (Fig. 5e). Although calcium indicators capture neuronal dynamics in large populations of neurons, they have limited temporal resolution due to the inherently slow dynamics of calcium ions and offer only a partial view of the neuron’s polarization state^137,138. Consequently, they fail to capture fast spiking activities and subthreshold neural dynamics that do not lead to action potentials.

These limitations have been addressed with the advent of genetically encoded voltage indicators (GEVIs)^12,137,139. GEVI-based fluorescent imaging enables high spatiotemporal resolution recordings of neuronal activities. Compared with calcium indicators, which are typically sampled at rates up to 100 Hz (ref. ¹⁴⁰), GEVIs offer much higher temporal resolutions, up to 1 kHz, which is essential to monitor fast neural dynamics^{141,142,143,144,145}. GEVIs directly report membrane potentials, whereas calcium indicators are secondary messengers, reflecting changes in intracellular calcium concentrations. Consequently, GEVIs can capture more detailed signals from the nervous system, including subthreshold depolarizing (Fig. 5f) and hyperpolarizing signals¹³⁷. Despite these advantages, several challenges persist in achieving population-scale voltage recordings with GEVIs. As GEVIs bind to cell membranes, their total concentration is limited compared with calcium indicators. Furthermore, conventional microscopy set-ups do not typically offer the high frame rates (that is, 1,000 fps) required for fast GEVIs, limiting their potential utility in large population imaging studies. However, ongoing efforts are aimed at improving the efficacy of these indicators. In 2023, for example, a high-speed, positive-going GEVI (a type of voltage indicator that exhibits increased fluorescent intensity upon membrane depolarization and a rise in intracellular potential) compatible with two-photon imaging was used to simultaneously record the activity of more than 100 densely labelled neurons in awake, behaving mice with high temporal resolution for durations extending up to 1 h (ref. ¹⁴⁶). Reducing both photobleaching of the indicator and photodamage to neurons, which are crucial factors for long-term neuronal voltage imaging, this approach demonstrated superior performance in low-light conditions compared with conventional negative-going GEVIs.

Whereas both calcium and GEVI-based imaging techniques require the introduction of exogenous indicators into neurons, advancements in the field have led to the development of several label-free neural recording techniques. These innovative approaches have the potential to enable optical electrophysiology without any modifications to the neurons. A particularly promising strategy for label-free neuron imaging involves exploiting the capacitive electric field within the electric double layer (EDL) of a sensing surface that is in close proximity to neurons (Fig. 5g). For example, a seminal work from 2009 demonstrated how localized surface plasmon resonance of patterned gold nanoparticles can be used to optically detect neural activity¹⁴⁷. This method relies on the plasmonic resonance shift induced by the electric field of the EDL when the neuron fires. Furthermore, electric field-sensitive electrochromic materials such as graphene and PEDOT:PSS can also be employed to enhance the transduction of neural electrical activity into optical signals through the change of dielectric permittivity induced by the changing electric fields within the EDL. For example, by coating a thin layer of PEDOT:PSS film onto plasmonic structures, the electric field sensitivity was enhanced by three orders of magnitude compared with bare plasmonic structures, enabling label-free detection of electrophysiology signals from cardiomyocytes¹⁴⁸. Using the electrochromic properties of PEDOT:PSS film, spontaneous action potentials from cultured hippocampal and dorsal root ganglion neurons, as well as brain slices, were captured¹⁴⁹. In another study, a graphene-based voltage imaging platform has been demonstrated to image electric field dynamics with a temporal resolution comparable with electrophysiology¹⁵⁰. This platform has also facilitated recording of spatiotemporal dynamics of extracellular electrical activity in cardiomyocytes¹⁵¹, underscoring its potential for label-free all-optical electrophysiology. Integrating these label-free optical electrophysiology techniques with flexible and transparent probes could enable label-free multimodal brain interfaces that do not require external modification to the neurons. Moreover, these interfaces are not subject to photobleaching or phototoxicity^149,152, potentially enabling long-term, continuous multimodal electrophysiological recording.

Multimodal experiments with integrated optical and electrical neural interfaces

Neural recording technologies with multimodal capabilities enable a broader scope of experiments, aimed at deepening our understanding of brain functions by combining the complementary advantages of various modalities.

Multimodal experiment with laminar probes

Conventional penetrating laminar electrodes have been widely used in animal studies to record electrical activity from deep cortical and subcortical layers, but their inherent mechanical rigidity and non-transparent nature are major drawbacks. Nevertheless, they have been successfully employed in several multimodal experiments. Researchers have meticulously managed the implantation process to minimize the probe’s interference with the FOV and to reduce light-induced artefacts. For example, in 2021, a 32-channel silicon probe, obliquely implanted at a 27˚ angle to the cranial window, was used to achieve simultaneous optical imaging and recording of calcium activity in somato-sensory neurons¹⁵³. In the following year, the integration of tetrodes with a gradient index lens was demonstrated to enable concurrent electrophysiology and calcium imaging in the hippocampus of head-fixed, awake mice¹⁵⁴. Additionally, a 16-channel silicon probe placed in the hippocampal CA1 region combined with voltage-sensitive indicators to record cortical activity enabled a detailed exploration of the interplay between the hippocampus and the cortex¹⁵⁵. This study discovered that ripple and gamma oscillations in the hippocampus precede and follow the cortical activation, respectively (Fig. 6a).

**Fig. 6: Multimodal datasets acquired from the integration of electrical and optical neural recording techniques.**

The introduction of flexible and transparent laminar probes has greatly improved accessibility and practicality of multimodal experiments. These advances have facilitated the seamless integration of electrical and optical recording methods in multimodal experiments. For example, in 2021, a transparent and flexible 32-channel array (Neuro-FITM), implanted into the hippocampus and bent to the side, was used for simultaneous wide-field calcium imaging from both hemispheres without casting shadows in the FOV⁷². This study identified unique cortical patterns occurring during SWRs, often preceding hippocampal activity, and employed machine learning techniques to analyse the association between hippocampal neural population activity and these distinct cortical patterns (Fig. 6b).

Multimodal experiment with planar probes

Transparent and flexible planar neural interfaces can also be used, in combination with optical recording or stimulation techniques, to investigate neural dynamics in different parts of the brain. For example, in 2023, a flexible and transparent probe made from hexagonal boron nitride and graphene (NeuroWeb), in conjunction with optical stimulation, enabled the exploration of neural interactions between the somato-sensory cortex and the cerebellum¹⁵⁶. The unique open lattice structure of the probe offers high flexibility and robust adhesion, ensuring a conformal fit with the brain’s uneven surface in mice. In addition to tracking the spiking activity of multiple neurons over a week, the probe revealed varying delays in neural signal transmission between the somato-sensory cortex and the cerebellum, depending on the involved pathways (Fig. 6c).

Another study, in 2022, combined 16-channel graphene array with an imaging cannula to simultaneously record SWRs from the hippocampus and perform two-photon calcium imaging in hippocampal CA1 region. This study demonstrated a link between spatiotemporal patterns of SWRs and the selective activation of orthogonal hippocampal CA1 cell assemblies⁶³. In the same year, simultaneous two-photon imaging and surface recordings with 16-channel graphene arrays were employed to examine the integration of human cortical organoids transplanted into the retrosplenial cortex (RSC) of mice¹⁰⁵, revealing vascularization within the organoid and the formation of functional connections with the cortex (Fig. 6d).

Although these studies have notably advanced neuroscience research, the full potential of multimodal experiments remains largely untapped. Future developments in neural probe technologies are expected to facilitate a broader spectrum of multimodal experiments, which could be instrumental in deciphering numerous unresolved aspects of brain function.

Beyond single modality: advancing neuroscience through multimodal data processing

The wealth of data garnered from complementary modalities provide an unparalleled opportunity to explore neural dynamics from new perspectives. Yet the analysis of such complex multimodal neural recordings necessitates tools specifically tailored to the unique characteristics of each experiment. Traditionally, various methods have been developed for examining unimodal electrical or optical recordings in isolation. For example, techniques for detecting and classifying action potentials recorded by neural probes generally use specific sets of features, including the shape of action potential waveforms, firing rate, cross-correlograms and auto-correlograms, to detect and cluster action potentials into putative neurons (SUA and MUA)^157,158,159. In optical recordings, software solutions have been developed to isolate fluorescence signals from selected regions of interest while simultaneously minimizing motion artefacts and background noise^160,161. Furthermore, these signals can be integrated with behavioural tasks, thereby correlating neural activities with cognitive and behavioural functions²⁴. Although these techniques are well-established and widely used by the neuroscience community, they are generally optimized for single-modality experiments. Hence, to fully exploit the rich and complex multimodal neural datasets, it is essential to modify existing methods and develop new algorithms and analytical pipelines tailored to multimodal neural data analysis.

Challenges and strategies in multimodal neural data analysis

Successfully integrating multiple modalities into a cohesive data analysis pipeline requires consideration of the unique characteristics inherent to different types of signals. This step is crucial for ensuring accurate interpretations in multimodal studies. Insights can be drawn from previous research in multimodal data analysis and multimodal machine learning, which have primarily been applied to multimedia datasets. These areas, predominantly focused on text, video and speech datasets, have identified several challenges, including data representation, integration and synchronization complexities, that are not only relevant but also pervasive in other domains¹⁶². In the context of neural recordings, where combining electrophysiological and optical data introduces additional layers of complexity, the insights from multimodal machine learning, as well as the strategies developed to address the challenges in other biological domains including multi-omics (the research area that integrates various types of information from genes, proteins and other molecules to provide a comprehensive understanding of an organism’s functions), offer valuable perspectives^163,164. This cross-domain knowledge is instrumental in devising advanced analytical strategies to unravel the complex dynamics in multimodal neural recordings.

Data representation in multimodal neural recordings

Analysing and understanding multimodal datasets, particularly those that include both electrical and optical neural recordings, begins with the extraction of features that effectively capture the full range of neural dynamics and brain states. Feature extraction techniques must be adept at handling the differing spatial and temporal resolutions inherent in multimodal datasets. When executed correctly, these techniques not only aid in understanding the underlying mechanisms of these complementary neural recordings but also yield invaluable data for downstream analyses and tasks. Over the years, various methods such as deep neural networks (DNNs), autoencoders, recurrent neural networks (RNNs) and graphical models have been employed to extract detailed unimodal representations in diverse datasets¹⁶⁵. In neuroscience, these techniques are used to identify a low-dimensional latent space, often termed the ‘neural manifold’, which represents simple, underlying patterns within high-dimensional, complex neural recordings^166,167. By capturing these neural manifolds, direct connections between the multidimensional neural recordings and various sensory, cognitive and motor functions are established^168,169. Although these methods were initially designed for single modalities, adaptations can be made to extract multimodal representations, either in a combined (joint) latent space or in separate yet interconnected (aligned) latent spaces (Fig. 7a). For example, DNNs and contrastive learning were used to create a non-linear representation of high-dimensional spiking activity, effectively encapsulating behavioural elements¹⁷⁰. Alternatively, variational autoencoders (VAEs) with Gaussian process priors and frequency representations of time-series data as variational parameters were applied to extract both shared and unique latents from a multimodal dataset, which included whole-brain calcium activity and limb position in Drosophila¹⁷¹. Although these models were originally developed for unimodal neural recordings and auxiliary behavioural data, their potential for processing multimodal datasets involving both electrical and optical recordings is promising. Used in an effective way, these methodologies can extract a wealth of information, offering deep insights into how various brain functions, including behavioural and cognitive processes, are encoded.

**Fig. 7: Analysis methods for complex, multidimensional datasets from multimodal experiments.**

A key challenge in handling multimodal datasets is identifying meaningful correlations and establishing links between distinct features extracted from these recordings. This goes beyond merely coordinating representations between different modalities and extends to aligning latent spaces or feature sets derived from neural recordings across different trials, sessions, days or subjects, leveraging the complementary information in multimodal datasets^172,173. Successfully addressing this challenge can improve BCI performance to provide consistent results in long-term chronic studies by detecting and compensating for electrode movement and variations in neural coding^172,174. Various techniques, originally developed for multimedia applications, have been employed to align latent spaces across modalities by minimizing distances between the extracted features¹⁶². Methods including, but not limited to, generative models (such as VAEs and diffusion models)^175,176, RNNs¹⁷⁴ and canonical correlation analysis (CCA)¹⁷⁷ were adapted for aligning latent spaces in neural recordings. These approaches can be further refined for multimodal neural studies, serving different purposes. For example, in addition to stabilizing neural representations for BCI applications, they can assist in identifying and correlating distinct features of various cell types in both optical and electrical recordings. Although distinguishing between activities of different cell types in electrical recordings is quite challenging, optical recording techniques, using cell-specific indicators, can selectively label and monitor diverse cell types^124,178. Thus, employing appropriate alignment methods can reveal correlated features between electrical and optical signals in experiments that include optical recordings with various indicators and extracellular signals. Achieving accurate representation and alignment of these features could pave the way for cell-specific electrophysiology (Fig. 7b) and optotagging — photostimulation-assisted identification of neuronal populations — using optical imaging modalities.

Translation and fusion of multimodal neural recordings

A unique feature of multimodal datasets is the presence of shared information, which can be leveraged to enable the translation of data across different modalities. Since the early 2010s, various techniques — particularly in the fields of computer vision and natural language processing — have been developed for transcription of visual or auditory datasets¹⁷⁹. Convolutional neural networks (CNNs), hidden Markov models, DNNs, autoencoders and RNNs have been widely used for such cross-modality translations¹⁶². These methods have been refined and optimized to enable translation of information in multimodal biological experiments (Fig. 7c). For example, a series of autoencoders was used for bidirectional information translation between single-cell imaging and sequencing data¹⁸⁰. CNNs were also applied on multimodal neural datasets, integrating fMRI and EEG, to achieve bidirectional transcoding between these modalities¹⁸¹. Building on this foundation, the subsequent integration and fusion of shared and modality-specific information from multimodal datasets play a pivotal role in enhancing the prediction and decoding of behavioural or cognitive measures in neural experiments (Fig. 7d). Extensively explored in multimedia and healthcare sectors, this approach has led to the development of various classification and regression techniques that integrate complementary modalities for optimal decoding and prediction performance^182,183. Adapted for multimodal neural research, they hold the potential to predict diverse behavioural measures, thereby advancing our understanding of complex cognitive functions such as memory formation and motor skill acquisition. Additionally, these methods can substantially improve the functionality of BCI systems by exploiting the rich, complementary information available in multimodal datasets. For example, a study that considered LFP and spiking activity as distinct modalities discovered neural embeddings of these multiscale (continuous and discrete) signals through generalized linear models (GLMs)¹⁸⁴. This approach revealed joint latent spaces that effectively predicted 3D motion dynamics in non-human primates, showing superior performance over unimodal dataset models. In another study, a deep learning model using an attention mechanism fused EEG and fNIRS features to decode motor imagery and mental arithmetic tasks¹⁸⁵. The spatial guidance provided by fNIRS resulted in enhanced performance in both tasks compared with unimodal models.

Implications of advanced data analysis methods

The evolution of data analysis pipelines has not only enabled a deeper understanding of the complex brain dynamics through multimodal recordings but also paved the way for substantial advancements in the field of neuroscience. For example, RNNs were trained to uncover the non-linear relationships between different modalities, predicting the calcium activity across the entire cortex using local surface potentials recorded from the somato-sensory cortex¹⁰¹ or predicting both the average and cellular calcium activity of neurons at depth using surface potential from the visual cortex¹⁰². These preliminary studies highlight the potential of multimodal experiments and data analysis techniques to improve the recording capabilities of neural interfaces.

Models trained on extensive multimodal datasets to unravel the relationship between local and global as well as surface and deep brain activity could extend the spatial range of electrical recordings from submillimetre to the centimetre scale within the brain’s 3D space. Such advancements set the stage for the development of a new generation of neural interfaces that are minimally invasive, provide expansive spatial coverage and can capture neural dynamics with exceptional temporal resolution. These cutting-edge neural recording technologies can revolutionize neuroscience animal studies by providing tools capable of capturing neural dynamics across extensive 3D volumes with high temporal precision over several months, overcoming limitations such as head fixation and photobleaching, which are common in current optical imaging studies. Furthermore, minimally invasive recording technologies with enhanced capabilities could lead to the development of BCI systems that are more intuitive, precise and responsive, and capable of functioning throughout a patient’s lifetime.

Outlook

Electrical and optical neural recording techniques are widely used in neuroscience due to their distinct capabilities and advantages. Electrophysiology offers exceptionally high temporal resolution, exceeding 20 kHz, capturing fast neural dynamics such as action potentials over extended periods. Nevertheless, the spatial resolution of electrical neural recording techniques is limited by the size of the electrodes and their geometry, thereby posing challenges in monitoring the activity of large neural populations. By contrast, optical imaging techniques can record neural activity with submicron spatial resolution from large neural ensembles on 2D planes or within 3D volumes of brain tissue. Despite advances in the development of new voltage indicators, optical imaging techniques mostly have limited temporal resolution when capturing neural dynamics from large populations, suffer from photobleaching and are typically limited to head-fixed, short-term recordings.

Integration of complementary modalities in multimodal set-ups addresses these limitations, offering advanced toolkits for neuroscience research. There is growing evidence that multimodal experiments can notably deepen our understanding of brain dynamics and cognitive functions¹⁸⁶ and facilitate causal studies¹⁵⁶, going beyond conventional correlative approaches to investigate the neural circuits and signal pathways that shape brain functions¹⁸⁷. Such multimodal experiments contribute to fundamental neuroscience research, addressing unresolved questions, and hold remarkable implications for brain–machine interfaces aimed at improving the quality of life for many¹⁸⁸. Furthermore, findings from multimodal studies might lead to the development of new biologically inspired artificial intelligence models¹⁸⁹.

Current electrical and optical neural recording technologies continue to evolve, to further push their limits and meet the demands of innovative multimodal experiments and studies. For example, the ongoing development of transparent and flexible neural interfaces with improvements in channel count, density, electrode size and noise reduction is geared towards offering new generations of neural probes. These probes must provide higher bandwidth and signal quality as well as compatibility with multimodal experiments. Additionally, the development of multifunctional neural probes, integrating chemical and optical sensors, represents a step forward in enhancing the functionality of current neural probes¹⁹⁰. Optical recording technologies are also evolving in parallel, facilitating their integration into multimodal experiments. For example, novel objective lenses have enabled optical imaging with an increased working distance, which can facilitate the integration of various transparent neural interfaces with optical imaging set-ups¹⁹¹. The integration of these systems with label-free imaging techniques might facilitate brain-wide optical recording of neural dynamics at single-cell resolution, which is considered the ‘holy grail’ in neuroscience¹⁹².

Advances in multimodal neural recording technologies lead to larger and more comprehensive datasets that call for new data analysis pipelines able to effectively capture the essence of neural dynamics in complex, multidimensional datasets, potentially transforming both unimodal and multimodal experiments. For example, models developed and trained to infer cellular activity from surface electrical recordings can augment single-modality experiments based on electrophysiology, providing a more comprehensive understanding of neuronal populations in deeper brain layers¹⁰². Moreover, capturing the non-linear dynamics across modalities has shown promising results in expanding the spatial reach of local electrophysiology recordings^101,193,194. Such progress not only offers insights into brain dynamics but also facilitates long-term chronic experiments overcoming certain constraints, such as the need for head fixation in optical recording, which limits the range of behavioural tasks that can be conducted.

The computational cost associated with these models increases with the size and complexity of multimodal datasets, presenting several challenges for multimodal studies. Conventional data acquisition systems and offline data processing pipelines fall short for real-time multimodal set-ups, highlighting the need for innovative solutions. Technologies such as in-memory computing and neuromorphic electronics have demonstrated potential in on-memory computation tasks by merging memory and processing units to surpass the efficiency of conventional systems^195,196,197. This integration not only offers more energy-efficient computation but also maintains high performance levels. The application of neuromorphic approaches to multimodal experiments could pave the way for cost-effective, efficient and real-time analysis of multimodal datasets, marking a substantial advance in the fields of neuroscience and brain–machine interfaces¹⁹⁸.