Damian Sendler: Multiple timescales are processed and integrated into a meaningful whole. Complex multiscale temporal organization appears to be present in the brain. Intrinsic neural timescales (INT) describe the different timescales observed in different regions of the brain, but their function and neural mechanisms remain a mystery. It is our contention that recent research on INT is critical for input processing. It is a form of input sharing, which means that they are used by a wide range of different species. Encoding inputs may be accomplished by matching inputs’ stochasticity with neural activity’s ongoing temporal statistics, i.e., input matching. For input processing, input integration versus segregation and sampling are the most important temporal mechanisms. This helps to connect the brain to its evolutionary and environmental context. It has significant implications for our understanding of mental characteristics and psychiatric disorders, as well as expanding our view of intelligence beyond the brain by incorporating timescales.
Damian Jacob Sendler: A wide range of regular and irregular inputs, occurring over a wide range of timescales, are constantly being fed into the brain by our environment. Consider music, one of the most complex temporal inputs. Even better, we are able to combine the music’s various timescales into a single meaningful whole, like a melody. Furthermore, the melody can be clearly discerned from the background accompaniment.
Dr. Sendler: Is it possible for our brains to process and integrate so many different types of information? Research suggests that the brain has its own internal timescales (INT)1–11. In resting state, lower-order unimodal sensory regions, such as the primary visual cortex, show short timescales compared to higher-order transmodal regions, such as the default-mode network (DMN). Even so, there is still no clear understanding of the INT’s specific function or role in the brain and neural processing.
By looking at the evidence from both human and non-human species, we believe that INTs play an important role in processing and structuring inputs over a range of timescales (Fig. 1 for a general framework). Our goal here is to describe basic dynamic principles that are shared across different types of input (see Box 1) rather than focusing on specific inputs (such as visual, somatosensory, or auditory; see Box 1). When it comes to processing and shaping extrinsic timescales of multiscale input, the brain uses its own INT to do so. That way, the brain is able to encode environmental inputs’ stochastic structure according to its own. Its unimodal–transmodal hierarchy and INT determine its stochastic structure. Those mechanisms include temporal integration/segregation and input sampling with subsequent shifts to lower frequency modes in the processing hierarchy, as we will explain.
Many different interpretations of the term “input” exist. Sensory functions are first and foremost associated with the concept of input. For example, somatosensory, visual, and auditory inputs can all be distinguished. It has been shown that these sensory input systems are related, with different input streams in the brain, both functionally and INT25. These input streams share a progression from unimodal (like primary sensory cortex) to more transmodal regions like the dorsolateral prefrontal cortex. Some of the transmodal regions are shared between various sensory input systems.
Hence, despite their distinctions in terms of their modality-specific sensory input, these input systems may nevertheless converge in the higher-order transmodal regions. Considering the unimodal–transmodal structure of INT, this suggests that some dynamic, i.e. temporal, features and mechanisms of information acquisition may be shared by the various sensory modalities, as discussed above.
In addition to sensory inputs from the external environment, the brain also receives inputs from the body, that is, its interoceptive inputs (and also its proprioceptive inputs) (and also its proprioceptive inputs). Taylor et al.116 demonstrate that the brain’s interoceptive input streams (with insula as the key region) again follow the hierarchical progression from unimodal to transmodal regions. Interoceptive inputs also converged with exteroceptive sensory inputs in the higher transmodal regions, which is remarkable. That suggests some commonality among intero- exteroceptive input processing.
In addition to the constant “bombardment” of exteroceptive and interoceptive inputs from the environment and the body, the brain exhibits spontaneous activity changes. Exteroceptive stimuli usually elicit activity changes of varying degrees of strength, but these changes can be weak or strong. Such changes in the auditory cortex’s spontaneous activity may be interpreted as “external voices,” even though they are actually hallucinatory in nature.77,117,118 Since healthy people can experience auditory hallucinations, these “healthy” brain activity changes may even exist and be referred to as “neuronal input”77. Mind wandering119,120, self-reference processing121,122, and mental time travel/episodic simulation67,112 have been linked to these spontaneous changes. More interestingly, higher-order transmodal regions have stronger neuronal inputs than lower-order transmodal regions106,123–125, which is consistent with the findings of previous studies in this area. Even the brain’s own neuronal input appears to follow the progression from a single-mode to a multi-mode hierarchy.
To sum up, it appears that the concept of input extends far beyond the distinction between different types of sensory input and the various sources of those inputs (such as the environment, one’s own body, or one’s own mind). There appears to be a deeper level that is shared among the various inputs, unlike the more superficial level where different inputs and sources are distinguished. There is a strong temporal and dynamic influence on this deeper level of input by the brain’s INT. In this paper, we describe the various features and mechanisms of input processing at a deeper temporal and dynamic level.
Does INT have any bearing on how we behave and think? Psychiatric disorders are a good indicator of this. In a recent fMRI resting state study14, the ACW was used with autistic subjects. Adults with autism spectrum disorder (ASD) had significantly shorter ACW in the primary sensory regions (visual, sensorimotor, and auditory) than healthy controls, and these changes were negatively correlated with the severity of autism. When it came to autism, however, ACW in the right caudate region was significantly longer in ASD patients, and this was also associated with the severity of repetitive restrictive behavior.
These findings were corroborated by an investigation into adolescent children with Autism Spectrum Disorder (ASD) using fMRI resting state ACW. This means that the intrinsic timescales have a developmental component. In addition, researchers looked into the neuro-anatomical basis by calculating the volume of gray matter in the area. Local gray matter volume showed a significant positive correlation with the duration of ACW in the same region, which was also true for regions affected by ASD. Finally, they performed a mediation analysis, which revealed that the duration of the ACW14 mediated its effect on autistic symptoms through the gray matter volume in the aforementioned regions.
Another fMRI resting state study in ASD126 supports the importance of intrinsic timescales in autism. Power-law exponent (PLE) was calculated by operating in frequency domain rather than time domain (and spectral entropy). They found that the salience network (insula, supragenual anterior cingulate cortex, and thalamus) showed increased PLE with stronger power in slow frequencies in ASD (see also ref. 127 showing that the salience network exhibits the highest variability and flexibility among the different networks). Furthermore, they found that schizophrenia did not have an increased PLE in the salience network. Finally, these findings were specific to PLE and not found in other areas of ASD61,128,129, such as regional homogeneity and neural variability.
An EEG study in schizophrenia found abnormally long ACW (and high PLE) in several electrodes during a task state that involved self-specificity (i.e., an enfacement task)130. The study included mostly post-acute first episode subjects. They also found that schizophrenia subjects had a significantly lower degree of change in ACW from rest to task, which means that, unlike healthy subjects, they barely shortened their ACW while performing the task. Interestingly, the same ACW prolongation during task and its reduced rest–task difference was not observed during a non-self task, i.e., auditory oddball—this suggests a close relationship of ACW with self-specificity.
In addition, they found that the degree of ACW mediated the relationship between self-disturbance and negative symptoms in schizophrenia participants. Changes in INT may be related to psychopathological symptoms and more generally to behavior or cognition, as demonstrated in the autism study by Watanabe and colleagues14. A recent study on psychosis in schizophrenia25 adds credence to this idea.
Researchers have found that the INT in patients with psychosis/schizophrenia is significantly reduced (i.e., shortened) compared to healthy controls. They also found specific changes in neural hierarchy of auditory and somatosensory input streams: an increase in INT at the lower levels of the neural hierarchies may reflect hallucinations (comparing psychotic subjects with severe vs. mild hallucinations). Increased timescales at higher levels of neural hierarchies might be a sign of delusional thinking (comparing psychotic subjects with severe vs. mild delusions). INT appears to play an important role in mediating certain psychopathological symptoms of psychosis, according to research.
Damian Jacob Markiewicz Sendler: We discovered that the brain’s intrinsic hierarchical organization, i.e., its temporo-spatial hierarchy, can recapitulate and thus model the environmental hierarchies of various events. The living don’t have to “represent” a mental model of their surroundings: “An agent doesn’t have a model of its world—it is a model. To put it another way, our embodied brains aren’t just a representation of the sensorium—they are it. 131. The human agent (and related non-human species) is a temporo-spatial model of its environment based on the temporal hierarchy of its INT, but in a miniature scale-free way.
There are many ways to think of the brain as a free energy-driven temporal model of its environmental context. That results in the brain’s temporal and spatial nestedness within the context of its surroundings. Scale-free self-similarity in their shape or form connects the body, brain, and environment, despite their different temporal (and spatial) scales. One scale-free self-similar way the brain and its temporo-spatial organization nestle in a larger environment is like the smaller Russian doll being contained within a larger one (same shape, different size). In light of this temporo-spatial resemblance, we might do better to concentrate on “what our head’s inside of” rather than “what inside our heads”86.
An intrinsic temporal and spatial hierarchy must be built into the design and architecture of artificial agents, which has major implications for the modeling of artificial agents. In future AI models, it may be possible to implement such intrinsic spatial and temporal organization in their artificial agents, including the different timescales and the core–periphery organization (see ref. 133 for first steps in this direction in artificial agents using what they describe as “multiple time scale recurrent neural network”).
Using spatial and temporal hierarchies, Tani’s compelling model of an artificial agent135,137,138 can be extended by combining top-down (providing the agent’s inner input) and bottom-up (providing the agent’s outer input) layers. The temporo-spatial architecture and the free energy principle can be combined to create a small-scale but self-similar model of the artificial agent’s environment. If you want to achieve this, you’ll need a dynamic and constantly changing temporo-spatial hierarchy for the agent. So in order for an agent’s spatiotemporal dynamics to minimize its variational free energy in relation to a given environmental context, the causal (or temporo-spatial) architecture of the environment must be recapitulated or installed.
What is the relationship between intra-regional INTs and inter-regional ties? All of the features of the INT can be found within the same region as well as between regions. Excitation–inhibition balance and its local recurrent wiring34, such as in supragranular feedforward and infra-granular feedback connections5,13,32,35, are examples of intra-regional cellular features (see also ref. 20 for demonstrating the relevance of population codes). There is considerable variability in the INT at the single neuron level even within regions, according to Cavanagh et al36. Working memory (see also ref. 24) and/or decision-making36–38 place demands on a neuron’s temporal receptive field, which can change over time. Furthermore, Spitmaan et al.37 find that they are less dependent on the task context, which further demonstrates their adaptability. The authors also mention that the timescales of different neurons during task-related activity suggest that they are independent, or flexible.
Inter-regional connectivity has a significant impact on INT’s intra-regional characteristics. Chaudhuri and colleagues16 demonstrated that local connectivity alone is not sufficient to produce the wide range of timescales found in the cortex. Even more importantly, they remove all of the long-range projections from their non-human primate model5, limiting the range of different timescales and eliminating the inherent temporal hierarchy. Non-human primates9 and humans13,18,29,39–42 share the same relationship between intra-regional INT and inter-regional functional connectivity.
Damien Sendler: What is the relationship between inter-regional functional connectivity and the INT? It has been shown in two recent human fMRI studies that the duration of INT in various regions, as measured by the resting state ACW, is positively correlated with the degree to which a given region’s functional connectivity changes during task18,43. Individual variability in ACW across different regions was found to be closely linked to the individual variation in functional connectivity patterns within those same areas, as demonstrated by Raut and colleagues13 (see also42,44–48).
Inter-regional connectivity patterns in the brain appear to be closely linked to INT, as long-range inter-regional connections are a major component of intra-regional temporal features. These timescales can interact and integrate with each other as a result of the close connection between intra-regional and inter-regional timescales As we’ll see later, this could increase the number of timescales that are available, i.e., the repertoire of timescales.
Do task states have an effect on the int of a resting state? An affirmative response indicates that they are involved in input processing. The excellent studies of Hasson and colleagues15,26,27,49–51 strongly suggest the relevance of INT for input processing (ref. 1 for review). It has been shown that lower-order unimodal sensory regions preferentially process temporal segments of external stimuli (like single words of stories or short episodes in movies). Higher-order transmodal regions are activated more frequently when there are longer pauses (like whole paragraphs in a story or a longer episode in a movie). Temporal receptive windows, which Hasson and colleagues1 refer to as temporal receptive fields on the cellular level, describe how external inputs are processed and structured in terms of time.
Is there an overlap between the INT’s spatial or topographical pattern in rest and task states, i.e., a rest–task pattern? In functional connectivity18,52–55, such rest–task overlap has been well demonstrated. However, in INT, this issue remains open. To a great extent, these studies on the brain’s temporal responsiveness windows confirm the hierarchical organization of INT at rest. Similarly, the ACW in the DMN is the longest when at rest. Longer input sequences are processed by the DMN in task states, but the shorter resting state ACW in unimodal sensory regions seems to find its equivalent in the short input sequences processed by these regions1, as shown by studies of task states. As a result, the topographical organization of rest ACW and task temporal receptive windows is similar. There appears to be a close relationship between rest and task, or rest–task modulation or interactions. From 56 to 60 (see below for the discussion of task-specific changes in INT).
Rest–task overlap suggests that the hierarchical organization of ACW in resting state ACW is carried over to, and thus present, in the temporal receptive windows during task states, as well. Computational modeling and brain imaging both show evidence of this rest–task overlap. They found that regions with longer ACW, such as those in the transmodal core, respond to external stimuli with lower and slower activity changes than sensory regions; on the other hand, sensory regions have a shorter ACW at the periphery, which is accompanied by a higher amplitude and a faster response to external stimuli (see also refs. 17,35). In Chaudhuri and colleagues5’s non-human primate-based network model, electrical stimulation of V1 in the visual cortex yielded similar results (see also ref. 29). Weakly connected regions to input regions show longer INT during stimulation, which is an interesting finding. Another example of how tasks have an impact that goes beyond the stimulated regions of the brain: Data from human brain imaging support these computational findings on INT rest–task overlap. Research by Ito and colleagues18 examined the ACW in a resting state and its amplitude during various task states. Resting state ACW duration (in different regions) was found to have a negative correlation with task-related activity, i.e., amplitude. There are transmodal regions that have lower task-related amplitudes if they have a longer resting state ACW. while unimodal regions exhibit higher peak intensity during different tasks because of their shorter ACW. Accordingly, these findings imply that resting state INT has a significant influence on task-related activity and input processing. This, however, remains a mystery as to how it works.
If the resting state’s INT influences the temporal features of task states as well as associated cognition, this is what we call rest–task interaction or modulation (see also refs. 56,58,62). Golesorkhi and colleagues22 have just addressed this issue (see also ref. 15 for initial steps). ACW-50 and ACW-0 (see above) were studied using MEG during both rest and three different task states (motor, story-math, working memory). Resting state’s ACW and its hierarchical core–periphery organization strongly predicted their task states: the resting state’s ACW core–periphery organization was essentially preserved during all three task states as topographical rest–task correlation yielded high values (0.8–0.9) 22. Results from this study suggest that, regardless of task, an individual’s INT’s hierarchical organization persists during task states.
Damian Sendler
When calculating the rest–task difference, Golesorkhi and colleagues22 found some task-specific changes (Fig. 2) (which subtracts and cancels out the shared, i.e., correlating temporal hierarchical organization). It was found that higher-order network regions, which were presented at 30-second intervals, had an ACW that shortened during the story-math task. In motor and working memory tasks, only minor alterations were observed. In lower-order network regions, the ACW was significantly shortened during working memory, but only minimally in story-math and motor tasks. These findings suggest that task-specific changes can be observed once the hierarchical temporal organization present in rest and task is removed. ACW and INT can be modulated during task states, making them dynamic and adaptive rather than static and non-adaptive, as previously believed. Tasked modulation appears to mainly concern the shortening of ACW in comparison to rest, but more studies are needed. ECoG24 in humans exhibits INT’s adaptability during a working memory task’s delay period (relative to the pre-stimulus baseline).
Behavior and cognition are influenced by INT in addition to task states. It has been found that non-human primates who spend more time in the resting state of their INT (as measured during baseline intervals sandwiched between tasks) perform better in a range of tasks. In a delay discounting task, for example, delays are longer, and spatial response coding is stronger in the delay period during a non-match-to-goal task63. As for the human side, recent fMRI and/or EEG studies have shown that the resting state’s ACW is directly linked to higher-order cognition, such as the level of consciousness (64), sleep stage 21, and the sense of self (66–69) and psychiatric disorders (70). (see Box 2 for details). Data suggest that INT influences behavior, including perception and higher-order cognition such as self awareness and self-consciousness. Because task states and perceptions and cognition are dependent on a variety of inputs, these data support the idea that INT is critical for input processing and structuring.
ACW, i.e., normal neural timescales accompanied by a balance of slow and fast frequencies, is associated with a normal capacity to encode inputs on a whole-brain level with healthy awake subjects and subjects with motor deficits but preserved input processing (amyotrophic lateral sclerosis, locked-in syndrome), while other physiological, pharmacological, and pathological conditions, e.g., sleep (N1-N2), unresponsive wakefulness, present short ACW (the EEG signals and the ACW representations are taken from the datasets investigated in Zilio et al.21).
It’s worth noting, however, that Zilio and colleagues21 only looked at activity during the resting state. As a result, the ACW’s relationship to input processing can only be inferred indirectly; further research into task states involving actual inputs is required to establish a direct link. As input processing is known to be deficient in disorders and altered consciousness72–76, their findings suggest that the resting state’s INT exerts the capacity for input processing, a neural predisposition77–80. A person’s capacity to process multiscale inputs can be seen even when they aren’t being exposed to external multiscale inputs. In sleep, for example, the brain’s capacity or predisposition to process inputs is preserved, allowing us to be awakened at any time by strong external stimuli. Total anesthesia and coma, on the other hand, make this impossible, as the brain’s capacity or predisposition for input processing is lost in these cases.
So far, we’ve shown how important INT is for handling input from the outside world. The assumption that all species share a common external environment would lead one to expect some degree of overlap or sharing in their INT amongst these various species as well. This “input sharing” across species is clearly evident in the evolutionary preservation of INT across species.
Both non-human primates4,5,20 and humans13,14,18,22 show regional differences in the INT along the transmodal–unimodal gradient. Cross-species studies on both the cellular81,82 and regional-systemic83 levels show that this can be applied to a wide range of species. As a first step, Shinomoto and coworkers81,82 show that the non-human primate brain has distinct regions of regular, random, and burst-like spiking patterns in different parts of its cortex. It is shown that the temporal fingerprinting in the regions’ temporal structure of their firing pattern holds across different species, including non-human primates, cats and rats; the differences in firing patterns between different regions within one species are larger than the firing pattern differences within the same region across different species82,84. These findings show that the temporal characteristics of neural firing patterns at the cellular level in specific brain regions are conserved across species.
When it comes to regional and systemic oscillations, there have been similar findings of cross-species preservation. There is evidence that various oscillatory rhythms such as alpha, spindles, and ripples are present in the same frequency range across a wide range of species, including humans, non-human primates, dogs and bats.83 (see also ref. 84). According to Buzsáki and colleagues83, the frequency range of the rhythmic pattern remains the same in different mammals, even if the brain size changes and grows larger during evolution. It is concluded that a priority in evolution is the preservation of the brain’s temporal constants: “In summary, it suggests the brain’ architectural aspects—scaling of the ratios between neuron types, modular growth, system size, inter-system connectivity, synaptic path lengths, and axon caliber—are subordinated to a temporal organizational prior.
Damian Jacob Sendler
Do you know how input is processed by the INT? Using the temporal receptive windows developed by Hasson and colleagues, various task state studies suggest that the INT may segment inputs into short and long sections, such as single words, sentences, and paragraphs 1, 15, 26, 27, 29, 90. This kind of “temporal smoothing”92,93 may imply that certain inputs are processed with a high degree of temporal integration. Higher levels of temporal segregation are required to process other inputs in a more segregated and therefore more precise manner (see refs. 93,94). Together, this represents a delicate balancing act between integrating and separating input data.
Is there a way for INT to control the amount of integration and segregation they perform during input processing? The ACW measures the degree of correlation between the patterns of neural activity at various points in time. Correlation is low if only a few distinct time points are correlated, indicating that ACW is short. There will be a high degree of temporal segregation but a low degree of temporal integration in the processing of inputs beyond the time points that correlate in ACW. It is also possible that, due to a low correlation with a low ACW, the processing of single inputs may be more or less limited to their actual durations, meaning that temporal smoothing or expansion95 is not applied. The low degree of “temporal smoothing” required by short INTs means that inputs are processed with high temporal precision, both in terms of specific time points and the actual durations they actually have.. Intrinsic timescales in single-modal regions such as sensorimotor cortex, which have both a short ACW at rest and an even shorter temporal receptive window when performing a task, strongly support such an input processing strategy
So far, we’ve shown how the INT modulates input timescales by performing temporal integration and segregation on the input data. Many different timescales in the environment are confronted by the brain, and the brain has limited capacity to process this information. 83–101,101,102 When it comes to bridging the gap between its own timescales and those of its environment, how can the brain do so effectively? The brain’s smaller-scale neural activity should encode all inputs from the larger-scale environment with the least amount of error possible, ideally.
A fast–slow gradient from uni- to transmodal regions shows hierarchical organization of timescales in the brain, according to the empirical data. Mathematicians call this “down-sampling” from fast input stochastics to slower input stochastics103 when transitioning from the faster unimodal to the slower transmodal regions. Down-sampling across the hierarchy of unimodal and transmodal regions, the INT acts as input samplers. Initial numerical simulations are used to support the differential response between unimodal and transmodal regions during data processing (this section). After that, the fast–slow gradient of down-sampling will be illustrated mathematically with new simulation data in a second step (next section).
Sensory networks would be the first to downsample under our fast–slow gradient assumption. The intrinsic timescales of sensory and unimodal networks are shorter than those of transmodal networks15,22. According to this, the initial sampling would be performed at a higher frequency, with subsequent sampling being performed on more widely spaced timescales (down-sampling). This means that unimodal regions, because of their shorter intermodulation time (INT), should have a faster and more transient, or fast-frequency response. There should be a longer-lasting response in transmodal regions.
Music and language, both of which have temporal complexity, are difficult for the brain to process, let alone bring together in meaningful wholes like melodies or sentences. One of the most important roles the brain plays in processing input is that of intrinsic neural timescales (INT). Because of INT’s central position during sleep and wakefulness, not to mention during task performance, this is a reasonable assumption. It is based on findings like this one to propose that the INT’s primary function is to shape and structure input processing, including aspects like cross-species input sharing and encoded stochasticity. This has to do with input sharing across species and input encoding by matching the stochastics of both the environment and the brain, respectively. Input integration vs. segregation on temporal grounds as well as (II) fast–slow down-sampling along the INT’s unimodal–transmodal hierarchy may both play a role in mediating this effect. INT is extremely relevant to current views on brain function, including its role in mental features and psychiatric disorders, when taken together because of their key role in input processing through distinct mechanisms.
