V.S. Ramachandran

Journal of Consciousness Studies, 8, No. 12, 2001, pp. 3-34

V.S. Ramachandran and E.M. Hubbard 
Synaesthesia - A Window into Perception, Thought and Language

Abstract: We investigated grapheme-colour synaesthesia and found that: 
(1) The induced colours led to perceptual grouping and pop-out, 
(2) a grapheme rendered invisible through 'crowding' or lateral masking induced synaesthetic colours—a form of blindsight—and 
(3) peripherally presented graphemes did not induce colours even when they were clearly visible. 

Taken collectively, these and other experiments prove conclusively that synaesthesia is a genuine perceptual phenomenon, not an effect based on memory associations from childhood or on vague metaphorical speech. We identify different subtypes of number-colour synaesthesia and propose that they are caused by hyperconnectivity between col-our and number areas at different stages in processing; lower synaesthetes may have cross-wiring (or cross-activation) within the fusiform gyrus, whereas higher synaesthetes may have cross-activation in the angular gyrus. 

This hyperconnectivity might be caused by a genetic mutation that causes defective pruning of connections between brain maps. The mutation may further be expressed selectively (due to transcription factors) in the fusiform or angular gyri, and this may explain the existence of different forms of synaesthesia. 

If expressed very diffusely, there may be extensive cross-wiring between brain regions that represent abstract concepts, which would explain the link between creativity, metaphor and synnesthesia (and the higher incidence of synnesthesia among artists and poets). 

Also, hyperconnectivity between the sensory cortex and amygdala would explain the heightened aversion synaesthetes experience when seeing numbers printed in the 'wrong'colour. Lastly, kindling (induced hyperconnectivity in the temporal lobes of temporal lobe epilepsy [TLE] patients) may explain the purported higher incidence of synaesthesia in these patients. 

We conclude with a synaesthesia-based theory of the evolution of language. Thus, our experiments on synaesthesia and our theoretical framework attempt to link several seemingly unrelated facts about the human mind. 

Far from being a mere curiosity, synaesthesia may provide a window into perception, thought and language.

Finally, we should note that it is possible that the distribution of gene expression and level of cross-activation is not bimodal; hence the heterogeneity of the phenomenon. Indeed, one might expect to encounter 'mixed' types rather than just higher and lower. Ironically it is this heterogeneity that has often caused researchers to avoid studying synaesthesia altogether or led them to conclude that the whole phenomenon is bogus.

Artists, Poets and Synaesthesia
Synaesthesia is purported to be more common in artists, poets and novelists (Dailey et al., 1997; Domino, 1989; Root-Bernstein & Root-Bernstein, 1999). For example, Domino (1989) reports that, in a sample of 358 fine-arts students, 84(23%) reported experiencing synaesthesia. This incidence is higher than any reported in the literature (see above), suggesting that synaesthesia may be more common among fine-arts students than the population at large. Domino then tested 61 of the self-reported synaesthetes and 61 control subjects (equated on gender, major, year in school and verbal intelligence) on four experimental mea-sures of creativity. He found that, as a group, synaesthetes performed better than controls on all four experimental measures of creativity. While this study has the advantage of using an experimental method to assess creativity, it suffers a severe limitation in that no experimental tests were conducted to assess synaesthetic experiences. Further studies making use of our objective experimental measures of synaesthesia are clearly required to confirm this result.

How can the cross-wiring hypothesis explain these results? One thing these groups of people have in common is a remarkable facility linking two seemingly unrelated realms in order to highlight a hidden deep similarity (Root-Bernstein & Root-Bernstein, 1999). When Shakespeare writes 'It is the East and Juliet is the sun', our brains instantly understand this. You don't say, 'Juliet is the sun. Does that mean she is a glowing ball of fire?' (Schizophrenics might say this; they often interpret metaphors literally). Instead, your brain instantly forms the right links, 'She is warm like the sun, nurturing like the sun, radiant like the sun' and so on. How is this achieved?

It has often been suggested that concepts are represented in brain maps in the same way that percepts (like colours or faces) are. One such example is the con-cept of number, a fairly abstract concept, yet we know that specific brain regions (the fusiform and the angular) are involved. Perhaps many other concepts are also represented in non-topographic maps in the brain. If so, we can think of meta-phors as involving cross-activation of conceptual maps in a manner analogous to cross-activation of perceptual maps in synaesthesia. If this idea is correct then it might explain the higher incidence of synaesthesia in artists and poets. If mutation-induced cross-wiring selectively affects the fusiform or angular gyrus someone may experience synaesthesia. However, if this mutation is more diffusely expressed it may produce a more generally cross-wired brain creating a greater propensity and opportunity for creatively mapping from one concept to another (and if the hyperconnectivity also involves Sensory-to-limbic connections the reward value of such mappings would also be higher among synaesthetes).

The Angular Gyrus and Synaesthetic Metaphors

In addition to its role in abstract numerical cognition, the angular gyrus has long been known to be concerned with cross-modal association (which would be consistent with its strategic location at the crossroads between the temporal, parietal and occipital lobes). Intriguingly, patients with lesions here tend to be literal minded (Gardner, 1975), which we would interpret as a difficulty with metaphor. However, no satisfactory explanation has yet been given for this deficit.

Based on what we have said so far, we would argue that the pivotal role of the angular gyrus in forming cross-modal associations is perfectly consistent with our suggestion that it is also involved in metaphors - especially cross-modal metaphors. (Indeed, we recently saw an anomic aphasic with left angular gyrus damage who, unlike normals, showed no propensity for the bouba/kiki effect described in the next section.) 

It is even possible that the angular gyrus was origi-nally involved only in cross-modal metaphor but the same machinery was then co-opted during evolution for other kinds of metaphor as well. Our idea that excess cross-wiring might explain the penchant for metaphors among artists and poets is also consistent with data suggesting that there may be a larger number of cross connections in specific regions of the right hemisphere (Scheibel et al., 1985), and the observed role of the right hemisphere in processing non-literal aspects of language (Anaki et al., 1998; Brownell et al., l990).

We realize that this is an unashamedly phrenological view of metaphor and synaesthesia. The reason it may seem slightly implausible at first is because of the apparent arbitrariness of metaphorical associations (e.g., 'a rolling stone gathers no moss'). Yet, metaphors are not arbitrary.

Lakoff and Johnson (1980) have systematically documented the non-arbitrary way in which metaphors are structured, and how they in turn structure thought. A large number of metaphors refer to the body and many more are inter-sensory (or synaesthetic).

Furthermore, we have noticed that synaesthetic metaphors (e.g., 'loud shirt') also respect the directionality seen in synaesthesia (Day, 1996; Ullman, 1945; Williams, 1976). That is, they are more frequent one direction than the other (e.g., from the audi-tory to the visual modality). We suggest that these rules are a result of strong anatomical constraints that permit certain types of cross-activation, but not others.

Evolution of Language

One of the oldest puzzles in psychology is the question of how language evolved. The problem is that several interlocking pieces needed to co-evolve. But how could this have happened given that evolution has no foresight? Alfred Russell Wallace was so frustrated in trying to answer this that he felt compelled to invoke divine intervention. 

More recently, even Chomsky, the founding father of modern linguistics, has expressed the view that, given the complexity of language, it could not have possibly evolved through natural selection.

Our solution to the riddle of language origins comes from synaesthesia. To understand this argument, we need to put together several ideas.

Figure 7. Demonstration of kiki and bouba. Because of the sharp inflection of the visual shape, sub-jects tend to map the name kiki onto the figure on the left, while the rounded contours of the figure on the right make it more like the rounded auditory inflection of bouba.

First, consider stimuli like those shown in figure 7, originally developed by Köhler (1929; 1947) and further explored by Werner (1934; 1957; Werner & Wapner, 1952). If you show fig. 7 (left and right) to people and say 'In Martian language, one of these two figures is a "bouba" and the other is a "kiki", try to guess which is which', 95% of people pick the left as kiki and the right as bouba, even though they have never seen these stimuli before." The reason is that the sharp changes in visual direction of the lines in the right-hand figure mimics the sharp phonemic inflections of the sound kiki, as well as the sharp inflection of the tongue on the palate. 


The bouba/kiki example provides our first vital clue for understanding the origins of proto-language, for it suggests that there may be natural constraints on the ways in which sounds are mapped on to objects.'11/12
[ 11 ] In his original experiments, Köhler ( 1929) called the stimuli takete 
and baluma. He later renamed the baluma stimulus maluma (Köhler, 1947). However, the results were essentially unchanged and 'most people answer[ed] without hesitation' (p. 133). (For further discussion, see Lindauer, 1990; Marks, 1996.) Our results again confirm these findings with a different set of stimuli and different names.
[ 12] This idea reminded us of the onomatopoeic theory of language origins ('bow-wow' = dog) but is quite different in that the relationship between the visual appearance of a dog and the sound mad e by a dog is completely arbitrary (unlike the kiki/bouba example). The case of 'suck', which is the actual sound produced when you suck mav be an interesting hybrid example.

Second, we propose the existence of a kind of sensory-to-motor synaesthesia, which may have played a pivotal role in the evolution of language. 

Dance and Rhythm

A familiar example of this is dance, where the rhythm of movements synaesthetically mimics the auditory rhythm. This type of synaesthesia may be based on cross- activation not between two sensory maps but between a sensory (i.e., auditory) and a motor map (i.e., Broca's area). This means that there would be a natural bias towards mapping certain sound contours onto certain vocalizations.

This somewhat speculative proposal gains credibility from recent work on 'mirror neurons' by Rizzolatti and colleagues (di Pellegrino, et al., 1992; Fadiga et al., 2000; Rizzolatti et al., 2001). 

These are neurons found in the ventral premotor area in monkeys and (possibly) humans (Altschuler et al., 1997; 2000; Iacoboni et al., 1999). Most neurons in this area will fire when the monkey per-forms complex manual tasks (e.g., grasping a peanut, pulling something or pushing something). But a subset of them, mirror neurons, will fire even when the monkey watches another 'actor' monkey or human performing the same action. We can think of these neurons as doing an internal simulation of such actions.
[ 13] With knowledge of these neurons, you have the basis for understanding a host of puzzling aspects of the human mind: 'mind reading', empathy, imitation learning (Iacoboni et a/., 1999, Ramachandran,2000b), and even the evolution of language (Rizzolatti & Arbib, 1998).

Another piece of circumstantial evidence for the notion of sensorimotor synaesthesia (and its possible link to mirror neurons) is the occurrence of a rare form of synaesthesia in which sounds evoke the automatic and uncontrollable adoption of certain, highly specific postures (Devereux, 1966).

Putting these ideas together, we conjecture that the representation of certain lip and tongue movements in motor brain maps may be mapped in non-arbitrary ways onto certain sound inflections and phonemic representations in auditory regions and the latter in turn may have non-arbitrary links to an external object's visual appearance 
(as in bouba and kiki). 14 
[14] Brent Berlin provides an especially relevant example (Berlin, 1994). He presented English speakers with fish and bird names from a language  completely unrelated to English (Huambisa, a language of the Jivoran language family in north central Peru). He found tbat English speakers were able to correctly discriminate bird names from fish names significantly more often than chance, even though they had never heard Huambisa, and it bears no family resemblance to English. After further analyses to rule out onomatopoeia, Berlin concludes that this is evidence for universal sound symbolism of the sort we describe here.

The stage is then set for a sort of 'resonance' or bootstrapping in the co-evolution of these factors, thereby making the origin of proto-language seem much less mysterious than people have assumed (see figure 8).

The origin of proto-language
Figure 8. A new synaesthetic bootstrapping theory of language origins.

Arrows depict cross-domain remapping of the kind we postulate for synaesthesia in the fusiform gyrus. 
(1) A non- arbitrary synaesthetic correspondence between visual object shape (as repre-sented in IT and other visual centers) and sonnd contours represented in the auditory cortex (as in our bouba/kiki example). Such synesthetic correspondence could be based on either direct cross-activation or mediated by the angular gyrus - long known to be involved in inter-sensory transformations. 
(2) Cross domain mapping (perhaps involving the arcuate fasiculus) between sound con-tours and motor maps in or close to Broca's area (mediated, perhaps, by mirror neurons). 
(3) Motor to motor mappings (synkinesia) caused by links between hand gestures and tongue, lip and mouth movements in the Penfield motor homunculus (e.g., the oral gestures for 'little' or 'diminutive' or 'teeny weeny' synkinetically mimic the small pincer gesture made by opposing thumb and index finger (as opposed to 'large' or 'enormous'). The cross-wiring would necessarily require trans-forming a map of two dimensional hand gestures into one-dimensional tongue and lip movements . . (e.g., the flexion of the fingers and palmar crease in 'come hither' is mimicked by the manner in which the tongue goes back progressively on the palate). 
(4) And you pout your lips to say 'you', 'vous' or 'thoo' as if to mimic pointing outward whereas 'me', 'mois' and 'I' mimic pointing inwards towards yourself. If such oral echoes of hand gestures are accompanied by emotional gut-tural utterances it would lead to the creation of early proto-words. Notice that each of these effects might be quite small but through progressive mutual bootstrapping they could have evolved into the shared vocabulary of early hominids. Add to this additional bootstrapping provided by co-opting the circuits originally used for symbol manipulation, semantics and tool manipulation, and you have fully modern language (e.g., the use of tools requires sub-assemblies such as attaching a head to a handle before hammering a nail - and this has the same formal logical structure as hierarchi-cal syntactic tree of language). We are currently testing these ideas by studying aphasics. 

We would also point out that lip and tongue movements and other vocalizations may be synaesthetically linked to objects and events they refer to in closer ways than we usually assume and this may have been especially true early in the evolution of the proto-language of ancestral hominids, e.g., words referring to some- thing small often involve making a synaesthetic small /rl with the lips and a narrowing of the vocal tracts (e.g., words such as 'little', 'petite', 'teeny' and 'diminutive') whereas the opposite is true for words denoting large or enormous.15
[15] This currently quite contentious issue is being studied within linguistics under the banner of 'phonesthemes' or sound symbolism (see, for example, Hinton et al., 1994). Recent research has sup-ported the concept of phonesthemes in English (Hutchins, 1999) and cross-linguistically (Berlin,1994). Our new studies on synaesthesia and our speculations on language origins obviously have considerable relevance to this issue of universal sound symbolism.

A third, important factor that may have contributed to this bootstrapping is synaesthesia caused by cross-activation between two motor maps rather than between two sensory maps (a better phrase might be 'synkinaesia'). 

For example, Darwin ( 1872) noted that when cutting something with a pair of scissors we often unconsciously clench and unclench our jaws, as if to sympathetically mimic the hand movements; in our scheme this would be an example of synkinaesia between the motor maps for the mouth and hand, which are right next to each other in the Penfield motor homunculus of the pre-central gyrus. In the example cited above, mouth shape for 'petite', 'teeny' and 'diminutive' might be synkinetic mimicry of the pincer-like opposition of thumb and forefinger to denote small size. Also, when pointing I use my index finger to point outward to you. I also produce a partial outward pout with my lips (as in English 'you', French 'tu' or 'vous' and Tamil 'thoo'), whereas when I point inward to myself, my lips and tongue move inwards (as in English 'me', French 'moi, and Tamil 'naan') In this , manner a primitive vocabulary of gesture and pantomime could evolve through synkinaesia into a corresponding vocabulary of tongue/palate/lip movements (causing vocalizations, especially if accompanied by guttural utterances). 

We are suggesting that these factors provided the initial impetus for language evolution, not that all modern language is synaesthetic in origin. 

The subsequent elaboration and refinement of the deep structure of language may have relied on synaesthetic metaphor (and, indeed, was probably guided by offline, hierarchic, symbol manipulation as well as semantic constraints, mediated by influences from the Wernicke's area). It is, however, the initial emergence of a complex multi-component trait that usually poses a challenge for evolution through natural selection, and that is what we are trying to explain here. 

That is, our theory really pertains to the origin of proto-language rather than Chomskyan universal grammar, but we believe that given the pre-adaptation provided by proto-language, Chomskyan UG could have evolved more readily. Additionally, numerous thinkers (Bickerton, 1995; Devlin, 2000; Lieberman, 1992) have pointed out that syntactic structure may have arisen from the pre-adaptation provided by syllabic structure. 

The key idea here is that each of these different effects (synaesthesia between -object appearance and sound contour, between sound contour and vocalizations, and synkinaesia) in isolation may have been too small to have exerted adequate selection pressure for the emergence of proto-language, but a bootstrapping between all of them acting together may have indeed been sufficient. (And then add to this an additional bootstrapping between the syllabic structure, symbol manipulation and the syntactic/hierarchical structure, and you have fully evolved language.) 

Another example of a 'synaesthetic metaphor' found in everyone is the use of the word 'disgusting'. We say this in response to unpleasant smells and tastes while at the same time raising our hands up and scrunching up our noses (Darwin showed that even a newborn infant would do this - suggesting that it is 'hard-wired'). The olfactory bulb projects to the orbito-frontal cortex, and olfactory and gustatory 'disgust' is almost certainly mediated by this part of the frontal lobes. _ But why do we use the same word, 'disgusting', and make the same face in response to someone whose behaviour is morally disgusting (e.g., a drunk mak-ing an unwelcome sexual pass at a woman)? This is unlikely to be coincidence since it is cross-cultural: The Tamil phrase for moral disgust means 'he smells bad' and the French word 'degoutant(e)' (used for social situations) literally means, 'bad tasting'. We would argue that this usage emerged because moral and social disgust is also mediated by the orbito-frontal cortex; i.e., it is yet another example of cross-wiring or even of the same brain map being used for two seemingly unrelated functions (given evolution's tendency to be opportunistic in using pre-existing hardware). 

Early mammals may have used the orbito-frontal cortex exclusively for olfactory and gustatory disgust, but as mammals became more social it came to communicate or signal olfactory disgust to others (stay away from that rotten food) and then eventually to communicate moral and social dis-gust (stay away from that rotten man). This is how evolution works, given that there is no master design ('God is a hacker', as Francis Crick has said).

Even the great apes may have some such synaesthetic scatological propensities. When Washoe wanted to 'sign' her disgust at someone's behaviour she used the same word as for faeces (and indeed apes throw facces at humans whom they are disgusted with).

One wonders, also, whether there exists a genetically based synaesthetic link between sex and aggression (and if so could this have anything to do with the proximity of nuclei concerned with sex with those concerned with aggression in the hypothalamus? Again, the use of sexually loaded words as aggressive swear words ('F*** you') appears to be cross-cultural. In French, the equivalent phrase is 'Va t'en faire f**tre', which translates to something like 'Go F*** yourself'. If there is no genetic basis related to anatomical/neural constraints, why do all (or ~-most) languages say 'F*** you' and one never hears 'Bite you', which would be the more logical choice given the obvious semantic associations between biting and aggression?

Hyperconnectivity and Emotions

Synaesthetes often report strong emotions in response to multi-sensory stimuli (both positive and negative depending on whether the associations are the 'right' or 'wrong' ones). Additionally, patients with temporal lobe epilepsy seem to have a propensity towards synaesthetic experiences. Why?

Despite these subjective reports, there is no clear experimental validation of the claim that synaesthetes have strong responses to 'discordant' sensory inputs, leading one to wonder, is their aversion to such stimuli any different from what a non-synaesthete experiences when confronted with, say, a blue carrot or green rose? Anecdotally this seems to be true; one of our synaesthetes claimed that incorrectly coloured numbers were 'ugly' and felt like 'nails scratching on the blackboard'. 

Conversely, when numbers were the correct colour it 'felt right, like the "aha" when the solution to a problem finally emerges'. Assuming that the claim is true, can we explain it in terms of our cross-wiring or cross-activation hypothesis of synaesthesia? Y-~

Visual information that is 'recognized' by the cortex of the temporal lobe (e.g. the fusiform) ordinarily gets relayed to the amygdala, nucleus accumbens and other parts of the limbic system (Amaral et al., 1992; LeDoux, 1992). 

These structures evaluate the significance of the object, so that we may speak of the amygdala and nucleus accumbens as developing an 'emotional salience map' of objects and events in the world. 

If the object is emotionally significant or salient such as a predator, prey or mate, the message gets relayed to the hypothalamic nuclei to prepare the body for fighting, fleeing or mating. Neural signals cascade from the limbic structures down the autonomic nervous system to decrease gastric motility and increase heart rate and sweating (e.g., Lang et al., 1964; Mangina & Beuzeron-Mangina, 1996). 

This autonomic arousal can be measured by monitoring changes in skin conductance caused by sweat - the skin conductance response (SCR) - which provides a direct measure of emotional arousal and limbic activa-tion. Typically, if you look at neutral objects such as a table or chair there is no arousal or change in SCR, but if you look at prey, mate or predator, there is.

We have suggested that a mutation that causes hyperconnectivity (either by defective pruning or reduced inhibition) may cause varying degrees and types of synaesthesia, depending on how extensively and where in the brain it is expressed (in turn modulated by transcription factors). 

Now imagine what would happen if there were hyperconnectivity between the fusiform gyrus (and other sensory cor-tices) and the limbic system (especially the amygdala and nucleus accumbens). If we assume that one's aesthetic and emotional responses to sensory inputs depend on these connections, then presenting a discordant input, such as a grapheme in the wrong colour, would produce a disproportionately large emotional aversion (like 'nails scratching on a blackboard') and, conversely, harmonious blends of colour and grapheme will be especially pleasant to look at (which may involve the nucleus accombens rather than the amygdala). 

The net result of this will also be a progressive 'bootstrapping' of pleasurable or aversive associations through limbic reinforcement of concordant and discordant inputs. This, by the way, allows us to also invoke a form of learning in the genesis of synaesthesia.

In order to test this idea, one could measure the SCR in synaesthetes in response to an incorrectly coloured number and compare this response to one pro-duced in a non-synaesthetic subject who is looking at blue carrots. A non-synaesthete might be a bit amusod or puzzled by the blue carrot but she is unlikely to say it feels like nails scratching on a blacktoard. We would therefore predict a bigger SCR in the synaesthete looking at the incorrectly coloured grapheme than in control subjects.

The hyperconnectivity explanation for synaesthesia is also consistent with the claim that the phenomenon is more common among patients with temporal lobe epilepsy (TLE). The repeated seizure activity is likely to produce 'kindling' (causing hyperconnectivity between different brain regions) which would explain reports of synaesthesia in TLE (see e.g., Jacome, 1999). Furthermore, if the sei-zures (and kindling) were to strengthen the sensory-amygdala connections, then TLE patients might also be expected to have heightened emotional reactions to specific sensory inputs. There are strong hints that this is the case (Ramachandran et al., 1997) 16
[ 16] There are also anecdotal reports that synaesthesia might be more common among individuals with per-fect pitch. Given that people with perfect pitch have an enlarged auditory representation in the superior temporal gyrus (planum temporale) (Schlaug et al., 1995), we would predict that this enlargement may allow hyperconnectivity to occur more readily between auditory and colour maps, producing a higher incidence of sound-colour synaesthesia. This explanation reverses the traditional causal arrow that perfect pitch may be more common in people with synaesthesia because the colours allow people to uniquely identify the tones.

Something along these lines may also explain why some famous artists have had TLE, Van Gogh being the most famous example (e.g., Kivalo, 1990; Meiss-ner, 1994). If our scheme is correct his heightened emotions in response to colours and visual attributes (resulting from kindling) might have indeed fuelled his artistic creativity.

Synaesthesia and the Philosophical Riddle of Qualia

Finally, the study of the unusual sensory experiences of synaesthetes may also shed light on the philosophical problem of qualia. There is now a growing consensus that the best way to solve this ancient philosophical riddle is to narrow down the neural circuitry (Crick & Koch, 1995; 1998; Metzinger, 2000) and, especially, the functional logic (Ramachandran & Blakeslee, 1998; Ramachandran & Hirstein, 1997) of those brain processes that are qualia laden as opposed to those that are not (e.g., the reflexive contraction of the pupil in response to light can occur in coma; however, there is no qualia as when you are awake and see a red rose). One strategy used to explore the neural basis of qualia is to hold the physi-cal stimulus constant, while tracking brain changes that co-vary with changes in the conscious percept (e.g., Sheinberg & Logothetis, 1997; Tong & Engel, 2001). In the case of synaesthesia, we are making use of the same strategy, but using pre-existing, stable differences in the conscious experiences of people who expe-rience synaesthesia compared with those who do not.

Ramachandran and Hirstein (1997) have suggested three 'laws' of qualia; functional criteria that need to be fulfilled in order for certain neural events to be associated with qualia (a fourth has recently been added; see Ramachandran & Blakeslee, 1998). 

Of course, this still doesn't explain why these particular events are qualia laden and others are not (Chalmer's 'hard problem') but at least it narrows the scope of the problem.
The four laws are:

1) Qualia are irrevocable and indubitable. You don't say 'maybe it is red but I can visualize it as green if I want to'. An explicit neural representation of red is created that invariably and automatically 'reports' this to higher brain centres.
2) Once the representation is created, what can be done with it is open-ended. You have the luxury of choice, e.g., if you have the percept of an apple you can use it to tempt Adam, to keep the doctor away, bake a pie, or even just to eat. Even though the representation at the input level is immutable and automatic, the output is potentially infinite. This isn't true for, say, a spinal reflex arc where the output is also inevitable and automatic. Indeed, a paraplegic can even have an erection and ejaculate without an orgasm.
3) Short-term memory. The input invariably creates a representation that persists in short-term memory - long enough to allow time for choice of output. Without this component, again, you get just a reflex arc. 
4) Attention. Qualia and attention are closely linked. You need attention to ful-fil criterion number two; to choose. A study of circuits involved in attention, therefore, will shed much light on the riddle of qualia.

Based on these laws, and the study of brain-damaged patients, we have suggested that the critical brain circuits involved in qualia are the ones that lead from sensory input to amygdala to cingulate gyrus (Ramachandran & Hirstein, 1997). 

Synaesthesia - the 'blending' of different sensory qualia - obviously has relevance to the qualia problem, as first pointed out by Jeffrey Gray (Gray et al., 1997; Gray, 1998). In particular, we would argue that the lower synaesthetes have the qualia of red evoked when they see a '5' or hear C-sharp. But when you and I experience red while looking at a black-and-white picture of an apple, the red does not fulfil all four criteria specified above, so there is very little qualia (leaving aside the question of whether you can have partial qualia if some criteria alone are fulfilled). And lastly, the higher synaesthetes may be a borderline case. As such, they can be used to shed light on the nature of qualia as well as metaphor (such borderline cases can be valuable in science; consider the manner in which viruses helped us to understand the chemistry of life).

To understand the importance of synaesthesia in illuminating the qualia problem consider the following thought experiment performed on your own brain. When you are asleep an evil East-coast genius, who we'll call DD, swaps or cross-wires the nerves coming into your brain from your ears and eyes. You then wake up. Consider the following places where the wiring could have been swapped.
1) If the swapping is done suffficiently early in sensory processing, the outcome is obvious: say the pathways from the auditory nuclens of the brain stem are 11 diverted to the visual cortex and the optic radiations to the auditory cortex. Then you would 'hear' sights and 'see' sounds.
2) If the swapping were done at or close to the output stage (e.g., in Broca's area) where you generate the word 'red' or 'C-sharp', again, the answer would be obvious. You might say, 'When you play me that tone I know it's a tone and experience it as such but I feel an irresistible urge to say red' (like a patient with Tourette's Syndrome).

But now we come to the key question: What if the swapping or cross-wiring is done at some stage in between these two extremes? Is there a critical boundary between these two extremes, so if you cross wires after the boundary you merely experience an urge whereas if you cross wires before that boundary you literally see red? Is it a fuzzy boundary or a sharp one? We would argue that this boundary corresponds exactly to the point where the transition is made from the four laws of qualia being fulfilled (before the boundary) to where they are not fulfilled (after the boundary).

Of considerable relevance to this philosophical conundrum is a new observation that we made on a grapheme-colour synaesthete (Ramachandran and Hub-bard, 2001a). This subject was colour anomalous (s-cone deficiency leading to a difficulty discriminating purples and blues) but intriguingly, he claimed to see numbers in colours that he could never see in the real world ('Martian colours'). This is yet another piece of evidence against the memory hypothesis - for how can you remember something you have never seen? On the other hand, the cross-wiring hypothesis explains it neatly. If we assume that the colour process-ing machinery in V4 in the fusiform is largely innate, then the genetically based cross-activation of cells in this area would evoke colour phosphenes even though the colours cannot be seen in the real world because of retinal cone deficiencies.

Indeed, even synaesthetes who are not colour blind sometimes say that the synaesthetically induced colours are somehow 'weird' or 'alien' and don't look quite the same as normal 'real world' colours. Previously, no satisfactory account has been proposed for this. The cross-wiring hypothesis explains this as well. For two reasons, the activation of cells in the visual centres caused by real world input is, in all likelihood, going to be somewhat different from the spurious or abnormal activation caused indirectly through numbers. First, given that it is abnormal, the cross-wiring is unlikely to be very precise. It might be slightly messy and this 'noise' may be experienced as weird Martian colours. This may be analogous to phantom limb pain (also caused by aboormal cross-wiring, Ramachandran & Hirstein, 1998).

Second, the cross-activation obviously skips the earlier levels of the colour-processing hierarchy which may ordinarily contribute to the final qualia - and this unnatural stimulation might cause the subject to see Martian colours. The implication of this is that the experience of qualia may depend on the activation of the whole visual hierarchy (or a large part of it), not just the pontifical cells at the end of the chain.17
[17] This point is consistent with current ideas of distributed processing. The six synaptic levels do not form a static hierarchy wherein neural transforrnations of one level are passed on to the next lovel in a conveyor-like fashion. Rather, three or four levels are co-active at any one time as expected in a distributed system (we are indebted to an anonymous reviewer for bringing this point to our attention; see also Churchland et al. 1994).

Summary and Conclusions

Synaesthesia has always been regarded as somewhat spooky. Even though it has been known for over 100 years, it has often been thought of as a curiosity - just a quirk based on early childhood memory associations or a mere metaphorical asso-ciation between different sensory terms. Indeed, it has been largely ignored by mainstream neuroscience and psychology despite the fact that both Cytowic ( 1989; 1997) and Marks (e.g., 1975; 1982; 2000) have repeatedly emphasized its potential importance for understanding normal sensory function. More recently, interest in this phenomenon has been revived by the intriguing experimental work and theo-retical speculations of Baron-Cohen, Harrison, Gray and colleagues (see above).

Although synaesthesia has been studied for over 100 years, our psychophysical experiments were the first to prove conclusively that synaesthesia is a genuine sensory phenomenon. Four lines of evidence support this: (1) Synaesthetically induced colours can lead to perceptual grouping, segregation and pop-out. (2) Synaesthetic colours are not seen with eccentric viewing even if the numbers are scaled in size to make them clearly visible. (3) A crowded grapheme that is not consciously perceived can nevertheless evoke the corresponding colour. (4) A colour-blind synaesthete sees colours in numbers that he cannot otherwise see in real-life visual scenes.

The results of Stroop-like interference tasks are sometimes cited as evidence for the view that synaesthesia is sensory (Mills et al., 1999) and sometimes for the conflicting view that synaesthesia is conceptual (Dixon et al., 2000; Mattingley et al., 2001 ) but neither inference is justified. Stroop interference merely shows that the association between the grapheme and the colour is automatic. Since Stroop-like interference can occur at any stage in the system—from perception all the way up to motor output (MacLeod, 1991 )—it is completely uninformative in determining whether synaesthesia is perceptual or conceptual. The main strength of our psychophysical approach to synaesthesia is that we make system-atic predictions instead of relying solely on the subjects' introspective reports. This is even more important for synaesthesia than for ordinary psychophysics since the subject is often trying to express the ineffable.

Having established the sensory nature of synaesthesia in our first two subjects, we propose a specific testable hypothesis: That grapheme-colour synaesthesia is caused by a mutation causing defective pruning and cross-activation between V4 (or V8) and the number area, which lie right next to each other in the fusiform gyrus. Although the cross-talk idea has been around for some time, no specific brain areas have been suggested and the idea is usually couched in vague terms that do not take advantage of known patterns of localization.

In addition to the lower synaesthetes (JC and ER) there also appear to be other types of number-colour synaesthetes in whom the effect may be more concept driven; i.e., the effect is conceptual rather than sensory. We suggest that in them the cross-activation occurs at a higher level - perhaps between the angular gyrus (known to be involved in abstract number representation) and a higher colour area in the vicinity that receives input from V4.

We suggest, further, that synaesthesia is caused by a mutation that causes defective pruning between areas that are ordinarily connected only sparsely. Various transcription factors may then influence the exact locus and extent to which the gene is expressed. If it is expressed only in the fusiform someone may be a lower synaesthete. If expressed in the angular gyrus someone may be a higher synaesthete. And, if expressed between primary gustatory cortex and adjoining hand and face regions of primary somatosensory cortex, the result might be a per-son who 'tastes shapes'. The distribution may not be bimodal, however, so there may be mixed types who combine features of several different types of synaesthesia.

One prediction would be that higher synaesthetes should experience colours even with tactile numbers or subitizable clusters of dots. Furthermore, if the higher colour area has different psychophysical properties then the induced colours will also have different psychophysical properties in higher synaesthetes. For example, in higher synaesthetes, the colours may not fall off with eccentric viewing of nurnbers and the induced colours may not give rise to grouping or pop-out, nor would numbers rendered invisible by crowding evoke colours in these higher synaesthetes. These predictions are all easy to test, but we must bear in mind that if the gene is expressed in a patchy manner in multiple locations, there might be mixed synaesthetes who complicate the picture. It remains to be seen whether days of the weck and months of the year - embodying the abstract rule of cardinality or sequence - would evoke colours ouly in higher synaesthetes, lower synaesthetes, or both.

We suggest, also, that the study of synaesthesia can help us understand the neural basis of metaphor and creativity. Perhaps the same mutation that causes cross-wiring in the fusiform, if expressed very diffusely, can lead to more extensive cross-wiring in their brains.
If concepts are represented in brain maps just as percepts are, then cross-activation of brain maps may be the basis for metaphor and this would explain the higher incidence of synaesthesia in artists, poets and novel-ists (whose brains may be more cross-wired, giving them greater opportunity for metaphors).

Our speculations on the neural basis of metaphor also lead us to propose a novel synaesthetic theory of the origin of language. 

We postulate that at least four earlier brain mechanisms were already in place before language evolved;
 1) a non-arbitrary synaesthetic link between object shapes and sound 
       contours (e.g., bouba and kiki), 
2) a synaesthetic mapping between sound contour and motor lip and  tongue movements (mediated, perhaps, by the recently discovered mirror neurons system in the ventral premotor area that must represent the movements of others, including vocal movements), 
3) a synaesthetic correspondence between visual appearance and vocalizations (e.g., 'petite', 'teeny' and 'little' for diminu-tive objects mimed synaesthetically by a small /i/ formed by the lips and a small vocal tract), and 
4) cross-activation between motor maps concerned with gesticulation and vocalizations. This would have allowed an autocatalytic bootstrapping culminating in the emergence of a vocal proto-language. Once this was in place other selection pressures could kick in to refine it (through the combined effects of symbol manipulation/semantics and of the exaptation provided by the syllabic structure for syntactic deep structure).


This raises the fascinating question of the relationship between language and thought - more specifically between the hierarchical/syntactic Chomskyan tree structure and abstract, off-line symbol manipulation (including logic). Did the latter accelerate the evolution of the former, or was it the other way around?
Or, did they co-evolve through mutual bootstrapping, as we suggest in this essay? Our neuro-logical approach to this problem will be to give non-linguistic logic puzzles to patients with Broca's aphasia, who have lost syntax. For example, we know that they cannot use 'if', 'then', 'but' and 'un-less', but can they play chess (which requires the tacit use of such relational concepts)? Can they still use a computer language or algebra (assuming tbat they could before the stroke)? 
And what about patients with Wernicke's aphasia - can they engage in symbol manipulation and logic, given that their syntax is intact? 
Another intriguing question is whether the hierarchical structure of tool use in early hominids provided an exaptation for the hierarchical structure of syntax (Greenfield, 1991). This seems very plausible to us: e.g., hammering a nail or stone core could give rise to distinctions such as 'active' and 'passive' or 'subject' and 'object'. Indeed, one wonders whether tool use may have even provided an exaptation for thought itself.

This idea is different from the two more traditional theories of language origins (Pinker, 1994): First, that language simply involves the specific implementation of a more general-purpose mechanism (such as thinking and symbol manipulation) or second, that it evolved exclusively as a specific adaptation for communication. 

On our scheme, neither of these extreme views is correct.
Instead, we postulate that language evolved through co-opting and finding novel uses for multiple mecha-nisms evolved originally for very different functions and by a fortuitous synergistic bootstrapping between these functions. This sort of co-opting of pre-existing machinery for novel uses is the rule, rather than the exception, in evolution, but this seems to have escaped the notice of even sophisticated psycholinguists.
The mutation-based hyperconnectivity hypothesis may also explain why many synaesthetes exhibit such strong emotional reactions to even trivial sensory discord or harmony. We suggest that this occurs because of hyperactivation of the amygdala, nucleus accumbens and other limbic structures by sensory inputs. A similar hyperconnectivity (based on kindling rather than mutation) could explain the purported higher incidence of synaesthesia as well as heightened emotions in response to sensory stimuli seen in TLE. Such hyperconnectivity (whether caused by genes or by TLE-induced kindling) would also increase the value of a reward or aversion, thereby strengthening pre-existing associative links (this would allow learning to play a role in synaesthesia).

Our scheme invokes limbic structures for explaining the emotional overtones of synaesthesia but it is very different from Cytowic's ( 1989; 1997) view that it all happens in the limbic system because the limbic system is phylogenetically ancient and everything must eventually converge on it. In our hyperconnectivity model, the gene is expressed at multiple sites along the sensory processing hierarchy (in a patchy or diffuse manner) including the sensory-to-amygdala connec-tions in some individuals—the limbic system is not the ouly player, nor even the most important one. 

But if one had to choose any single neuroanatomical locus for synaesthesia, better candidates would be the insula (where there is pre-existing convergence of information from many sensory modalities, includ-ing visceral sensations and pain) or the angular gyrus (as discussed above). 'Anat-omy is destiny' was one of Freud's few insightful remarks and finds resonance with the main ideas expressed in this paper (e.g., the rare form of pain-colour synaesthesia may be due to cross-wiring in the insular cortex, and perhaps the same might be trne for a woman that we recently encountered who reports orgasm-colour synaesthesia).

Finally, we discuss the relevance of this scheme for more subjective aspects of consciousness such as mental imagery and qualia. While both mental imagery and synaesthesia are paradigmatic examples of internal mental states, we have shown how the relation between the two might be fruitfully explored. In addition, we have shown how the cross-wiring hypothesis can explain synaesthetes' intro-spective reports. Because neural activation in the fusiform gyrus bypasses normal stages of processing at the retina, synaesthetes can experience qualia that are unavailable to non-synaesthetes. In addition, these results suggest that the entire perceptual pathway (or large portion) is essential for the experience of qualia, not merely the final stages.

The ideas we have presented in this essay are highly speculative but we hope they will provide a springboard for future speculations and experimental work on synaesthesia. Whether all of our ideas turn out to be correct or not, one thing is clear. Far from being an oddity, synaesthesia allows us to proceed (perhaps) from a single gene to a specific brain area (e.g. fusiform or angular) to phenotype - systematic psychophysics (e.g., pop-out fall off with eccentricity, masking, flicker and so on) - and perhaps even to metaphor, Shakespeare, and the evolution of language, all in a single experimental subject.


We thank Geoffrey Boynton, Francis Crick, Patricia Churchland, Julia Fuller-Kindy, Jeffley Gray, Richard Gregory, William Hirstein, Nick Humphrey, Mike Morgan and Diane Rogers-Ramachandran for helpful cornments.