The STM/working memory module plays the central part for performing the interactive processes between other associated modules within the AMS. In cognitive scientific/psychological studies, it is generally acknowledged that the STM (or working memory) is the “seat” for describing consciousness. (Further discussion of consciousness is left until Chap. 11).
8.3 Short-Term/Working Memory Module 137 In AMS, since both the functionalities of STM and working memory are rather considered to be complementary to each other, both the notions of STM and working memory can be treated within a single module; the term STM implies relatively short duration of retaining the information, in contrast to the LTM modules; whereas, under the name “working memory”, such infor- mation can be dealt, or even coordinated/deviated from the original, within the “working memory”, due to the interactive processes with the associated modules. Hence, the name “STM/working memory”.
Moreover, with respect to the short-term retention of information in mem- ory, it is considered in some studies in cognitive science/psychology (cf. Atkin- son and Shiffrin, 1968; Gazzaniga et al., 2002) that the notion of sensory mem- ory is also taken into account besides the STM. In the AMS context, however, whether such a further distinction is necessary or not may, again, be merely confined within the issue of implementation, as it can be seen that the no- tion of sensory memory in the structural sense is subsumed under the concept of the STM/working memory module and/or is already implemented within the sensation module; for instance, the length of the feature data in each pre-processing unit in Fig. 6.1 may be closely tied to the capacity of sensory memory. (The issue of implementation within the kernel memory concept will also be discussed later in Sect. 8.3.4.)
Although the full account/justifications for the functionality of the STM/
working memory in a cognitive scientific/psychological view point cannot be given in this book, we next consider one of the most influential working mem- ory models describing the “phonological loop” concept, which was originally developed by Baddeley and Hitch (Baddeley and Hitch, 1974), and how such a model can be interpreted within the AMS context.
8.3.1 Interpretation of Baddeley & Hitch’s Working Memory Concept in Terms of the AMS
In the psychological study (Baddeley and Hitch, 1974), Baddeley and Hitch proposed the model of working memory which extends the concept of STM such as the one in (Atkinson and Shiffrin, 1968), by introducing the concept of the so-called “phonological loop”, with some supportive neuropsychological arguments by the studies of patients with specific brain lesions (for the detail, see e.g. Gazzaniga et al., 2002). Their working memory is divided into three parts, i.e. a central executive mechanism and the two subordinate systems, namely, the phonological loop and visuospatial sketchpad, the latter two of which are controlled by the central executive system. Then, they explained both the forgetting mechanism of STM and the relation between the STM and LTM, e.g. the notion of how the transfer of memory from the STM to LTM can be performed, in terms of their working memory model. As the name “phonological loop” implies, the subordinate system is a mechanism for acoustically (or verbally) coding the information (i.e. sound inputs) in work- ing memory and is considered to perform the coding by subvocally rehearsing
138 8 Memory Modules and the Innate Structure
the items to be remembered over the short-term. In contrast, the “visuospa- tial sketchpad” functions separately from (but in parallel to) the phonological loop and performs the coding of the pure visual (or visuospatial) counterpart of the information within the working memory.
Moreover, it is anatomically considered that, apart from the well-known Brodmann’s area 40 (Brodmann, 1909), the rehearsal process in the phonolog- ical loop involves a region in the left premotor region (area 44), i.e. both the lateral frontal and inferior parietal lobes, whilst for the visuospatial sketchpad the parieto-occipital regions of both the left and right hemispheres of brain are the keys (for a concise review, cf. Gazzaniga et al., 2002)1.
As in Fig. 5.1 (on page 84), the STM/working memory module has the bi-directional connections with the three modules, i.e. 1)attention, 2)emo- tion, and 3) explicit LTM module, whilst the sensation, implicit LTM module, and the two output modules, i.e. both the primary output and perception(i.e.secondary output) modules, are all connected with mono- directional data flows. The latter two represent the feedback inputs to the STM/working memory module. Moreover, the two modules, i.e. 1)thinking and 2)intentionmodule, are considered to function in parallel.
Hence, it is considered that the model of the aforementioned STM/working memory concept (Atkinson and Shiffrin, 1968; Baddeley and Hitch, 1974;
Gazzaniga et al., 2002) is directly relevant to the interactive data process- ing between the STM/working memory and LTM (and/or the LTM oriented) modules, within the AMS context.
Then, it is considered that the model of working memory proposed by Baddeley and Hitch (Baddeley and Hitch, 1974; Baddeley, 1986) involves the following two data processes:
1) Thedata-fusionof both the auditory and visual sensory data within the STM/working memory module ;
2) The transfer of the outcome within the STM/working memory to the LTM module.
In the AMS context, the two processes in the above can be justified within the interactive data processing between the STM/working memory and LTM modules, as described next.
1In general AI, it is considered that, although such an anatomical placement for each functionality as described in the above is not always a crucial matter for mod- elling various cognitive/psychological functionalities, specifying the area/region for a certain function (i.e. the phonological loop/visuospatial sketchpad in the working memory) can greatly facilitate in “understanding” of such function. However, since not only a real brain is a totally complex system but the measurements currently available are limited in the capacity, to elucidate precisely the functionalities, such area/regional specification still remains a hard task. Nevertheless, where appropri- ate, we consider this sort of anatomical place justifications in this book.
8.3 Short-Term/Working Memory Module 139 8.3.2 The Interactive Data Processing:
the STM/Working Memory ←→LTM Modules
In the data process 1) above, it is firstly considered that both the auditory and visual sensory data, which are received from theperceptionmodule and/or recalled from theLTMmodules (i.e. due to the requests from other associated modules such as attention or emotion), reside within the STM/working memory module over a certain (short) period of time. Imagine a situation e.g. that the STM/working memory module receives the auditory sensory (encoded) data from the sensation module, which has not yet been stored within a specific area of the LTM, whilst the visual data corresponding to the auditory counterpart have already been stored in advance (by the prior learning process; see Chap. 7) and recalled from the (modality-specific area of) LTM within the STM/working memory. (Thus, the former process represents the data flow;sensation→STM/working memorymodule, whereas the latter;LTM→STM/working memorymodule)
Then, a reinforcement (or target) signal is given (in a certain manner, i.e. by the interactive processes between the memory modules, as described in the previous chapter) to associate the auditory data received from the sensation module with the visual counterpart via the learning process. In the sequel, this can cause the “data-fusion” of both the auditory and visual data.
In terms of the kernel memory, this data-fusion process can be ultimately interpreted as (merely) establishing a connection between one kernel unit with the template vector set to the auditory data and another with the visual counterpart, within the STM/working memory module. For representing this establishment, the principle of SOKM (in Chap. 4), in which the simultaneous activation of the kernel units can eventually lead to the formation of the link weight(s) in between, can be exploited. Hence, it is also said that this process simulates a general notion of learning, e.g. the situation where a child is about to learn/associates the visual part of a new word (“learnt by heart” in advance) with the auditory counter part.
Next, for the data process 2) above, the data transfer, which represents the data flow, i.e.STM/working memory →LTMmodule(s), can occur, if (as in the aforementioned phonological loop concept) the outcome of the data-fusion, which can be given in the form of a kernel network consisting of multiple kernel units within the STM/working memory, resides within the STM/working memory for a certain (sufficiently long) period of time. In this regard, it is said that the data transfer, i.e. the STM/working memory → LTM modules, simulates the role of the hippocampus in the neurophysiological context (for a concise review of the studies, see e.g. Gazzaniga et al., 2002).
Therefore, in summary, by examining the two data processes 1) and 2) above, the following three data flows between the three modules, i.e. the STM/working memory, LTM, and the input: sensation modules, can be drawn, as depicted in Fig. 5.1:
140 8 Memory Modules and the Innate Structure
• Sensation−→ STM/Working Memory Module
Represents the receipt of the (encoded) data from the sensation module; the sensory data will be used for the data-fusion within the STM/working memory module.
• STM/Working Memory −→LTM Modules
Represents the transfer of the transient data or consolidation of the kernel networks (i.e. composed by multiple kernel units and the link weights in between), which have survived after a sufficiently long period of time, within the STM/working memory module to the LTM module(s). In addition, this sort of transfer/consolidation can be occurred intermittently.
• LTM Modules−→ STM/Working Memory Module Represents the memory recall of the data stored within the LTM module(s); as in the first data flow:sensation−→STM/working memory module, the recalled data will also be used for the data- fusion within the STM/working memory module, where necessary.
Although the description of the three data flows in the above is limited to the case of the data-fusion where both the auditory and visual data are only considered, within the AMS context, this can be generalised to any combina- tion of the sensory data, without loss of generality.
8.3.3 Perception of the Incoming Sensory Data in Terms of AMS In AMS, it is considered that, once sensory data are received by the AMS, the perception is (normally) performed via the STM/working memory module;
after receiving the sensory data from the sensation module, the data are directly transformed into the respective kernel units within the STM/working memory module and also sent to the corresponding modality-specific area of the implicit LTMmodule. Then, the data transfer to the implicit LTM module immediately yields (a series of) the perceptual outputs obtained as the pattern recognition results from theperceptionmodule (as described in Chap. 6. Hence, in such a case, it can also be seen that the STM/working memory acts as the sensory memory). Eventually, the recognition results are fed back to the STM/working memory module; the perceptual outputs which are given as the feedback inputs to the STM/working memory module may be alternatively represented by the symbolic kernel units (with the kernel function given as (3.11)).
Therefore, performing the perception of the sensory data in terms of AMS involves the following four data flows:
1) Sensation −→STM/Working Memory 2) STM/Working Memory−→Implicit LTM
3) Implicit LTM −→Perception
4) Perception −→STM/Working Memory
8.3 Short-Term/Working Memory Module 141 Normally, it is considered that the perception of the incoming data in 1–4) above can be immediately performed. However, how rapidly/correctly the data processing within 1) and 2) can be performed also depends upon the current states of the STM/working memory and the associated modules (i.e.attention,emotion,intention, and/orthinkingmodule), as described later.
Although the descriptions of the data flows between the STM/working memory and other associated modules, such as attention or emotion, are left to the later chapters, we are now ready to consider modelling the STM/working memory module in terms of the kernel memory, as described in the next subsection.
8.3.4 Representation of the STM/Working Memory Module in Terms of Kernel Memory
Figure 8.1 shows an illustration of the STM/working memory module in terms of the kernel memory representation and the relationship between a total of the nine associated modules, i.e. 1)attention, 2)emotion, 3,4) bothexplicit and implicit LTM, 5)intention, 6,7) bothprimary and secondary (per- ceptual) outputs, 8) sensation, and 9)thinking module (also, compare Fig. 8.1 with Fig. 5.1 on page 84).
As in the figure, the STM/working memory module consists of multiple kernel units, as well as the explicit/implicit LTM modules, and is (partially) connected to both the LTM modules, by means of the link weights between the kernel unitsKiS (i= 1,2, . . . , NS)2andKjEand/orKkI (j= 1,2, . . . , NE, k= 1,2, . . . , NI), where, in each memory module, the number of kernel units is (in practice) assumed to be upper limited, i.e.NS ≤NS,max,NE ≤NE,max, andNI ≤NI,max.
In Fig. 8.1, as indicated by the corresponding data flows, the STM/working memory also receives the feedback inputs from both the primary and sec- ondary (i.e. perceptual) outputs (albeit not explicitly shown for the latter in Fig. 8.1), apart from the sensory inputs; in practice, the STM/working mem- ory module is initially considered as an empty kernel memory space, and, whenever either the incoming data from the sensation module or the feedback inputs from the primary/secondary (i.e. perceptual) output modules are given to the STM/working memory, we may i) create new kernel units one by one or ii) replace some existing ones (i.e. by taking into account the factorNs).
2For convenience, in Fig. 8.1, the kernel units with the superscript “S” stands for those within the “STM/working memory”, whereas the superscripts “E” and
“I” denote respectively the “explicit LTM” and “implicit LTM”. In addition, note that, as aforementioned, since here both the sensory memory and STM are treated within a single module in the AMS context, the maximum number of the kernel unitsNS,maxmay be set to a relatively large value, by taking into account the large capacity of sensory memory compared to the STM (for this argument, see p.305 of Gazzaniga et al., 2002).
142 8 Memory Modules and the Innate Structure
.. .
K4S K1S
Primary Output:
Behaviour, Motion, Direction, Endocrine
.. .
K2I K3I K1I
K4I
...
...
...
.. . .. . .. .
.. .
.. .
K2E K1E
4 KE
K3E K3S K2S
Implicit LTM Modules Functioning in Parallel
Interactive Modules /
Output Inputs
Sensory
STM / Working Memory
KI Input: Sensation
KS
NS NI
Explicit LTM
KE NE
Secondary (Perceptual) 4) Thinking Module
3) Intention Module 2) Emotion Module 1) Attention Module
Fig. 8.1.An illustration of the STM/working memory module in terms of the kernel memory, consisting of multiple kernel units, and the relationship between the nine associated modules, i.e. 1)attention, 2)emotion, 3,4) bothexplicit and implicit LTM, 5)intention, 6,7) bothprimary and secondary (perceptual) outputs, 8)sensation, and 9)thinkingmodule
For both the cases i) and ii), such kernel units are formed, with the tem- plate vectors (or matrices) identical to those incoming data/feedback inputs within the STM/working memory module. Then, the data, which are stored in the form of the template vectors within the kernel units so formed, will be im- mediately sent to the areas corresponding to the respective modality-specific areas of the kernel units within the LTM modules. Thus, in the case of pre- senting them to the implicit LTM, we may obtain (a series of) the perceptual outputs (e.g. of a particular object(s)) from the secondary output module, which can be given as the cause of the activations of the kernel units within such areas of the implicit LTM module.
For the feedback inputs, it is also possible that they can be (alternatively) represented in terms of symbolic kernel units, instead of exploiting the regular kernel units.
8.3 Short-Term/Working Memory Module 143 8.3.5 Representation of the Interactive Data Processing Between the STM/Working Memory and Associated Modules
In the later part in Sect. 8.3.2, the three data flows relevant to the STM/
working memory module; i.e. 1)sensation −→ STM/working memory;
2) STM/working memory −→ LTM modules; and 3) LTM modules
−→STM/working memory, were established, by examining Baddeley and Hitch’s working memory concept. In this subsection, we consider how these processes can be actually represented within the kernel memory principle.
1) Data flow: Sensation−→STM/Working Memory
In Fig. 8.1, the data processing 1) sensation −→ STM/working mem- oryis represented by the data flow from thesensationmodule (which con- sists of a cascade of the pre-processing units, as described in Chap. 6) to the STM/working memory module; the encoded data obtained through a series of the pre-processing units are directly i) given as the input to or ii) used as the respective template vectors to form the kernel units within the STM/working memory. (For the former i), if we consider a Gaussian kernel unit as given by (3.8), the input vectorxcorresponds to such encoded data. For either the case i) or ii), we may consider the principle similar to the construction of the SOKM given in Sect. 4.2.4.
2) Data flow: STM/Working Memory −→LTM
Then, for representing the data flow 2)STM/working memory−→ LTM modules, it is considered that there are the two types of processing involved;
i) generation of the perceptual outputs via the LTM modules, due to the activations of the kernel units within the STM/working memory module as aforementioned in the previous subsections, i.e. by the incoming sensory data or thinking process, and ii) the transfer (or transition) of the kernel units from the STM/working memory to the LTM modules (as in the Baddeley and Hitch’s working memory described in Sect. 8.3.1).
For ii), a condition must be given to the STM/working memory module;
the kernel units swiftly disappear from the STM/working memory module3, or are replaced by those with different parameter settings, as aforementioned, unless they are transferred to the LTM modules within a certain period of time.
3In the case of hardware representation, it does not imply that such “disappear- ance” of the kernel units can actually occur, but rather, the parameters of some kernel units, i.e. the template vectors, link weights, etc, can be reset/become com- pletely different, e.g. when new incoming data arrive at the STM/working memory module.
144 8 Memory Modules and the Innate Structure 3) Data flow: LTM−→ STM/Working Memory
Thirdly, the data flow 3)LTM modules−→STM/working memoryde- picts the recall of the data stored within the LTM modules, due to e.g. the request by the other associated modules.
However, as aforementioned in Sect. 8.2, the third data flow doesnot al- ways imply that the kernel units are actually transferred back (or copied) from the LTM to the STM/working memory module, but, rather, the acti- vated kernel units within the LTM modules are just monitored by marking them and then holding the information of the absolute locations, etc, within the auxiliary memory space4 that may alternatively represent the STM part of the STM/working memory module. In the AMS context, it is also possible to consider that such auxiliary memory can be represented within theinten- tionandthinking modules, both of which are considered to work in parallel with the STM/working memory module. (We will then return to this issue in Chaps. 9 (Sect. 9.3) and 10 (Sect. 10.4)).
Within a similar context as above, both the two feedback inputs, i.e. the data flow primary output −→ STM/working memory and that sec- ondary output−→ STM/working memory, are depicted (dashed lines) in both Figs. 5.1 (on page 84) and 8.1 (i.e. for the former only, as described ear- lier), since these feedbacks are already represented by the monitoring process of the activations from the kernel units within the LTM modules, the process of which is performed by the STM/working memory module.
8.3.6 Connections Between the Kernel Units within the STM/Working Memory, Explicit LTM, and Implicit LTM Modules
Now, consider a situation where there are multiple kernel units KiS (i = 1,2, . . . , Ns) formed within the STM/working memory, as in Fig. 8.1, and each kernel unitKiS is represented in either form depicted in Fig. 3.1 (on page 32) or Fig. 3.2 (on page 37). Then, as illustrated in Fig. 8.1, it is considered that there can be the following five types of the connections between the kernel units (via the link weights):
i) Connection betweenKiS andKjS (i=j) ; ii) Connection between KiS andKjE orKkI ; iii) Connection between KiE andKjE (i=j) ; iv) Connection betweenKiE andKjI ;
v) Connection betweenKiI andKjI (i=j)
The establishment of the connections as in the above can be achieved by e.g. following the Hebbian learning principle as in the SOKM (in Chap. 4);
4Here, the notion of auxiliary memory is different from that of a kernel unit.