In Hebb (1949) (p.62), Hebb postulated, “When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased.”
In the SOKM, the “link weights” (or simply, “weights”) between the ker- nels are defined in this neuropsychological context. Namely, the following con- jecture can be firstly drawn:
Conjecture 1: When a pair of kernelsKi andKj (i=j,i, j∈ {all indices of the kernels}) in the SOKM are excited repeatedly, a new link weightwij betweenKi and Kj is formed. Then, if this occurs intermittently, the value of the link weightwij is increased.
In the above, Hebb’s original postulate for the adjacent locations of cell A and B is not considered; since, in actual hardware implementation of the proposed scheme (e.g. within the memory system of a robot), it may not always be crucial for such place adjustment of the kernels. Secondly, Hebb’s postulate implies that the excitation of cell A may occur due to thetransfer of activations from other cells via the synaptic connections. This can lead to the following conjecture:
Conjecture 2: When a kernel Ki is excited and one of the link weights is connected to the kernelKj, the excitation ofKi is trans- ferred toKjvia the link weightwij. However, the amount of excita- tion depends upon the (current) value of the link weight.
4.2.1 An Algorithm for Updating Link Weights Between the Kernels
Based uponConjectures 1 and 2above, the following algorithm for updat- ing the link weights between the kernels is given:
[The Link Weight Update Algorithm]
1) If the link weightwij is already established, decrease the value according to:
wij=wij×exp(−ξij) (4.1)
4.2 The Link Weight Update Algorithm (Hoya, 2004a) 61 2) If the simultaneous excitation of a pair of kernelsKi and
Kj (i=j) occurs (i.e. when the activation is above a given threshold as in (3.12);Ki≥θK) and is repeatedptimes, the link weightwij is updated as
wij =
winit ; ifwij does not exist wmax ; else ifwij > wmax wij+δ; otherwise.
(4.2) 3) If the activation of the kernelKi unit does not occur dur- ing a certain period p1, the kernel unit Ki and all the link weights connected to the kernel unitwi(= [wi1, wi2, . . .]) are removed from the SOKM (thus, representing theextinction of a kernel).
whereξij, winit, wmax, and δare all positive constants. In 2) above, after the weight update, the excitation counters for both Ki and Kj, i.e. εi and εj, may be reset to 0, where appropriate. Then, both conditions 1) and 2) in the algorithm above also moderately agree with the rephrasing of Hebb’s principle (Stent, 1973; Changeux and Danchin, 1976):
1. If two neurons on either side of a synapse are activated asynchronously, then that synapse is selectively weakened or eliminated2.
2. If two neurons on either side of a synapse (connection) are activated si- multaneously (i.e. synchronously), then the strength of that synapse is selectively increased.
4.2.2 Introduction of Decay Factors
Note that, to meet the second rephrasing above, a decaying factor is intro- duced within the link weight update algorithm (in Condition 1), to simulate the synaptic elimination (or decay). In the SOKM context, the second rephras- ing is extended and interpreted such that i) the decay can always occur in time (though the amount of such decay is relatively small in a (very) short period of time) and ii) the synaptic decay can also be caused when the other kernel(s) is/are activated via the transmission of the activation of the kernel. In terms of the link weight decay within the SOKM, the former is represented by the factor ξij, whereas the latter is under the assumption that the potential of the other end may be (slightly) lower than the one.
At the neuro-anatomical level, it is known that a similar situation to this occurs, due to the changes in the transmission rate of the spikes (Hebb, 1949;
Gazzaniga et al., 2002) or the decay represented by e.g. long-term depression
2To realise the kernel unit connections representing the directional flows as de- scribed in Sect. 3.3.4, this rephrasing may slightly be violated.
62 4 The Self-Organising Kernel Memory (SOKM)
(LTD) (Dudek and Bear, 1992). These can lead to modification of the above rephrasing and the following conjecture can also be drawn:
Conjecture 3: When kernel Ki is excited by inputx andKi also has connection to kernelKj via the link weightwij, the activation ofKj is computed by the relation
Kj =γwijKi(x) (4.3)
or
Kj=γwijIi (4.4)
where γ (0 << γ ≤1) is the decay factor, and Ii is defined as an indicator function
Ii= 1 ; if the kernelKi(x) is excited (i.e. whenKi(x)≥θK)
0 ; otherwise. (4.5)
In the above, the indicator functionIi is sufficient to imitate the situation where an impulsive spike (or action potential) generated from one neuron is transmitted to the other via the synaptic connection (for a thorough discus- sion, see e.g. Gazzaniga et al., 2002), due to the excitation of the kernelKiin the context of modelling the SOKM. The above also indicates that, apart from the regular input vectorx, the kernel can be excited by the secondary input, i.e. the transfer of the activations from other nodes, unlike conventional neural architectures. Thus, this principle can be exploited further for multi-domain data processing (in Sect. 3.3.1) by SOKMs, where the kernel can be excited by the transfer of the activations from other kernels so connected, without having such regular inputs.
In addition, note that another decay factor γ is introduced. This decay factor can then be exploited to represent a loss during the transmission.
4.2.3 Updating Link Weights Between (Regular) Kernel Units and Symbolic Nodes
In Figs. 3.4, 3.5, 3.7, and 3.8, various topological representations in terms of kernel memory have been described. Within these representations, the final network output kernel units are newly defined and used, in addition to regular kernel units, and it has been described that these output kernel units can be defined in various manners as in (3.16), (3.17), (3.18), (3.25), (3.28), or (3.30), without directly affecting the contents of the memory within each kernel unit. Such output units can thus be regarded as symbolic nodes (as in conventional connectionist models) representing the intermediary/internal states of the kernel memory and, in practice, exploited for various purposes,
4.2 The Link Weight Update Algorithm (Hoya, 2004a) 63 e.g. to obtain the pattern classification result(s) in a series of cognitive tasks (for a further discussion, see also Sects. 4.6 and 7.2).
Then, within the context of SOKM, the link weights between normal kernel units and such symbolic nodes as those representing the final network outputs can be either fixed or updated by [The Link Weight Update Algorithm]
given earlier, depending upon the applications. In such situations, it will be sufficient to define the evaluation of the activation from such symbolic nodes in a similar manner to that in (3.12).
Thus, it is also said that the conventional PNN/GRNN architecture can be subsumed and evolved within the context of SOKM.
4.2.4 Construction/Testing Phase of the SOKM
Consequently, both the construction of an SOKM (or the training phase) and the manner of testing the SOKM are summarised as follows:
[Summary of Constructing A Self-Organising Kernel Memory]
Step 1)
• Initially (cnt= 1), there is only a single kernel in the SOKM, with the template vector identical to the first input vector presented, namely,t1=x(1) (or, for the Gaussian kernel,c1=x(1)).
• If a Gaussian kernel is chosen, a unique setting of the radiusσmay be used and determineda priori(Hoya, 2003a).
Step 2)
For cnt = 2 to {num. of input data to be presented}, do the following:
Step 2.1)
• Calculate all the activations of the kernels Ki (∀i) in the SOKM by the input data x(cnt), (e.g. for the Gaussian case, it is given as (3.8)).
• Then, if Ki(x(cnt)) ≥ θK (as in (3.12)), the kernelKi is excited.
• Check the excitation of kernels via the link weights wi, by following the principle in Conjecture 3.
• Mark all the excited kernels.
Step 2.2)
If there is no kernel excited by the input vector x(cnt), add a new kernel into the SOKM by setting its template vector tox(cnt).
64 4 The Self-Organising Kernel Memory (SOKM)
Step 2.3)
Update all the link weights by following [The Link Weight Update Algorithm] given above.
In Step 1 above, initially there is no link weight but a single kernel in the SOKM and, later in Step 2.3, a new link weight may be formed, where appropriate.
Note also that Step 2.2 above can implicitly prevent us from generating an exponentially growing number of kernels, which is not taken into con- sideration by the original PNN/GRNN approaches. In another respect, the above construction algorithm can be seen as the extension/generalisation of the resource-allocating (or constructive) network (Platt, 1991), in the sense that 1) the SOKM can be formed to deal with multi-domain data simulta- neously (in Sect. 3.3.1), which can potentially lead to more versatile applica- tions, and 2) lateral connections are also allowed between the nodes within the sub-SOKMs responsible for the respective domains.
[Summary of Testing the Self-Organising Kernel Memory]
Step 1)
• Present input dataxto the SOKM, and compute all the kernel activations (e.g. for the Gaussian case, this is given by (3.8)) within the SOKM.
• Check also the activations via the link weightswi, by following the principle in the aforementioned Con- jecture 3.
• Mark all the excited kernels.
Step 2)
• Obtain the maximally activated kernel Kmax (for instance, this is defined in (3.17)) amongst all the marked kernels within the SOKM.
• Then, if performing a classification task is the objec- tive, the classification result can be obtained by sim- ply restoring the class label ηmax from the auxiliary memory attached to the corresponding kernel (or, by checking the activation of the kernel unit indicating the class label, in terms of the alternative kernel unit representation in Fig. 3.2).