Comparisons of Stimulus Learning and Response Learning in a Punishment Situation

Bolles, R.C., Holtz, R., Dunn, T., & Hill, W. (1980). Comparisons of Stimulus Learning and Response Learning in a Punishment Situation. Learning and Motivation, 11, 78-96.

Early on in the study of learning, learning was believed to consist of the attachment of a response to a stimuli (an S-R association). Later, in contrast with the long-held conventional view that all learning was of the S-R form, alternative forms were proposed by Pavlov and others. Examples include stimulus learning (S-S*) and response learning (R-S*). What type of learning underlied punishment, for example? This stimulated debate. In common punishment paradigms, it became obvious that is was unclear whether the animal was learning that shock is correlated there contextually with the bar (stimulus) or whether shock is correlated with its behavior of pressing the bar (response). The purpose of this paper was to attempt to disentangle these different forms of learning experimentally. Four novel experimental paradigms were explored.

Experiment 1 contained one single bar in the chamber which could either be pressed or pulled. Animals had to alternate their behavioral response (press to pull, and back again) in order to be rewarded. Punishment was delivered on every tenth press for half the animals or every tenth pull for the other half. Results showed a rapid initial suppression (fear to environmental stimuli), but later a return to baseline for the unpunishment response and continued suppression for the punished response. This seems to be evidence for both types of learning taking place within one paradigm. Experiment 2 simply adjusted the response contingencies (up and down) to see if results would be sensitive to this type of experimental manipulation. Instead of an FR-10 punishment schedule, rats were shocked on FR-4 or FR-25. FR-4 showed dramatic differences in responding from the outset. FR-25 differences only emerged in the second day of punishment.

Experiment 3 used two bars, each of which could be either pressed or pulled. Thus, four punishment conditions were possible, punishing a left-press, a left-lift, a right-press, or a right-lift. Results showed that when a rat was punished for a left-lift, for example, it quickly stopped lifting AND pressing the left bar. However, it continued to lift AND press the right bar. Thus, learning, in this case, seems to be mostly about stimuli. Experiment 4, like Experiment 2, changed the contingencies. Punishment was shifted from an FR-1 schedule to an FR-10 schedule. Punishment conditions were: press or lift left bar, press or lift right bar, lifting left or right, and pressing left or right. Across the conditions, the general trend that emerged was a rapid emergence of stimulus learning and then a slower but undeniable development of response learning.

Symmetrical effects of amphetamine and alpha-flupenthixol on conditioned punishment and conditioned reinforcement: contrasts with midazolam

Killcross, A.S., Everitt, B.J., & Robbins, T.W. (1997). Symmetrical effects of amphetamine and alpha-flupenthixol on conditioned punishment and conditioned reinforcement: contrasts with midazolam. Psychopharmacology, 129, 141-152.

There is evidence to suggest that forebrain dopaminergic systems are likely to be involved in both appetitive and aversive motivation. These authors studied the effects of dopamine (DA) agents on conditioned punishment (aversive learning) and conditioned reinforcement (appetitive learning) paradigms using DA agonists and antagonists injected systemically. In conditioned punishment, a Pavlovian CS predicting punishment is added to an instrumental bar-pressing paradigm, but only on one of the bars. Normal animals will adjust their bar-pressing away from this lever. (Note: These authors, however, implemented a punishment procedure by presenting a CS and shock upon bar-press, rather than a conditioned punishment procedure which would only present the CS upon bar-press). In conditioned reinforcement, a CS predicting reward is added to the instrumental bar-pressing paradigm on one of the bars. Animals will naturally come to favor this bar paired with the appetitive CS.

DA agonists increased the effect of a punishing CS, causing the animals to further decrease their bar pressing. DA agonists also enhanced the effect of an appetitive CS, increasing bar pressing. DA antagonists, on the other hand, decreased the effect of a punishing CS. They also reduced the effect of an appetitive CS. Thus, it appears dopaminergic agents modulate the behavioral impact of both appetitively and aversively motivated conditioned stimuli on instrumental performance. Systemic benzodiazepene administration was also explored with results showing a selective impact on aversively-motivated stimuli (i.e. no effect on the appetitive CS).

Different lateral amygdala outputs mediate reactions and actions elicited by a fear-arousing stimulus

Amorapanth, P., LeDoux, J.E., & Nader, K. (2000). Different lateral amygdala outputs mediate reactions and actions elicited by a fear-arousing stimulus. Nature Neuroscience, 3, 1, 74-79.

One common reaction to threat is elicitation of unlearned species-typical defense reactions. Another is the ability to take novel actions in threatening situations, which may also prove advantageous. In escape-from-fear (EFF) tasks, animals come to learn that an arbitrary response (e.g. stepping into the adjoining chamber) terminates a CS which predicts shock. Authors placed lesions throughout the fear circuit to see try to elucidate function. Lateral amygdala (LA) is believed to be the sensory interface where CS information enters the amygdala; lesions here block acquisition of both conditioned freezing responses, as expected, but also block acquisition of the CS's reinforcement of a new response in the EFF task. Central nucleus (CE) lesions block expression of hard-wired motoric output (i.e. freezing itself), but not the EFF. Basal nucleus (B) lesions had no effect on conditioned freezing, but did block the EFF. By way of interactions between the B and striatal circuits, reinforcement in the amygdala may come to reinforce novel motor responses. Thus, it may be that activation of LA by a CS triggers a reactive response output system via the CE and an active output system via the B.

Conflict Monitoring and the ACC

Botvinick, M.M., Cohen, J.D., & Carter, C.S. (December 2004). Conflict monitoring and anterior cingulate cortex: An update. Trends in Cognitive Sciences, Vol. 8, No. 12, 539-546.

Activity in the dorsal anterior cingulate cortex (ACC) shows up in a variety of tasks. For example, a transient potential (known as the error-related negativity, or ERN) is elicited from the posterior ACC in response to error commission. And a similar evoked potential, the feedback-related negativity (FRN), occurs in response to error feedback and may derive from the same portion of the cingulate that generates the ERN. These activity patterns during commission of errors led researchers to suggest an 'error detection' function for the ACC.

However, tasks which require overriding of habitual responses and tasks which require selecting among a set of equally permissible responses also yield ACC activation. This led researchers to put forth a 'conflict detection' theory of ACC function, with the structure being especially responsible for selecting between competing motor responses.

However, recent studies have proposed other unifying theories to explain the role of the ACC beyond just error detection and conflict monitoring. Some suggest the ACC serves to evaluate action outcomes, performing cost-benefit analyses on possible outcomes and using reward-related information to guide action selection. This 'action-outcome evaluation' view is particularly consonant with other research connecting the mesencephalic dopamine system with the ACC in reinforcement learning.

Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops

Tanaka, S.C. et al. (August 2004). Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nature Neuroscience, Volume 7, Number 8, 887-893.

Curiously, lesions in the nucleus accumbens in rats result in a tendency to choose small immediate rewards over larger future rewards. And low activity in the central serotonergic system is also associated with impulsive behavior in humans. This article argues that the lateral orbit-frontal cortex (OFC) takes on the role of predicting immediate rewards, while the dorsolateral prefrontal cortex (dlPFC), dorsal pre-motor cortex (dMC), and inferior parietal cortex (IPC) are involved in prediction of future outcomes. Therefore, different sub-loops of the cortico-basal ganglia network are specialized for reward prediction at different time scales. The last piece of the puzzle, the dorsal raphe nucleus, is thought to use serotonin influence to control the effective time scale of reward prediction, allowing flexible selection of a relevant time-scale appropriate for the task at the time of decision-making.

The Neural Basis of Human Error Processing

Holroyd, C.B. and Coles, M.G.H. (2002). The Neural Basis of Human Error Processing: Reinforcement Learning, Dopamine, and the Error-Related Negativity, Psychological Review, Vol. 109, No. 4, 679-709.

It is clear that our superior ability to learn from consequences makes us quite exceptional animals. But what are the neural substrates for reinforcement learning? This article takes a stab at that question.

Researchers have inferred the existence of a generic, high-level error-processing system in the brain for some time. When human participants commit errors in a wide variety of psychological tasks, a negative deflection is witnessed in EEG data, deemed the error-related negativity (ERN), which appears to be generated from anterior cingulate cortex (ACC). On the other hand, researchers have argued that the mesencephalic dopamine system conveys reinforcement learning signals to the basal ganglia and frontal cortex, where they are used to facilitate development of adaptive behavioral programs. This article proposes a hypothesis which unifies the two – specifically, when human subjects commit errors the dopamine system conveys a negative reinforcement learning signal to the frontal cortex where it generates an ERN by disinhibiting the dendrites of motor neurons in the ACC (hence the negative potential seen).

To dig a bit deeper... first the mesencephalic dopamine system. This is a small collection of nuclei including the substantia nigra pars compacta and the ventral tegmental area (VTA) that project diffusely to the basal ganglia and frontal cortex. The consequence of stimulation from this area appears to reinforce learning, solidifying behavior. After learning how to complete a task properly, presentation of a reward elicits a phasic response in dopamine neurons. When a reward is better than predicted, a positive dopamine signal is elicited. And, as expected, when (i) an expected reward is not delivered, (ii) a reward is worse than predicted, or (iii) punishment is administered instead, mesencephalic dopamine neurons decrease their firing rate, falling below baseline. Interestingly, over time and practice on a task, the presentation of the reward no longer elicits the phasic dopaminergic response; instead, the conditioned stimulus predicting delivery of the reward elicits the phasic activity. As such, the phasic dopaminergic activity is said to propagate “back in time” from the reward to the conditioned stimulus with learning. Thus, the mesencephalic dopamine system can be understood to produce predictive and critical error signals which can be used by other parts of the brain for reinforcement learning.

Now the ERN. The ERN is a negative wave pattern witnessed during commission of an error, essentially the brain’s “Oh shit!” signal. The amplitude of the ERN increases with incentive (e.g. financial reward) and with the degree of error (i.e. very wrong errors as opposed to only slightly incorrect). The ERN can be elicited (i) by presentation of negative feedback to the participant, or (ii) by detection of error commission itself.

The theory that this paper offers is that the ACC, which generates the ERN and receives input from numerous semi-independent command structures, is responsible for conflict monitoring in the brain – detecting competing choices from these multiple motor controllers and resolving the response conflict. Its job is therefore to identify which of its input are best suited for carrying out the task – serving as a motor control filter – and finally transforming multiple intentions into a unitary action. They contend that the ACC is “trained” into choosing the correct controller by the mesencephalic dopamine system’s reinforcement learning signals, and that the ERN essentially reflects transmission of this reinforcement learning signal to the ACC. The mesencephalic dopamine system, then, plays the role of an adaptive critic, assigning a “goodness” or “badness” to witnessed response outcomes, and communicates its opinion to the ACC to bias future decision-making.

The article used behavioral data (a probabilistic learning task) to support this hypothesis, as well as computer simulation modeling which predicted the observed behavioral results.