Modernization Hub

Modernization and Improvement

The Gamification Report | Episode 10: Cognitive/Emotional Relationships, Neurotourism, and more!


David Chandross: Welcome to the Gamification
Report, episode 10. This is a special issue. We’re going to be looking at neuroscience
today. We’re going to start with some really interesting
papers looking at cognitive-emotional perspectives in serious games. Some material that’s … Some of its prepublication
manuscripts we’ve been able to obtain. We’re going to look at this area of neuro-tourism. Remember, we spoke in our last episode about
virtual diving. Talk about neuro-tourism and how it’s picking
up. We’re going to be looking at brain monitoring
in neuroscience, neuroscience gamification, and mirror neurons and serious simulation
games. These are some really fascinating areas which
all converge on this area of higher learning using gamification and the neuroscience digital
neuropsychology. We’re going to start with a paper by Jeffrey
K. Mullins and Rajiv Sabherwal at the University of Arkansas, 2018. What they looked at here was the connection
of emotion to serious educational games, what they call a cognitive-emotional perspective. This is on based on the fact that there are
three dominant emotional theories, what’s called the differential emotions theory, the
cognitive emotion theory, and appraisal theory. Let’s dig a little bit deeper. Theories of emotion, and we know this is an
area that’s just exploding. For example, cannabis is becoming legalized
over the next couple of months here in Canada. Cannabis operates by stimulating the anandamide
receptors in the cortex, and different areas of the brain. There’s actually many locations for these
receptors. We don’t have these cannabinoid receptors
for getting high on weed, that’s not the reason nature designed them. That’s a side effect of the drug use. What the anandamide receptor seems to do is
it links emotion to learning, and this is why in many different studies, every study
that we looked and we’ve used showed that cannabis interfered with learning in rats
and in all kinds of test animals, because those receptors actually connect emotion to
learning. Emotion and learning are something that we’re
beginning to understand much more on a molecular level now. Let’s just start with a bit of a shallow dive
to look at basic theories of emotion and understand how gamification can fit into emotional base-learning
in a more nuanced way that we might’ve talked about a few years ago. The first theory of emotion is called differential
emotion, and that is what we have what are called none cognitive emotions, which are
innate and develop early, and that we develop social cognitive emotions later in life. Then cognitive emotions are various things
like, “Hey, you took my toy,” or, “Geeze, I’m hungry,” or, “I miss you Mommy.” Social-cognitive emotions might have to do
with how you interact with other people and might have to do with developing more complex
social structures. The cognitive-emotional theory of emotion
is that all emotions result from cognitive appraisal, whether it’s automatic or volition. All of our emotions have to do with some kind
of cognitive, that is thinking or intellectual tasks, and emotion arises in response to the
task. Being chased by the saber-tooth tiger, if
something threatens your survival, so the emotion arises out of the cognitive content. Then the appraisal theory of emotion is that
emotions result from the unconscious strategies we develop through coping, so that emotions
actually arise from the uncomfortable and that they all have to do with basic biological
coping mechanisms. Again, we’re not going to get too much into
the psychology, but it gives us a bit of a groundwork to explore the topic a bit further. Let’s talk a little bit about time bending. The amygdala is a part of the brain which
is a critical hub which regulates flow and integration between the brain regions and
cognitive-emotional interactions. Emotion involves connecting different parts
of the brain which might have to deal with visual perception or some outer-sensory perception,
and thought, and experience. We know an amygdala, we’ve known this for
a long time, that things like fear processing and focusing attention to potential threats
occurs in the amygdala. Encoding, consolidation, and subjective recollection
of memories are all links to emotional stimulus. Emotions are very, very strongly linked to
learning in terms of the areas such as the ventral striatum. The ventral striatum is the emotional reward
learning center of the brain. When we learn to appreciate a certain wine
or we like a certain type of music, or we want to connect some kind of more physical,
more professional skill to learning, this all has to do with attaching reward to learning. The perspective they’re using as we look at
this paper is that serious educational gains can activate the amygdala more effectively
than other experiences in learning, and that’s the core that I think we’re all arriving towards. You know we’re all marching in this field
of gamification towards some kind of distant goal post, and where it appears that we’re
moving now is trying to understand how the amygdala becomes engaged. How games actually connect emotional learning
in ways that factual learning doesn’t really occur, in kind of ways that we may have spoken
in the past. There’s the amygdala, we can see in this little
rotating image, the amygdala right here, it looks like an almond. This is the hippocampus, this blue area, which
is our kind of consolidation and memory area of the brain. All of these networks are quite complex, but
we’re again, looking at the primary drive. What they conclude, and spelling out a very
well-reasoned hypothesis, is by integrating a map of the structure emotions within game
mechanics, you can elicit various emotions, and therefore designers can draw on this perspective
to develop gamified design. We can talk about gamified design of learning
through manipulating emotional circuitry, and I think this cognitive emotional bit of
the iceberg is something that’s really going to help us move forward in the field because
quite frankly I don’t think this level of psychological analysis has been applied to
serious educational games in the past, and now that we see a more microscopic kind of
dissection or the breakdown of different types of emotion that occur during learning and
being able to use a game element to pull out a fear, or competitiveness, or reassurance,
or perhaps storification some of these elements. We want to keep our eyes on this ball, and
it gives us something very real to work on. When we want to teach at a higher level and
more effective level, we have this complex emotional experience and we want to persuade
learners, we want to use as close as a persuasion through what we would call engineered experience. They would then conclude in this paper that
there was a call for serious experience in serious games, and the idea that persuasive
health messages. For example when we talk about smoking cessation
campaigns, or drug avoidance campaigns, et cetera, all use persuasive messaging. We’re looking at it in a more nuanced way
that there are ways through using serious educational games to trigger specific sets
of emotional responses that are more effective for transference of learning, and locking,
and learning. This is fascinating stuff that’s developing. Now I think you all remember this great little
footage from National Lampoon’s Family Vacation, and the idea that we are moving toward neuro-tourism,
and I think this is an interesting segue that was explored by Emese Panyik and José Gonçalves
in 2018. Again, this takes the idea of emotionality
and brings it to something that we can all relate to, that which is neuro-tourism. Tourist behaviors and emotions are very, very
key. If you have a really great trip down to a
wonderful place, like you visit Quinovic or you go down to Africa and you explore Zambezi,
you can understand that that creates an emotional impact. What underlies tourism as in many activities
is delivering an emotional sense of satisfaction. As we talked about in our last podcast, augmented
reality and virtual reality can then deliver tourist experience. Where you are actually going to be becomes
secondary to the actual tourism experience. I know this is probably not the best example,
but think about how good you feel after visiting a friend who’s got a great pool in their backyard
and spending a day and then having a nice barbecue. Even if it’s a one hour drive to visit them,
it’s that we actually switch our neurological circuitry to get the emotional experience
equated with what we call vacation or tourism, and that we can actually recreate those experiences
by some laymen, which is why a lot of people might have swimming pools. For example, Elon Musk produced something
called Neuralink, and that’s an implantable brain computer interface which will eventually
produce a non-invasive brain to brain communication. You can look at his work, I mean it’s received
quite a bit of … I mean even a Wikipedia kind of entry on it would take you fairly
deep into this idea of using Neuralink. Then this word, this concept is explored by
some other authors. We found the work of Diego Sempreboni and
Luca Viganò at Kings College 2018 in which they look at the what they call Mind Mining. This is the idea again, of taking these ideas
we talked about in neuro-tourism, but taking these in terms of persuasive interaction with
a website. Anger, frustration, and curiosity drive websites,
they’re designed for maximum customer visits. We know that the reward systems that you build
when you’re building a website have to do with health and wealth-related. One fantasy was Black Mirror’s Nosedive. Black Mirror was a British serious shown on
Netflix in which we can actually start to report on each other’s behavior, and we actually
enter a very kind of dark space right now called crypto jacking, and this is where we
actually take over your brain by using digital neuropsychology to take a certain percentage
of brain, cognitive, time, and attention, and behaviors related to core messages. So when you go to any specific website as
our colleague Brian Cugelman who works with AlterSpark which is a leader in the area of
digital neuropsychology. There are digital neuropsychologist being
hired by very large companies now, and this is the idea of crypto jacking, so essentially
you get apps on your phone that then drive you to specific behaviors, and they’re essentially
moving you into purchase decisions, and to nail your cognitive attention and behaviors. I said that this is reflected in some very
dystopic kind of visions, and this is what we call this idea of Mind Mining, of trying
to take any bit of cognitive space that we have left over, well beyond all the things
we have to do in our life, taking that cognitive space and having us focus on doing something
very small related to a purchase behavior. It’s really insidious kind of stuff, but we
need to understand that that’s kind of at the forefront. There are of course some other developments
in this area, the idea of this active neuroscience called Brain Monitoring Devices. Bill Byron et al publishes in 2018 this idea
of wellness development, which is going to be a $612 billion industry by 2024. In 2015, wellness development is a $123.2
billion dollar industry. These are essentially wearable devices that
measure your brain activity which are around the forehead, and the signals are collected
as you think and process by a series of dry electrodes which are then filtered and interpreted
by firmware to provide a continuous EEG signal trace. You’re able to do things like access your
own brain waves and use that feedback to achieve different states of consciousness. To give us maybe a little more to bite into,
examples that are already on the market and being developed would include MUSE, which
is being developed in InteraXon in Toronto, Canada. Emotiv EPOC in Sydney, Australia. ZenZone, developed by NeuroSky in San Jose,
California. The MUSe device consists of a headband that
you wear across your forehead with seven sensors positioned across the forehead and it’s behind
each year. This, essentially you put on this MUSE device,
so that’s collecting electro-physiological data, and by using an element of game-like
elements in terms of setting targets, you could use it for example pain control, or
you could use it for wakefulness. You could use it for hunger control and in
different appetite and drive control. This whole idea of these early Neuralink ideas,
but being used for the individual’s wellness development. EEG devices and pain are being used more and
more. PainQx, coming out of New York, New York,
uses research-grade and portable EEG, electroencephalograms again, to assess neural brain activity, and
it uses algorithms to describe pain states. I worked in pain research for a number of
years and one problem we’ve always had is how to know the kind of pain that a patient
is in, because eventually we use a visual analog scale. You might have a sheet of paper, and the person
has to point on the sheet of paper how bad is the pain today? Is it a 50? Is it a 70? Is it 100? What they’re doing with EEG, the PainQx for
example is they’re able to assess these and produce an algorithm to give you a different
kind of display. There are many different types of these that
are coming along in the pain management and wellness area. The New York University Brain Research Laboratory
Normative Database provides an exclusive license for PainQx, and essentially this laboratory
database was based on 20,000 patient database over the past 30 years. It’s culture, ethnic, and gender-free, but
not age free. Then they fuse that with looking at what’s
called electroencephalogram localization, which allowed it to decode neural activity
to localize activated brain region. You ended up with a three-dimensional representation
of regions of the brain that seem to be active when you’re experiencing pain. Then they did what was called a third step
of this technology, it was pain matrix correlation, which allowed it to filter out areas not correlated
to the perception of pain. Essentially you end up with some kind of high/low
pain-state. In the upper figures you see above have to
do with patients that are in quite a bit of pain, and you can see that it shows on the
sensors these activations at parts of the brain, and the bottom, you don’t see as much
activity. That would be a pain free subject. You’re not measuring from the thalamus, you’re
not measuring from the actual pain receptor regions of the brain. You’re trying to find correlates of brain
activity which seem to connect to pain experiences. There’s trauma in footballers, this was another
interesting study developed by Cerora in Bethlehem, Pennsylvania, and that is objective measures
of disease stage and outcome. This occurs in people that play football and
hockey and other contact sports. You have mild brain contusions if you’re hit,
let’s say during a hockey game and smashed into the boards, or you might have really
serious problems that you might see in rugby, in soccer, in football with less protection. The diagnosis and management of contusion
is performed using subjective tools and self-reports, similar to the pain model, but company Cerora
have developed bio-sensors which use, again, these kind of devices you place on the forehead. When you combine them together with traditional
cognitive tests empowered clinicians to look at the degree of brain injury. All of this neuroscience that’s developing
using these kind of tools is, again, contributing to this wellness and the self-monitoring in
areas that were very, very difficult, and it’s important for us to look at early stage
development of these lesions, and to be able to look at the actual critical hits and what
contribution they have to player performance down the road and long term recovery prospects. There are again, also other systems. The Moticon system is an insole system that’s
been designed in which you can look at timing, balance, pressure, force, and motion by having
a bio-sensor that’s located in a footpad. That allows you feedback that you could imagine
would be useful in many different ways. Everything from rehabilitation to improving
sports activity performance. There’s also a good deal of eye tracking technology
that’s developing to looks at what you look at when you engage a computer or a website,
and by using eye tracking, then you’re able to correlate that with different behavioral
activities. The SensoMatic and Tobii Eye Tracking technology
leaders, SMI in Teltow, Germany, is now owned by Apple, and Tobii, which is based on Stockholm,
Sweden, are the leaders in this space which are lab-based camera systems. Again, we’re not only getting feedback from
looking at the activities in the brain, but we’re allowing this neurophysiological feedback
to arrive using sensors located in different areas of the body. Gaze Capture by Apple is something that some
of you might have heard that’s coming on board. Researchers at MIT and the University of Georgia
and the Max Planck Institute have produced Gaze Capture, and it works on your Apple mobile
device, and essentially displays a sequence of dots that you’re able to track and fixate. You have front facing cameras which kept your
eyes moving during performing tasks, and then that’s put a machine learning algorithm, and
then you can predict gaze with low errors in smartphone and tablet devices. You can end up to the point where you can
operate machinery and devices by simply moving your eyes in very, very subtle ways, and you’re
going to see a lot more of this technology. Here we have the basic breakdown of Gaze Capture
by MIT showing the Gaze Capture on different images of faces, how we scan the face, how
it tracks the image, and how that can be used then to activate different kinds of applications
and technologies that are wearable. We’re now going to take a bit of a switch
from a lot of this work in bio-sensors and advanced tracking technologies and look at
neuroscience gamification. We’re going to look at the work now of Aleksandra
Przegalińska in 2015 in neuroscience. Now as early as 1961, Erving Goffman had argued
along with other sociologists that play was a form of serious social interaction that
required nuanced definition. Play has, we can really say the gamification
is a reboot of Erving Goffman’s work, so essentially we would argue from an anthropological standpoint
that gamification is a way of modernizing our view of play and that play is actually
how we develop interactions with other people in the world. So essentially everything we do is a form
of play. Now Scott Nicholson of the University of Sir
Wilfrid Laurier talks about play as being exploration within boundaries. He’s a board game design professor, I think
he teaches gamification, a well-known name in the industry. We’ve had a chat several years ago about defining
play, but this is taking play and looking at it in a much more anthropological standpoint
than it is actually the primary way in which we interact with other people. The Melon Headset is a headband and mobile
app duo that is a self-tracking device. What it essentially does is it allows us to
play and interact with data and with different situations or simulations, and then to measure
brain activity. The Melon device will develop algorithms that
allow us to detect what the user is focusing on and then provide personalized feedback
in how to pursue, improve. Essentially you’re using a device which is
based on the idea of you being in a playful, exploratory interaction with the environment
and giving you feedback on how to interact with the environment more effectively. For example, Melon was a great little device
in which you were able to create Origami creatures using focus. You can see on the screen, if you focus your
attention, you can fold your bird, and if you focus your attention really well, then
you can fold a little bear, you can have a little birds. What it’s doing is actually measuring a specific
state of consciousness. Other games that have been used for bio-feedback
that have been around for a number of years, but this is one that really looks at focus
and our ability to do things in a virtual space like virtual Origami by giving us personalized
feedback on how focused we are at the task. This is really a kind of idea of positive
neuropsychology which is called the positivity ratio, and let’s talk about this for a moment
as we begin to wind down here. The Losada ratio or the Losada line, derived
from Marcial Losada in 2015, is largely discredited now in positive psychology, and that is that
we have what are called flourishing people and languishing people. Flourishing is this idea that we want to be
a Losada level of 2.9 or above. Again, we flourish when we’re in a playful
states. That’s the basic idea. This goes back thousands of years. If we look at the work of Aristotle and Socrates,
they wrote on something called eudemonia, and eudaimonia is flourishing in life. We have what’s called hedonia, which is pleasure
in life, and we have what’s called eudaimonia, which is meaning and life. When we have a high pleasure of meaning and
a high pleasure of purpose and satisfaction, then we are in a flourish state. Think of gamification as a way, a technique
to boost productivity by moving as closer to flourishing states through play. I know we’re taking on quite a journey today,
but we’re going to conclude by looking at kind of what’s probably going on in how we
interact with other people in play states, and how this is going to imply working in
digital psychology and gamification moving forward. Gamified neuroscientists uses this term mirror
neurons a lot, and some of you may be aware of mirror neurons, so we’re going to take
a little bit of a deep-dive into this. Basically when we see somebody do something
else we mirror them. The mirror neurons are doing that. So if I’m around somebody who’s expert and
I do what they do, then I’m occurring it. And mirror neurons can be used for mirroring
undesirable behaviors as well. So Yukti Pro specialized in productivity enhancement. It tries to build using mirror neuron interactions
to progress, so it’s a way of actually digitizing how we watch others progress. You have a new karate student entering the
dojo for the first time, or aikido student watching their master doing aikido or karate,
and they begin to mirror those actions. Again, it’s a form of playful interaction. Playing not as in frivolous, but playful as
in this exploratory space in which we learn. Mirror neurons are specifically located in
regions of the brain in which we use simulation games. Think about it, if we built a game for simulating
work in any kind of industry, we would essentially be trying to trigger mirror neurons through
the game’s system, rather than having to rely on watching the person that would normally
teach us. You would engage mirror neurons in the game. You would observe an action which would trigger
simulation of that action. Neurons in the premotor cortex would fire
during these actions, and therefore this empathy called inner imitation would then kick in. Essentially in the game’s system you would
be in a virtual world dealing with virtual problems, and each one of them triggering
mirror neurons in a storified context. Mirror neurons connect a lot of different
regions of the brain and we’re not going to go into this in too much detail, but we can
see here that we have motor outputs, that which you do. Your intention of reading, your goal sequencing
and representation, and we develop a generative model based on prediction error. Again, to make this simple, if I were spending
my first day on the job learning a skill like handling a customer return and I watch the
more experienced person teach me how to manage a complex return, they lost a receipt or something
like this, I might do the wrong things, and that would give the wrong output, the customer’s
unhappy, or things wouldn’t resolve properly in the system, so we develop a generative
model by mirroring different activities. The big question here is how we actually trigger
mirror neurons. The idea that we would have some kind of cognitive
apprenticeship that would develop. This goes back to the work of RajMohan and
Mohandas 2007, and it was funny. They discovered mirror neurons serendipitously
by Rizzolatti and his work when they were looking at the grasp response in macaque monkey. There was an area in the premotor cortex called
area F5, so whenever one animal was doing something and the other monkey see, monkey
do, those neurons would fire all the time. It turns out that there’s all kinds of diverse
populations of mirror neurons contained in the brain, mostly in the premotor cortex. The premotor cortex is a fascinating area
of the brain. I remember the premotor cortex tells you to
move before you actually move. If you say “I want to walk across the room,”
the premotor cortex would prime the brain for that activity. There’s a lot of questions about what fires
the premotor cortex which is the seat of human will, it goes right back to Descartes’ original
questions going to the 17th century of what is the will. I remember our meetings with Nobel laureate
Sir John Eccles who later on his life started to look at what the human soul could be in
terms of human interaction, when we have a desire to undertake any interaction, it occurs
in this premotor cortex. Of course, Eccles was being somewhat fanciful. He was talking about probability of waves
occurring in the cortex which had to do with quantum release of neurotransmitters. What’s interesting is that mirror neurons
seem to drive the very basis of out behavior, and that is fascinating but insidious. It means if you were raised in a family with
all the dysfunctional behaviors then you would mirror those behaviors. It’s not that you’re consciously learning
them, you’re actually triggering these neurons by seeing how your parents or other family
members engage. Mirror neurons are located in many different
parts of the brain and we’ll just step back one slide here. It’s our second to last slide. This is again, looking at the superior temporal
sulcus, and this is the mirror neuron system that we see. This is the primary motor cortex, this is
where we decide to move, and so the premotor cortex is right in here. The mirror neurons are in a specific area
of the brain that’s picking up activities that we see in either simulation or in real
people living around us. This picks us up to I would say a good point
to conclude today we’d be how does this connect with empathy, because empathy is an interesting
subject in itself. Empathy is how we would then take social cognition
and add it to the mirror neuronal understanding. How we actually deal with others and understand
people has to do with our ability understanding their emotional processing. Empathy is not always a good thing. In fact, it’s been argued by some scholars
that empathy was the basis of the Third Reich, that the German people arose up with kind
of an anger at how they’d been treated in the first world war, and that Hitler, a part
of his appeal, and all populist totalitarian leaders have depended upon empathy, us trying
to empathize with this great cause to move forward with, that we must fight the great
battle in that case for emerging Nazi Germany. Empathy is not the benign activity that we
might say, that we like empathy. Empathy is a way of actually seeing the world
or experiencing the world through these motor neuron networks. By understanding motor neuron, you know mirror
neuron networks under, we can probably understand ways that we inherit hatred and understand
populism, and understand isolation from others as well. We’ve gone around the bend today in taking
you very deep into understanding how serious games can affect us on such deep levels of
emotion and cognition. I hope today has been helpful for you, it’s
certainly been a deep-dive for us. We’ll see you next week. David Chandross from the Center for Teaching
and Learning at Humber College. Have a great week.

Leave a Reply

Your email address will not be published. Required fields are marked *