Ethics statement

The study was approved by and performed in accordance with the guidelines and regulations of the Wroclaw Medical University, Poland; approval no. 149/2020. The submission to the Ethical Committee covered, among others, participant consent, research plans, recruitment strategy, data management procedures, and GDPR issues. Participants provided written informed consent, in which they declared that they (1) were informed about the study details, (2) understand what the research involves, (3) understand what their consent was needed for; (4) may refuse to participate in the research at any time during the research project; (5) had the opportunity to ask questions of the experimenter and receive answers to those questions. Finally, participants gave informed consent to participate in the research, agreed to be recorded during the study, and consented to the processing of their personal data to the extent necessary for the implementation of the research project, including sharing their psycho-physiological and behavioral data with other researchers.

The participants were recruited via a paid advertisement on Facebook. Seventy people responded to the advertisement. We have excluded ten non-Polish speaking volunteers. An additional 15 could not find a suitable date, and two did not show up for the scheduled study. As a result, we collected data from 43 participants (21 females) aged between 19 and 29 (M=22.37, SD=2.25). All participants were Polish.

The exclusion criteria were significant health problems, use of drugs and medications that might affect cardiovascular function, prior diagnosis of cardiovascular disease, hypertension, or BMI over 30 (classified as obesity). We asked participants to reschedule if they experienced an illness or a major negative life event. The participants were requested (1) not to drink alcohol and not to take psychoactive drugs 24hours before the study; (2) to refrain from caffeine, smoking, and taking nonprescription medications for two hours before the study; (3) to avoid vigorous exercise and eating an hour before the study. Such measures were undertaken to eliminate factors that could affect cardiovascular function.

All participants provided written informed consent and received a 50 PLN (c.a., $15) online store voucher.

We used short film clips from databases with prior evidence of reliability and validity in eliciting targeted emotions19,20,21,22,23. The source film, selected scene, and stimulus duration are provided in Table 1.

We used two types of self-assessment for manipulation checks that accounted for discrete and dimensional approaches to emotions. For the discrete approach, participants reported retrospectively, using single-item rating scales, on how much of the targeted emotions they had experienced while watching the film clips21. The questionnaire was filled in electronically with a tablet, see Fig.1a. It included nine items corresponding to the selected stimuli. Each emotion-related scale ranged from 1 (not at all) to 5 (extremely). The questionnaire was modeled after the instruments used in previous studies with similar methodology24,25,26,27.

The English version of the self-reports used in the study: (a)questionnaire for discrete emotions; (b)questionnaire for valence, arousal, and motivation. The original Polish version can be found in the Supp. Mat. Fig 3.

For the dimensional approach, participants reported retrospectively, using single-item rating scales, on how much valence, arousal, and motivation they experienced while watching the film clips. The 3-dimensional emotional self-report was collected with the Self-Assessment Manikin SAM28. The SAM is a validated nonverbal visual assessment developed to measure affective responses. Participants reported felt emotions using a graphical scale ranging from 1 (a very sad figure) to 9 (a very happy figure) for valence, Fig.1b; and from 1 (a calm figure) to 9 (an agitated figure) for arousal, Fig.1b. We also asked participants to report their motivational tendency using a validated graphical scale modeled after the SAM29, i.e., whether they felt the urge to avoid or approach while watching the film clips, from 1 (figure leaning backward) to 9 (figure leaning forward)30, Fig.1b. The English versions of the self-reports used in the study are illustrated in Fig.1.

The behavioral and physiological signals were gathered using three wearable devices and a smartphone:

An EEG headband Muse 2 equipped with four EEG electrodes (AF7, AF8, TP9, and TP10), accelerometer (ACC), and gyroscope (GYRO). The data was transmitted to a smartphone in real-time using the Mind Monitor (https://mind-monitor.com) application. At the end of each day, data from the smartphone was transferred to the secure disk;

A wristband Empatica E4 monitoring blood volume pulse (BVP), interbeat interval (IBI), electrodermal activity (EDA), acceleration, and skin temperature (SKT). The Empatica E4 was mounted on the participants dominant hand. The device was connected wirelessly via Bluetooth to the tablet using a custom-made Android application with Empatica E4 link SDK module31. The data was streamed in real-time to the tablet and after the study to the secure server. The signals obtained with the Empatica E4 were synchronized with the stimuli presented on the tablet;

A smartwatch Samsung Galaxy Watch SM-R810 providing heart rate (HR), peak-to-peak interval (PPI), raw BVP the amount of reflected LED light, ACC, GYRO, and rotation data. A custom Tizen application was developed and installed on the watch to collect and store data locally. At the end of each day, data was downloaded to the secure disk;

A smartphone Samsung Galaxy S20+5G recording participants upper-body head, chest, and hands. The footage also included a small mirror reflecting the tablet screen to enable later synchronization with stimuli. At the end of each day, recordings were moved to the encrypted offline disk.

The Muse 2 has lower reliability than medical devices but sufficient for nonclinical trial settings32. It has been successfully used to observe and quantify event-related brain potentials33, as well as to recognize emotions34. The Empatica E4 has been compared with a medical electrocardiograph (ECG), and proved to be a practical and valid tool for studies on HR and heart rate variability (HRV) in stationary conditions35. It was also likewise effective as the Biopac MP150 in the emotion recognition task36. Moreover, we have used the Empatica E4 for intense emotion detection with promising results in a field study37,38. The Samsung Watch devices were successfully utilized (1) to track the atrial fibrillation with an ECG patch as a reference39, and (2) to assess the sleep quality with a medically approved actigraphy device as a baseline40. Moreover, Samsung Watch 3 performed well in detecting intense emotions41.

Additionally, a 10.4-inch tablet Samsung Galaxy Tab S6 was used to guide participants through the study. A dedicated application was developed to instruct the participants, present stimuli, collect self-assessments, as well as gather Empatica E4 signals, and synchronize them with the stimuli.

The sampling rate of the collected signals is provided in Table2. The devices and the experimental stand are illustrated in Fig.2.

Devices used to gather the physiological data and the experimental stand.

The study was conducted between the 16th of July and the 4th of August, 2020. It took place in the UX Wro Lab - a laboratory at the Wrocaw University of Science and Technology. Upon arrival, participants were informed about the experimental procedure, Fig.3. They then signed the written consent. The researcher applied the devices approximately five minutes before the experiment so that the participants could get familiar with them. It also enabled a proper skin temperature measurement. From this stage until the end of the experiment, the physiological signals were recorded. Next, participants listened to instructions about the control questionnaire and self-assessments. The participants filled out the control questionnaire about their activity before the experiment, e.g., time since the last meal or physical activity and wake-up time. Their responses are part of the dataset.

The experiment procedure.

The participants were asked to avoid unnecessary actions or movements (e.g., swinging on the chair) and not to cover their faces. They were also informed that they could skip any film clip or quit the experiment at any moment. Once the procedure was clear to the participants, they were left alone in the room but could ask the researcher for help anytime. For the baseline, participants watched dots and lines on a black screen for 5minutes (physiological baseline) and reported current emotions (emotional baseline) using discrete and dimensional measures. The main part of the experiment consisted of ten iterations of (1) a 2-minute washout clip (dots and lines), (2) the emotional film clip, and (3) two self-assessments, see Fig.3. The order of film clips was counterbalanced using a Latin square, i.e., we randomized clips for the first participant and then shifted by one film clip for each next participant so that the first film clip was placed as the last one.

After the experiment, participants provided information about which movies they had seen before the study and other remarks about the experiment. Concluding the procedure, participants received the voucher. The whole experiment lasted about 50minutes, depending on the time spent on the questionnaires.

Empatica E4 was synchronized with the stimuli out-of-the-box using a custom application and Empatica E4 SDK. Samsung Watch and Muse 2 devices were synchronized using accelerometer signals. All three devices were placed on the table, which was then hit with a fist. The first peak in the ACC signal was used to find the time shift between the devices, Fig.4. All times were synchronized to the Empatica E4 time.

The time difference between the devices used in the study identified by recording the ACC signal when devices were moved according to the synchronization procedure.

Each device stored data in a different format and structure. We unified the data to JSON format and divided the experiment into segments covering washouts, film clips, and self-assessment separately. We provide the raw recordings from all used devices. Additionally, we performed further preprocessing for some devices/data and provide it alongside the raw data.

For EEG, the raw signal represents the signal filtered with a 50Hz notch frequency filter, which is a standard procedure to remove interference caused by power lines. Besides the raw EEG, the Mind Monitor application provides the absolute band power for each channel and five standard frequency ranges (i.e., delta to gamma, see Table2). According to the Mind Monitor documentation, these are obtained by (1) using a fast Fourier transform (FFT) to compute the power spectral density (PSD) for frequencies in each channel, (2) summing the PSDs over a frequency range, and (3) taking the logarithm of the sum, to get the result in Bels (B). The Mind Monitor documentation presents details https://mind-monitor.com.

The processing of BVP signal from the Samsung Watch PPG sensor consisted of subtracting the mean component, eight-level decomposition using Coiflet1 wavelet transform, and then reconstructing it by the inverse wavelet transform based only on the second and third levels. Amplitude fluctuations were reduced by dividing the middle value of the signal by the standard deviation of a one second long sliding window with an odd number of samples. The final step was signal normalization to the range of [1,1].

The upper-body recordings were processed with the OpenFace toolkit42,43,44 (version 2.2.0, default parameters) and Quantum Sense software (Research Edition 2017, Quantum CX, Poland). The OpenFace library provides facial landmark points and action units values, whereas Quantum Sense recognizes basic emotions (neutral, anger, disgust, happiness, sadness, surprise) and head pose.

Some parts of the signals were of lower quality due to the participants movement or improper mounting. For example, the quality of EEG signal can be investigated using Horse Shoe Indicator (HSI) values provided by the device, which represent how well the electrodes fit the participants head. For video clips, OpenFace provides information about detected faces with their head pose per one frame. We have not removed low-quality signals so that users of the dataset can decide how to deal with them. Any data-related problems that we identified are included in the data_completeness.csv file.

Read the original post:
Emognition dataset: emotion recognition with self-reports, facial expressions, and physiology using wearables | Scientific Data - Nature.com