Studies have investigated electroencephalogram (EEG)-based emotion recognition using hand-crafted EEG features (e.g., differential entropy) or the annotated emotion categories without any additional emotion factors (e.g., context). The effectiveness of raw EEG-based emotion recognition remains for further investigation. In this study, we investigated the effectiveness of multi-task learning (MTL) for raw EEG-based convolutional neural networks (CNNs) in emotion recognition with auxiliary context information. Thirty subjects participated in this study, where their brain signals were collected when watching six types of emotion images (social/nonsocial-fear, social/nonsocial-sad, and social/nonsocial-neutral). For the MTL architecture, we utilized temporal and spatial filtering layers from raw EEG-based CNNs as shared and task-specific layers for emotion and context classification tasks. Subject-dependent classifications and five repeated five-fold cross-validation were performed to test the classification accuracy for all comparison models. Our results showed that (1) the MTL classifier had a significantly higher classification accuracy and improved the performance of the single-task learnings (STLs) for both emotion and context, and (2) the ShallowConvNet was the best network architecture among the considered CNNs for the MTL with statistically significant improvement to the raw EEG-based STLs. This shows that the MTL can be a promising method for emotion recognition in utilizing the raw EEG-based CNN classifiers and emphasizes the importance of considering context information.