Promptly predicting defects during an additive manufacturing process using only copious log data provides many advantages, albeit with computational limitations. We focus on predicting defects during electron beam melting with the black box nature of the manufacturing machine. For an accurate prediction of defects, which are rare (<2%), we extract temporal information to track abnormalities and formulate a feature selection algorithm that maximizes the expected value of a cost-sensitive accuracy. Correct identification of features responsible for the defects increases predictive power and informs manufacturers of potential corrective/preventive actions for process improvement. We solve the feature selection through resampling strategies integrated with ensemble procedures to handle data uncertainty and imbalance. Exploiting data uncertainty in our search leads to finding robust features with consistent predictive power. Our proposed methodology shows a 43% improvement in predicting defects (recall) without losing precision. Beyond additive manufacturing, this methodology has general application for rare-event prediction and imbalanced datasets.