In the past decade, deep learning has revolutionized machine problem solving. Advances in computing power and data access have led this technology to grow from distinguishing between cats and dogs to playing Go at a superhuman level. Medical applications were enabled by AlexNet, a convolutional neural network that provided breakthroughs allowing for computer vision tasks, such as image labeling, identification, or segmentation. These advances are responsible for new clinical use cases, such as classifying computed tomography tumor images as benign or malignant. However promising these new advances may appear, major challenges exist in acquiring data sets large enough to be clinically relevant.