Solid deuterium (sD $ _2$ is used as an efficient converter to produce ultracold neutrons (UCN). It is known that the sD $ _2$ must be sufficiently cold, of high purity and mostly in its ortho-state in order to guarantee long lifetimes of UCN in the solid from which they are extracted into vacuum. Also the UCN transparency of the bulk sD $ _2$ material must be high because crystal inhomogeneities limit the mean free path for elastic scattering and reduce the extraction efficiency. Observations at the UCN sources at Paul Scherrer Institute and at Los Alamos National Laboratory consistently show a decrease of the UCN yield with time of operation after initial preparation or later treatment (“conditioning”) of the sD $ _2$ . We show that, in addition to the quality of the bulk sD $ _2$ , the quality of its surface is essential. Our observations and simulations support the view that the surface is deteriorating due to a build-up of D $ _2$ frost-layers under pulsed operation which leads to strong albedo reflections of UCN and subsequent loss. We report results of UCN yield measurements, temperature and pressure behavior of deuterium during source operation and conditioning, and UCN transport simulations. This, together with optical observations of sD $ _2$ frost formation on initially transparent sD $ _2$ in offline studies with pulsed heat input at the North Carolina State University UCN source, results in a consistent description of the UCN yield decrease.