A discrete Gaussian distribution over the integers is a Gaussian distribution restricted so that its support is the set of all the integers. This paper studies the problem of sampling exactly from discrete Gaussian distributions over the integers. It is required to generate integers according to a given discrete Gaussian distribution without any statistical discrepancy. In 2016, Karney proposed an exact sampling algorithm for discrete Gaussian distributions whose parameters are rational numbers. This algorithm uses rejection sampling, and it is a discretization of his algorithm for sampling exactly from the standard normal distribution. In this paper, we give a rigorous and complete analysis of the rejection rate of this algorithm, which was not given by Karney, and show that it cannot generate integers efficiently in the case where the standard deviation of the distribution is very small (e.g. much smaller than 1/2). Then, we present an alternative algorithm for this special case, which can sample exactly and efficiently from discrete Gaussian distributions with very small standard deviations.