Intensity shot noise in digital holograms distorts the quality of the phase images after phase retrieval, limiting the usefulness of quantitative phase microscopy (QPM) systems in long term live cell imaging. In this paper, we devise a hologram-to-hologram neural network, Holo-UNet, that restores high quality digital holograms under high shot noise conditions (sub-mW/cm2 intensities) at high acquisition rates (sub-milliseconds). In comparison to current phase recovery methods, Holo-UNet denoises the recorded hologram, and so prevents shot noise from propagating through the phase retrieval step that in turn adversely affects phase and intensity images. Holo-UNet was tested on 2 independent QPM systems without any adjustment to the hardware setting. In both cases, Holo-UNet outperformed existing phase recovery and block-matching techniques by ∼ 1.8 folds in phase fidelity as measured by SSIM. Holo-UNet is immediately applicable to a wide range of other high-speed interferometric phase imaging techniques. The network paves the way towards the expansion of high-speed low light QPM biological imaging with minimal dependence on hardware constraints.