I understand how a prism works and how a single raindrop can scatter white light into a rainbow, but it seems to me that in normal atmospheric conditions, we should not be able to see rainbows.

When multiple raindrops are side-by-side, their emitted spectra will overlap. An observer at X will see light re-mixed from various originating raindrops. The volume of rain producing a rainbow typically has an angular diameter at least as wide as the rainbow itself, does it not?