It is proposed that the ground-state manifold of the neutral nitrogen-vacancy center in diamond could be used as a quantum two-level system in a solid-state-based implementation of a broadband noise-free quantum optical memory. The proposal is based on the same-spin Λ-type three-level system created between the two E orbital ground states and the A1 orbital excited state of the center, and the cross-linear polarization selection rules obtained with the application of a transverse electric field or uniaxial stress. Possible decay and decoherence mechanisms of this system are discussed, and it is shown that high-efficiency, noise-free storage of photons as short as a few tens of picoseconds for at least a few nanoseconds could be possible at low temperature.