The complex range of interactions between electrons and electromagnetic fields gave rise to countless scientific and technological advances. A prime example is photon-induced nearfield electron microscopy (PINEM), enabling the detection of confined electric fields in illuminated nanostructures with unprecedented spatial resolution. However, PINEM is limited by its dependence on strong fields, making it unsuitable for sensitive samples, and its inability to resolve complex phasor information. Here, we leverage the nonlinear, overconstrained nature of PINEM to present an algorithmic microscopy approach, achieving far superior nearfield imaging capabilities. Our algorithm relies on free-electron Ramsey-type interferometry to produce orders-of-magnitude improvement in sensitivity and ambiguity-immune nearfield phase reconstruction, both of which are optimal when the electron exhibits a fully quantum behavior. Our results demonstrate the potential of combining algorithmic approaches with state-of-the-art modalities in electron microscopy and may lead to various applications from imaging sensitive biological samples to performing full-field tomography of confined light.