Physical neural networks made of analog resistive switching processors are promising platforms for analog computing and for emulating biological synapses. State-of-the-art resistive switches rely on either conductive filament formation or phase change. These processes suffer from poor reproducibility or high energy consumption, respectively. Our work, on one hand, focuses on understanding and controlling the variability of the conductive filament formation in insulating oxide materials. On the other hand, we are innovating alternative synapse designs that rely on a deterministic charge-controlled mechanism, modulated electrochemically in a solid state, and that consists of shuffling the smallest cation, the proton. As typical throughout our research, here, too, we combine experimental synthesis, fabrication, and characterization with first principles-based computational modeling to gain a deep understanding and control of these promising devices.