We report here on the magnetic properties of ZnO:Mn- and ZnO:Co-doped nanoparticles. We have found that the ferromagnetism of ZnO:Mn can be switched on and off by consecutive low-temperature annealings in O2 and N2, respectively, while the opposite phenomenology was observed for ZnO:Co. These results suggest that different defects (presumably n-type for ZnO:Co and p-type for ZnO:Mn) are required to induce a ferromagnetic coupling in each case. We will argue that ferromagnetism is likely to be restricted to a very thin, nanometric layer at the grain surface. These findings reveal and give insight into the dramatic relevance of surface effects to the occurrence of ferromagnetism in ZnO-doped oxides.