The confinement that COVID-19 has brought about has had a negative influence on people’s psychological health. However, this impact is not widespread throughout the population, and men and women may be affected differently and it is not known what protective factors may exist. In this sense, physical activity has classically been shown to be a habit associated with psychological health. The study aimed to analyze the impact of confinement on psychological health (psychological well-being, coping, emotions, and perception of daily difficulties), taking into account gender, and perceived changes in physical activity. After the project was approved by the University’s Ethics Commission, the participants, after signing the informed consent, completed the online questionnaires during the days from 6 to 20 April, the time when, in Spain, confinement was in place and the highest peak of deaths and infections from COVID-19 occurred. A total of 457 Spanish participants (247 men and 210 women) were evaluated in psychological well-being, in its adaptation to Spanish, in coping, with the Spanish adaptation of the COPE Inventory, in daily habits and difficulties (ad hoc questionnaire) and the level of physical activity they had (sedentary, active, and federated players) was recorded. Besides, the perceived change in physical activity due to confinement was recorded. The results showed that perceived emotions, difficulties for certain routines, psychological well-being, and coping differed according to sex. On the other hand, depending on the change in physical activity habits, it was observed that participants who increased their physical activity responded differently in the perception of emotions, and difficulties for routines and in psychological well-being. Finally, differences were also observed in most variables between sedentary, active, and federated participants. Results are discussed highlighting the importance of physical activity as a moderating factor of the impact of confinement.