Parks and town squares can play an important role by offering spaces for cognitive restorativeness in urban contexts. Therefore, it is important that these spaces be designed in a way that encourages restorativeness. Indeed, their perceived quality should motivate users to stay and take advantage of them. Yet, it is not clear whether perceptions as to the quality of these spaces is relevant in promoting restorativeness. Thus, the aim of this study is to analyze whether elements of environmental quality perceived by users of public spaces favor restorativeness both in parks and squares. Environmental and social aspects are taken into consideration, since restorative experiences involve cognitive and physiological recovery, as well as a component of interaction with the environment. In this research, 519 users of 32 urban public spaces—town squares and parks—on the island of Tenerife (Spain) participated. Participants evaluated these spaces using four dimensions that focused on spaces’ perceived environmental quality: design of spaces, care of spaces, social interaction, and presence of sensorial elements. Additionally, we evaluated the perceived restorativeness of each space. The results showed that the design of spaces, care of the spaces, social interaction, and presence of sensorial elements explain the variance in perceived restorativeness, although with different weights for parks and squares. We found that perceived quality of a space is a key predictor of its restorativeness. This means that maintaining parks and town squares is a relevant task given that they contribute to reducing cognitive overload, increasing sustainability, and facilitating health care in urban settings.