Sunspot activity has not always correlated with temperature rise and fall.
"In their Third Assessment Report, the IPCC state that the measured magnitude of recent solar variation is much smaller than the amount of predicted climate change due to greenhouse gases. [44] In 2002, Lean et al.[45] stated that while "There is ... growing empirical evidence for the Sun's role in climate change on multiple time scales including the 11-year cycle", "changes in terrestrial proxies of solar activity (such as the 14C and 10Be cosmogenic isotopes and the aa geomagnetic index) can occur in the absence of long-term (i.e., secular) solar irradiance changes ... because the stochastic response increases with the cycle amplitude, not because there is an actual secular irradiance change." They conclude that because of this, "long-term climate change may appear to track the amplitude of the solar activity cycles," but that "Solar radiative forcing of climate is reduced by a factor of 5 when the background component is omitted from historical reconstructions of total solar irradiance ...This suggests that general circulation model (GCM) simulations of twentieth century warming may overestimate the role of solar irradiance variability." More recently, a study and review of existing literature published in Nature in September 2006 suggests that the evidence is solidly on the side of solar brightness having relatively little effect on global climate, with little likelihood of significant shifts in solar output over long periods of time.[11][46] Lockwood and Fröhlich, 2007, find that there "is considerable evidence for solar influence on the Earth’s pre-industrial climate and the Sun may well have been a factor in post-industrial climate change in the first half of the last century," but that "over the past 20 years, all the trends in the Sun that could have had an influence on the Earth’s climate have been in the opposite direction to that required to explain the observed rise in global mean temperatures."[47]
Studies that have shown that the sun may indeed have an effect on present-day climate include those by Scafetta & West, [48]. Scafetta and West claim that solar variability is a major, if not dominant climate forcing. Based on correlations between specific climate and solar forcing reconstructions, they argue that a 'realistic climate scenario is the one described by a large preindustrial secular variability (e.g., the paleoclimate temperature reconstruction by Moberg et al.)[49] with the total solar irradiance experiencing low secular variability (as the one shown by Wang et al.)[50]. Under this scenario the Sun might have contributed 50% of the observed global warming since 1900."
There's also the fact that 11 of the hottest years on record have occurred during the past 13 years.
Decreasing polar ice coverage would also seem to contradict a cooling trend.