I think "Winter's Coming" is a double entendre like many other popular recurring Game of Thrones mottos, "words", or phrases, which is used convey the notion of "in preparation of" (a warning) and/or something that is inevitably beyond one's control (ie: death), which in itself could play back into Game of Thrones/A Song of Ice and Fire's notions of existing in a "predetermined" universe with "cycle" cosmology with "seasons" and "songs" being metaphor.)
The general tenor of a bad winter season does indicate harsh living conditions and generally bad or longer winters have existed more "naturally" in the universe.
There is also the bigger Winter, which most likely [metaphysically] correlates between House Stark, The North, and the arrival of The Night King, White Walkers, [ice] Wights, and "Army of the Dead, deriving from a "winter cycle" started during the Age of Heroes when the Night King (TV Show Only) was made by the Children of Forest, as some kind of retaliation against The First Men or The Andals for being betrayed. This is something that plays into to the metaphysical nature of Game of Thrones universe, but also, as the Game of Thrones Showrunners have expressed, The Night King is one personification of "death" (See Also: Azor Ahai).
Thus when Arya says, 'Winter has come for House Frey.', it's clear that it retains an association with death and extinction in metaphorical terms too.
So winter's plots on the story does effect the crops (there was food shortage, which then Dany furthered by accidentally burning them all while on Drogon's back, which may also played a role on how Cersei was able to retain some power by making Dany look bad), but the bigger Winter also disrupted the 'Game of Thrones', as the Night King was [finally] able to breach the [magically enchanted] wall, threatening the existence of every living thing, but it also tells us that because The Knight King was finally killed, that a cycle may have ended and broken into a new,because the show ends in a such a way where the Starks, nearly extinct, are closer to how they were when they arrived/united/ruled the North during the Age of Heroes and being in touch/close with nature (and/or The Children of the Forest's beliefs), the Kingdoms are being ruled differently (Because Bran is seemingly taking a backseat to let his council rule), and where Westoros feels like it's entering a time of peace!
The final two books that have yet to be released are titled, The Winds of Winter and the final book, A Dream of Spring, implying a hopeful future with spring onto a promising summer! Bran's Direwolf was named Summer and when Jon leaves with the Wildlings North of the Wall, there is a sprout of some plant piercing through the snow!