During the flash backflashback,when odin when Odin picks up baby lokiLoki, loki'sLoki's blue skin turns to asgardianAsgardian skin. When he is grabbed by the frost giants, his skin reverted back to the original form, not only changing the color, but the texture as well. (if you noticenoticed, the frost giants have ice like skin, presumably from being in jhodenhimJotunheim so long.). Loki's size is actually talked about in the original norseNorse myths.
Frost giants are different from humans in many ways, one of them being that the males carry the children and give birth. Loki had two older brothers, (their names escape me at this moment), both of whom laufeyLaufey (loki'sLoki's father) birthed. They are much older than lokiLoki. There was much pain in laufey'sLaufey's labor of the two eldest because they were so big. Laufey was the king of the frost giants,and and so naturally he was huge, even for a frost giant. When laufeyLaufey was giving birth to lokiLoki he had no pain which concerned him. It turned out, lokiLoki was a runt and so the maid asked if he wanted to get rid of lokiLoki. Though sometimes crulecruel, laufeyLaufey was not heartless, and kept the little frost giant.
Not long after odinOdin came and the war began, killing both of loki'sLoki's older brothers, who lokiLoki never got to know. Laufey had to leave his now olnyonly child and join in the battle against the norseNorse god. the rest of the story is shown in the movie. Odin takes the small frost giant and raises him as his own alongside his biological son thorThor who is older than lokiLoki. They do not know they are not related untilluntil many years later as adults. That wouldn't have mattered to them any wayanyway. They were brothers by bond, not blood, which in a way can be much better.
So there you go, sorry it was long, but I love mythology. I hope I explained it well enough and answeranswered the question. :)