In the mighty British Navy at the time of Empire building, every sailing ship had cannon (the plural of cannon) for protection. Cannon of the times required round iron cannonballs.
A ship's master wanted to store the cannonballs such that they could be available for instant use when needed, but in a manner that would not let them roll around the gun deck.
The solution devised was to stack them up in a square-based pyramid next to the cannon. The top level of the stack had one ball, the next level down had three, the next had nine, the next had sixteen, and so on.
Four levels would provide a stack of 30 cannonballs. The only real problem was how to keep the bottom level from sliding out from under the weight of the higher levels. To do this, they devised a small brass plate referred to as a "brass monkey," with one rounded indentation for each cannonball in the bottom layer. Brass was used because the cannonballs wouldn't rust on the brass monkey, where they would rust on an iron one. When temperature falls, brass contracts faster than iron. As it got cold on the gun decks, the indentations in the brass monkey would get smaller than the iron cannonballs they were holding. If the temperature got cold enough, the bottom layer of cannonballs would pop out of the indentations, spilling the entire pyramid over the deck.
Thus it was, quite literally, "cold enough to freeze the balls off a brass monkey." And so, a familiar phrase became part of the language.