Naggar was the centre of power for nearly two centuries. The Naggar Castle, built by Raja Sidh Singh around 1460 AD, served as the royal residence and state headquarters until the 17th century, till the capital was moved to Sultanpur (later renamed Kullu). After that, it became the summer palace of the rajas till the British arrived in 1846. The castle was then sold to a British army officer called Major Hay. Later in 1978, it was converted into a hotel by the Himachal Pradesh Tourism Corporation and renamed Castle Hotel.

The Naggar Castle is a fine edifice made of stone and wood depicting the indigenous craft of the masons of the western Himalayas. The castle is built around a central courtyard surrounded by verandahs. The beautifully done temple of Jagati Pat stands in the central courtyard. It houses a stone slab with an inscription about how it was carried from the peak of Deo Tibba (6,001m) by Gods in the form of bees. There is also a small museum on the premises. Exhibits include the traditional garb of the women of this area. Some musical instruments like karnal (broad bell horn) and singa (long curled horn) are also on display.