George Hardman owned a textile mill at New Hall Hey, where he built his wealth in the cotton and woolen textile trade. In the 1840s, he constructed this building as his family home, which overlooked his mill—a site still visible today.

Hardman later donated the building to the people of Rawtenstall to serve as a museum and funded the expansion of the gardens to create Whitaker Park. The museum and park officially opened in 1902.

In 2019, The Whitaker received a grant from the National Lottery Heritage Fund to expand the museum. The adjacent barn and stables were transformed into a café bar and a function space, enhancing the visitor experience.