The primary settlement in town, where 2,757 people resided at the 2010 census, is defined as the Jaffrey census-designated place (CDP) and is located along the Contoocook River at the junction of U.S. Route 202 and New Hampshire routes 124 and 137.
First granted by the Massachusetts General Court in 1736 to soldiers from Rowley, Massachusetts, returning from the war in Canada, the town was known as Rowley-Canada. In 1749, the town was re-chartered by the Mason proprietors as Monadnock No. 2, sometimes called Middle Monadnock or Middletown. It was one of the first towns established following the Masonian proprietors' purchase of undivided lands under the claim.
Settled about 1758, the town was regranted in 1767. It would be incorporated in 1773 by Governor John Wentworth, and named for George Jaffrey, member of a wealthy Portsmouth family. Jaffrey's son was a life trustee of Dartmouth College, and designer of the official college seal. The Contoocook River provided water power for mills. Village prosperity would be expressed in fine early architecture, including the Town Meetinghouse, built in 1775.
Beginning in the 1840s, the area's scenic beauty attracted tourists, and several summer hotels were built at the base of Mount Monadnock, enduringly popular with hikers. Some who scaled the summit were Ralph Waldo Emerson, Henry David Thoreau and Rudyard Kipling. The experience inspired Emerson in 1845 to write the poem, Monadnoc.
Jaffrey was the setting for a 1950 biography by Elizabeth Yates entitled Amos Fortune, Free Man, winner of the 1951 Newbery Medal. Amos Fortune was an African-born slave who purchased his freedom and that of his wife, and established a tannery in the village. He is buried in the local cemetery, together with bandbox craftswoman Hannah Davis, and author and summer resident Willa Cather.