The town is in the east part of Columbia County.
Ellis, Capt. Franklin, "History of Columbia County, New York", (1878), page 381:
The town was organized from parts of the towns of Hillsdale, Chatham, and Canaan, March 28, 1818. A little more than one-fifteenth of the present town was taken from Chatham, a little over one-eighth from Canaan, and a little less than five-sixths from Hillsdale. From the fact that among the first settlers there were no less than twelve families of Spencers, the north part of Hillsdale had been known from the first as "Spencer's-town." This name finally attached itself simply to the village, and when the division of the town was being talked up it was proposed to call the new town "New Ulm." When the bill erecting it passed the Legislature, however, Martin Van Buren, then a State Senator, and who, being an ardent admirer of the great Napoleon, was somewhat incensed at one of his political opponents (Elisha Williams, if we mistake not), who had succeeded in having a town in Seneca county christened "Waterloo," leaped to his feet and moved to amend by calling the new town "Austerlitz." Having carried his point, he retired to his seat, saying "There's an Austerlitz for your Waterloo."
Past residents of note
Edna St. Vincent Millay, Pulitzer-prize winning poet, lived at Steepletop, a more than property that had previously been a farm. Her sister, Norma, inherited the property at Millay's death in 1950. Norma and her husband, Charles Ellis, in 1973 founded a non-profit artist residency program, the Millay Colony for the Arts, and deeded a small portion of the land to the organization. In 1978, Norma created the non-profit Edna St. Vincent Millay Society that now oversees the house and remainder of the property. Tours are available.