Granville has been called the "Colored Slate Capital of the World." Quarries in the town mine slate that comes in colors such as green, gray, gray black, purple, mottled green and purple, and red.
From evidence discovered around 1850, the St. Francis Indians appear to have used the town for hunting and making tools in the past.
This border area between Vermont and New York was for a long time not clearly under control of either state. Arrivals from New England settled here hoping to gain the benefits of New England in areas such as land ownership and voting rights. Early settlers arrived before 1770, but the state line was not established until 1790, leaving settlers in this town within New York State.
The town was founded in 1780.
Early agricultural efforts included dairy herds and sheep raising. Extensive slate deposits were located around 1850 in Middle Granville, and the first quarries opened around 1853. The first slate deposits had been located in 1839 and were used for local construction.