The area was originally known as Greenwood, and was noted for the iron works belonging to Robert and Peter P. Parrott, of Parrott gun fame. The Greenwood Furnace was established in 1810; during the American Civil War, the furnace produced the iron for the famous Parrott Gun, which was built in Cold Spring, New York by the Parrott brothers. The Parrotts built St. John's Episcopal Church in Arden in 1863. By the 1890s, the iron industry in New York was in decline due to the discovery of surface beds of iron in Minnesota.
The hamlet takes its current name from the estate built in the area by railroad magnate Edward H. Harriman in the late 19th century. It is today a National Historic Landmark, but not open to the public. The Open Space Institute acquired the building and 540 surrounding acres from Columbia University for $4.5 million in 2007. Reportedly, the land is to be preserved and the building sold.