Sag Harbor is about three fifths in Southampton and two fifths in East Hampton. The dividing line is Division Street which becomes Town Line Road just south of the village. Most of the defining landmarks of the village — including its Main Street, the Whalers Church, Jermain Library, Whaling Museum, the Old Burying Ground, Oakland Cemetery, Mashashimuet Park, and Otter Pond are in Southampton. However, almost all the Bay Street marina complex, including Sag Harbor Yacht Club and Breakwater Yacht Club, at the foot of Main Street, is in East Hampton, as are the village's high school, the Sag Harbor State Golf Course, and the freed slave community of Eastville.
Sag Harbor was settled sometime between 1707 and 1730. The first bill of lading using the name Sag Harbor was recorded in 1730. While some accounts say it was named for neighboring Sagaponack, which at the time was called "Sagg", Sagaponack and Sag Harbor both got their name from a tuber the Metoac Algonquins raised. One of the first crops that was sent back to England, the tuber-producing vine is now called the Apios americana. The Metoac called it sagabon. That is how the harbor and neighboring village got its name. Such namings were not unusual. Tuckahoe in Westchester County, about from Sag Harbor, got its name from the aboriginal term for the Peltandra virginica, the Arrow Arum.
The port supplanted the East Hampton community of Northwest which is about east of Sag Harbor. International ships and the whaling industry had started in Northwest, but its port was too shallow. The most valuable whale product was whale oil which was used in lamps; thus it could be said that Sag Harbor was a major oil port.
By 1789 Sag Harbor had "had more tons of square-rigged vessels engaged in commerce than even New York City." It had become an international port. After the Second Session of Congress on July 31, 1789 Sag Harbor was declared the first official port on entering the United States. This turned Sag Harbor into a melting pot of different cultures. With Sag Harbor being the first stop on entering United States territory, this is where ships would stop before sailing to New York City to finish their trip. With all the ships coming in and out of Sag Harbor the United States government decided that they should place a customs house in the town. Long Island did not have a customs house before the one in Sag Harbor.
During the American Revolutionary War, American raiders under Return Jonathan Meigs attacked a British garrison on May 23, 1777, on a hill at what today is the Old Burying Ground next to the Whaler's Church, killing six and capturing 90 British soldiers in what was called Meigs Raid.
During the War of 1812, it was claimed that the British attacked the village on July 11, 1813, but were driven back. In reality, several open boats from the British squadron that dominated the Sound during the war entered the harbor at night without any advance planning because the young midshipman in command of these small boats was curious. That officer, C. Claxton R.N., later described his youthful misadventures years later as editor of The Naval Monitor. Upon landing on the wharf and planning some mischief they heard an alarm gun fired before they could set fire to the coasting vessel docked there and beat a hasty retreat. There were no injuries, and Claxton and his men made it safely back to HMS Ramillies (1785) anchored off Gardiners Island.
The whaling industry in Sag Harbor peaked in the 1840s. Sag Harbor is mentioned in Chapters 12, 13, 57 and 83 of Moby-Dick, including this passage:
Relics of this period include the Old Whaler's Church, a Presbyterian church that sported a steeple which was claimed to be the tallest structure on Long Island when it opened in 1843. The steeple collapsed during the Great Hurricane of 1938.
The Sag Harbor Whaling & Historical Museum owns and occupies the handsome, 1845 Greek Revival home designed by Minard Lafever for whaling merchant Benjamin Huntting II. The Masonic Lodge (Wamponamon 437) occupies the second floor and celebrated its 150th anniversary in 2008. The museum is open to the public, and the staircase alone is worth a visit.
The design of the Whaler's Church and the Masonic Temple are attributed to prominent 19th century American architect Minard Lafever. The broken mast monument in Oakland Cemetery is the most visible of several memorials to those who died at sea.
The whaling business collapsed after 1847, initially with the discovery of other methods to create kerosene, with the first being coal oil. The discovery of petroleum in Titusville, Pennsylvania, in 1859 sealed the end. Many of the ships based in Sag Harbor were sailed to San Francisco where they were simply abandoned during the California Gold Rush. The last whaleship — the Myra — sailed from Sag Harbor in 1871.
One sailor who continued on other endeavors was Mercator Cooper who sailed out of Sag Harbor on November 9, 1843, on the Manhattan on a voyage that would make him the first American to visit Tokyo Bay. Aboard the ship was Pyrrhus Concer, a former slave who was the first black man the Japanese had seen. Cooper's adventures were to continue on another voyage out of Sag Harbor when on January 26, 1853, sailing the Levant, he became the first person to set foot on East Antarctica.
During World War I the E. W. Bliss Company tested torpedoes in the harbor a half mile north of the village. As part of the process, Long Wharf in Sag Harbor was reinforced with concrete, and rail spurs were built along the wharf as the torpedoes were loaded onto ships for testing. The torpedoes were shipped via the Long Island Rail Road along Sag Harbor to the wharf which was owned by the railroad at the time. Among those observing the tests was Thomas Alva Edison. Most of today's buildings on the wharf, including the Bay Street Theatre, were built during this time. The torpedoes which did not have live warheads are occasionally found by divers on the bay floor.
Various industries have operated locally, the last of which was the Bulova Watchcase Factory, which closed in 1981.
Sag Harbor was author John Steinbeck's residence from 1955 until his death in 1968. Steinbeck did some of his writings in a little house on the edge of his property in Sag Harbor. His view from the writing house overlooked the Upper Sag Harbor Cove. In the novel, Travels With Charley, Mr. Steinbeck starts an 11 week trip from Sag Harbor with his dog, Charley.
Writer William Demby lived in Sag Harbor his last years till his death on May 24, 2013.