Sag Harbor is about three fifths in Southampton and two fifths in East Hampton. The dividing line is Division Street which becomes Town Line Road just south of the village. Most of the defining landmarks of the village — including its Main Street, the Whalers Church, Jermain Library, Whaling Museum, the Old Burying Ground, Oakland Cemetery, Mashashimuet Park, and Otter Pond are in Southampton. However, almost all the Bay Street marina complex, including Sag Harbor Yacht Club and Breakwater Yacht Club, at the foot of Main Street, is in East Hampton, as are the village's high school, the Sag Harbor State Golf Course, and the freed slave community of Eastville.
Sag Harbor was settled sometime between 1707 and 1730. The first bill of lading using the name Sag Harbor was recorded in 1730. While some accounts say it was named for neighboring Sagaponack, which at the time was called "Sagg", Sagaponack and Sag Harbor both got their name from a tuber the Metoac Algonquins raised. One of the first crops that was sent back to England, the tuber-producing vine is now called the Apios americana. The Metoac called it sagabon. That is how the harbor and neighboring village got its name. Such namings were not unusual. Tuckahoe in Westchester County, about from Sag Harbor, got its name from the aboriginal term for the Peltandra virginica, the Arrow Arum.
The port supplanted the East Hampton community of Northwest which is about east of Sag Harbor. International ships and the whaling industry had started in Northwest, but its port was too shallow. The most valuable whale product was whale oil which was used in lamps; thus it could be said that Sag Harbor was a major oil port.
By 1789 Sag Harbor had "had more tons of square-rigged vessels engaged in commerce than even New York City." It had become an international port.
During the American Revolutionary War, American raiders under Return Jonathan Meigs attacked a British garrison on May 23, 1777, on a hill at what today is the Old Burying Ground next to the Whaler's Church, killing six and capturing 90 British soldiers in what was called Meigs Raid.
During the War of 1812, the British attacked the village on July 11, 1813, but were driven back.
The whaling industry in Sag Harbor peaked in the 1840s. Sag Harbor is mentioned in Chapters 12, 13, 57 and 83 of Moby-Dick, including this passage:
Relics of this period include the Old Whaler's Church, a Presbyterian church that sported a steeple which was claimed to be the tallest structure on Long Island when it opened in 1843. The steeple collapsed during the Great Hurricane of 1938.
The Sag Harbor Whaling & Historical Museum owns and occupies the handsome, 1845 Greek Revival home designed by Minard Lafever for whaling merchant Benjamin Huntting II. The Masonic Lodge (Wamponamon 437) occupies the second floor and celebrated its 150th anniversary in 2008. The museum is open to the public, and the staircase alone is worth a visit.
The design of the Whaler's Church and the Masonic Temple are attributed to prominent 19th century American architect Minard Lafever. The broken mast monument in Oakland Cemetery is the most visible of several memorials to those who died at sea.
The whaling business collapsed after 1847, initially with the discovery of other methods to create kerosene, with the first being coal oil. The discovery of petroleum in Titusville, Pennsylvania, in 1859 sealed the end. Many of the ships based in Sag Harbor were sailed to San Francisco where they were simply abandoned during the California Gold Rush. The last whaleship — the Myra — sailed from Sag Harbor in 1871.
One sailor who continued on other endeavors was Mercator Cooper who sailed out of Sag Harbor on November 9, 1843, on the Manhattan on a voyage that would make him the first American to visit Tokyo Bay. Aboard the ship was Pyrrhus Concer, a former slave who was the first black man the Japanese had seen. Cooper's adventures were to continue on another voyage out of Sag Harbor when on January 26, 1853, sailing the Levant, he became the first person to set foot on East Antarctica.
During World War I the E.W. Bliss Company tested torpedoes in the harbor a half mile north of the village. As part of the process, Long Wharf in Sag Harbor was reinforced with concrete, and rail spurs were built along the wharf as the torpedoes were loaded onto ships for testing. The torpedoes were shipped via the Long Island Rail Road along Sag Harbor to the wharf which was owned by the railroad at the time. Among those observing the tests was Thomas Alva Edison. Most of today's buildings on the wharf, including the Bay Street Theatre, were built during this time. The torpedoes which did not have live warheads are occasionally found by divers on the bay floor.
Various industries have operated locally, the last of which was the Bulova Watchcase Factory, which closed in 1981.
Sag Harbor was author John Steinbeck's residence from 1955 until his death in 1968.
The Sag Harbor-North Haven Bridge, renamed The LCpl Jordan Haerter Veterans' Memorial Bridge in November 2008, is notable as the site of Pop artist Ray Johnson's presumed suicide in 1995 as well as two abortive suicide attempts by monologist Spalding Gray, in September 2002 and October 2003.