Sag Harbor is about three fifths in Southampton and two fifths in East Hampton. The dividing line is Division Street which becomes Town Line Road just south of the village. Most of the defining landmarks of the village — including its Main Street, the Whalers Church, Jermain Library, Whaling Museum, the Old Burying Ground, Oakland Cemetery, Mashashimuet Park, and Otter Pond are in Southampton. However, almost all the Bay Street marina complex, including Sag Harbor Yacht Club and Breakwater Yacht Club, at the foot of Main Street, is in East Hampton, as are the village's high school, the Sag Harbor State Golf Course, and the freed slave community of Eastville.
Sag Harbor was settled sometime between 1707 and 1730. The first bill of lading using the name Sag Harbor was recorded in 1730. While some accounts say it was named for neighboring Sagaponack, which at the time was called "Sagg", Sagaponack and Sag Harbor both got their name from a tuber the Metoac Algonquins raised. One of the first crops that was sent back to England, the tuber-producing vine is now called the Apios americana. The Metoac called it sagabon. That is how the harbor and neighboring village got its name. Such namings were not unusual. Tuckahoe in Westchester County, about from Sag Harbor, got its name from the aboriginal term for the Peltandra virginica, the Arrow Arum.
The port supplanted the East Hampton community of Northwest which is about east of Sag Harbor. International ships and the whaling industry had started in Northwest, but its port was too shallow. The most valuable whale product was whale oil which was used in lamps; thus it could be said that Sag Harbor was a major oil port.
By 1789 Sag Harbor had "had more tons of square-rigged vessels engaged in commerce than even New York City." It had become an international port.
During the American Revolutionary War, American raiders under Return Jonathan Meigs attacked a British garrison on May 23, 1777, on a hill at what today is the Old Burying Ground next to the Whaler's Church, killing six and capturing 90 British soldiers in what was called Meigs Raid.
During the War of 1812, the British attacked the village on July 11, 1813, but were driven back.
The whaling industry in Sag Harbor peaked in the 1840s. Sag Harbor is mentioned in Chapters 12, 13, 57 and 83 of Moby-Dick, including this passage:
Relics of this period include the Old Whaler's Church, a Presbyterian church that sported a steeple which was claimed to be the tallest structure on Long Island when it opened in 1843. The steeple collapsed during the Great Hurricane of 1938.
The Sag Harbor Whaling & Historical Museum owns and occupies the handsome, 1845 Greek Revival home designed by Minard Lafever for whaling merchant Benjamin Huntting II. The Masonic Lodge (Wamponamon 437) occupies the second floor and celebrated its 150th anniversary in 2008. The museum is open to the public, and the staircase alone is worth a visit.
The design of the Whaler's Church and the Masonic Temple are attributed to prominent 19th century American architect Minard Lafever. The broken mast monument in Oakland Cemetery is the most visible of several memorials to those who died at sea.
The whaling business collapsed after 1847, initially with the discovery of other methods to create kerosene, with the first being coal oil. The discovery of petroleum in Titusville, Pennsylvania, in 1859 sealed the end. Many of the ships based in Sag Harbor were sailed to San Francisco where they were simply abandoned during the California Gold Rush. The last whaleship — the Myra — sailed from Sag Harbor in 1871.
One sailor who continued on other endeavors was Mercator Cooper who sailed out of Sag Harbor on November 9, 1843, on the Manhattan on a voyage that would make him the first American to visit Tokyo Bay. Aboard the ship was Pyrrhus Concer, a former slave who was the first black man the Japanese had seen. Cooper's adventures were to continue on another voyage out of Sag Harbor when on January 26, 1853, sailing the Levant, he became the first person to set foot on East Antarctica.
During World War I the E.W. Bliss Company tested torpedoes in the harbor a half mile north of the village. As part of the process, Long Wharf in Sag Harbor was reinforced with concrete, and rail spurs were built along the wharf as the torpedoes were loaded onto ships for testing. The torpedoes were shipped via the Long Island Rail Road along Sag Harbor to the wharf which was owned by the railroad at the time. Among those observing the tests was Thomas Alva Edison. Most of today's buildings on the wharf, including the Bay Street Theatre, were built during this time. The torpedoes which did not have live warheads are occasionally found by divers on the bay floor.
Various industries have operated locally, the last of which was the Bulova Watchcase Factory, which closed in 1981.
Sag Harbor was author John Steinbeck's residence from 1955 until his death in 1968.
The Sag Harbor-North Haven Bridge, renamed The LCpl Jordan Haerter Veterans' Memorial Bridge in November 2008, is notable as the site of Pop artist Ray Johnson's presumed suicide in 1995 as well as two abortive suicide attempts by monologist Spalding Gray, in September 2002 and October 2003.
Sag Harbor is the birthplace of Iraq war hero Marine LCpl Jordan Christian Haerter who was killed on April 22nd, 2008 in Ramadi,Iraq at the age of 19 defending his post at Joint Security Station Nasser. While on guard duty, LCpl Haerter and Cpl Jonathan Yale of Meherrin VA stopped a suicide bomber from breaking through a barrier protecting an Iraqi Police station housing 33 Marines and 21 Iraqi Police. Their actions saved everyone inside the compound but cost them their lives and earned both Marines the Navy Cross posthumously for extraordinary combat heroism. President Obama said of the Marines: "These two Marines stood their ground. These two Marines opened fire. And these two Marines stopped that truck. When the thousands of pounds of explosives detonated, they had saved fifty Marines and Iraqi police who would have been in the truck’s path, but Corporal Yale and Lance Corporal Haerter lost their own lives. Jonathan was 21. Jordan was 19."
Writer William Demby lived in Sag Harbor his last years till his death on May 24, 2013.