In 1842, the Austin-Johnson House was erected on what is now Johnson Ferry Road. It is the oldest house in Sandy Springs. In 1851, Wilson Spruill donated five acres (two hectares) of land for the founding of the Sandy Springs United Methodist Church, near the natural spring for which the city is named. In 1905, the Hammond School was built at Johnson Ferry Road and Mt. Vernon Highway, across the street from the church.
In 1950, the state legislature blocked Atlanta from annexing the community, which remained rural until the Interstate Highway System was authorized by the Federal-Aid Highway Act of 1956. In 1959, after a fire at Hammond Elementary School, Atlanta Mayor William Hartsfield urged residents to support annexation so that the area would have better firefighting protection. Community opposition killed the proposal. In the early 1960s, Georgia 400 and Interstate 285 were constructed, connecting Sandy Springs to metro Atlanta and initiating a housing boom that brought new residents and major land development. In 1966, annexation by Atlanta was defeated in a referendum, with two-thirds voting against.
Debate over incorporation began in the 1970s when the city of Atlanta attempted to use a state law to force annexation of Sandy Springs. The attempt failed when the Supreme Court of Georgia ruled that the law was unconstitutional. In response, the Committee for Sandy Springs was formed in 1975. In every legislative session, state legislators representing the area introduced a bill in the Georgia General Assembly to authorize a referendum on incorporation. Legislators representing Atlanta and southwestern Fulton County, who feared tax revenue that would be lost from incorporation, blocked the bills using the procedural requirement that all local legislation be approved first by a delegation of representatives from the affected area. In 1989, a push was made for Sandy Springs to join neighboring Chattahoochee Plantation in Cobb County. This move was blocked by Speaker of the House Tom Murphy.
When the Republican Party gained a majority in both houses of the General Assembly in 2005, the procedural rules previously used to prevent a vote by the full chamber were changed so that the bill was handled as a state bill and not as a local bill. The assembly also repealed the requirement that new cities must be at least three miles (4.8 km) from existing cities, because the new city limits border both Roswell and Atlanta. The bill allowing for a referendum on incorporation was introduced and passed as HB 37. The referendum initiative was approved by the Assembly and signed by Governor Sonny Perdue.
The referendum was held on June 21, 2005, and residents voted 94% to 6% in favor of incorporation. Many residents expressed displeasure with county services, claiming, based upon financial information provided by the county, that the county was redistributing revenues to fund services in less financially stable areas of the county, ignoring local opposition to rezoning, and allowing excessive development. Many residents of unincorporated and less-developed south Fulton County strongly opposed incorporation, fearing the loss of tax revenues which fund county services. County residents outside Sandy Springs were not allowed to vote on the matter. Efforts such as requesting the U.S. Justice Department to reject the plan were unsuccessful.
A mayor and six city council members were elected in early November 2005, with Eva Galambos, who had initiated and led the charge for incorporation, elected mayor by a wide margin. Formal incorporation occurred on December 1, making Sandy Springs the third-largest city ever to incorporate in the U.S. The city's police force and fire department began service in 2006. In 2010, the city became the first jurisdiction in Georgia to successfully "bail out" from the preclearance requirements of Section 5 of the Voting Rights Act.