Swan was born in Cape Girardeau, Missouri. As a child, he learned drums, piano and guitar, and began writing songs. His first big break was in 1962 when Clyde McPhatter recorded "Lover Please", a song written by Swan when he was in a local band called Mirt Mirly & the Rhythm Stoppers, and which had first been recorded by Bill Black.[1] McPhatter's version quickly became a No. 7 pop hit.
Swan moved to Memphis to work with Black, but this was cut short with Black's illness and subsequent death in 1965. He also for a time worked on Elvis Presley's Graceland estate. He then moved to Nashville, which enabled him to write hit country songs for numerous artists, including Conway Twitty, Waylon Jennings, and Mel Tillis. In 1969, Swan first took on the role of record producer, producing Tony Joe White's Top Ten hit "Polk Salad Annie". Swan also played bass guitar for Kris Kristofferson, and then signed a solo recording deal with Monument Records.