Jimmy Hart was brought into wrestling by Jerry "The King" Lawler , with whom he had attended Memphis Treadwell High School. Having been asked to sing back-up with Lawler, Hart later became Lawler's manager.

After splitting from Lawler, Hart created a stable known as Hart's First Family of Wrestling to attack Lawler, which among others included King Kong Bundy, "Ravishing" Rick Rude, Lanny Poffo, Jim "The Anvil" Neidhart, Ox Baker, Kamala, Randy Savage, "Hot Stuff" Eddie Gilbert, The Iron Sheik and Kevin Sullivan.

In 1982, Hart earned national headlines doing a program with comedian and television star Andy Kaufman. Hart, Lawler and Kaufman would continue this feud for over a year. Around this time, Hart became known as "The Wimp", a nickname given to him by Lawler and chanted by fans, and was the subject of the song "Wimpbusters," which was sung by Lawler to the tune of "Ghostbusters" by Ray Parker, Jr.; a music video was also made featuring Lawler, announcer Lance Russell, and wrestlers such as Randy Savage, Jimmy Valiant, Dutch Mantel, Tommy Rich, and Rufus R. Jones, along with footage of Lawler beating Hart and his "First Family".

In 1981 through 1984, Hart led Austin Idol, Masao Ito, and Gilbert to NWA/AWA International titles.