Jarvi studied music first in Tallinn, and later in Leningrad at the Leningrad Conservatory under Yevgeny Mravinsky, and Nikolai Rabinovich, among others. Early in his career, he held posts with the Estonian Radio and Television Symphony Orchestra, the Estonian National Symphony Orchestra and the Opera in Tallinn. In 1971 he won first prize in the International Conductors Competition at the Accademia Nazionale di Santa Cecilia in Rome. Jarvi emigrated to the United States in 1980 with his family. He became an American citizen in 1987.

In 1982, he became the principal conductor of the Gothenburg Symphony, and held the post for 22 years, the longest-serving principal conductor in the orchestra's history. During his Gothenburg tenure, the recording profile and reputation of the orchestra greatly increased. He also helped to secure corporate sponsorship from Volvo that allowed the orchestra to increase in size from 80 to 110 players. He retained his post in Gothenburg until 2004, and now holds the title of Principal Conductor Emeritus (Chefdirigent Emeritus) with the orchestra.