Arlen was born Hyman Arluck, in Buffalo, New York, the child of a Jewish cantor. His twin brother died the next day. He learned the piano as a youth and formed a band as a young man. He achieved some local success as a pianist and singer and moved to New York City in his early 20s. He worked as an accompanist in vaudeville. At this point, he changed his name to Harold Arlen. He performed on record with the "Buffalodians" orchestra, as well as those of Red Nichols, Henny Hendrickson and Arnold Johnson.
Between 1926 and about 1934, Arlen appeared occasionally as band vocalist on records by The Buffalodians, Red Nichols, Joe Venuti, Leo Reisman and Eddie Duchin, usually singing his own compositions.
In 1929, Arlen composed his first well-known song: "Get Happy" (with lyrics by Ted Koehler). Throughout the early and mid-1930s, Arlen and Koehler wrote shows for the Cotton Club, a popular Harlem night club, as well as Broadway musicals and Hollywood films. Arlen also continued to perform with some success, most notably on records with Leo Reisman's society dance orchestra.
Arlen's compositions have always been popular with jazz musicians because of his facility at incorporating a blues feeling into conventional American popular songs.
Arlen and Koehler wrote several hit songs during the early and mid-1930s.
In the mid-1930s, Arlen married, and spent increasingly more time in California, writing for movie musicals. It was at this time that he began working with lyricist Yip Harburg. In 1938, the team was hired by Metro-Goldwyn-Mayer to compose songs for The Wizard of Oz. The most famous of these is the song "Over the Rainbow" for which they won the Academy Award for Best Music, Original Song. They also wrote "Down with Love", a song later featured in the 2003 movie Down with Love.
Arlen was a longtime friend and former roommate of actor Ray Bolger who would star in The Wizard of Oz, the film for which "Over the Rainbow" was written.
In the 1940s, Arlen teamed up with lyricist Johnny Mercer, and continued to write hit songs like "Blues in the Night" ("My Mama Done Tol' Me") and "Ac-Cent-Tchu-Ate the Positive"