After leaving the school, Herald and Weissberg moved to New York, where they were soon joined by Bob Yellin. Yellin had studied classical violin, piano, and vocal harmony as a child and trumpet at the High School of Music and Art. After listening to a recording of Flatt & Scruggs, Yellin had been converted to bluegrass and bought himself a banjo. Using Pete Seeger's instructional record How to Play the 5 String Banjo, Yellin mastered the three-finger Scruggs style of banjo picking. In 1958, Yellin and Mike Seeger (of the New Lost City Ramblers) attended the Old Time Fiddlers Convention in Galax, VA, and received a second prize for their double banjo rendition of the traditional folk tune "Old Joe Clark."

Herald, Weissberg, and Yellin had few commercial aspirations when they formed the Greenbriar Boys, and their performances were limited to concerts at American Youth Hostels and the Sunday sessions at Washington Square Park. When Gerde's Folk City opened, the Greenbriar Boys were one of the first acts to play the club.

The Greenbriar Boys became more serious after Weissberg left to join the Tarriers and was replaced by Paul Prestopino and then Ralph Rinzler. A mandolin player and folklorist, Rinzler owned a large collection of tapes that included recordings by Riley Puckett, Charlie Poole, and Uncle Dave Macon. Rinzler encouraged the Greenbriar Boys to rehearse on a regular basis, and the group's sound began to meld. In 1960, the Greenbriar Boys traveled to Union Grove, NC, where they became the first northern group to win the band competition at the Old Time Fiddlers Convention. A year later, Yellin returned to the festival and became the first northerner to win first prize in the banjo competition.

1962 proved a pivotal year for the Greenbriar Boys. In addition to accompanying Joan Baez on two songs -- "Pal of Mine" and "The Banks of the Ohio" -- on her second album, the group was signed by Vanguard Records. After being featured along with Jackie Washington, Bob Gude, and Hedy West on a multi-artist sampler, New Folks, the Greenbriar Boys released a self-titled debut album. They subsequently released three other albums -- Ragged but Right! in 1964, Dian & the Greenbriar Boys in 1965, and Better Late Than Never in 1966. With Rinzler leaving the group to accept a job as director of the folk department at the Smithsonian Institute, the Greenbriar Boys were joined by bluegrass mandolinist and vocalist Frank Wakefield and fiddler Jim Buchanan on Better Late Than Never. The Greenbriar Boys' arrangement of "Different Drum," by Mike Nesmith (later of the Monkees), was re-recorded and turned into a pop hit by Linda Ronstadt and the Stone Poneys in 1967. However, the Greenbriar Boys disbanded in 1967, although they have occasionally reunited in the years since.