In this study, we investigated the barley yellow mosaic virus (BaYMV, genus Bymovirus) factor(s) responsible for breaking eIF4E-mediated recessive resistance genes (rym4/5/6) in barley. Genome mapping analysis using chimeric infectious cDNA clones between rym5-breaking (JT10) and rym5-non-breaking (JK05) isolates indicated that genome-linked viral protein (VPg) is the determinant protein for breaking the rym5 resistance. Likewise, VPg is also responsible for overcoming the resistances of rym4 and rym6 alleles. Mutational analysis identified that amino acids Ser-118, Thr-120, and His-142 in JT10 VPg are the most critical residues for overcoming rym5 resistance in protoplasts. Moreover, the rym5-non-breaking JK05 could accumulate in the rym5 protoplasts when eIF4E derived from a susceptible barley cultivar was expressed from the viral genome. Thus, the compatibility between VPg and host eIF4E determines the ability of BaYMV to infect barley plants.