Freshman Jacob Robson broke a 2-2 deadlock in the eighth inning with a sacrifice line-drive to left, setting the stage for Holder's first win of the campaign.
Mississippi State's 6-0 start is its best since 2006.
"We played a solid, quality Purdue ball club and had some pretty good swings tonight... we just didn't have a lot to show for it," said State coach John Cohen. "I thought we did a nice job of focusing after the rain delay."
The Bulldogs had two runners on base with two outs in the seventh inning in a 1-1 game when rain forced suspension of play. Some 91 minutes later the game resumed, and PU pitcher Mike Lutz struck the next two batters he faced, Mitch Slauter and Hunter Renfroe, to load the bases and drive in the go-ahead run for the Bulldogs.
The lead was short lived. Purdue got a pair of two-out hits in the ninth off relief pitcher Luis Pollorena, the second an RBI double into the right field corner by Kyle Wood that tied the score at 2-2.
In the home half of the eight, Alex Detz drew a lead-off walk. Pinch-runner Derrick Armstrong then stole second, advanced to third on an errant throw to second base, and then raced home with the winning run on Robson's smash line-drive to the leftfielder. McElroy (0-1), the third of four Boilermaker hurlers in the game, took the loss.
Holder earned his third career win and first of the season, fanning three of the four batters he faced. He now has two wins and 10 saves in his last 14 mound career appearances of 2012-13. This year he has retired 13 of 14 batters faced, 11 with strikeouts.
Starting pitchers Jacob Lindgren and Jordan Minch allowed one hit and three hits respectively as the duo dueled in shutout fashion over the first five frames.
Purdue broke the deadlock when Kyle Johnson opened the sixth with a single and touched home later on Sean McHugh's bases-loaded infield groundout. Rookie relief man Myles Gentry got the final two outs in the frame to douse the Boilermaker rally.
Minutes later in the bottom of the sixth Bulldog junior outfielder Hunter Renfroe evened the score, launching a season-first home run to left.
The game, whose start was delayed 24 minutes by rain, was then suspended by rain with two outs in the bottom of the seventh and State threatening.
Shortstop Adam Frazier collected a pair of singles and Renfroe drove in two of MSU's three runs while Purdue lead-off hitter Brandon Krieg singled twice for Purdue.
Lindgren struck out six and did not walk a batter in 5.1 innings of no-decision mound work.
Minch gave up a run on four hits and struck out five in over innings.
In the earlier game, Purdue erupted for five runs in the eighth inning to top Samford 9-4. Second baseman Brandon Krieg and Conner Hudnall collected four hits each to lead the Boilermaker's 16-hit attack.
The Bulldogs resume play Saturday in the inaugural Diamond Classic presented by Polk's Meat Products, taking on Samford (4-2) at 3 p.m. and Purdue at 6 p.m. A pair of right-handers, junior Evan Mitchell (0-0, 1.80) and senior Kendall Graveman (1-0, 0.00), have been designated for Saturday starts.