After playing 103 games for Philadelphia in 1953, he was a part of a large deal between the A's and Yankees that sent him back to New York's system. He didn't play for the Yankees OR their minor leagues, as he was sent straight to independent Toronto.
The Lethbridge (Alberta) Herald of Jan. 5, 1954 explains...
"TORONTO (CP) Toronto Maple Leafs of the International League Monday night, announced the purchase for $15,000 of infielder Loren Babe from New York Yankees of the American League. Babe will be used at second base. He played third base for Syracuse Chiefs of the International League in 1951 and 1952. In those seasons he hit .272 and .305 playing under manager Bruno Betzel. He was purchased by the Leafs on the. recommendation of Betzel, signed recently as Toronto coach, , Babe, sold to Philadelphia Athletics last, season for $25,000 after playing in five games for the Yankees, batted .230 in 103 games for the A's. The Yankees bought him back at their original selling price to complete the recent. 11-player deal they made with the A's."
Babe stayed with the independent Maple Leafs until 1958, when he went to Seattle in the PCL.
“Former New York Yankee infielder Loren Babe was purchased yesterday from Toronto by the Seattle Rainiers of the Pacific Coast Baseball league, Dewey Soriano, the club’s general manager, announced.”
His stay in Seattle was a short, injury-plagued one, and he was soon in a Rochester uniform.
"ROCHESTER, N. Y., May 20.— (5)—Infielder Loren Babe today was signed by the Rochester Red Wings in a move to add batting power to the club. Babe, who played with Seattle earlier this year, had been a free agent after suffering an ankle injury. Red Wing General Manager George Sisler said Babe's contract was conditioned on his ability to play and to produce. Babe, a left-handed hitter, batted .262 last year with Toronto. He hit five homers and drove in 54 runs. He was sold to Seattle at the end of the season. He injured his ankle in Spring training and was batting .220 when he left the club."
The wonderful collection of work at "On A Silver Diamond" mentions that it was a free agent signing.
Other than one at-bat in 1961, Babe finished his career in Rochester.