I think every president from at least Jefferson (who wanted to annex Florida) to Polk (who annexed just about everything else) and beyond had problems with the Seminoles. The battles of the Seminoles were legendary.
Van Buren did less harm than most, which is to his credit. But harm was there, nevertheless.
America was on the move, which the Jeffersonians were more than happy to encourage it, and kept as an element of their sense of national agenda, such as it was. Whether it was the failed goal of turning the Carribean into an American lake, or the success in conquering (both through bribes and violence) much of the land claimed by Mexico and desired by Britain.
While not inevitable, the journey flowed onward, like the rivers oft used for travel. None could stop the flow for very long.