If the amount of granules were not great, there will be no problem.
If you are concerned about this, you may add for example 1 teaspoon baking soda (mixed in 2-3 gallons of water before adding) into the tank (sodium bicarbonate, NaHCO3). This will help buffering the water against acidification.
Peat pellets will not hurt your water.
You could have dumped a whole box of them in there, and I'm sure your fish will love it.
However, I'm sure you don't want a bunch of pellets floating around in your tank for aesthetic reasons, so what is not easily "Fished out" will be sucked up into your filters and can either be extracted there, or left in the filters.
I sometimes pack my filters with them.
They soften the water and lower pH to simulate the natural environment nicely.
Don't sweat it... you dropped "Good stuff" into your tank.