If I understand your top and bottom borders are too long, I would leave the ends alone to line up with the sides and then make a slightly wider seam on several of the other blocks to get the fit.
if it is too short, I would add a block and then do the same.