Keep in mind that plot plans for walls are measured to center of the wall (not the edge surface). That could be the few inches of discrepancy.
Otherwise, you'll have to work with your builder. If the builder isn't cooperative, you'll probably have to involve the city inspectors and/or code...