A firm based at a Herefordshire farm has been told it will have to rethink its plan to replace redundant farm buildings with four new houses.