Anger is mounting over plans for a lavish barn conversion in Nelson's Norfolk village.