I think ultimately Chris was always going to be a polarising character. Personally, I think he is no more annoying than his predecessor - Chris's ranting is nothing different to Clarkson's ranting and sometimes racists and outdated views of the world.
I would think the BBC looked at the numbers and saw that people will just hate Chris for being Chris, much moreso than Clarkson, regardless of how well the show is produced. So it was probably a mutual decision to step away for the good of the programme and the other presenters. No doubting four of the six episodes have been epic, much better than any of the last few series presented by CHM. Its still a great visual treat (which many doubted it would be) it just needs presentation refinement, but of course it does, things do not just switch on and off that easily.