The branch He mentions is probably an Asherah, a tree or a wooden pole that stood for the fertility goddess. It was essentially a phallic symbol. God says that their wickedness was the same as sticking that idol (and what they were doing) right in His nose—right in His face.
However, we need to understand that they were committing their idolatry in private. In their minds, they thought it did not make any difference; it was not hurting anyone. If nobody could see them doing it, they thought, "Hey, I'm getting away with the perfect crime. I can have my cake and eat it too. I can go to the Temple and look like I'm really somebody in the community. But, you see, here in the dark nobody sees me bowing down and worshipping these idols."
Does it make any difference? It makes a great difference to God! He was offended by what was going on. In this context, these people were in captivity because of what they were doing—it was a matter of cause and effect. What they sowed earned captivity.
John W. Ritenbaugh
Every Action Has a Reaction