Is He not implying that they did not love Him? Do you not think that if they loved Him they would have been obedient to what it says in the Old Testament? Is He not saying, very strongly, here that these people were not obeying God?
You see, in reality, they belonged to someone else. They belonged to another "family." Jesus very clearly represented the Family of God. These people, who were supposed to represent the Family of God, were not representing the Family of God. They were representing someone else.