Is He not implying that they did not love Him? Do you not think that if they loved Him they would have been obedient to what it says in the Old Testament? Is He not saying, very strongly, here that these people were not obeying God?
You see, in reality, they belonged to someone else. They belonged to another "family." Jesus very clearly represented the Family of God. These people, who were supposed to represent the Family of God, were not representing the Family of God. They were representing someone else.
Let us turn to John 8. It is quite a long dialogue that takes place here between Christ and the Pharisees. These people were claiming they were Abraham's descendants, and Jesus arguing back to them that they could not possibly be Abraham's descendants (except physically) because they did not reflect the same kind of works that Abraham did. Therefore, they did not reflect the same kind of spirit that Abraham did. You see, the works reflect the spirit.