First, he refused through words. He explained his belief to her in no uncertain terms. "Look, I will not do this because not only is it disloyal to my master, it is a sin against God," and against her, and it would be a sin against his wife to come.
Second, he bore all the temptations she laid before him, and avoided them as well as he could. When that ceased to be effective and she trapped him, he took physical action and fled. He got out of there no matter what the consequences to himself, leaving his garment behind, which was to them proof that he did what he was accused of. It landed him in prison. But he was able to go with a clear conscience, because he had done what was right. He is a sterling example of crucifying the flesh in order to walk in the Spirit.
He had to go in there. He could not ignore his job. What was he going to do? Pit her against her husband while she was in the act of trying to commit adultery against her husband? Is he going to say to his boss, "Well, I won't go into the house because your wife is trying to seduce me"?
No. There eventually came the time when she was playing her cards. She was watching things. Now is the time for the rattlesnake to strike. "Because nobody else is in the house, I've got him." The serpent was there—inside.