The Hebrews did not fulfill the promise. It is still open. Paul then goes on into Joshua and the people who entered the Land, which should have been the fulfillment but was not. After Joshua died, the whole nation went down spiritually until everybody (as it says in the last verse of Judges) was doing what was right in his own eyes. There was no king. There was no central authority. There was nobody to point these people in the right direction. They did not enter into the "rest."
Psalm 95 is generally conceded to be a psalm of David or of Asaph, who was looking back in time. David lived roughly 300 years after Joshua, and the promise of entering God's rest had not been fulfilled in David's time either. Was it fulfilled in any other historical time? No. That is why the apostle is writing this: It still remains! God's promise has not been fulfilled.
Who will it be fulfilled by? Paul is hoping it will be fulfilled by these people who were drifting away, that is, the church. The promise of entering into that Sabbath rest has not yet been fulfilled.
John W. Ritenbaugh
The Fourth Commandment (Part 4)