They were given a specific job. They were not even numbered to go to war or any of those other things. Their job was to take care of the Tabernacle, the dwelling place of God.
Now the Levites were not numbered because God had chosen them. And that was their job. Matthew Henry says they belonged to God. They were God's tribe. That is a high calling. They were to set it up when necessary, take it down when necessary, and brethren, they were to guard it with their lives. If any outsider were to approach the Tabernacle, they were to be killed. This offense was punishable by death. Access to God was cut off and not any old Joe could wander up to the Tabernacle. This penalty of death is repeated at least five times in the book of Numbers if I counted correctly.