Please do not scrap the tree bark. It is as good as injuring it.
Instead, paint the number using dissolved limestone. Mix the limestone powder with water. Paint the numbers, just one paints milestones.
If it is erased, you have to repaint. But, it is environment friendly and cheap. Also...