There is probably some need for concern. Depending on the area you are in the fungus can cause root rot. In southern California, the cinnamon fungus is prevalent on avocado trees. It comes mostly from too much moisture in the soil. To help only water the areas around the tree that are the driest.

I have many avocado trees and sometimes it take 10 - 15 years for them to bear fruit. Normally around 7 years.

If it is root rot. This fungus spreads throughout the root system and Poor drainage and over-watering cause root rot, leading the death of the tree. Small sized, pale colored, wilting leaves are present along with growth failure and, when detected, the tree should be destroyed.