Do I have to apply sealant at the shelf-to-wall transitions or can I use grout instead?

The use of sealant at these transitions is required for two purposes. First, the material actually seals the transitions against surface water penetration at the joints that could otherwise fuel mold growth. Remember, the most effective means to control mold is to manage moisture effectively. Second, the sealant provides strength to the application to resist typical loads. If grout were substituted for sealant in this application, it could easily crack and pop out from the joints with even limited movement. In general, the use of sealant at such a transition between such fixtures and shower walls is standard practice.