The leaves are considered a food item but currently many tea bags may contain small amounts of plastic
If you use loose leaf tea then this can be emptied directly into the food waste recycling bin, if you are using bags you could tear them open placing the the bag part in the general waste bin and contents in the food waste recycling bin.
The small amount of plastic used make the glue holding the bag together heat resistant. Most manufactures are researching alternatives which should be on the market soon
Why not get yourself a tea strainer and buy loose leaf tea. Loose leaf tea can be purchased by weight so you could bring along your own caddy so no waste is created.