In this paper, we present HH-GNN, a parameter-efficient graph neural network that operates on documents encoded as tree-like graphs. HH-GNN learns word, sentence and document representations in a hierarchical manner, in the hyperbolic space, whose curvature better fits the structure of these graphs in comparison to the Euclidean space. The evaluation conducted on five well-known datasets against representative CNNs, RNNs and GNNs highlights the relevancy and consistency of HH-GNN. We also show that it can match or outperform DistilBERT when distillating knowledge from BERT-large despite having 160× fewer parameters.