Transformers have revolutionized sequential data processing but lack explainability, particularly problematic in regulated fields like healthcare. Our work introduces a graph-based visualization of attention learning and a metric for validating learned connections against ground truth. Testing on Behrt (a diagnostic prediction model) demonstrates how our method reveals inter-diagnosis relationships and dataset biases.