The prediction of patient disease trajectories using Electronic Health Records (EHRs) is challenging due to non-stationarity, the granularity of medical codes, and difficulties in integrating multimodal data. Current models often overlook critical insights in unstructured data, primarily relying on structured diagnosis codes. In this paper, we propose a novel approach that incorporates unstructured clinical notes into deep learning models for sequential disease prediction. By embedding clinical notes in Transformer-based models, we provide a richer representation of patient histories, enhancing accuracy in predicting future diagnoses. Our experiments show significant improvements in predictive performance compared to traditional models relying solely on structured codes.