In response to escalating energy demands and environmental concerns, the imperative promotion of sustainable practices is explored in this paper. The focus is on employing RL techniques to optimize energy consumption and associated costs, within energy management systems. A three-step approach is introduced to efficiently manage charging cycles in building energy storage units. The strategy involves clustering load curves, incorporating domain knowledge into the learning algorithm, and predicting future observations for real-time decision-making. The method enables controlled exploration and efficient training of EMS agents. In comparison to the benchmark, our model reduces energy costs by up to 15%, decreases consumption during peak periods, and showcases adaptability across diverse building consumption profiles.