Am I just worrying too much?

If your blood tests came back perfect, you eat healthy, and exercise I would not worry about developing diabetes. Your height and weight aren't in obese range, which is when obesity becomes an issue. I would not worry about this.