Researcher Benchmarks 15 Blood Glucose Forecasting Models on Personal and Public Data

A researcher with type 1 diabetes developed a 'model zoo' to test 15 different forecasting architectures, finding that classical machine learning models like XGBoost and LightGBM currently outperform deep learning models in predicting blood glucose levels 2 hours into the future.

Model Performance and Benchmarking

The evaluation compared a wide range of models, including classical regressions (ARIMA, SARIMAX), deep learning architectures (LSTM, GluNet), and large-scale foundation models (Chronos-2, TimesFM 2.5). The testing was conducted on two distinct datasets: a personal dataset of 115 days of continuous glucose monitor (CGM) data and the public MetaboNet dataset.

On the personal dataset, XGBoost led with an RMSE of 22.2 mg/dL at the 60-minute horizon. The author's custom transformer, T1DMAI D16 DILATE, followed closely in 3rd place with an RMSE of 22.8 mg/dL. On the MetaboNet dataset, while XGBoost remained the leader, the T1DMAI model achieved the lowest RMSE of all tested models at the 90-minute mark.

This figure illustrates the primary benchmarking results where XGBoost and the custom T1DMAI model are compared against other architectures.

“All models suck. XGBoost is the best one but my own model comes close.”

— GitHub - 0xdeadf1sh/T1DMAI: A model zoo of blood glucose forecasters trained & tested on my own data

Architecture and Training Methodology

The T1DMAI model is an encoder-only transformer designed for efficiency, utilizing 16 layers of width 16 and a total of 31,253 parameters. It processes CGM data, carbohydrate absorption curves, and insulin action curves to output seven quantiles (representing the 5% to 95% range) for every 5-minute step over a 2-hour window.

To improve performance, the researcher used a Low-Rank Adaptation (LoRA) adapter—a small 3,771-parameter component—to fine-tune the models on real training data. This adapter significantly improved accuracy, reducing the 60-minute RMSE on the personal dataset from 26.1 mg/dL to 22.8 mg/dL.

Chart drawn by HN Digest from figures reported by github.com; values are shown as written in the source.

Trade-offs in Complexity and Speed

A significant trade-off exists between the computational cost of foundation models and the speed of classical models. While foundation models like Chronos-Bolt mini contain over 21 million parameters and require several seconds per forecast, the T1DMAI model can perform inferences in roughly 5.6 to 7.3 milliseconds.

This image supports the discussion on the efficiency of the T1DMAI model compared to larger foundation models.

Despite the high accuracy of some deep learning models, the researcher noted that many of the low-RMSE models failed to catch 'lows' (hypoglycemic events) on the MetaboNet dataset, where linear extrapolation actually performed better at catching those specific events. This highlights the difficulty of balancing general accuracy with the detection of critical, rare clinical events.

This visual helps explain the difficulty of balancing general accuracy with the detection of specific clinical events like hypoglycemic lows.

Research Limitations and Safety

The study is currently limited by a small sample size for the personal data (N=1 over 35 days) and the fact that the models are research artifacts. The researcher explicitly states that the software is not a clinically validated medical device and must not be used to make medical, diagnostic, or treatment decisions.

The results also show that model performance varies significantly across different subjects. For example, on the MetaboNet dataset, the 90% prediction band held the truth 75-78% of the time for some models, while it reached 81-85% when using the LoRA adapter.