/kaggle/input/global-weather-repository/GlobalWeatherRepository.csv
/kaggle/input/global-weather-repository/state.db

Dataset Shape: (104453, 41)

Dataset Info:
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 104453 entries, 0 to 104452
Data columns (total 41 columns):
 #   Column                        Non-Null Count   Dtype  
---  ------                        --------------   -----  
 0   country                       104453 non-null  object 
 1   location_name                 104453 non-null  object 
 2   latitude                      104453 non-null  float64
 3   longitude                     104453 non-null  float64
 4   timezone                      104453 non-null  object 
 5   last_updated_epoch            104453 non-null  int64  
 6   last_updated                  104453 non-null  object 
 7   temperature_celsius           104453 non-null  float64
 8   temperature_fahrenheit        104453 non-null  float64
 9   condition_text                104453 non-null  object 
 10  wind_mph                      104453 non-null  float64
 11  wind_kph                      104453 non-null  float64
 12  wind_degree                   104453 non-null  int64  
 13  wind_direction                104453 non-null  object 
 14  pressure_mb                   104453 non-null  float64
 15  pressure_in                   104453 non-null  float64
 16  precip_mm                     104453 non-null  float64
 17  precip_in                     104453 non-null  float64
 18  humidity                      104453 non-null  int64  
 19  cloud                         104453 non-null  int64  
 20  feels_like_celsius            104453 non-null  float64
 21  feels_like_fahrenheit         104453 non-null  float64
 22  visibility_km                 104453 non-null  float64
 23  visibility_miles              104453 non-null  float64
 24  uv_index                      104453 non-null  float64
 25  gust_mph                      104453 non-null  float64
 26  gust_kph                      104453 non-null  float64
 27  air_quality_Carbon_Monoxide   104453 non-null  float64
 28  air_quality_Ozone             104453 non-null  float64
 29  air_quality_Nitrogen_dioxide  104453 non-null  float64
 30  air_quality_Sulphur_dioxide   104453 non-null  float64
 31  air_quality_PM2.5             104453 non-null  float64
 32  air_quality_PM10              104453 non-null  float64
 33  air_quality_us-epa-index      104453 non-null  int64  
 34  air_quality_gb-defra-index    104453 non-null  int64  
 35  sunrise                       104453 non-null  object 
 36  sunset                        104453 non-null  object 
 37  moonrise                      104453 non-null  object 
 38  moonset                       104453 non-null  object 
 39  moon_phase                    104453 non-null  object 
 40  moon_illumination             104453 non-null  int64  
dtypes: float64(23), int64(7), object(11)
memory usage: 32.7+ MB
None

First few rows:

Missing Values Summary:

Dropping columns with >30% missing values: []

Shape after cleaning: (104209, 47)

Shape after creating target variable: (103942, 48)
Target variable range: 0.17 to 1614.10

Feature matrix shape: (103942, 17)
Target vector shape: (103942,)
Shape after removing outliers: (95076, 17)
Percentage of data retained: 91.47%

Training set size: 76060
Test set size: 19016

Linear Regression:
  RMSE: 9.7052
  MAE: 6.8813
  R²: 0.5035
--------------------------------------------------
Random Forest:
  RMSE: 8.8431
  MAE: 6.3565
  R²: 0.5877
--------------------------------------------------
Gradient Boosting:
  RMSE: 8.6075
  MAE: 5.9873
  R²: 0.6094
--------------------------------------------------
XGBoost:
  RMSE: 9.9760
  MAE: 6.9492
  R²: 0.4754
--------------------------------------------------
SVR:
  RMSE: 8.6992
  MAE: 6.0515
  R²: 0.6011
--------------------------------------------------

Model Performance Comparison:

Best Model: Gradient Boosting

Fitting 5 folds for each of 48 candidates, totalling 240 fits
Best parameters for Gradient Boosting: {'learning_rate': 0.05, 'max_depth': 3, 'min_samples_leaf': 2, 'min_samples_split': 2, 'n_estimators': 100}

Final Performance after Tuning:
RMSE: 8.6123
MAE : 5.9449
R²  : 0.6090

Top 10 Most Important Features:

Model Performance by PM2.5 Category:

Cross-Validation Results for Best Model:
RMSE: 8.0816 (±0.3249)
R²: 0.6521 (±0.0244)

FINAL MODEL SUMMARY
==================================================

Predictions saved to 'pm25_predictions_results.csv'

Model Performance by Country (Top 10):

Model Performance by Country (R² ≥ 0, Top 10):

Plots will be saved to: /kaggle/working/pm25_forecasting_plots
============================================================
GEOGRAPHICAL PERFORMANCE ANALYSIS
============================================================
Countries analyzed: 35
Total test samples in analysis: 19016

📊 MODEL PERFORMANCE BY COUNTRY (Top 15 by RMSE):

✅ Country performance data saved to: /kaggle/working/pm25_forecasting_plots/country_performance_analysis.csv

📈 PERFORMANCE SUMMARY STATISTICS:
==================================================
Total Countries Analyzed: 35
Countries with R² > 0.7: 1
Countries with R² > 0.5: 13
Countries with R² > 0.3: 26
Average R² across countries: 0.3904
Average RMSE across countries: 8.2057
Average MAE across countries: 5.9942

🏆 TOP 5 COUNTRIES BY PERFORMANCE:
1. Tuvalu: R²=0.4396, RMSE=3.5814, Samples=535
2. Tonga: R²=0.6304, RMSE=3.6660, Samples=523
3. Timor-Leste: R²=0.5072, RMSE=3.8521, Samples=532
4. Tanzania: R²=0.6317, RMSE=3.8660, Samples=535
5. Vanuatu: R²=0.5459, RMSE=4.0564, Samples=530

✅ All geographical analysis plots saved to: /kaggle/working/pm25_forecasting_plots/
✅ Country performance data saved to CSV

============================================================
INTRODUCTION & BACKGROUND SECTION PLOTS
============================================================

🌍 GLOBAL PM2.5 BACKGROUND STATISTICS:
==================================================
Global Average PM2.5: 25.55 µg/m³
WHO Guideline (annual mean): 5 µg/m³
WHO Interim Target 1: 15 µg/m³
Locations exceeding WHO guideline: 83.3%
Locations exceeding interim target: 49.2%
Maximum observed PM2.5: 1614.1 µg/m³
Minimum observed PM2.5: 0.2 µg/m³

📊 REGIONAL AVERAGES:
Americas: 53.6 ± 130.6 µg/m³ (n=2657.0)
Asia: 62.8 ± 66.3 µg/m³ (n=8414.0)
Europe: 16.9 ± 16.1 µg/m³ (n=12379.0)
Middle East: 48.6 ± 61.8 µg/m³ (n=4271.0)
Other: 20.6 ± 24.1 µg/m³ (n=76221.0)

🔬 RESEARCH MOTIVATION STATISTICS:
==================================================
Next-day autocorrelation: 0.8406
Average PM2.5 variability (std): 39.82 µg/m³
Number of unique locations: 221
Number of unique countries: 186
Date range: 2024-05-16 to 2025-11-03
Total observations: 103,942

✅ All introduction/background plots saved to: /kaggle/working/pm25_forecasting_plots/

============================================================
SHAP ANALYSIS - MODEL INTERPRETABILITY
============================================================
✅ SHAP already installed

🔍 Creating SHAP explainer for Gradient Boosting...
Using TreeExplainer for Gradient Boosting
Calculating SHAP values for 1000 samples...
✅ SHAP values calculated successfully!
SHAP values shape: (1000, 17)
Expected value: 16.9517
✅ SHAP values saved to /kaggle/working/pm25_forecasting_plots/shap_values.npy

📊 Generating SHAP dependence plots for top features...
Top 6 features by SHAP importance: ['air_quality_PM2.5', 'air_quality_Carbon_Monoxide', 'air_quality_Sulphur_dioxide', 'pressure_mb', 'air_quality_PM10', 'air_quality_Ozone']

🔄 Analyzing feature interactions...

👤 Generating individual prediction explanations...

📋 High PM2.5 Prediction:
   Actual PM2.5: 60.87 µg/m³
   Predicted PM2.5: 46.18 µg/m³
   Error: -14.69 µg/m³

<Figure size 1200x400 with 0 Axes>

📋 Low PM2.5 Prediction:
   Actual PM2.5: 4.40 µg/m³
   Predicted PM2.5: 1.95 µg/m³
   Error: -2.45 µg/m³

<Figure size 1200x400 with 0 Axes>

📋 Most Accurate Prediction:
   Actual PM2.5: 10.36 µg/m³
   Predicted PM2.5: 10.36 µg/m³
   Error: -0.00 µg/m³

<Figure size 1200x400 with 0 Axes>

📋 Least Accurate Prediction:
   Actual PM2.5: 55.30 µg/m³
   Predicted PM2.5: 5.37 µg/m³
   Error: -49.93 µg/m³

<Figure size 1200x400 with 0 Axes>

📋 Random Typical Case:
   Actual PM2.5: 9.44 µg/m³
   Predicted PM2.5: 12.90 µg/m³
   Error: 3.47 µg/m³

<Figure size 1200x400 with 0 Axes>

📈 Generating decision plots...

<Figure size 1200x800 with 0 Axes>

✅ Decision plot generated successfully!

📊 SHAP QUANTITATIVE ANALYSIS
==================================================
Top 10 Features by SHAP Importance:

Comparison: Model Feature Importance vs SHAP Importance

📈 FEATURE DIRECTION ANALYSIS:
Positive SHAP values increase PM2.5 predictions
Negative SHAP values decrease PM2.5 predictions
----------------------------------------
• air_quality_PM2.5:
  Average SHAP effect: 2.5056 (increases predictions)
  Correlation with PM2.5: 1.0000
  ✅ SHAP direction matches correlation

• air_quality_Carbon_Monoxide:
  Average SHAP effect: -0.1465 (decreases predictions)
  Correlation with PM2.5: 0.6160
  ⚠️  SHAP direction differs from correlation

• air_quality_Sulphur_dioxide:
  Average SHAP effect: 0.2141 (increases predictions)
  Correlation with PM2.5: 0.3128
  ✅ SHAP direction matches correlation

• pressure_mb:
  Average SHAP effect: 0.1779 (increases predictions)
  Correlation with PM2.5: -0.0042
  ⚠️  SHAP direction differs from correlation

• air_quality_PM10:
  Average SHAP effect: -0.0151 (decreases predictions)
  Correlation with PM2.5: 0.6459
  ⚠️  SHAP direction differs from correlation

📊 Analyzing feature impact across PM2.5 levels...

💡 KEY SHAP INSIGHTS FOR IEEE PAPER
============================================================
1. MOST INFLUENTIAL FEATURES:
   • air_quality_PM2.5 (SHAP importance: 6.7646)
   • air_quality_Carbon_Monoxide (SHAP importance: 0.7200)
   • air_quality_Sulphur_dioxide (SHAP importance: 0.4951)

2. FEATURE EFFECTS ON PM2.5 PREDICTIONS:
   • air_quality_PM2.5: INCREASES predicted PM2.5
   • air_quality_Carbon_Monoxide: DECREASES predicted PM2.5
   • air_quality_Sulphur_dioxide: INCREASES predicted PM2.5

3. MODEL INTERPRETABILITY:
   • Expected value (baseline): 16.95 µg/m³
   • Top 5 features explain 87.9% of predictions
   • Model shows consistent feature importance across different PM2.5 levels

4. PRACTICAL IMPLICATIONS:
   • Current day PM2.5 is the strongest predictor of next day PM2.5
   • Meteorological factors (temperature, humidity, wind) significantly influence predictions
   • Model captures non-linear relationships between features and PM2.5

✅ All SHAP analysis completed and saved to: /kaggle/working/pm25_forecasting_plots/
✅ SHAP values and sample data saved for reproducibility

🔍 SHAP ANALYSIS DATA SOURCE VERIFICATION
==================================================
📊 Data Source: YOUR Global Weather Repository dataset
📁 Dataset shape: (103942, 53)
🔢 SHAP sample size: 1000 instances from your test set
📈 Features analyzed: 17 features from your data
🎯 Target variable: Next-day PM2.5 from your measurements

📋 Sample of features used in SHAP analysis:
    1. temperature_celsius
    2. humidity
    3. pressure_mb
    4. wind_kph
    5. precip_mm
    6. visibility_km
    7. air_quality_PM2.5
    8. air_quality_PM10
    9. air_quality_Ozone
   10. air_quality_Nitrogen_dioxide

📊 SHAP values calculated from:
   • X_test shape: (19016, 17)
   • Sample used: (1000, 17)
   • Model: Gradient Boosting trained on YOUR data
   • Expected value (baseline): 16.95 µg/m³

✅ VERIFICATION: All SHAP analysis was performed on YOUR real weather and air quality data

	country	location_name	latitude	longitude	timezone	last_updated_epoch	last_updated	temperature_celsius	temperature_fahrenheit	condition_text	...	air_quality_PM2.5	air_quality_PM10	air_quality_us-epa-index	air_quality_gb-defra-index	sunrise	sunset	moonrise	moonset	moon_phase	moon_illumination
0	Afghanistan	Kabul	34.52	69.18	Asia/Kabul	1715849100	2024-05-16 13:15	26.6	79.8	Partly Cloudy	...	8.4	26.6	1	1	04:50 AM	06:50 PM	12:12 PM	01:11 AM	Waxing Gibbous	55
1	Albania	Tirana	41.33	19.82	Europe/Tirane	1715849100	2024-05-16 10:45	19.0	66.2	Partly cloudy	...	1.1	2.0	1	1	05:21 AM	07:54 PM	12:58 PM	02:14 AM	Waxing Gibbous	55
2	Algeria	Algiers	36.76	3.05	Africa/Algiers	1715849100	2024-05-16 09:45	23.0	73.4	Sunny	...	10.4	18.4	1	1	05:40 AM	07:50 PM	01:15 PM	02:14 AM	Waxing Gibbous	55
3	Andorra	Andorra La Vella	42.50	1.52	Europe/Andorra	1715849100	2024-05-16 10:45	6.3	43.3	Light drizzle	...	0.7	0.9	1	1	06:31 AM	09:11 PM	02:12 PM	03:31 AM	Waxing Gibbous	55
4	Angola	Luanda	-8.84	13.23	Africa/Luanda	1715849100	2024-05-16 09:45	26.0	78.8	Partly cloudy	...	183.4	262.3	5	10	06:12 AM	05:55 PM	01:17 PM	12:38 AM	Waxing Gibbous	55

	Model	RMSE	MAE	R²
2	Gradient Boosting	8.607536	5.987339	0.609420
4	SVR	8.699182	6.051479	0.601058
1	Random Forest	8.843118	6.356464	0.587747
0	Linear Regression	9.705164	6.881306	0.503455
3	XGBoost	9.976024	6.949152	0.475352

	PM2.5 Category	Classification Accuracy
1	Moderate (12.1-35.4)	0.791891
0	Good (0-12)	0.722001
2	Unhealthy for Sensitive (35.5-55.4)	0.463061
3	Unhealthy (55.5-150.4)	0.000000

	Value
Best Model	Gradient Boosting
Final RMSE	8.612281
Final MAE	5.944932
Final R²	0.608989
CV RMSE	8.081556
CV R²	0.652092
Number of Features	17
Training Samples	76060
Test Samples	19016

	Country	Samples	RMSE	R²	Avg_PM2.5
20	Tuvalu	535	3.581427	0.439625	6.556479
15	Tonga	523	3.665990	0.630439	9.450956
14	Timor-Leste	532	3.852080	0.507236	8.394445
12	Tanzania	535	3.866014	0.631654	9.504206
28	Vanuatu	530	4.056362	0.545891	9.859943
0	Somalia	468	4.781026	0.583343	14.048248
6	Suriname	532	4.916826	0.580398	9.430912
8	Sweden	534	5.257851	0.045622	7.400833
16	Trinidad and Tobago	532	5.539427	0.474136	10.148536
25	United States of America	524	5.725299	-0.204446	6.387366

	Missing Count	Percentage (%)
country	0	0.0
sunrise	0	0.0
gust_kph	0	0.0
air_quality_Carbon_Monoxide	0	0.0
air_quality_Ozone	0	0.0
air_quality_Nitrogen_dioxide	0	0.0
air_quality_Sulphur_dioxide	0	0.0
air_quality_PM2.5	0	0.0
air_quality_PM10	0	0.0
air_quality_us-epa-index	0	0.0
air_quality_gb-defra-index	0	0.0
sunset	0	0.0
uv_index	0	0.0
moonrise	0	0.0
moonset	0	0.0

	feature	importance
6	air_quality_PM2.5	0.944895
7	air_quality_PM10	0.014701
11	air_quality_Carbon_Monoxide	0.011048
10	air_quality_Sulphur_dioxide	0.005760
2	pressure_mb	0.005591
1	humidity	0.003787
8	air_quality_Ozone	0.003349
9	air_quality_Nitrogen_dioxide	0.003186
4	precip_mm	0.002493
12	month	0.001761

	Country	Samples	RMSE	MAE	R²	MAPE (%)	Bias	Avg_PM2.5
0	Tuvalu	535	3.5814	2.5848	0.4396	68.33	0.5393	6.5565
1	Tonga	523	3.6660	2.6712	0.6304	52.93	0.3052	9.4510
2	Timor-Leste	532	3.8521	2.8006	0.5072	58.35	0.7500	8.3944
3	Tanzania	535	3.8660	2.8829	0.6317	54.82	0.9681	9.5042
4	Vanuatu	530	4.0564	2.9095	0.5459	48.33	0.2120	9.8599
5	Somalia	468	4.7810	3.0264	0.5833	27.04	-0.3357	14.0482
6	Suriname	532	4.9168	3.4325	0.5804	70.66	0.8207	9.4309
7	Sweden	534	5.2579	4.0587	0.0456	115.78	2.7386	7.4008
8	Trinidad and Tobago	532	5.5394	3.7818	0.4741	68.83	0.8505	10.1485
9	United States of America	524	5.7253	4.1448	-0.2044	139.90	2.3592	6.3874
10	Switzerland	557	6.4989	4.7486	0.5955	75.76	0.9636	13.2352
11	Zambia	534	6.9050	4.8656	0.3504	50.99	1.4557	12.9758
12	Ukraine	532	7.1152	5.0985	0.6299	57.19	0.7218	16.4857
13	Sri Lanka	496	7.2563	4.9923	0.7429	40.29	-0.0913	21.1231
14	Vatican City	534	8.0787	5.6293	0.4983	45.48	-0.3622	16.9394

	feature	shap_importance
6	air_quality_PM2.5	6.7646
11	air_quality_Carbon_Monoxide	0.7200
10	air_quality_Sulphur_dioxide	0.4951
2	pressure_mb	0.4321
7	air_quality_PM10	0.3701
8	air_quality_Ozone	0.2377
12	month	0.2117
9	air_quality_Nitrogen_dioxide	0.2015
1	humidity	0.1688
15	country_encoded	0.1335

	feature	shap_importance	model_importance	rank_diff
0	air_quality_PM2.5	6.7646	0.9449	0.0
1	air_quality_Carbon_Monoxide	0.7200	0.0110	-1.0
2	air_quality_Sulphur_dioxide	0.4951	0.0058	-1.0
3	pressure_mb	0.4321	0.0056	-1.0
4	air_quality_PM10	0.3701	0.0147	3.0
5	air_quality_Ozone	0.2377	0.0033	-1.0
6	month	0.2117	0.0018	-3.0
7	air_quality_Nitrogen_dioxide	0.2015	0.0032	0.0
8	humidity	0.1688	0.0038	3.0
9	country_encoded	0.1335	0.0006	-3.0
10	precip_mm	0.1158	0.0025	2.0
11	temperature_celsius	0.0748	0.0013	1.0
12	location_encoded	0.0400	0.0011	1.0
13	visibility_km	0.0256	0.0005	0.0
14	day_of_week	0.0000	0.0000	0.0