Chapter 7: Conclusion, Limitations and Further Work
Conclusion, Limitations and Further Work
The relationship between the GCM output variables with global solar radiation observations have stronger auto-correlation for month averaged values in comparison to the daily auto-correlations. The influence of lagged values reflects the seasonality of the observed global solar radiation with lags having higher correlation between 4 and 6 lagged months as well as 10 to 12 lagged months.
The CNN-LSTM modelling approach demonstrates the ability to automatically produce features that enable it to learn the mapping between GCM outputs and target radiation observations without requiring more complicated feature selection procedures. Although is a more complex model requiring additional search for hyper-parameters as well as model configuration. A search procedure is necessary in order to identify suitable network configuration as well as parameters for the training optimisation process itself (especially the learning rate). The type of regularisation applied in the network has an impact on network accuracy. Regularisation permits the network to generalise across the variation present in the input domain and assists the network in learning suitable internal representations. This is one of the qualities of the deep learning architectures that help to provide robust learning even in the case where feature engineering and selection is not performed prior to training (this is not to dismiss feature engineering altogether however). The experimentation undertaken in this thesis demonstrates that the CNN-LSTM architecture is able to learn a mapping from coarse scale GCM output variables to radiation observations that is more accurate than those of the baseline models in an end to end manner. After the initial network configuration search, the findings on this data set indicate that max-pooling after each Convolutional layer assists the network in producing better results, as well as applying batch-normalisation post to the convolution operation. The best performing regularisation applied to the weights at each layer was L2 regularisation as opposed to L1 regularisation and drop-out was not applied in the training of the network as it was shown to produce lower performance metrics on this data set. Batch normalisation was not applied on the output layer, as this appeared to impede the ability of the network to produce the range of outputs matching those of the observations.
The CNN-LSTM models exhibit good generalisation capability across multiple sites in comparison to the baseline models for a majority of sites, however it is evident that there is some local dependency given the difference in performance between sites. Recommendations in the next section suggest further work in the direction of ensemble modelling to address these differences.
The CNN-LSTM exhibits lower uncertainty on the test set as well as the RCP4.5 and RCP8.5 climate warming scenarios for the period between 2006 to 2020. Seasonal differences between the error occurred in the local model, exhibiting higher error during summer months as opposed to the universal model which exhibited larger residuals during the winter months. Projections produced by the predictions of the models over the study area indicate small predicted decreases in surface availability of global solar radiation under the RCP8.5 scenario, up until 2100.
Limitations and Recommendations of Further Work
Downscaling of GCM model output is inherently different to the forecasting auto-correlation setting, where the relationship between previous values of the target variable and the next values can be learnt. Instead, the relationship between the GCM model and the observation must be learnt. The relationship is impacted both by the grid point scale and the bias produced by the GCM model itself. It appears that the GCM models are strongly connected to longer term seasonal trends, as opposed to short term high resolution daily observations, however such an assertion would be a potential area of further investigation. In the literature review section of this document, a large number of studies focused on temperature and precipitation rather than radiation. It would be of interest to determine whether there are differences in linkage of GCM outputs with different types of observed climate variables in the downscaling task.
In approaching the downscaling task, the outputs of three GCM models were averaged to form an ensemble prediction of the output variables. Instead of applying downscaling to the output derived from GCM models, it is common to use a reanalysis product based on historical observations in the downscaling task. It would be of interest to train the same architectures produced over the same study area and compare evaluation metrics as well as the mean prediction intervals in order to determine the influence of the reanalysis product on the accuracy of the same models.
In this study, the GCM outputs were not augmented with additional features other than variables representing the month of year, and lags of the same output variables. However instead of allowing the mapping between GCM outputs and radiation to be learnt directly from the data, it is possible to combine these outputs with existing models for the estimate of global solar radiation, such as those described by Ertkin and Yaldiz, 1999 [1], in order to generate an additional training feature that may provide good correlation with the target variable.
The study focused on the use of the 1-dimensional Convolutional module in feature extraction, rather than the 2-dimensional convolution, which did not perform as well in initial experiments on the downscaling task. There is room for further work in terms of investigating whether larger numbers of input grid points are required to support the extraction of spatial relationships by 2-dimensional convolutions in the downscaling task. It appears that the super-resolution scaling methods are useful in the imaging domain perhaps due to the self-similarity of localised features within the input image, however, such architectures may not be appropriate in the use of a small number of grid points in the downscaling task. An empirical algorithm can be applied to select the number of input points surrounding the site of interest. When considering multiple sites, it would be of interest to determine whether a large number of shared input grid points impact the ability of the model to learn against observations due to the increasing uniformity of shared inputs. Additional considerations concern hardware limitations where different network architectures are concerned, especially in terms of available GPU memory when increasing the dimensionality of the network.
Additional research in the area of network architecture search is also warranted, since developing a network requires search over the possible configuration space, the ability to search over potential architectures in order to select a suitable combination of modules would be highly advantageous. Such an area of research would need to address such challenges in the search algorithms use of available resources, potentially in terms of best use of distributed hardware. This thesis approached the task of selecting best configurations using a single metric (\(R^{2}\)) in both the CNN depth search as well as the network configuration search. Further work could investigate the use of a vector of multiple unitless evaluation metrics (including \(R^{2}\), Willmott’s index of agreement, Nash-Sutcliffe efficiency and KGE) in order to take advantage of the various strengths of each benchmark and a distance measure between the “ideal” values of each resulting for each step of the search procedure.
Projections obtained from the network indicate slight decrease in global solar radiation under RCP8.5 in comparison to RCP4.5. However, neural network models do not readily provide interpretable results, whereas methods for more general machine learning interpretability have been devised, tools such as Local Interpretable Model-Agnostic Explanations (LIME), Ribero et al. [2] which can be applied to ascertain the influence of covariates on the predicted values of the network. Such methods would be beneficial in supporting the use of deep learning methods as models which support interpretation of the role of the predictors.
Local variation between sites is indicated in the evaluation of the CNN-LSTM models, especially in terms of Carpentaria Downs Station. Noting that Carpentaria Downs Stations is situated further North-West and is relatively isolated from the other sites in this study, and that the model performed well on other sites, it would be of interest to determine whether a more uniform spatial sampling for locations of interest would influence the resulting models’ ability to generalise between sites.
During the development of both local and universal CNN-LSTM models, a number of configurations provided different results for performance metrics on each site in the downscaling task. The process of model development required selection of the final model out of several alternatives, attempting to select a model that performed well across all sites. As the performance varies with geographic location, it should be possible to combine multiple models that exhibit slightly different biases for sites having similar characteristics. Rather than building a single model for each site, it would be possible to build models that perform well for different extremes and combine these models either in an ensemble (as recommended in [3]), or potentially in a parallel network architecture by designing a network graph with multiple branches that can be merged at the output node permitting end-to-end training. Such an architecture speaks to the advantage of deep learning models which through their modular composition provide a mechanism permitting the synthesis of a high number of covariates in the learning process.
References
Downscaling Global Climate Models with Convolutional and Long-Short-Term Memory Networks for Solar Energy Applications by C.P. Davey is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.