The modeling and prediction of the covariance matrix is very important for financial risk management and investment portfolio management. To solve the problem of low prediction accuracy of time series models for highdimensional variables long short memory neural network model LSTM is used to propose a covariance matrix prediction model using high-frequency data based on deep learning. The model uses financial high-frequency data to obtain the realized covariance matrix performs DRD decomposition on the realized covariance matrix vectorizes the correlation coefficient matrix R and uses the vector heterogeneous autoregressive model HAR to predict the realized correlation coefficient matrix R. Based on the realized volatility matrix D this paper uses the idea of semi covariance combined with the LSTM model obtains the deep learning prediction model of the realized volatility matrix D and constructs the dynamic prediction model of realized covariance matrix LSTM-SDRD-HAR. The LSTM and HAR model can capture the long-term memory of actual data and the semi-covariance is conducive to capturing the leverage of financial data. The empirical analysis shows that compared with the traditional vector HAR prediction model LSTM-SDRD-HAR has a more accurate prediction of the realized covariance matrix. The effective frontier portfolio investment structured by LSTM-SDRD-HAR prediction is better.