In view of gene expression characteristics of high dimension and high noise，orthogonal subspace matrix decomposition algorithm is proposed based on beta divergence matrix decomposition by introducing the orthogonal constraint to objective function for optimization and solution. The iteration rules of the matrix decomposition is given by the gradient descent method，the decomposition items are used to reduce the dimension of feature space，and the derived vector is used for kmeans clustering. Five tumor gene expression data are chosen for the experiment，and the results show that the improved algorithm matrix decomposition clustering is obviously better than other methods.