Abstract: With the rapid development of science and technology, the risk of image encryption algorithm being cracked is gradually increasing. Although AES (advanced encryption standard) algorithm has superior performance, it also has the problem of key fixation, which poses a major threat to the advent of quantum computers in the future. First, discrete cosine transform (DCT) was used to transform plaintext images in DCT domain. Second, the probability distribution matrix was constructed by alternating quantum random walk, and the probability distribution matrix was preprocessed to perform bit by bit XOR with the plaintext after DCT. Finally, the singular value of the probability distribution matrix was calculated, and the corresponding hexadecimal expression was given as the AES key. An element of the probability distribution matrix was randomly extracted, and the round constant Rcon was constructed according to the coding rules of the AES algorithm, thereby improving the encryption iteration function of the AES algorithm. The test results show that the algorithm has good encryption effect and high robustness, and the correlation coefficient of adjacent pixels in the ciphertext image in three directions can reach 0.001 5.