SQL-DFS: A Massive Small File Storage System Based on HDFS

In order to solve the problem of high occupancy rate of Name Node memory while using Hadoop distributed file system( HDFS) to store massive small files,this paper analyzed the HDFS storage structure and presented a SQL-DFS file system based on metadata storage cluster. In SQL-DFS,in order to move small file metadata from Name Node memory to metadata storage cluster a small file processing module was added in Name Node. In order to improve the reading and writing speed of the metadata,relational database cluster was used,and in order to reduce the time of request for Name Node the reading process of the small file was optimized. To further reduce the load pressure of Name Node,the checking of file block from Data Node was completed by metadata storage cluster. Finally the contrast experiments were carried out between HDFS and SQL-DFS experimental platform. The experimental results show that SQL-DFS in the file average cost( FAC) and memory occupancy rate are significantly better than that of the original HDFS architecture and has better small file storage capacity. It can be used for the storage of massive small files.