1. To start a new data manager, need parameters data_dir: the dir to store the data and metadata format_version: the version of index and data file size_per_index: size in byte per index index_num_per_file: number of index per index file max_size_data_file: maximum size per data file data_blk_size: block size for data file

data_dir +

• metadata [bin file] #generated by data manager
• transid [dir] #transid for every backend
• 0...n [dir]
  • index.0
  • data.0

metadata structrue: [format_version, size_per_index, index_num_per_file, index_file_num_per_dir,max_size_data_file, data_blk_size, need_compress], size: 2+4+4+4+8+4+2 = 28 bytes index structrue: [transid, type_no, dir_no, file_no, data_blk_no, data_len, checksum], size: 8+4+4+4+8+4+4 = 36 data structure: [transid, checksum, raw_datapack]

format_version: 16-bits unsigned integer (S) size_per_index: 32-bits unsigned integer (L) index_num_per_file: 32-bits unsigned integer (L) index_file_num_per_dir: 32-bits unsigned integer (L) max_size_data_file: 64-bits unsigned integer (Q) data_blk_size: 32-bits unsigned integer (L) need_compress: 16-bits unsigned integer (S)

transid: 64-bits unsigned integer (Q) type_no: 32-bits unsigned integer (L) dir_no: 32-bits unsigned integer (L) file_no: 32-bits unsigned integer (L) data_blk_no: 64-bits unsigned integer (Q) data_len: 32-bits unsigned integer(L) check_sum: 32-bits unsigned integer(L) CRC check sum

NOTICE:

1. transid start from 1, 0 stands for nil