我找到了一种似乎可行的解决方案!
看一下:用h5py增量写入hdf5!
为了将数据附加到特定数据集,必须首先在相应的轴上调整特定数据集的大小,然后在“旧” nparray的末尾附加新数据。
因此,解决方案如下所示:
with h5py.File('.\PreprocessedData.h5', 'a') as hf:
hf["X_train"].resize((hf["X_train"].shape[0] + X_train_data.shape[0]), axis = 0)
hf["X_train"][-X_train_data.shape[0]:] = X_train_data
hf["X_test"].resize((hf["X_test"].shape[0] + X_test_data.shape[0]), axis = 0)
hf["X_test"][-X_test_data.shape[0]:] = X_test_data
hf["Y_train"].resize((hf["Y_train"].shape[0] + Y_train_data.shape[0]), axis = 0)
hf["Y_train"][-Y_train_data.shape[0]:] = Y_train_data
hf["Y_test"].resize((hf["Y_test"].shape[0] + Y_test_data.shape[0]), axis = 0)
hf["Y_test"][-Y_test_data.shape[0]:] = Y_test_data
但是,请注意,您应该使用创建数据集maxshape=(None,)
,例如
h5f.create_dataset('X_train', data=orig_data, compression="gzip", chunks=True, maxshape=(None,))
否则无法扩展数据集。