Introduction
In this blog we will create cheat-sheet, or reference material where in we will cover the main functionalities for accessing the data from HDF5 Files in Python using pandas HDFStore.
HDFStore
► The HDFStore class is the pandas abstraction, responsible for dealing with HDF5 data.
- Under the hood HDFStore uses the PyTables library, so make sure that it is installed, if you want to use this format.
► Give the HDFStore constructor the path to a demo file and create a store:
filename = "pytable_df_file.h5"
store = pd.io.pytables.HDFStore(filename)
print(store)
► HDFStore has a dictionary like interface, which means we can store key-value pairs, for example: a pandas DataFrame with a corresponding lookup key.
► Store a DataFrame which contains your data in HDFStore as follows. The key is 'df' here (same is highlighted below), and the DataFrame object is df.
store['df'] = df
print(store)
► Save the DataFrame back to HDF5
- This is quite simple, simply assign back the df to HDF5 file, as mentioned in below line
store['df'] = changed_df
► We can access the DataFrame in three ways:
1. First with the get() method
print("Get", store.get('df').shape)
2. Dictionary like lookup
print("Lookup", store['df'].shape)
3. And finally the dotted access
print("Dotted", store.df.shape)
► We can delete an item from the store by calling the remove() method, or with the del operator.
del store['df']
print("After del\n", store)
► The is_open attribute indicates whether the store is open or not.
► The store can be closed with the close() method. In below sample we will close the store and check again that it got closed or not:
print("Before close", store.is_open)
store.close()
print("After close", store.is_open)
► Pandas also provides a DataFrame to_hdf() method, and a top-level read_hdf() function to read and write HDF5 data. In below snippet we will call the to_hdf() method and read the data:
df.to_hdf(tmpf.name, 'data', format='table')
print(pd.read_hdf(tmpf.name, 'data', where=['index>363']))
Summary
Using this cheat-sheet you should be able to start coding the needed functionality.
References
Your feedback
Your feedback is of paramount importance to us, provide your valuable inputs; what else you would like to read about, or if you would like the author to elaborate on any specific topic from this article, or any other article on this blog.
Disclaimer
The material presented in this blog is to help educate the developers, and the author(s) & owner(s) of this blog are not responsible of any damage arising out of use of the code snippets or any other contents of this blog.
Comments
Post a Comment