Version Data¶

With log_data_version and log_s3_data_version helpers you can log data location and data hash to Neptune. It will be stored as property and can be viewed both in the Details section of an experiment:

and in the experiment dashboard as a column.

Check this example project to see more.

Prerequisites¶

Initialize Neptune

[ ]:

import neptune
neptune.init('USER_NAME/PROJECT_NAME')

File data version¶

[ ]:

from neptunecontrib.versioning.data import log_data_version

FILEPATH = '/path/to/data/my_data.csv'
with neptune.create_experiment():
    log_data_version(FILEPATH)

Folder data version¶

[ ]:

from neptunecontrib.versioning.data import log_data_version

DIRPATH = '/path/to/data/folder'
with neptune.create_experiment():
    log_data_version(DIRPATH)

S3 bucket data version¶

We can log both a version of a particular key which is similar to file versioning.

[ ]:

BUCKET = 'my-bucket'
PATH = 'training_dataset.csv'
with neptune.create_experiment():
    log_s3_data_version(BUCKET, PATH)

We can log a combined version of all the keys that start with a particular string which is similar to versioning a directory

[ ]:

BUCKET = 'my-bucket'
PATH = 'train_dir/'
with neptune.create_experiment():
    log_s3_data_version(BUCKET, PATH)

Prefixing¶

If you want to track multiple data sources make sure to prefix them before logging. For example:

[ ]:

from neptunecontrib.versioning.data import log_data_version

FILEPATH_TABLE_1 = '/path/to/data/my_table_1.csv'
FILEPATH_TABLE_2 = '/path/to/data/my_table_2.csv'

with neptune.create_experiment():
    log_data_version(FILEPATH_TABLE_1, prefix='table_1_')
    log_data_version(FILEPATH_TABLE_2, prefix='table_2_')