Data Versioning
Last updated
Last updated
You might already be familiar with versioning from working with version control or source control for your code. The basic idea is that, as you work, you create static copies of your work that you can refer back to later. (This is what the "commit" button in kernels does.) This is particularly important when youâre collaborating because if you and a collaborator are working on the same file, you can compare each of your versions and decide which changes to make.
So version control for code is a good idea⌠but what about data?
The idea that you should version your data is actually a somewhat controversial.
at all. And there are some good reasons for this:
Most version control software is designed for files with code in them, and these files generally arenât very big. Trying to use version control software for large files (more than a couple MB) means that a lot of their useful features, like showing line differences, are no longer as useful.
Version control can mean storing multiple copies of files, and if you have a large dataset this can quickly get very expensive.
If youâre routinely backing up your data and are using version control for the queries or script youâre using to extract data, then it may be redundant to store specific subsets of your data separately.
I agree that you donât need to save separate versions of your data for every single task you do. However, if youâre training machine learning models, I believe you should version both the exact data you used to train your model and your code. This is because, without the code, data and environment, you wonât be able to reproduce your model. (I wrote .)
This becomes a really thorny problem if you want to run experiments and compare models to each other. Does this new model architecture outperform the one youâre currently using? Without knowing what data you trained the original model on, itâs hard to tell.
When should you version your data?
When making schema/metadata changes, like adding or deleting columns or changing the units that information is stored in.
When youâre training experimental machine learning models. The smallest reproducible unit for machine learning models is training data + model specification code. (I talk more about this .)
When should you consider not versioning your data?
When your data isnât being used to train models. For example, itâs more space efficient to just save the SQL query you used to make a chart than it is to save all the transformed data.
When your data is large enough that storing a versioned copy would be prohibitively expensive. In this case, Iâd recommend versioning both the scripts you used to extract the data and enough descriptive statistics that you could re-generate a very similar dataset.
When your project lives entirely on GitHub. Versioning large datasets via GitHub can quickly become unwieldy. (GitLFS can help with this, but if youâre storing very large datasets, in general GitHub probably isnât the best tool for the job. A database or blog storage hosting service specifically designed for large data will generally give you fewer headaches. Most cloud services will generally already have some form of versioning built in.)
Of course, whether or not you should version data eventually comes down to a judgement call on your part.
If you're already using cloud tools to store your data, most platforms will have versioning built in.You should probably use these rather than setting up your own system, if for no other reason than that it will be someone elseâs problem when it inevitably breaks. đ
The biggest difference between these tools and Git is that they
Similarities:
Both are based on Git and are designed to interface well with existing Git toolchains.
They focus on versioning whole pipelines, versioning the data, code and trained models together.
Differences:
DVC is only available as a command line tool. Pachyderm also has a graphical user interface (GUI).
DVC is open source and free. Pachyderm does have an open source version, but to get all the bells and whistles you'll need to shell out for the enterprise edition.
Pachyderm is fully containerized (Docker and Kubernetes). You can use containers with DVC but they're not the default.
This section is a little bit theoretical, so I've got some discussion questions for you.
For each of these datasets consider whether it makes sense to version this data.
You have a streaming datasets of sensor data with more than five billion columns. It's updated every five seconds. Youâre using it to create a dashboard to let stakeholders monitor anomalies.
Youâve got a .CSV with a few thousand rows of student data. Youâre storing it in a computer that conforms with your country's laws around data privacy. You want to build a model to see if thereâs an effect of when tests are administered on test scores.
Youâre working with a customer database of one million pet owners who have ordered custom dog food through your startup. You want to create a slide deck with visualizations that summarize information about your customers to help the marketing team decide where to buy newspaper ads.
In [1]:
OK, now that weâve got some theory out of the way and you should have a better idea of whether versioning is appropriate, letâs make some new datasets!
Today, weâre going to be making datasets from GitHub repos (short for ârepositoriesâ). First, youâll need to pick a repo.
Once youâve picked a repo, itâs fairly easy to create your dataset.
Go to www.kaggle.com/datasets (or just click on the "Datasets" tab up near the search bar).
Click on "New Dataset".
In the modal that pops up, click on the circle with a silhouette of an Octocat (the GitHub mascot) in it.
Enter a dataset title and the URL of the GitHub repository you're interested in. If the URL is valid, you should see a list of all the files that will be included in your dataset.
Hit create.
Success!
Go to the page for your dataset. (You can find it by clicking on the "Datasets" tab next to the search bar and then clicking on the "Your Datasets" tab.)
Click on the "Settings" tab. (The end of the URL will be /settings.)
To turn off versioning, select "Latest version only" in the "Versioning" drop down.
To automatically update your dataset, choose your preferred frequency from the "Update" dropdown.
That's all there is to it! đ
If your data is on Kaggle, we already take care of the data versioning for you; you can scroll to the bottom of your dataset landing page to see the "History" tab and check out previous versions of the dataset, what changes were made between versions and any updates that have been made to the metadata. .
But some data shouldn't be put on Kaggle (, for example) and sometimes you might prefer to work locally. What tools can you use in that case?
The ecosystem of data versioning tools is still pretty young, but you do have some options. Right now, two of the more popular are , which is short for Data Version Control, and . These aren't coding environments like Kaggle Kernels are. Rather, they're similar to Git: they let you save specific versions of your code and data along with comments on them. This allows you to revert your work back later and track what you did and to collaborate with others on the same code/data.
Other options for versioning data and pipelines include, in alphabetical order: , , , , and . These are a mix of closed and open source and a lot of them are in development, so it's a good idea to do some shopping around before you commit to a tooling system.
Currently, we only support creating datasets from public repos. If you donât have any public data of your own, you can check out to see if something tickles your fancy.