Model versioning#

Use-cases:

  • A/B testing between branches.

  • Updating a model in production is as easy as merging it.

  • We can mount the models folder or load it explicitly.

ML experimentation today#

Currently, many organizations manage their experiments by saving snapshots of the data, the logs, the metrics and the models. The best performing models are copied to a different location for deployment to dev and production environments, which are ometimes managed by a database, or with docker-image registry. Often, another third-party tool is used to manage experiments, which then needs to be integrated with the rest of the development stack.

An example:

s3://project-data/
├── data/
│ ├── snapshot-210322.csv
│ ├── snapshot-210323.csv

s3://project-models/
├── v1/
│ ├── model-210322.pkl
│ ├── model-210323.pkl
├── prod/
│ ├── model.pkl
├── dev/
│ ├── model.pkl
├── a-b-test-210322/
│ ├── model.pkl
├── a-b-test-210323/
│ ├── model.pkl

s3://project-metrics/ (accuracy, etc.)
├── v1/
│ ├── 210322.json
│ ├── 210323.json

s3://project-logs/ (bugs, errors, etc.)
├── v1/
│ ├── 210322.log
│ ├── 210323.log

s3://model-monitoring/ (model drift, etc.)
├── prod/
│ ├── 210322.csv
│ ├── 210323.csv
├── dev/
│ ├── 210322.csv
│ ├── 210323.csv
├── a-b-test
│ ├── 210322.csv
│ ├── 210323.csv

https://github.com/org/project-inference
├── app.py
├── dockerfile

https://github.com/org/project-training
├── train.py
├── notebooks/
│ ├── 210322.ipynb
│ ├── 210323.ipynb

https://github.com/org/project-ops
├── docker-compose.yml
├── teraform.tf

...

And this doesn’t even cover other common use cases like sharing data or models with other teams!

ML experimentation with pyxet and XetHub#

With XetHub, we can use Git to manage every part of your ML experiments by storing models and assest alongside your code. Optionally, use Git submodules to manage only large data on XetHub.

xet://org/project (branch-prod/dev/ab-test-210322/ab-test-210323)
├── data/ xet://org/project/data (submodule)   ├── data.csv
├── models/
│   ├── model.pkl
├── metrics/
│   ├── model.json
├── logs/
│   ├── app.log
├── monitoring/
│   ├── predictions.csv
│   ├── drift.csv
├── src (Always fit the right model and technologies)   ├── train.py
│   ├── serve.py 

This means that we can save a single model (or a whole pipeline) per branch — and merge it to the environment of our choosing. We can even write logs of bugs, drift, etc., which are always fitting the model in production, to the repository for easy review.

Experiment with feature engineering#

Checkout a new branch, add the feature engineering, train the model and commit the new model, logs, and metrics.
Good enough to go to production? Merge it all to prod in a single step.

Reproduce a model#

We can always reproduce results and compare models by checking out branches and re-running.

Retraining with more data?#

Start a new experiment branch from any existing one, pull new data with git submodule update data and run the training again. XetHub will overwrite the model, metrics, and checkpoints in the new branch, and if the pull request review is successful, merge the changes back to prod. This ensures that the model in prod is always up-to-date, and that its always stored alongside its metrics, inference code and relevant logs.

Everything this model needs is saved in the repo. If changes to the app are needed, they are managed together.