# Model versioning

Use-cases:

* A/B testing between branches.
* Updating a model in production is as easy as merging it.
* We can mount the models folder or load it explicitly.

## ML experimentation today

Currently, many organizations manage their experiments by saving snapshots of the data, the logs, the metrics and the
models. The best performing models are copied to a different location for deployment to dev and production environments, 
which are ometimes managed by a database, or with docker-image registry. Often, another third-party tool is used to 
manage experiments, which then needs to be integrated with the rest of the development stack.

*An example:*

```bash
s3://project-data/
├── data/
│ ├── snapshot-210322.csv
│ ├── snapshot-210323.csv

s3://project-models/
├── v1/
│ ├── model-210322.pkl
│ ├── model-210323.pkl
├── prod/
│ ├── model.pkl
├── dev/
│ ├── model.pkl
├── a-b-test-210322/
│ ├── model.pkl
├── a-b-test-210323/
│ ├── model.pkl

s3://project-metrics/ (accuracy, etc.)
├── v1/
│ ├── 210322.json
│ ├── 210323.json

s3://project-logs/ (bugs, errors, etc.)
├── v1/
│ ├── 210322.log
│ ├── 210323.log

s3://model-monitoring/ (model drift, etc.)
├── prod/
│ ├── 210322.csv
│ ├── 210323.csv
├── dev/
│ ├── 210322.csv
│ ├── 210323.csv
├── a-b-test
│ ├── 210322.csv
│ ├── 210323.csv

https://github.com/org/project-inference
├── app.py
├── dockerfile

https://github.com/org/project-training
├── train.py
├── notebooks/
│ ├── 210322.ipynb
│ ├── 210323.ipynb

https://github.com/org/project-ops
├── docker-compose.yml
├── teraform.tf

...
```

And this doesn't even cover other common use cases like sharing data or models with other teams!

## ML experimentation with pyxet and XetHub

With XetHub, we can use Git to manage every part of your ML experiments by storing models and assest alongside your code.
Optionally, use [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to manage only large data on XetHub.


```bash
xet://org/project (branch-prod/dev/ab-test-210322/ab-test-210323)
├── data/ xet://org/project/data (submodule)
│   ├── data.csv
├── models/
│   ├── model.pkl
├── metrics/
│   ├── model.json
├── logs/
│   ├── app.log
├── monitoring/
│   ├── predictions.csv
│   ├── drift.csv
├── src (Always fit the right model and technologies)
│   ├── train.py
│   ├── serve.py 
```

This means that we can save a single model (or a whole pipeline) per branch — and merge it to the environment of our
choosing. We can even write logs of bugs, drift, etc., which are always fitting the model in production, to the repository 
for easy review.

### Experiment with feature engineering

Checkout a new branch, add the feature engineering, train the model and commit the new model, logs, and metrics.   
Good enough to go to production? Merge it all to prod in a single step.

### Reproduce a model

We can always reproduce results and compare models by checking out branches and re-running.

### Retraining with more data?

Start a new experiment branch from any existing one, pull new data with `git submodule update data` and run the training 
again. XetHub will overwrite the model, metrics, and checkpoints in the new branch, and if the pull request review is 
successful, merge the changes back to prod. This ensures that the model in prod is always up-to-date, and that its
always stored  alongside its metrics, inference code and relevant logs.

Everything **this** model needs is saved in the repo. If changes to the app are needed, they are managed together.