Do you have any reference on how to use tensorboard on tfevents files that are saved in Domino – inevitably under Datasets because they're too big to put under Domino project files?
Yes, you can use datasets in Domino to write these large log (tfevents) files to. You can either write the logs to a dataset scratch space or dataset snapshot. The below example discusses the approach to take to write this to a dataset scratch space.
By default each workspace is launched with a datasets scratch space that you can use to write these logs to. The scratch space is located at /domino/datasets/<user-name>/<project-name>/scratch
To configure this in your tensorboard, you can follow the below steps:
Add the below lines of code to the dockerfile instructions of your compute environment. You may skip this step if you have already configured the tensorboard extension.
RUN pip install ipython jupyter jupyter-tensorboard --upgrade
RUN jupyter tensorboard enable --system
From inside your Jupyter workspace, navigate to New -> Tensorboard -> Custom Directory. Specify the path of your scratch space here.
That's it! Your configuration is complete. Now run code in your notebook and when you log new events, these should get captured in the /logs directory in your scratch space. You can view the logs in the Tensorboard UI by navigating to Tensorboard -> Current directory
Once your work in the workspace is complete, you can choose to stop and commit changes in the workspace, post which if you navigate to the Datasets dashboard under the project, you'll be able to see the logs in the scratch space:
Note: scratch space is a temporary space that is mounted by default to each workspace. If you want to save the logs for future reference or reproducibility, you should create a new snapshot in an existing dataset by clicking the blue 'Create a Dataset Snapshot from Scratch Space' button.
Alternatively, you can also choose to write the output logs directly to a dataset snapshot, instead of the scratch space, by following the instructions mentioned here.