Sync files programatically

pankhuri.verma

How can we sync files programmatically in between our code runs.

john.joo

Hi @pankhuri.verma , why would you want to sync files programmatically during your code runs?

Is this to make sure you're backing up files so that you don't lose progress on some buggy code? Is this to monitor the progress of your run, say loss metrics, in the realtime way?

katie.shakman

We had a support ticket that sparked this request initially, and it seems the use case had to do with running multiple experimental configurations from the same job or workspace, and syncing after each iteration/configuration change.

joaquim.benseny

We would also appreciate this. Use case:

1. From workspace A we run some code and create some data that we store in the workspace.
2. Trigger multiple workspaces that should use this data (i.e. workspace B, workspace C...)
3. Wait from workspace A and wait until they finish.
4. Bring files from workspace B, workspace C...
5. Collate all results and carry on some calculations

Although we can use blobs_get and files_upload this is not great when you have multiple files and we would like to use sync between step 1 and 2 and get latest domino files for step 4.