Automatic resubmission of jobs

This option is primarily intended for CCRT users. Since batch job walltime is limited to 3 days, it may be necessary to relaunch long inversions automatically. To enable this, set two optional arguments in obsoperator as described below. The inversion will stop automatically after autokill_time and a new job will be submitted. Set a sufficient margin between autokill_time and the cluster walltime, because the kill/resubmit process occurs at the end of the current transform loop. This margin must be large enough to ensure all running transforms finish before the job is killed. To enable automatic resubmission:

1. Modify the YAML to enable resubmission:

   • in obsoperator, set option autokill_time to the duration after which the job is resubmitted.

   • in obsoperator, set option max_resubmissions.

obsoperator:
  plugin:
    name: standard
    version: std
    type: obsoperator
  autokill_time: 71H
  max_resubmissions: 5

2. Depending on your CIF configuration at CCRT, you may need to add a Python userbase path via python_userbase and a virtual environment path via python_venv.

3. A new job will be submitted when autokill_time is reached, enabling inversions that span periods longer than 3 days.