Debugging

If a pipeline job is failing and I need to debug it I’ll want to know what variables are being used. Often these variables are defined and inherited by a top level project for which I’m not an admin and don’t have access to.

So this is how I deal with it.

  1. Create a job that logs information that I need to know for debugging.
some_job:
  stage: some_stage
  after_script:
    - |
      set +e
      mkdir debug
      export TIMESTAMP=$(date +"%Y-%m-%dT%H%M%S")
      env > debug/env.sh
      env | grep ^CI_ | tee -a debug/ci-pipeline-env.sh
      echo "in working directory $(pwd) with files:" && ls -al
      find $PWD -type f > debug/all.fl
      echo "All files: $(wc -l debug/all.fl) files found" 
  artifacts:
    paths:
      - debug/
    when: always
  allow_failure: true  

NOTE: Change the some_job and some_stage as appropriate to debug the job you’re trying to debug.

NOTE2: If this job (some_job) already has a after_script defined then you will need to merge the original after_script to your debug version because only one after_script runs. The last one defined wins so for example:

some_job:
  after_script:
    - echo "original cleanup"
  after_script:
    - echo "debug stuff"

Only echo “debug stuff” would run. The first after_script is simply overwritten. No error, no warning — it’s just not used.

  1. Sometimes I’ll need to comment out rules that would otherwise prevent the pipeline jobs or steps from running. ``` some_job: rules:
    • if: ‘$SOME_JOB_DISABLED == “true”’ when: never
    • if: ‘$CI_PIPELINE_SOURCE == “push”’
    • if: ‘$CI_PIPELINE_SOURCE == “merge_request_event”’
    • if: ‘$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH’
    • if: ‘$CI_COMMIT_TAG’
    • when: never ``` The above job will only run if there’s a merge request or a push onto the main (or master) branch (aka the $CI_DEFAULT_BRANCH).

I’ll often need to comment out rules that are preventing the job from running on an arbitrary branch as my commits start being made in an arbitrary branch.

To comment out, I simply use a hash # at the start of the line

    # - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'

Be aware of the yaml rules for indentation. Your comment must be at the appropriate indentation level in order to be syntactically valid for your debugging.


<
Previous Post
Start Graphical Environment from tty login
>
Blog Archive
Archive of all previous blog posts