Debugging pipelines

Pipeline executions can fail for many reasons, Flow.bio has many ways to help you interact with executions to solve any problems.

1. Cancelling executions

If you already know that you configured something wrong, you can cancel a running execution by clicking on the "Cancel" button in the top right of the execution page.

2. Retrying executions

Sometimes processes fail due to random issues with software connections and simply retrying the process is all that is needed to move ahead. You can retry failed or cancelled executions by clicking on the "Retry" button in the top right of the execution page.

On doing this, you will be taken to an execution page that is populated with all of your existing data and parameters. If you want to try it exactly the same way again then scroll down and click "Run Pipeline" without making any changes. However you can also use this opportunity to change some parameters if you think this is the reason why your execution failed.

3. Inspecting failed processes

When an execution has failed it is marked with a red "Error" tag. By looking through the execution page you will often be able to find processes that have failed, marked by the same tag. You can inspect the input data to the process to establish if this looks correct.

By clicking on the [>_] button you can see the code error message - this is often helpful in diagnosing the problem. For example, here we can see that a parameter needs to be changed for STAR to successfully align this data.

Clicking the [$ ] button lets you see exactly the code that was executed - sometimes it is possible to spot errors here. You can also download the input data and try this command yourself locally to diagnose the issue.

4. Inspecting the Nextflow Report and Logs

Other useful sources of information are the Nextflow Report and the Logs, which you can find at the top of the execution page.

• Nextflow Report This is useful for showing the resources used by each process, and at the bottom the 'Tasks' table lets you sort tasks by status, allowing you to quickly see which processes have failed.

• Nextflow Log

If you are a Nextflow whiz then the full log can be helpful for debugging issues, but oftentimes it can be a bit hard to find what you need here!

4. Improving your pipelines as a developer

As an Admin user on Flow.bio you can monitor your pipeline usage by all Flow.bio users over time and see where your pipeline commonly fails. This can be a great way to identify a process that needs better error handling or documentation.

You can find this by going to the "Usage" tab, scrolling down and selecting the pipeline you are interested in.

In this example, we can see that the pipeline has failed twice and both times it was the "IDENTIFY_PSITES" process that failed, telling us that this process might need some attention.

5. If you get stuck

If you get stuck feel free to post your issue on our Forum, where we will respond as soon as possible.