EXPERIMENTAL SimpleMarkDuplicatesWithMateCigar (Picard)

Examines aligned records in the supplied SAM/BAM/CRAM file to locate duplicate molecules. All records are then written to the output file with the duplicate records flagged.

Category Read Data Manipulation

Overview

This is a simple tool to mark duplicates using the DuplicateSetIterator, DuplicateSet, and SAMRecordDuplicateComparator. Users should continue to use MarkDuplicates in general. The main motivation of this tool was the fact that MarkDuplicates has many, many, many useful test cases, but few unit tests for validating individual duplicate sets. To test the DuplicateSetIterator, DuplicateSet, and SAMRecordDuplicateComparator, the most expedient method was to write this tool and make sure it behaves similarly to MarkDuplicates. Not the best, I know, but good enough. NH 06/25/2015. See MarkDuplicates for more details. @author nhomer

SimpleMarkDuplicatesWithMateCigar (Picard) specific arguments

This table summarizes the command-line arguments that are specific to this tool. For more details on each argument, see the list further down below the table or click on an argument name to jump directly to that entry in the list.

Argument name(s)	Default value	Summary
Required Arguments
--INPUT -I		One or more input SAM, BAM or CRAM files to analyze. Must be coordinate sorted.
--METRICS_FILE -M		File to write duplication metrics to
--OUTPUT -O		The output file to write marked records to
Optional Tool Arguments
--arguments_file		read one or more arguments files and add them to the command line
--ASSUME_SORT_ORDER -ASO		If not null, assume that the input file has this order even if the header says otherwise.
--ASSUME_SORTED -AS	false	If true, assume that the input file is coordinate sorted even if the header says otherwise. Deprecated, used ASSUME_SORT_ORDER=coordinate instead.
--BARCODE_TAG		Barcode SAM tag (ex. BC for 10X Genomics)
--CLEAR_DT	true	Clear DT tag from input SAM records. Should be set to false if input SAM doesn't have this tag. Default true
--COMMENT -CO		Comment(s) to include in the output file's header.
--DUPLEX_UMI	false	Treat UMIs as being duplex stranded. This option requires that the UMI consist of two equal length strings that are separated by a hyphen (e.g. 'ATC-GTC'). Reads are considered duplicates if, in addition to standard definition, have identical normalized UMIs. A UMI from the 'bottom' strand is normalized by swapping its content around the hyphen (eg. ATC-GTC becomes GTC-ATC). A UMI from the 'top' strand is already normalized as it is. Both reads from a read pair considered top strand if the read 1 unclipped 5' coordinate is less than the read 2 unclipped 5' coordinate. All chimeric reads and read fragments are treated as having come from the top strand. With this option is it required that the BARCODE_TAG hold non-normalized UMIs. Default false.
--DUPLICATE_SCORING_STRATEGY -DS	SUM_OF_BASE_QUALITIES	The scoring strategy for choosing the non-duplicate among candidates.
--FLOW_DUP_STRATEGY	FLOW_QUALITY_SUM_STRATEGY	Use specific quality summing strategy for flow based reads. Two strategies are available: FLOW_QUALITY_SUM_STRATEG: The selects the read with the best total homopolymer quality. FLOW_END_QUALITY_STRATEGY: The strategy selects the read with the best homopolymer quality close to the end (10 bases) of the read. The latter strategy is recommended for samples with high duplication rate
--FLOW_EFFECTIVE_QUALITY_THRESHOLD	15	Threshold for considering a quality value high enough to be included when calculating FLOW_QUALITY_SUM_STRATEGY calculation.
--FLOW_MODE	false	enable parameters and behavior specific to flow based reads.
--FLOW_Q_IS_KNOWN_END	false	Treat position of read trimming based on quality as the known end (relevant for flow based reads). Default false - if the read is trimmed on quality its end is not defined and the read is duplicate of any read starting at the same place.
--FLOW_SKIP_FIRST_N_FLOWS	0	Skip first N flows, starting from the read's start, when considering duplicates. Useful for flow based reads where sometimes there is noise in the first flows (for this argument, "read start" means 5' end).
--FLOW_UNPAIRED_END_UNCERTAINTY	0	Maximal difference of the read end position that counted as equal. Useful for flow based reads where the end position might vary due to sequencing errors. (for this argument, "read end" means 3' end)
--FLOW_UNPAIRED_START_UNCERTAINTY	0	Maximal difference of the read start position that counted as equal. Useful for flow based reads where the end position might vary due to sequencing errors. (for this argument, "read start" means 5' end in the direction of sequencing)
--FLOW_USE_END_IN_UNPAIRED_READS	false	Make the end location of single end read be significant when considering duplicates, in addition to the start location, which is always significant (i.e. require single-ended reads to start andend on the same position to be considered duplicate) (for this argument, "read end" means 3' end).
--FLOW_USE_UNPAIRED_CLIPPED_END	false	Use position of the clipping as the end position, when considering duplicates (or use the unclipped end position) (for this argument, "read end" means 3' end).
--help -h	false	display the help message
--MAX_FILE_HANDLES_FOR_READ_ENDS_MAP -MAX_FILE_HANDLES	8000	Maximum number of file handles to keep open when spilling read ends to disk. Set this number a little lower than the per-process maximum number of file that may be open. This number can be found by executing the 'ulimit -n' command on a Unix system.
--MAX_OPTICAL_DUPLICATE_SET_SIZE	300000	This number is the maximum size of a set of duplicate reads for which we will attempt to determine which are optical duplicates. Please be aware that if you raise this value too high and do encounter a very large set of duplicate reads, it will severely affect the runtime of this tool. To completely disable this check, set the value to -1.
--MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP -MAX_SEQS	50000	This option is obsolete. ReadEnds will always be spilled to disk.
--MOLECULAR_IDENTIFIER_TAG		SAM tag to uniquely identify the molecule from which a read was derived. Use of this option requires that the BARCODE_TAG option be set to a non null value. Default null.
--OPTICAL_DUPLICATE_PIXEL_DISTANCE	100	The maximum offset between two duplicate clusters in order to consider them optical duplicates. The default is appropriate for unpatterned versions of the Illumina platform. For the patterned flowcell models, 2500 is moreappropriate. For other platforms and models, users should experiment to find what works best.
--PROGRAM_GROUP_COMMAND_LINE -PG_COMMAND		Value of CL tag of PG record to be created. If not supplied the command line will be detected automatically.
--PROGRAM_GROUP_NAME -PG_NAME	SimpleMarkDuplicatesWithMateCigar	Value of PN tag of PG record to be created.
--PROGRAM_GROUP_VERSION -PG_VERSION		Value of VN tag of PG record to be created. If not specified, the version will be detected automatically.
--PROGRAM_RECORD_ID -PG	MarkDuplicates	The program record ID for the @PG record(s) created by this program. Set to null to disable PG record creation. This string may have a suffix appended to avoid collision with other program record IDs.
--READ_NAME_REGEX		MarkDuplicates can use the tile and cluster positions to estimate the rate of optical duplication in addition to the dominant source of duplication, PCR, to provide a more accurate estimation of library size. By default (with no READ_NAME_REGEX specified), MarkDuplicates will attempt to extract coordinates using a split on ':' (see Note below). Set READ_NAME_REGEX to 'null' to disable optical duplicate detection. Note that without optical duplicate counts, library size estimation will be less accurate. If the read name does not follow a standard Illumina colon-separation convention, but does contain tile and x,y coordinates, a regular expression can be specified to extract three variables: tile/region, x coordinate and y coordinate from a read name. The regular expression must contain three capture groups for the three variables, in order. It must match the entire read name. e.g. if field names were separated by semi-colon (';') this example regex could be specified (?:.;)?([0-9]+)[^;];([0-9]+)[^;];([0-9]+)[^;]$ Note that if no READ_NAME_REGEX is specified, the read name is split on ':'. For 5 element names, the 3rd, 4th and 5th elements are assumed to be tile, x and y values. For 7 element names (CASAVA 1.8), the 5th, 6th, and 7th elements are assumed to be tile, x and y values.
--READ_ONE_BARCODE_TAG		Read one barcode SAM tag (ex. BX for 10X Genomics)
--READ_TWO_BARCODE_TAG		Read two barcode SAM tag (ex. BX for 10X Genomics)
--REMOVE_DUPLICATES	false	If true do not write duplicates to the output file instead of writing them with appropriate flags set.
--REMOVE_SEQUENCING_DUPLICATES	false	If true remove 'optical' duplicates and other duplicates that appear to have arisen from the sequencing process instead of the library preparation process, even if REMOVE_DUPLICATES is false. If REMOVE_DUPLICATES is true, all duplicates are removed and this option is ignored.
--SORTING_COLLECTION_SIZE_RATIO	0.25	This number, plus the maximum RAM available to the JVM, determine the memory footprint used by some of the sorting collections. If you are running out of memory, try reducing this number.
--TAG_DUPLICATE_SET_MEMBERS	false	If a read appears in a duplicate set, add two tags. The first tag, DUPLICATE_SET_SIZE_TAG (DS), indicates the size of the duplicate set. The smallest possible DS value is 2 which occurs when two reads map to the same portion of the reference only one of which is marked as duplicate. The second tag, DUPLICATE_SET_INDEX_TAG (DI), represents a unique identifier for the duplicate set to which the record belongs. This identifier is the index-in-file of the representative read that was selected out of the duplicate set.
--TAGGING_POLICY	DontTag	Determines how duplicate types are recorded in the DT optional attribute.
--version	false	display the version number for this tool
Optional Common Arguments
--ADD_PG_TAG_TO_READS	true	Add PG tag to each read in a SAM or BAM
--COMPRESSION_LEVEL	5	Compression level for all compressed files created (e.g. BAM and VCF).
--CREATE_INDEX	false	Whether to create an index when writing VCF or coordinate sorted BAM output.
--CREATE_MD5_FILE	false	Whether to create an MD5 digest for any BAM or FASTQ files created.
--MAX_RECORDS_IN_RAM	500000	When writing files that need to be sorted, this will specify the number of records stored in RAM before spilling to disk. Increasing this number reduces the number of file handles needed to sort the file, and increases the amount of RAM needed.
--QUIET	false	Whether to suppress job-summary info on System.err.
--REFERENCE_SEQUENCE -R		Reference sequence file.
--TMP_DIR		One or more directories with space available to be used by this program for temporary storage of working files
--USE_JDK_DEFLATER -use_jdk_deflater	false	Use the JDK Deflater instead of the Intel Deflater for writing compressed output
--USE_JDK_INFLATER -use_jdk_inflater	false	Use the JDK Inflater instead of the Intel Inflater for reading compressed input
--VALIDATION_STRINGENCY	STRICT	Validation stringency for all SAM files read by this program. Setting stringency to SILENT can improve performance when processing a BAM file in which variable-length data (read, qualities, tags) do not otherwise need to be decoded.
--VERBOSITY	INFO	Control verbosity of logging.
Advanced Arguments
--showHidden	false	display hidden arguments

Argument details

Arguments in this list are specific to this tool. Keep in mind that other arguments are available that are shared with other tools (e.g. command-line GATK arguments); see Inherited arguments above.

--ADD_PG_TAG_TO_READS

Add PG tag to each read in a SAM or BAM

boolean true

--arguments_file

read one or more arguments files and add them to the command line

List[File] []

--ASSUME_SORT_ORDER / -ASO

If not null, assume that the input file has this order even if the header says otherwise.

Exclusion: This argument cannot be used at the same time as ASSUME_SORTED.

The --ASSUME_SORT_ORDER argument is an enumerated type (SortOrder), which can have one of the following values:

unsorted
queryname
coordinate
duplicate
unknown

SortOrder null

--ASSUME_SORTED / -AS

If true, assume that the input file is coordinate sorted even if the header says otherwise. Deprecated, used ASSUME_SORT_ORDER=coordinate instead.

Exclusion: This argument cannot be used at the same time as ASSUME_SORT_ORDER.

boolean false

--BARCODE_TAG

Barcode SAM tag (ex. BC for 10X Genomics)

String null

--CLEAR_DT

Clear DT tag from input SAM records. Should be set to false if input SAM doesn't have this tag. Default true

boolean true

--COMMENT / -CO

Comment(s) to include in the output file's header.

List[String] []

--COMPRESSION_LEVEL

Compression level for all compressed files created (e.g. BAM and VCF).

int 5 [ [ -∞ ∞ ] ]

--CREATE_INDEX

Whether to create an index when writing VCF or coordinate sorted BAM output.

Boolean false

--CREATE_MD5_FILE

Whether to create an MD5 digest for any BAM or FASTQ files created.

boolean false

--DUPLEX_UMI

Treat UMIs as being duplex stranded. This option requires that the UMI consist of two equal length strings that are separated by a hyphen (e.g. 'ATC-GTC'). Reads are considered duplicates if, in addition to standard definition, have identical normalized UMIs. A UMI from the 'bottom' strand is normalized by swapping its content around the hyphen (eg. ATC-GTC becomes GTC-ATC). A UMI from the 'top' strand is already normalized as it is. Both reads from a read pair considered top strand if the read 1 unclipped 5' coordinate is less than the read 2 unclipped 5' coordinate. All chimeric reads and read fragments are treated as having come from the top strand. With this option is it required that the BARCODE_TAG hold non-normalized UMIs. Default false.

boolean false

--DUPLICATE_SCORING_STRATEGY / -DS

The scoring strategy for choosing the non-duplicate among candidates.

The --DUPLICATE_SCORING_STRATEGY argument is an enumerated type (ScoringStrategy), which can have one of the following values:

SUM_OF_BASE_QUALITIES
TOTAL_MAPPED_REFERENCE_LENGTH
RANDOM

ScoringStrategy SUM_OF_BASE_QUALITIES

--FLOW_DUP_STRATEGY

Use specific quality summing strategy for flow based reads. Two strategies are available: FLOW_QUALITY_SUM_STRATEG: The selects the read with the best total homopolymer quality. FLOW_END_QUALITY_STRATEGY: The strategy selects the read with the best homopolymer quality close to the end (10 bases) of the read. The latter strategy is recommended for samples with high duplication rate

The --FLOW_DUP_STRATEGY argument is an enumerated type (FLOW_DUPLICATE_SELECTION_STRATEGY), which can have one of the following values:

FLOW_QUALITY_SUM_STRATEGY
FLOW_END_QUALITY_STRATEGY

FLOW_DUPLICATE_SELECTION_STRATEGY FLOW_QUALITY_SUM_STRATEGY

--FLOW_EFFECTIVE_QUALITY_THRESHOLD

Threshold for considering a quality value high enough to be included when calculating FLOW_QUALITY_SUM_STRATEGY calculation.

int 15 [ [ -∞ ∞ ] ]

--FLOW_MODE

enable parameters and behavior specific to flow based reads.

boolean false

--FLOW_Q_IS_KNOWN_END

Treat position of read trimming based on quality as the known end (relevant for flow based reads). Default false - if the read is trimmed on quality its end is not defined and the read is duplicate of any read starting at the same place.

boolean false

--FLOW_SKIP_FIRST_N_FLOWS

Skip first N flows, starting from the read's start, when considering duplicates. Useful for flow based reads where sometimes there is noise in the first flows (for this argument, "read start" means 5' end).

int 0 [ [ -∞ ∞ ] ]

--FLOW_UNPAIRED_END_UNCERTAINTY

Maximal difference of the read end position that counted as equal. Useful for flow based reads where the end position might vary due to sequencing errors. (for this argument, "read end" means 3' end)

int 0 [ [ -∞ ∞ ] ]

--FLOW_UNPAIRED_START_UNCERTAINTY

Maximal difference of the read start position that counted as equal. Useful for flow based reads where the end position might vary due to sequencing errors. (for this argument, "read start" means 5' end in the direction of sequencing)

int 0 [ [ -∞ ∞ ] ]

--FLOW_USE_END_IN_UNPAIRED_READS

Make the end location of single end read be significant when considering duplicates, in addition to the start location, which is always significant (i.e. require single-ended reads to start andend on the same position to be considered duplicate) (for this argument, "read end" means 3' end).

boolean false

--FLOW_USE_UNPAIRED_CLIPPED_END

Use position of the clipping as the end position, when considering duplicates (or use the unclipped end position) (for this argument, "read end" means 3' end).

boolean false

--help / -h

display the help message

boolean false

--INPUT / -I

One or more input SAM, BAM or CRAM files to analyze. Must be coordinate sorted.

R List[String] []

--MAX_FILE_HANDLES_FOR_READ_ENDS_MAP / -MAX_FILE_HANDLES

Maximum number of file handles to keep open when spilling read ends to disk. Set this number a little lower than the per-process maximum number of file that may be open. This number can be found by executing the 'ulimit -n' command on a Unix system.

int 8000 [ [ -∞ ∞ ] ]

--MAX_OPTICAL_DUPLICATE_SET_SIZE

This number is the maximum size of a set of duplicate reads for which we will attempt to determine which are optical duplicates. Please be aware that if you raise this value too high and do encounter a very large set of duplicate reads, it will severely affect the runtime of this tool. To completely disable this check, set the value to -1.

long 300000 [ [ -∞ ∞ ] ]

--MAX_RECORDS_IN_RAM

When writing files that need to be sorted, this will specify the number of records stored in RAM before spilling to disk. Increasing this number reduces the number of file handles needed to sort the file, and increases the amount of RAM needed.

Integer 500000 [ [ -∞ ∞ ] ]

--MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP / -MAX_SEQS

This option is obsolete. ReadEnds will always be spilled to disk.
If more than this many sequences in SAM file, don't spill to disk because there will not be enough file handles.

int 50000 [ [ -∞ ∞ ] ]

--METRICS_FILE / -M

File to write duplication metrics to

R File null

--MOLECULAR_IDENTIFIER_TAG

SAM tag to uniquely identify the molecule from which a read was derived. Use of this option requires that the BARCODE_TAG option be set to a non null value. Default null.

String null

--OPTICAL_DUPLICATE_PIXEL_DISTANCE

The maximum offset between two duplicate clusters in order to consider them optical duplicates. The default is appropriate for unpatterned versions of the Illumina platform. For the patterned flowcell models, 2500 is moreappropriate. For other platforms and models, users should experiment to find what works best.

int 100 [ [ -∞ ∞ ] ]

--OUTPUT / -O

The output file to write marked records to

R File null

--PROGRAM_GROUP_COMMAND_LINE / -PG_COMMAND

Value of CL tag of PG record to be created. If not supplied the command line will be detected automatically.

String null

--PROGRAM_GROUP_NAME / -PG_NAME

Value of PN tag of PG record to be created.

String SimpleMarkDuplicatesWithMateCigar

--PROGRAM_GROUP_VERSION / -PG_VERSION

Value of VN tag of PG record to be created. If not specified, the version will be detected automatically.

String null

--PROGRAM_RECORD_ID / -PG

The program record ID for the @PG record(s) created by this program. Set to null to disable PG record creation. This string may have a suffix appended to avoid collision with other program record IDs.

String MarkDuplicates

--QUIET

Whether to suppress job-summary info on System.err.

Boolean false

--READ_NAME_REGEX

MarkDuplicates can use the tile and cluster positions to estimate the rate of optical duplication in addition to the dominant source of duplication, PCR, to provide a more accurate estimation of library size. By default (with no READ_NAME_REGEX specified), MarkDuplicates will attempt to extract coordinates using a split on ':' (see Note below). Set READ_NAME_REGEX to 'null' to disable optical duplicate detection. Note that without optical duplicate counts, library size estimation will be less accurate. If the read name does not follow a standard Illumina colon-separation convention, but does contain tile and x,y coordinates, a regular expression can be specified to extract three variables: tile/region, x coordinate and y coordinate from a read name. The regular expression must contain three capture groups for the three variables, in order. It must match the entire read name. e.g. if field names were separated by semi-colon (';') this example regex could be specified (?:.*;)?([0-9]+)[^;]*;([0-9]+)[^;]*;([0-9]+)[^;]*$ Note that if no READ_NAME_REGEX is specified, the read name is split on ':'. For 5 element names, the 3rd, 4th and 5th elements are assumed to be tile, x and y values. For 7 element names (CASAVA 1.8), the 5th, 6th, and 7th elements are assumed to be tile, x and y values.

String

--READ_ONE_BARCODE_TAG

Read one barcode SAM tag (ex. BX for 10X Genomics)

String null

--READ_TWO_BARCODE_TAG

Read two barcode SAM tag (ex. BX for 10X Genomics)

String null

--REFERENCE_SEQUENCE / -R

Reference sequence file.

PicardHtsPath null

--REMOVE_DUPLICATES

If true do not write duplicates to the output file instead of writing them with appropriate flags set.

boolean false

--REMOVE_SEQUENCING_DUPLICATES

If true remove 'optical' duplicates and other duplicates that appear to have arisen from the sequencing process instead of the library preparation process, even if REMOVE_DUPLICATES is false. If REMOVE_DUPLICATES is true, all duplicates are removed and this option is ignored.

boolean false

--showHidden / -showHidden

display hidden arguments

boolean false

--SORTING_COLLECTION_SIZE_RATIO

This number, plus the maximum RAM available to the JVM, determine the memory footprint used by some of the sorting collections. If you are running out of memory, try reducing this number.

double 0.25 [ [ -∞ ∞ ] ]

--TAG_DUPLICATE_SET_MEMBERS

If a read appears in a duplicate set, add two tags. The first tag, DUPLICATE_SET_SIZE_TAG (DS), indicates the size of the duplicate set. The smallest possible DS value is 2 which occurs when two reads map to the same portion of the reference only one of which is marked as duplicate. The second tag, DUPLICATE_SET_INDEX_TAG (DI), represents a unique identifier for the duplicate set to which the record belongs. This identifier is the index-in-file of the representative read that was selected out of the duplicate set.

boolean false

--TAGGING_POLICY

Determines how duplicate types are recorded in the DT optional attribute.

The --TAGGING_POLICY argument is an enumerated type (DuplicateTaggingPolicy), which can have one of the following values:

DontTag
OpticalOnly
All

DuplicateTaggingPolicy DontTag

--TMP_DIR

One or more directories with space available to be used by this program for temporary storage of working files

List[File] []

--USE_JDK_DEFLATER / -use_jdk_deflater

Use the JDK Deflater instead of the Intel Deflater for writing compressed output

Boolean false

--USE_JDK_INFLATER / -use_jdk_inflater

Use the JDK Inflater instead of the Intel Inflater for reading compressed input

Boolean false

--VALIDATION_STRINGENCY

Validation stringency for all SAM files read by this program. Setting stringency to SILENT can improve performance when processing a BAM file in which variable-length data (read, qualities, tags) do not otherwise need to be decoded.

The --VALIDATION_STRINGENCY argument is an enumerated type (ValidationStringency), which can have one of the following values:

STRICT
LENIENT
SILENT

ValidationStringency STRICT

--VERBOSITY

Control verbosity of logging.

The --VERBOSITY argument is an enumerated type (LogLevel), which can have one of the following values:

ERROR
WARNING
INFO
DEBUG

LogLevel INFO

--version

display the version number for this tool

boolean false

Return to top

GATK version 4.6.2.0 built at Sun, 13 Apr 2025 13:21:43 -0400.

**EXPERIMENTAL** SimpleMarkDuplicatesWithMateCigar (Picard)

Category Read Data Manipulation

Overview

SimpleMarkDuplicatesWithMateCigar (Picard) specific arguments

Argument details

--ASSUME_SORT_ORDER / -ASO

--ASSUME_SORTED / -AS

--COMMENT / -CO

--DUPLICATE_SCORING_STRATEGY / -DS

--help / -h

--INPUT / -I

--MAX_FILE_HANDLES_FOR_READ_ENDS_MAP / -MAX_FILE_HANDLES

--MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP / -MAX_SEQS

--METRICS_FILE / -M

--OUTPUT / -O

--PROGRAM_GROUP_COMMAND_LINE / -PG_COMMAND

--PROGRAM_GROUP_NAME / -PG_NAME

--PROGRAM_GROUP_VERSION / -PG_VERSION

--PROGRAM_RECORD_ID / -PG

--REFERENCE_SEQUENCE / -R

--showHidden / -showHidden

--USE_JDK_DEFLATER / -use_jdk_deflater

--USE_JDK_INFLATER / -use_jdk_inflater

EXPERIMENTAL SimpleMarkDuplicatesWithMateCigar (Picard)