NAME
uniq - report or filter out repeated lines in a file
SYNOPSIS
uniq [-c|-d|-u][-f fields][-s char][input_file [output_file]]
DESCRIPTION
The uniq utility shall read an input file comparing adjacent lines, and
write one copy of each input line on the output. The second and
succeeding copies of repeated adjacent input lines shall not be
written.
Repeated lines in the input shall not be detected if they are not
adjacent.
OPTIONS
The uniq utility shall conform to the Base Definitions volume of
IEEE Std 1003.1-2001, Section 12.2, Utility Syntax Guidelines.
The following options shall be supported:
-c Precede each output line with a count of the number of times the
line occurred in the input.
-d Suppress the writing of lines that are not repeated in the
input.
-f fields
Ignore the first fields fields on each input line when doing
comparisons, where fields is a positive decimal integer. A field
is the maximal string matched by the basic regular expression:
[[:blank:]]*[^[:blank:]]*
If the fields option-argument specifies more fields than appear on an
input line, a null string shall be used for comparison.
-s chars
Ignore the first chars characters when doing comparisons, where
chars shall be a positive decimal integer. If specified in
conjunction with the -f option, the first chars characters after
the first fields fields shall be ignored. If the chars option-
argument specifies more characters than remain on an input line,
a null string shall be used for comparison.
-u Suppress the writing of lines that are repeated in the input.
OPERANDS
The following operands shall be supported:
input_file
A pathname of the input file. If the input_file operand is not
specified, or if the input_file is ’-’ , the standard input
shall be used.
output_file
A pathname of the output file. If the output_file operand is not
specified, the standard output shall be used. The results are
unspecified if the file named by output_file is the file named
by input_file.
STDIN
The standard input shall be used only if no input_file operand is
specified or if input_file is ’-’ . See the INPUT FILES section.
INPUT FILES
The input file shall be a text file.
ENVIRONMENT VARIABLES
The following environment variables shall affect the execution of uniq:
LANG Provide a default value for the internationalization variables
that are unset or null. (See the Base Definitions volume of
IEEE Std 1003.1-2001, Section 8.2, Internationalization
Variables for the precedence of internationalization variables
used to determine the values of locale categories.)
LC_ALL If set to a non-empty string value, override the values of all
the other internationalization variables.
LC_COLLATE
Determine the locale for ordering rules.
LC_CTYPE
Determine the locale for the interpretation of sequences of
bytes of text data as characters (for example, single-byte as
opposed to multi-byte characters in arguments and input files)
and which characters constitute a <blank> in the current locale.
LC_MESSAGES
Determine the locale that should be used to affect the format
and contents of diagnostic messages written to standard error.
NLSPATH
Determine the location of message catalogs for the processing of
LC_MESSAGES .
ASYNCHRONOUS EVENTS
Default.
STDOUT
The standard output shall be used only if no output_file operand is
specified. See the OUTPUT FILES section.
STDERR
The standard error shall be used only for diagnostic messages.
OUTPUT FILES
If the -c option is specified, the output file shall be empty or each
line shall be of the form:
"%d %s", <number of duplicates>, <line>
otherwise, the output file shall be empty or each line shall be of the
form:
"%s", <line>
EXTENDED DESCRIPTION
None.
EXIT STATUS
The following exit values shall be returned:
0 The utility executed successfully.
>0 An error occurred.
CONSEQUENCES OF ERRORS
Default.
The following sections are informative.
APPLICATION USAGE
The sort utility can be used to cause repeated lines to be adjacent in
the input file.
EXAMPLES
The following input file data (but flushed left) was used for a test
series on uniq:
#01 foo0 bar0 foo1 bar1
#02 bar0 foo1 bar1 foo1
#03 foo0 bar0 foo1 bar1
#04
#05 foo0 bar0 foo1 bar1
#06 foo0 bar0 foo1 bar1
#07 bar0 foo1 bar1 foo0
What follows is a series of test invocations of the uniq utility that
use a mixture of uniq options against the input file data. These tests
verify the meaning of adjacent. The uniq utility views the input data
as a sequence of strings delimited by ’\n’ . Accordingly, for the
fieldsth member of the sequence, uniq interprets unique or repeated
adjacent lines strictly relative to the fields+1th member.
1. This first example tests the line counting option, comparing each
line of the input file data starting from the second field:
uniq -c -f 1 uniq_0I.t
1 #01 foo0 bar0 foo1 bar1
1 #02 bar0 foo1 bar1 foo0
1 #03 foo0 bar0 foo1 bar1
1 #04
2 #05 foo0 bar0 foo1 bar1
1 #07 bar0 foo1 bar1 foo0
The number ’2’ , prefixing the fifth line of output, signifies that the
uniq utility detected a pair of repeated lines. Given the input data,
this can only be true when uniq is run using the -f 1 option (which
shall cause uniq to ignore the first field on each input line).
2. The second example tests the option to suppress unique lines,
comparing each line of the input file data starting from the second
field:
uniq -d -f 1 uniq_0I.t
#05 foo0 bar0 foo1 bar1
3. This test suppresses repeated lines, comparing each line of the
input file data starting from the second field:
uniq -u -f 1 uniq_0I.t
#01 foo0 bar0 foo1 bar1
#02 bar0 foo1 bar1 foo1
#03 foo0 bar0 foo1 bar1
#04
#07 bar0 foo1 bar1 foo0
4. This suppresses unique lines, comparing each line of the input file
data starting from the third character:
uniq -d -s 2 uniq_0I.t
In the last example, the uniq utility found no input matching the above
criteria.
RATIONALE
Some historical implementations have limited lines to be 1080 bytes in
length, which does not meet the implied {LINE_MAX} limit.
FUTURE DIRECTIONS
None.
SEE ALSO
comm , sort
COPYRIGHT
Portions of this text are reprinted and reproduced in electronic form
from IEEE Std 1003.1, 2003 Edition, Standard for Information Technology
-- Portable Operating System Interface (POSIX), The Open Group Base
Specifications Issue 6, Copyright (C) 2001-2003 by the Institute of
Electrical and Electronics Engineers, Inc and The Open Group. In the
event of any discrepancy between this version and the original IEEE and
The Open Group Standard, the original IEEE and The Open Group Standard
is the referee document. The original Standard can be obtained online
at http://www.opengroup.org/unix/online.html .