Introduction to Unix Command-line

Part 2

Natalie Gill

Gladstone Bioinformatics Core

Press the ? key for tips on navigating these slides

Introductions

Natalie Gill Bioinformatician II

Setup

Run the following commands if you did not attend part 1:

mkdir unix_workshop

cd unix_workshop

curl -L -o unix_workshop.tar.gz 'https://www.dropbox.com/scl/fi/tdzpoivf7mienlenunqhf/unix_workshop.tar.gz?rlkey=6bfxnqgc5n4lgc9mc80ld75z4&dl=0'

tar -xzf unix_workshop.tar.gz

cd unix_workshop

curl -o part_2/homo_sapiens.refseq.tsv.gz https://ftp.ensembl.org/pub/current_tsv/homo_sapiens/Homo_sapiens.GRCh38.115.refseq.tsv.gz

File Compression

Command-line tools for compression

• Compression reduces the size of a file
• gzip : compresses a file and replaces it with a compressed version (.gz)
• tar : create and manipulate archive files

Archive: a single file that contains one or more files and/or folders that have been compressed

gzip/gunzip: compress/uncompress a file

gunzip part_2/homo_sapiens.refseq.tsv.gz
du -h part_2/homo_sapiens.refseq.tsv

 39M    part_2/homo_sapiens.refseq.tsv

• The uncompressed file is 33 megabytes

gzip part_2/homo_sapiens.refseq.tsv
du -h part_2/homo_sapiens.refseq.tsv.gz

3.7M    part_2/homo_sapiens.refseq.tsv.gz

• Compressing it makes it a 10th of the size

Note

• The magnitude of the compression depends on type of data
• The units for file sizes are not the same across all systems
  • Some systems define a kilobyte as 1000 bytes, while others define it as 1024 bytes

tar: compressing folders into archives

• Does not provide compression on its own, it uses gzip to create compressed archive files

tar -czf part_1.tar.gz part_1
ls -l

total 8
drwx---r--@ 4 nelphick  staff  128 Nov 10 15:00 part_1
-rw-r--r--  1 nelphick  staff  801 Nov 10 15:00 part_1.tar.gz
drwxr-xr-x@ 4 nelphick  staff  128 Nov 10 15:00 part_2

• -c: create a new archive
• -f: specify the name of the archive file
• -z: compress the archive with gzip

Unarchiving

• We did this in part 1 to unarchive the workshop folders

tar -xzf part_1.tar.gz

• -x: extract an archive
• -z: uncompress the archive with gzip
• -f: specify the name of the archive file

gunzip -c: cat compressed files

• To avoid uncompressing a large file just to read its contents, we can use gunzip -c
• This will output the the file to the terminal

gunzip -c part_2/homo_sapiens.refseq.tsv.gz | head

gene_stable_id  transcript_stable_id    protein_stable_id   xref    db_name info_type   source_identity xref_identity   linkage_type
ENSG00000142611 ENST00000378391 ENSP00000367643 NP_955533   RefSeq_peptide  DIRECT  100 100 -
ENSG00000142611 ENST00000378391 ENSP00000367643 NM_199454   RefSeq_mRNA DIRECT  99  62  -
ENSG00000142611 ENST00000270722 ENSP00000270722 NP_071397   RefSeq_peptide  DIRECT  100 100 -
ENSG00000142611 ENST00000270722 ENSP00000270722 NM_022114   RefSeq_mRNA DIRECT  100 100 -
ENSG00000232596 ENST00000420522 -   NR_147025   RefSeq_ncRNA    DIRECT  100 66  -
ENSG00000231510 ENST00000443270 -   NR_183671   RefSeq_ncRNA    DIRECT  100 64  -
ENSG00000149527 ENST00000449969 ENSP00000397289 NP_001289941    RefSeq_peptide  DIRECT  100 100 -
ENSG00000149527 ENST00000449969 ENSP00000397289 NM_001303012    RefSeq_mRNA DIRECT  99  99  -
ENSG00000149527 ENST00000449969 ENSP00000397289 XM_047435038    RefSeq_mRNA_predicted   DIRECT  96  92  -

System Variables

What are system variables?

• Special variables that contain information about the system’s configuration and state
• Used by the OS and programs to change their behavior based on the system’s state

Example:

echo $HOME

/Users/nelphick

Common System Variables

• $PWD : The working directory
• $HOME : The current user’s home directory
• $PS1 : the shell prompt string
• $TMPDIR : location of temporary files

These can change depending on the specific OS or program, TMPDIR can also be TEMP, TEMPDIR and TMP.

PATH: locations of executable files

• When you enter a command, the OS searches the directories in the $PATH to find its associated executable file

echo $PATH

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/usr/local/mysql/bin

• The OS will check these directories in the order they appear and use the first executable it finds

export: set system variables

• Useful for setting variables you want to be used across programs
• You can add new software to your $PATH like this:

export PATH="/path/to/new/software:$PATH"

• This will modify the $PATH for the current terminal session

Modifying the PATH for all future terminal sessions

• Add the export line to your ~/.bashrc or ~/.zshrc
• Proceed with caution
  • Make backups of these and read this guide
  • Changing the $PATH incorrectly can break system functionality

which: locate the executable associated with a command

• This command shows the location of the executable that the OS finds

which ls

/bin/ls

• Useful to check if there are multiple versions of a software installed

Shell Scripting

What is a script?

• Scripts are executable files for reusing code
• By convention scripts end in .sh
• This first line of the script is called the shebang

nano part_2/example_script.sh

#!/bin/bash

• The text that follows #! tells the OS where the interpreter is

which bash

/bin/bash

chmod: making a script executable

• By default, files are not executable

ls -l part_2/example_script.sh

-rw-r--r--  1 nelphick  staff  287 Nov 10 15:00 part_2/example_script.sh

• We can set the execute bit like this

chmod u+x part_2/example_script.sh
ls -l part_2/example_script.sh

-rwxr--r--  1 nelphick  staff  287 Nov 10 15:00 part_2/example_script.sh

Example

#!/bin/bash

# This is a comment. Comments are ignored by the shell.

# $1 is the first argument passed to the script
echo "Counting the genes in $1"

# count the unique genes in the file
u_genes=$(gunzip -c $1 | cut -f 1 | sort -u | wc -l)

echo "There are $u_genes unique genes in $1"

Let’s run it

./part_2/example_script.sh part_2/homo_sapiens.refseq.tsv.gz

Counting the genes in part_2/homo_sapiens.refseq.tsv.gz
There are    36406 unique genes in part_2/homo_sapiens.refseq.tsv.gz

Loops

• Useful for iterating over lines of a file or lists

for i in {1..3}
do

echo $i

done

1
2
3

While loops

count=0

while [ $count -lt 5 ]        # loop while count is less than 5
do
    echo $count
    count=$((count+1))
done

0
1
2
3
4

If statements

x=5

if [ $x -gt 10 ]                      # check if x is greater than 10
then
    echo "x is greater than 10"
else
    echo "x is not greater than 10"
fi                                    # end if statement

x is not greater than 10

Other Useful Commands

sed : stream editor

• Parses and transforms text, using a compact programming language
• It reads and modifies text line by line from a file or input stream
• Supports regular expressions
• Useful for replacing text

Example:

sed 's/search_string/replace_string/g' input.txt > output.txt

ssh : secure shell - conect to remote server

• Logging in to a remote server
• Remote desktop for the terminal

ssh username@remote

• The username would be your user on the remote server and remote is the hostname or IP address of the remote server or computer

scp : secure copy

• Copy files from a remote server or computer

scp [options] [source] [destination]

• Copy from local to remote

scp /path/to/local/file.txt username@remote:/path/to/remote/directory/

• Copy from remote to local

scp username@remote:/path/to/file.txt /path/to/local/directory/

• -r : copy a whole folder

AWK

awk : processing structured data

• A small programming language that is designed to work with structured data
• Has more complicated syntax but is faster at processing large files
• Designed to read a file or input stream line by line
• Operates on records (lines) and fields (columns)

Basic command:

awk options 'pattern {action}' input_file

Example : Sum the first 2 columns of a file

 awk -F '\t' '{print $1+$2}' part_1/list_numbers.tsv

4
15
17

• -F : provides the field separator
• $1,$2 : the first and second fields

Example : Find the average of a column

• For this example we only want the average if the 5th column equals “RefSeq_mRNA”

gunzip -c part_2/homo_sapiens.refseq.tsv.gz | \
awk -F '\t' '$5 == "RefSeq_mRNA" {sum += $7; count++} \
END {print sum / count}'

60.5871

Resources for learning AWK and sed

• The GNU AWK manual
• AWK Tutorial by Bruce Barnett
• Sed Tutorial by Bruce Barnett

End of Part 2

Additional learning materials

• Software carpentry provides a self paced course:
  • The Unix Shell
• Free online books:
  • The Unix Workbench
  • The Linux Command Line

Survey

• Please take some time to fill out the workshop survey: https://www.surveymonkey.com/r/DY7K5ZY

Upcoming Data Science Training Program Workshops

Introduction to RNA-Seq Analysis February 13-February 14, 2025 1:00-4:00pm PST

Intermediate RNA-Seq Analysis Using R February 20, 2025 9:00am-12:00pm PST

Introduction to Statistics, Experimental Design and Hypothesis Testing February 24-February 25, 2025 1:00-3:00pm PST

Complete Schedule