Strings or File Manipulation

root's picture

How to replace tab with newline:


gz@linux:~$ cat list
str1    str2    str3    str4    str5    str6    str7    str8
gz@linux:~$ cat list | tr '\t' '\n' >list2
gz@linux:~$ cat list2
str1
str2
str3
str4
str5
str6
str7
str8

How to get different parts of a string, for example with path and file name:


gz:tips# foo=/tmp/dir/filename.tar.gz
gz:tips# echo ${foo%/*}
/tmp/dir
gz:tips# echo ${foo##*/}
filename.tar.gz
gz:tips# echo ${foo%%.*}
/tmp/dir/filename
gz:tips# echo ${foo%.tar.gz}
/tmp/dir/filename
gz:tips# echo ${foo#*.}
tar.gz

How to find and rename recursively. First example is for files and second for directories:


gz:tips# for FILE in $(find . -type f -name "*foo*"); do NEW=`echo $FILE | sed -e 's/foo/bar/'`; mv "$FILE" "$NEW"; done
gz:tips# for DIR in $(find . -type d -name "*foo*"); do NEW=`echo $DIR | sed -e 's/foo/bar/'`; mv "$DIR" "$NEW"; done

How to display with awk from a desired column to the end. In the examples below we will display the free space for all vgs in AIX system:


root@gzaix:~# for i in $(lsvg -o); do printf "$i ==>\t";lsvg $i| grep -i free| awk '{ print substr($0, index($0,$4)) }'; done
datavg06 ==>    FREE PPs:       3 (384 megabytes)
datavg ==>      FREE PPs:       489 (62592 megabytes)
datavg01 ==>    FREE PPs:       549 (70272 megabytes)
datavg04 ==>    FREE PPs:       4 (512 megabytes)
datavg03 ==>    FREE PPs:       109 (13952 megabytes)
datavg02 ==>    FREE PPs:       159 (20352 megabytes)
datavg07 ==>    FREE PPs:       4 (256 megabytes)
datavg08 ==>    FREE PPs:       125 (8000 megabytes)
datavg05 ==>    FREE PPs:       143 (18304 megabytes)
rootvg ==>      FREE PPs:       13 (1664 megabytes)

How to search for a string in multiple files on linux:


root@linux:~# find . -type f -print0 | xargs -0 grep -i book

How to search and replace a string in multiple files on linux:


root@linux:~# find . -type f -print0 | xargs -0 sed -i 's/book/BOOK/g'

How to display the last column from an output:


root@aix:/ # lssrc -s nimsh
Subsystem         Group            PID          Status
 nimsh            nimclient                     inoperative
root@aix:/ # lssrc -s nimsh | awk '{print $NF}'
Status
inoperative

How to extract only numbers from a string with bash:


root@linux:~# TEST="1238ksds98amnsa62knb2gi2v23v"
root@linux:~# echo ${TEST//[!0-9]/}
123898622223
root@linux:~# echo ${TEST//[^0-9]/}
123898622223

How to extract only numbers from a string with sed:


root@linux:~# echo 1238ksds98amnsa62knb2gi2v23v | sed 's/[^0-9]//g'
123898622223

Passing variable to awk:


root@linux:~# for i in 21684 26667 20991; do ps -ef | awk -v ss="$i" '$2 ~ ss { print }'; done
root     21684 10157  0 Nov22 tty1     00:00:00 -bash
root     26667     1  0 Nov22 ?        00:00:00 dhclient3 eth2
vbox     20991 20983  0 Oct24 ?        00:00:00 gnome-session

another example:
florian@florian:~$ for i in 1 2 3 4 5; do echo $i | awk -v ss="awkvar$i" '{print ss" from "$0}'; done
awkvar1 from 1
awkvar2 from 2
awkvar3 from 3
awkvar4 from 4
awkvar5 from 5

How to count a string's characters:


root@linux:~$ STRING="12345"
root@linux:~$ echo "Chars: ${#STRING}"
Chars: 5

How to remove blank lines from a file using grep and awk:


root@linux:~# cat /tmp/test.txt
1234567890

abcde
fgh

ijklmn

root@linux:~# grep -v ^$ /tmp/test.txt
1234567890
abcde
fgh
ijklmn

root@linux:~# awk 'NF != 0 {print} ' /tmp/test.txt
1234567890
abcde
fgh
ijklmn

How to add a line after some string


(to add "BOOOOOOO!" if the line contains "abc" (first example) or if the line is "fgh" (second example)):
root@linux:~# awk '{print ; if ($0 ~ "abc") print "BOOOOOOO!"}' /tmp/test.txt
1234567890

abcde
BOOOOOOO!
fgh

ijklmn

root@linux:~# awk '{print ; if ($0 == "fgh") print "BOOOOOOO!"}' /tmp/test.txt
1234567890

abcde
fgh
BOOOOOOO!

ijklmn

How to transform a line from lowercase to uppercase with sed:


root@linux:~# sed 's/abcde/ABCDE/g' /tmp/test.txt
1234567890

ABCDE
fgh

ijklmn

How to split a string after the second delimiter with sed:


gz@linux:~$ echo "1:2:3:4:5:6:7:8:9" | sed "s/\([^:]*:[^:]*\):/\1\n/g"
1:2
3:4
5:6
7:8
9

or

gz@linux:~$ echo "1:2:3:4:5:6:7:8:9" | sed "s/[^:]*:[^:]*:/& /g" | sed "s/: /\n/g" 
1:2
3:4
5:6
7:8
9

or in ksh

gz@aix~# echo "1:2:3:4:5:6:7:8:9" | sed "s/\([^:]*:[^:]*\):/\1 /g" | tr " " "\n"
1:2
3:4
5:6
7:8
9

How to replace the 3rd same (repetitive) character with another:


srt@gz$ echo "00:21:26:400 --> 00:21:28:400" | sed "s/\([^00:]*:[^:]*:[^:]*\):/\1\,/g"
00:21:26,400 --> 00:21:28,400

How to convert uppercase to lowercase characters in a file with TR:


florian@florian:~$ cat b
ASLDJSLDJSLKJD
SLKJDLKSJDLK
SOIDUSOIDUOIS
SDOSDSJLD
florian@florian:~$ cat b | tr '[:upper:]' '[:lower:]' 
asldjsldjslkjd
slkjdlksjdlk
soidusoiduois
sdosdsjld
florian@florian:~$ cat b | tr '[A-Z]' '[a-z]'  
asldjsldjslkjd
slkjdlksjdlk
soidusoiduois
sdosdsjld

How to rename filenames in different order (for example files like "This is file 001.jpg" to "001 file is This.jpg"):


1. Put them to a file like ls.txt.
2. Run this script in the same directory.
#!/bin/bash
#
cat ls.txt | sed s/.jpg//g | while read line;
  do
	arr=($line)
	mv "${arr[0]} ${arr[1]} ${arr[2]} ${arr[3]}.jpg" "${arr[3]} ${arr[2]} ${arr[1]} ${arr[0]}.jpg"
done

How to calculate and display in AIX disk space in GB:


Total from bootinfo:
root@aix:~# for i in $(lspv|awk '{print $1}' | xargs); do bootinfo -s $i; done | awk '{ s = s+$1/1024 } END { printf "Total disk space: %dGB\n",s }'
Total disk space: 4125GB

Total from lsvg info:
root@aix:~# for i in $(lsvg -o); do lsvg $i| grep -i total| awk '{ print substr($0, index($0,$7)) }'; done | sed "s/[^0-9]//g" | awk '{ s = s+$1/1024 } END { printf "Total: %dGB\n",s }'
Total: 559GB

Total used and free from lsvg info:
root@aix:~# for i in $(lsvg -o); do lsvg $i| grep -i free| awk '{ print substr($0, index($0,$7)) }'; done | sed "s/[^0-9]//g" | awk '{ s = s+$1/1024 } END { printf "Free: %dGB\n",s }'
Free: 20GB
root@aix:~# for i in $(lsvg -o); do lsvg $i| grep -i used| awk '{ print substr($0, index($0,$6)) }'; done | sed "s/[^0-9]//g" | awk '{ s = s+$1/1024 } END { printf "Used: %dGB\n",s }'
Used: 538GB

How to search for files which contains a specific text within a tar archive:

root@aix:~# for f in `ls archives*.tar.Z `; do echo $f; zcat $f | tar -tvf - | grep WHAT_TO_SEARCH_FOR; done;
archives_20141007.tar.Z
archives_20141006.tar.Z
archives_20141009.tar.Z
-rw-rw-r-- 35000 3500    3120 Oct 06 15:00:05 2014 ./00/THE_FILE_INSIDE_ARCHIVE_WHICH_CONTAINS_WHAT_YOU_ARE_SEARCHIG_FOR

How to display the archives which have files with a specific text inside:


root@aix:~# ls archives*.tar.Z
archives_20141007.tar.Z
archives_20141006.tar.Z
archives_20141009.tar.Z
root@aix:~# ./test.sh "archives*.tar.Z" WHAT_TO_SEARCH_FOR
archives_20141009.tar.Z

The script:
root@aix:~# cat test.sh
STRING=$2
FILE=$1

if [ -z  "$STRING" ]; then
        echo "What are you looking for?"
        exit 99
fi

for f in $(ls ${FILE});
        do
        zcat $f | tar -tvf - | grep ${STRING} >/dev/null 2>&1;
        RC=$?
        if [ $RC == 0 ]; then
        echo $f
        fi
done

How to remove all whitespace (including tabs) from the beginning of the line to the first word:


# echo " Test, BABY!" | sed -e 's/^[ \t]*//'

How to change the extension on files in bulk with rename command:


# find . -name "*.php5" -exec rename -f 's/\.php5/.php/' {} \;

How to remove the extension on files in bulk with rename command:


# find . -name "*.php5" -exec rename -f 's/\.php5/.php/' {} \;

How to print field number X only on lines matching something, otherwise print field number Y with awk:


In this example below, as you can see, I am using the same awk command to print the field I want: virtual disk path
vbox@core:~$ vbm showvminfo irc | grep ^SATA | grep -v Empty
SATA (0, 0): /home/vbox/.VirtualBox/dynamic/irc/irc.vdi (UUID: 6e9e97ff-f312-45ca-8b3e-fa18b9032dd0)
vbox@core:~$ vbm showvminfo irc | grep ^SATA | grep -v Empty | awk '/Controller/{print $5;next}{print $4}'
/home/vbox/.VirtualBox/dynamic/irc/irc.vdi

vbox@core:~$ vbm showvminfo egg | grep ^SATA | grep -v Empty
SATA Controller (0, 0): /home/vbox/.VirtualBox/dynamic/egg/egg.vdi (UUID: 6977dea1-e224-4d04-8649-b688baed2430)
SATA Controller (1, 0): /home/vbox/.VirtualBox/dynamic/egg/egg-swap.vdi (UUID: 5b039f89-ed53-4ce8-809b-9aeb84273b36)
vbox@core:~$ vbm showvminfo egg | grep ^SATA | grep -v Empty | awk '/Controller/{print $5;next}{print $4}'
/home/vbox/.VirtualBox/dynamic/egg/egg.vdi
/home/vbox/.VirtualBox/dynamic/egg/egg-swap.vdi

Thou shalt not steal!

If you want to use this information on your own website, please remember: by doing copy/paste entirely it is always stealing and you should be ashamed of yourself!
Have at least the decency to create your own text and comments and run the commands on your own servers and provide your output, not what I did!

Or at least link back to this website.