Shell Scripting – Expansion
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
In the last blog post, we have discussed functions in shell.
https://cloudaffaire.com/shell-scripting-function/
In this blog post, we will discuss shell expansion. When we type something on the shell and hit enter, shell 1st checks if the line or block of code is commented. If the line or block is not commented, shell divides it into words and operators, obeying the quoting rules. These words are separated by metacharacters. After the command has been split into words, these words are expanded or resolved.
Shell Scripting – Expansion
Brace expansion:
Brace expansion is a mechanism by which arbitrary strings may be generated. Patterns to be brace-expanded take the form of an optional PREAMBLE, followed by a series of comma-separated strings between a pair of braces, followed by an optional POSTSCRIPT. The preamble is prefixed to each string contained within the braces, and the postscript is then appended to each resulting string, expanding left to right.
Brace expansions may be nested. The results of each expanded string are not sorted; left to right order is preserved. Brace expansion is performed before any other expansions, and any characters special to other expansions are preserved in the result. It is strictly textual. Bash does not apply any syntactic interpretation to the context of the expansion or the text between the braces. To avoid conflicts with parameter expansion, the string “${” is not considered eligible for brace expansion. A correctly-formed brace expansion must contain unquoted opening and closing braces, and at least one unquoted comma. Any incorrectly formed brace expansion is left unchanged.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
################################# ## Shell Scripting | Expansion ## ################################# ## Prerequisites: One Unix/Linux/POSIX-compliant operating system with bash shell ## --------------- ## Brace expansion ## --------------- ## Syntax ## {string1,string2,..,stringN} ## {range_start..range_end} ## {range_start..range_end..increment/decrement} ## ## {string or range} ## ## Comma-separated lists echo {hello,world} ## returns hello world echo {"hello","world"} ## returns hello world echo {'hello','world'} ## returns hello world echo {"hello ","welcome "}{"debjeet","cloudaffaire"} ## returns hello debjeet hello cloudaffaire welcome debjeet welcome cloudaffaire ## Range echo {1..5} ## returns 1 2 3 4 5 echo {a..e} ## returns a b c d e echo {1..5}{a..e} ## retuns 1a 1b 1c 1d 1e 2a 2b 2c 2d 2e 3a 3b 3c 3d 3e 4a 4b 4c 4d 4e 5a 5b 5c 5d 5e echo {1..10..2} ## returns 1 3 5 7 9 echo {10..1..3} ## returns 10 7 4 1 ## Preamble echo "hello "{debjeet,cloudaffaire} ## returns hello debjeet hello cloudaffaire echo c{at,ow,rab} ## returns cat cow crab ## Postscript echo {AWS,Azure,GCP}" cloud" ## returns AWS cloud Azure cloud GCP cloud echo {inter,intra,extra}net ## returns internet intranet extranet ## Preamble and Postscript echo "hello "{deb,alex}" welcome" ## returns hello deb welcome hello alex welcome ## With other commands touch myfile{1..5}.txt ## creates files with name myfile1.txt myfile2.txt etc. mkdir mydir{1..5} ## create directory with name mydir1 mydir2 etc. rm -r my* |
Tilde expansion:
If a word begins with an unquoted tilde character (“~”), all of the characters up to the first unquoted slash (or all characters, if there is no unquoted slash) are considered a tilde-prefix. If none of the characters in the tilde-prefix are quoted, the characters in the tilde-prefix following the tilde are treated as a possible login name. If this login name is the null string, the tilde is replaced with the value of the HOME shell variable. If HOME is unset, the home directory of the user executing the shell is substituted instead. Otherwise, the tilde-prefix is replaced with the home directory associated with the specified login name.
If the tilde-prefix is “~+”, the value of the shell variable PWD replaces the tilde-prefix. If the tilde-prefix is “~-“, the value of the shell variable OLDPWD, if it is set, is substituted. If the characters following the tilde in the tilde-prefix consist of a number N, optionally prefixed by a “+” or a “-“, the tilde-prefix is replaced with the corresponding element from the directory stack, as it would be displayed by the dirs built-in invoked with the characters following tilde in the tilde-prefix as an argument. If the tilde-prefix, without the tilde, consists of a number without a leading “+” or “-“, “+” is assumed.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
## --------------- ## Tilde expansion ## --------------- ## Syntax ## ~tilde-prefix/ ## tilde-prefix: ## 1. ~ ==> Home directories ## 2. ~ with + or - ==> Current or Previous Working Directory ## 3. ~ with + or - with an integer N ==> Directories from directory stack ## ~ represents your home directory echo ~ ## returns /home/cloudaffaire echo $HOME ## returns /home/cloudaffaire export HOME=/sbin ## changed the home directory echo ~ ## returns /sbin unset HOME echo ~ ## returns /home/cloudaffaire again ## You can reference your home directory using ~ mkdir -p ~/mydir/subdir ## create a new directory in your home directory echo "hello world" > ~/mydir/subdir/hello cat ~/mydir/subdir/hello ## ~+ represents your current working directory echo ~+ ## same data as PWD variable ## ~+ represents your previous working directory echo ~- ## same data as OLDPWD variable ## ~+N and ~-N represents directories from directory stack ## Create some directory and push them in directory stck mkdir mydir1 pushd mydir1 mkdir mydir2 pushd mydir2 mkdir mydir3 pushd mydir3 ## Current directory stack dirs -v ## Returns ## 0 ~/mydir1/mydir2/mydir3 ## 1 ~/mydir1/mydir2 ## 2 ~/mydir1 ## 3 ~ pwd ## returns /home/cloudaffaire/mydir1/mydir2/mydir3 cd ~+2 pwd ## returns /home/cloudaffaire/mydir1 cd ~-2 pwd ## returns /home/cloudaffaire/mydir1/mydir2 cd ~ pwd ## returns /home/cloudaffaire rm -r my* |
Shell Parameter Expansion:
The $ character introduces parameter expansion, command substitution, or arithmetic expansion. The parameter name or symbol to be expanded may be enclosed in braces, which are optional but serve to protect the variable to be expanded from characters immediately following it which could be interpreted as part of the name. When braces are used, the matching ending brace is the first ‘}’ not escaped by a backslash or within a quoted string, and not within an embedded arithmetic expansion, command substitution, or parameter expansion.
The basic form of parameter expansion is ${parameter}. The value of parameter is substituted. The parameter is a shell parameter as described above (see Shell Parameters) or an array reference (see Arrays). The braces are required when parameter is a positional parameter with more than one digit, or when parameter is followed by a character that is not to be interpreted as part of its name.
If the first character of parameter is an exclamation point (!), and parameter is not a nameref, it introduces a level of indirection. Bash uses the value formed by expanding the rest of parameter as the new parameter; this is then expanded and that value is used in the rest of the expansion, rather than the expansion of the original parameter. This is known as indirect expansion. The value is subject to tilde expansion, parameter expansion, command substitution, and arithmetic expansion. If parameter is a nameref, this expands to the name of the variable referenced by parameter instead of performing the complete indirect expansion. The exceptions to this are the expansions of ${!prefix*} and ${!name[@]} described below. The exclamation point must immediately follow the left brace in order to introduce indirection.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 |
## ------------------------- ## Shell Parameter Expansion ## ------------------------- ## If parameter is unset or null, the expansion of word is substituted. Otherwise, the value of parameter is substituted. ## Syntax ## ${parameter:-word} name=${var1:-"one"} ## since var1 not set, name takes the default value (one) provided echo $name ## returns one var1="two" ## set var1 to two name=${var1:-"one"} ## since var1 is set, name takes the value of var1 (two) instead of default value (one) echo $name ## returns two #also works with positional parameter myfun() { name=${1:-"debjeet"} ## 1 represent 1st parameter that is passed to the function echo "hello $name" } myfun ## returns hello debjeet myfun cloud ## returns hello cloud ## If parameter is unset or null, the expansion of word is assigned to parameter. The value of parameter is then substituted. ## Positional parameters and special parameters may not be assigned to in this way. ## Syntax ## ${parameter:=word} name=${var1:-"one"} ## since var1 not set, name takes the default value (one) provided echo $name ## returns one var1="two" ## set var1 to two name=${var1:-"one"} ## since var1 is set, name takes the value of var1 (two) instead of default value (one) echo $name ## returns two # but below will not work myfun() { name=${1:="debjeet"} ## $1: cannot assign in this way echo "hello $name" } myfun ## returns $1: cannot assign in this way myfun cloud ## returns hello cloud ## If parameter is null or unset, the expansion of word is written to the standard error and the shell, ## if it is not interactive, exits. Otherwise, the value of parameter is substituted. ## Syntax ## ${parameter:?word} unset var1 echo ${var1:?"var1 not set or null"} ## since var1 not set, returns var1 not set or null var1="debjeet" ## set var1 echo ${var1:?"var1 not set or null"} ## since var1 is set to debjeet, returns debjeet unset var1 ## If parameter is null or unset, nothing is substituted, otherwise the expansion of word is substituted. ## Syntax ## ${parameter:+word} echo ${var1:+"cloudaffaire"} ## since var1 not set, returns empty var1="debjeet" ## set var1 echo ${var1:+"cloudaffaire"} ## since var1 is set, returns cloudaffaire ## substring expansion expands up to length characters of the value of parameter starting at the character specified by offset. ## If parameter is ‘@’, an indexed array subscripted by ‘@’ or ‘*’, or an associative array name, the results differ as ## described below. If length is omitted, it expands to the substring of the value of parameter starting at the character ## specified by offset and extending to the end of the value. length and offset are arithmetic expressions ## If offset evaluates to a number less than zero, the value is used as an offset in characters from the end of the value ## of parameter. If length evaluates to a number less than zero, it is interpreted as an offset in characters from the end ## of the value of parameter rather than a number of characters, and the expansion is the characters between offset and that ## result. Note that a negative offset must be separated from the colon by at least one space to avoid being confused with ## the ‘:-’ expansion. ## Syntax ## ${parameter:offset:length} var1=123456789 ## declare an variable echo ${var1:5} ## returns 6789 echo ${var1:2:3} ## returns 345 echo ${var1:4:-3} ## returns 56 echo ${var1: -2} ## returns 89 echo ${var1: -5:-2} ## returns 567 array1=(1 2 3 4 5 6 7 8 9) ## declare an array echo ${array1[*]:5} ## returns 6 7 8 9 echo ${array1[*]:2:3} ## returns 3 4 5 echo ${array1[*]: -3} ## returns 7 8 9 ## If parameter is ‘@’, the result is length positional parameters beginning at offset. A negative offset is taken relative ## to one greater than the greatest positional parameter, so an offset of -1 evaluates to the last positional parameter. ## It is an expansion error if length evaluates to a number less than zero. set -- 1 2 3 4 5 6 7 8 9 echo ${@:3} ## returns 3 4 5 6 7 8 9 echo ${@:3:3} ## returns 3 4 5 echo ${@:3:-2} ## -2: substring expression < 0 echo ${@: -3:3} ## returns 7 8 9 echo ${@:0} ## returns -bash 1 2 3 4 5 6 7 8 9 ## Expands to the names of variables whose names begin with prefix, separated by the first character of the IFS special variable. ## When ‘@’ is used and the expansion appears within double quotes, each variable name expands to a separate word. ## Syntax ## ${!prefix*} ## ${!prefix@} var1="debjeet" var2="cloudaffaire" name=${!var*} echo $name ## returns var1 var2 name=${!var@} echo $name ## returns var1 var2 ## If name is an array variable, expands to the list of array indices (keys) assigned in name. If name is not an array, ## expands to 0 if name is set and null otherwise. When ‘@’ is used and the expansion appears within double quotes, ## each key expands to a separate word. ## Syntax ## ${!name[@]} ## ${!name[*]} array1=(1 2 3 4 5 6 7 8 9) notarray="debjeet" name=${!array1[@]} echo $name ## returns 0 1 2 3 4 5 6 7 8 name=${!notarray[@]} echo $name ## returns 0 ## The length in characters of the expanded value of parameter is substituted. If parameter is ‘*’ or ‘@’, the value ## substituted is the number of positional parameters. If parameter is an array name subscripted by ‘*’ or ‘@’, ## the value substituted is the number of elements in the array. ## Syntax ## ${#parameter} var1="debjeet" echo ${#var1} ## returns 7 var2=(1 2 3 4 5) echo ${#var2[@]} ## returns 5 set -- 1 2 3 4 5 6 7 echo ${#@} ## returns 7 ## The word is expanded to produce a pattern and matched according to the rules described below. ## If the pattern matches the beginning of the expanded value of parameter, then the result of the expansion is the ## expanded value of parameter with the shortest matching pattern (the ‘#’ case) or the longest matching pattern ## (the ‘##’ case) deleted. If parameter is ‘@’ or ‘*’, the pattern removal operation is applied to each ## positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable ## vsubscripted with ‘@’ or ‘*’, the pattern removal operation is applied to each member of the array in turn, ## and the expansion is the resultant list. ## Syntax ## ${parameter#word} ## ${parameter##word} var1="/this/is/some/path/where/file/is/located/myfile.txt" echo ${var1#/this/is/some/path} ## returns /where/file/is/located/myfile.txt echo ${var1##*/} ## returns myfile.txt ## The word is expanded to produce a pattern and matched according to the rules described below. ## If the pattern matches a trailing portion of the expanded value of parameter, then the result ## of the expansion is the value of paraeter with the shortest matching pattern (the ‘%’ case) ## or the longest matching pattern (the ‘%%’ case) deleted. If parameter is ‘@’ or ‘*’, the pattern ## removal operation is applied to each positional parameter in turn, and the expansion is the ## resultant list. If parameter is an array variable subscripted with ‘@’ or ‘*’, the pattern removal ## operation is applied to each member of the array in turn, and the expansion is the resultant list. ## Syntax ## ${parameter%word} ## ${parameter%%word} var1="this/is/some/path/where/file/is/located/myfile.txt" echo ${var1%/where/file/is/located/myfile.txt} ## returns this/is/some/path echo ${var1%%/*} ## returns this ## The pattern is expanded to produce a pattern just as in filename expansion. Parameter is expanded and the longest ## match of pattern against its value is replaced with string. The match is performed according to the rules described below. ## If pattern begins with ‘/’, all matches of pattern are replaced with string. Normally only the first match is replaced. ## If pattern begins with ‘#’, it must match at the beginning of the expanded value of parameter. ## If pattern begins with ‘%’, it must match at the end of the expanded value of parameter. ## If string is null, matches of pattern are deleted and the / following pattern may be omitted. ## If the nocasematch shell option is enabled, the match is performed without regard to the case of alphabetic characters. ## If parameter is ‘@’ or ‘*’, the substitution operation is applied to each positional parameter in turn, and the expansion ## is the resultant list. If parameter is an array variable subscripted with ‘@’ or ‘*’, the substitution operation is ## applied to each member of the array in turn, and the expansion is the resultant list. ## Syntax ## ${parameter/pattern/string} var1="hello world welcome" echo ${var1/world/all} ## returns hello all welcome ## This expansion modifies the case of alphabetic characters in parameter. The pattern is expanded to produce a pattern ## just as in filename expansion. Each character in the expanded value of parameter is tested against pattern, and, if ## it matches the pattern, its case is converted. The pattern should not attempt to match more than one character. ## The ‘^’ operator converts lowercase letters matching pattern to uppercase; the ‘,’ operator converts matching uppercase ## letters to lowercase. The ‘^^’ and ‘,,’ expansions convert each matched character in the expanded value; the ‘^’ and ‘,’ ## expansions match and convert only the first character in the expanded value. If pattern is omitted, it is treated like ## a ‘?’, which matches every character. If parameter is ‘@’ or ‘*’, the case modification operation is applied to each ## positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted ## with ‘@’ or ‘*’, the case modification operation is applied to each member of the array in turn, and the expansion is ## the resultant list. ## SYntax ## ${parameter^pattern} ## ${parameter^^pattern} ## ${parameter,pattern} ## ${parameter,,pattern} var1="hello world welcome" echo ${var1^} ## returns Hello world welcome echo ${var1^^} ## returns HELLO WORLD WELCOME |
Arithmetic Expansion:
Arithmetic expansion allows the evaluation of an arithmetic expression and the substitution of the result. The format for arithmetic expansion is: $(( expression )). The expression is treated as if it were within double quotes, but a double quote inside the parentheses is not treated specially. All tokens in the expression undergo parameter and variable expansion, command substitution, and quote removal. The result is treated as the arithmetic expression to be evaluated. Arithmetic expansions may be nested. If the expression is invalid, Bash prints a message indicating failure to the standard error and no substitution occurs.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
## -------------------- ## Arithmetic Expansion ## -------------------- ## Syntax ## $(( expression )) echo $(( 20 + 3 )) ## returns 23 echo $(( 20 - 3 )) ## returns 17 echo $(( 20 * 3 )) ## returns 60 echo $(( 20 / 3 )) ## returns 6 echo $(( 2 ** 3 )) ## returns 8 echo $(( 20 % 3 )) ## returns 2 |
Filename Expansion:
After word splitting, unless the -f option has been set, Bash scans each word for the characters ‘*’, ‘?’, and ‘[’. If one of these characters appears, then the word is regarded as a pattern, and replaced with an alphabetically sorted list of filenames matching the pattern. If no matching filenames are found, and the shell option nullglob is disabled, the word is left unchanged. If the nullglob option is set, and no matches are found, the word is removed. If the failglob shell option is set, and no matches are found, an error message is printed and the command is not executed. If the shell option nocaseglob is enabled, the match is performed without regard to the case of alphabetic characters.
When a pattern is used for filename expansion, the character ‘.’ at the start of a filename or immediately following a slash must be matched explicitly, unless the shell option dotglob is set. The filenames ‘.’ and ‘..’ must always be matched explicitly, even if dotglob is set. In other cases, the ‘.’ character is not treated specially. When matching a filename, the slash character must always be matched explicitly by a slash in the pattern, but in other matching contexts it can be matched by a special pattern character.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
## ------------------ ## Filename Expansion ## ------------------ touch {1,2,3}.{zip,mp3,txt,tar,sh} ## create some files ls -l * ## returns all files ls -l 1* ## returns all file name starting with 1 ls -l *.txt ## returns all files with extension .txt ls -l *.t?? ## returns all files with extension .txt or .tar ls -l [12]* ## returns all files with name staring with 1 or 2 ls -l [^12]* ## returns all files whoes name does not start with 1 or 2 rm [123]* ## deletes all files with name starting with 1 2 3 |
Hope you have enjoyed this article, in the next blog post, we will discuss shell substitution.