How do I iterate over a range of numbers defined by variables in Bash?

Asked 2023-09-20 20:56:39 View 9,273

How do I iterate over a range of numbers in Bash when the range is given by a variable?

I know I can do this (called "sequence expression" in the Bash documentation):

for i in {1..5}; do echo $i; done

Which gives:

1
2
3
4
5

Yet, how can I replace either of the range endpoints with a variable? This doesn't work:

END=5
for i in {1..$END}; do echo $i; done

Which prints:

{1..5}

  • Hi all, the informations and hints I have read here are all really helpful. I think it is best to avoid the use of seq. The reason is that some scripts need to be portable and must run on a wide variety of unix systems, where some commands may not be present. Just to do an example, seq is not present by default on FreeBSD systems. - anyone
  • Related discusions: bash for loop: a range of numbers and unix.stackexchange.com - In bash, is it possible to use an integer variable in the loop control of a for loop? - anyone
  • I don't remember since which version of Bash exactly but this command supports trailing zeros as well. Which sometimes is really helpful. Command for i in {01..10}; do echo $i; done would give numbers like 01, 02, 03, ..., 10. - anyone
  • For those like me who just want to iterate over the range of indices of an array, the bash way would be: myarray=('a' 'b' 'c'); for i in ${!myarray[@]}; do echo $i; done (note the exclamation mark). It's more specific than the original question, but could help. See bash parameter expansions - anyone
  • Brace expansion is also used for expressions like {jpg,png,gif} which isn't directly addressed here, though the answer will be identical. See Brace expansion with variable? [duplicate] which is marked as a duplicate of this one. - anyone

Answers

for i in $(seq 1 $END); do echo $i; done

edit: I prefer seq over the other methods because I can actually remember it ;)

Answered   2023-09-20 20:56:39

  • seq involves the execution of an external command which usually slows things down. This may not matter but it becomes important if you're writing a script to handle lots of data. - anyone
  • Just fine for a one-liner. Pax's solution is fine too, but if performance were really a concern I wouldn't be using a shell script. - anyone
  • seq is called just once to generate the numbers. exec()'ing it shouldn't be significant unless this loop is inside another tight loop. - anyone
  • The external command isn't really relevent: if you're worried about the overhead of running external commands you don't want to be using shell scripts at all, but generally on unix the overhead is low. However, there is the issue of memory usage if END is high. - anyone
  • Note that seq $END would suffice, as the default is to start from 1. From man seq: "If FIRST or INCREMENT is omitted, it defaults to 1". - anyone

The seq method is the simplest, but Bash has built-in arithmetic evaluation.

END=5
for ((i=1;i<=END;i++)); do
    echo $i
done
# ==> outputs 1 2 3 4 5 on separate lines

The for ((expr1;expr2;expr3)); construct works just like for (expr1;expr2;expr3) in C and similar languages, and like other ((expr)) cases, Bash treats them as arithmetic.

Answered   2023-09-20 20:56:39

  • This way avoids the memory overhead of a large list, and a dependency on seq. Use it! - anyone
  • @MarinSagovac Make sure to make #!/bin/bash the first line of your script. wiki.ubuntu.com/… - anyone
  • just a very short question on that: why ((i=1;i<=END;i++)) AND NOT ((i=1;i<=$END;i++)); why no $ before END? - anyone
  • @Baedsch: for the same reason i is not used as $i. bash man page states for arithmetic evaluation: "Within an expression, shell variables may also be referenced by name without using the parameter expansion syntax." - anyone
  • I've included this answer in my performance comparison answer below. stackoverflow.com/a/54770805/117471 (This is a note to myself to keep track of which ones I have left to do.) I created that answer specifically to address @bobbogo's unqualified claim and those that upvoted it. SPOILER: Memory overhead of a large list is not nearly as bad as the slow performance of c-style loops in bash. Comment there if you have thoughts. Let's not hijack this thread. - anyone

discussion

Using seq is fine, as Jiaaro suggested. Pax Diablo suggested a Bash loop to avoid calling a subprocess, with the additional advantage of being more memory friendly if $END is too large. Zathrus spotted a typical bug in the loop implementation, and also hinted that since i is a text variable, continuous conversions to-and-fro numbers are performed with an associated slow-down.

integer arithmetic

This is an improved version of the Bash loop:

typeset -i i END
let END=5 i=1
while ((i<=END)); do
    echo $i
    …
    let i++
done

If the only thing that we want is the echo, then we could write echo $((i++)).

ephemient taught me something: Bash allows for ((expr;expr;expr)) constructs. Since I've never read the whole man page for Bash (like I've done with the Korn shell (ksh) man page, and that was a long time ago), I missed that.

So,

typeset -i i END # Let's be explicit
for ((i=1;i<=END;++i)); do echo $i; done

seems to be the most memory-efficient way (it won't be necessary to allocate memory to consume seq's output, which could be a problem if END is very large), although probably not the “fastest”.

the initial question

eschercycle noted that the {a..b} Bash notation works only with literals; true, accordingly to the Bash manual. One can overcome this obstacle with a single (internal) fork() without an exec() (as is the case with calling seq, which being another image requires a fork+exec):

for i in $(eval echo "{1..$END}"); do

Both eval and echo are Bash builtins, but a fork() is required for the command substitution (the $(…) construct).

Answered   2023-09-20 20:56:39

  • The only drawback to the C style loop that it cannot use command line arguments, as they begin with "$". - anyone
  • @karatedog: for ((i=$1;i<=$2;++i)); do echo $i; done in a script works fine for me on bash v.4.1.9, so I don't see a problem with command line arguments. Do you mean something else? - anyone
  • It seems that eval solution is faster than built in C-like for: $ time for ((i=1;i<=100000;++i)); do :; done real 0m21.220s user 0m19.763s sys 0m1.203s $ time for i in $(eval echo "{1..100000}"); do :; done; real 0m13.881s user 0m13.536s sys 0m0.152s - anyone
  • Yes, but eval is evil... @MarcinZaluski time for i in $(seq 100000); do :; done is a lot quicker! - anyone
  • The performance must be platform specific since the eval version is quickest on my machine. - anyone

Here is why the original expression didn't work.

From man bash:

Brace expansion is performed before any other expansions, and any characters special to other expansions are preserved in the result. It is strictly textual. Bash does not apply any syntactic interpretation to the context of the expansion or the text between the braces.

So, brace expansion is something done early as a purely textual macro operation, before parameter expansion.

Shells are highly optimized hybrids between macro processors and more formal programming languages. In order to optimize the typical use cases, the language is made rather more complex and some limitations are accepted.

Recommendation

I would suggest sticking with Posix1 features. This means using for i in <list>; do, if the list is already known, otherwise, use while or seq, as in:

#!/bin/sh

limit=4

i=1; while [ $i -le $limit ]; do
  echo $i
  i=$(($i + 1))
done
# Or -----------------------
for i in $(seq 1 $limit); do
  echo $i
done


1. Bash is a great shell and I use it interactively, but I don't put bash-isms into my scripts. Scripts might need a faster shell, a more secure one, a more embedded-style one. They might need to run on whatever is installed as /bin/sh, and then there are all the usual pro-standards arguments. Remember shellshock, aka bashdoor?

Answered   2023-09-20 20:56:39

  • I don't have the power, but I would move this quite a bit up the list, above all the bash navel-gazing but immediately after the C-style for loop and arithmetic evaluation. - anyone
  • An implication is that brace expansion does not save much memory in comparison to seq for large ranges. E.g., echo {1..1000000} | wc reveals that the echo produces 1 line, a million words, and 6,888,896 bytes. Trying seq 1 1000000 | wc yields a million lines, a million words, and 6,888,896 bytes and is also more than seven times faster, as measured by the time command. - anyone
  • Note: I had mentioned the POSIX while method previously in my answer: stackoverflow.com/a/31365662/895245 But glad you agree :-) - anyone
  • I've included this answer in my performance comparison answer below. stackoverflow.com/a/54770805/117471 (This is a note to myself to keep track of which ones I have left to do.) - anyone
  • @mateor I thought C-style for loop and arithmetic evaluation are the same solution. Am I missing something? - anyone

The POSIX way

If you care about portability, use the example from the POSIX standard:

i=2
end=5
while [ $i -le $end ]; do
    echo $i
    i=$(($i+1))
done

Output:

2
3
4
5

Things which are not POSIX:

Answered   2023-09-20 20:56:39

  • Just had 4 upvotes on this answer, which is highly unusual. If this was posted on some link aggregation website, please give me a link, cheers. - anyone
  • The quote refers to x, not the entire expression. $((x + 1)) is just fine. - anyone
  • While not portable, and differing from GNU seq (BSD seq allows you to set a sequence termination string with -t), FreeBSD and NetBSD also have seq since 9.0 and 3.0, respectively. - anyone
  • @CiroSantilli @chepner $((x+1)) and $((x + 1)) parse exactly the same, as when the parser tokenizes x+1 it will be split into 3 tokens: x, +, and 1. x isn't a valid numerical token, but it is a valid variable name token, yet x+ is not, hence the split. + is a valid arithmetic operator token, yet +1 is not, so the token is again split there. And so forth. - anyone
  • I've included this answer in my performance comparison answer below. stackoverflow.com/a/54770805/117471 (This is a note to myself to keep track of which ones I have left to do.) - anyone

You can use

for i in $(seq $END); do echo $i; done

Answered   2023-09-20 20:56:39

  • seq involves the execution of an external command which usually slows things down. - anyone
  • It doesn't involve the execution of an external command for each iteration, just once. If the time to launch one external command is an issue, you're using the wrong language. - anyone
  • So is nesting the only case where this matters? I was wondering if there was a performance difference, or some unknown technical side effect? - anyone
  • @Squeaky That's a separate question whichtis answered here: stackoverflow.com/questions/4708549/… - anyone
  • I've included this answer in my performance comparison answer below. stackoverflow.com/a/54770805/117471 (This is a note to myself to keep track of which ones I have left to do.) - anyone

Another layer of indirection:

for i in $(eval echo {1..$END}); do
    ∶

Answered   2023-09-20 20:56:39

  • +1: Also, eval 'for i in {1..'$END'}; do ... ' eval seems the natural way to solve this issue. - anyone

I've combined a few of the ideas here and measured performance.

TL;DR Takeaways:

  1. seq and {..} are really fast
  2. for and while loops are slow
  3. $( ) is slow
  4. for (( ; ; )) loops are slower
  5. $(( )) is even slower
  6. Worrying about N numbers in memory (seq or {..}) is silly (at least up to 1 million.)

These are not conclusions. You would have to look at the C code behind each of these to draw conclusions. This is more about how we tend to use each of these mechanisms for looping over code. Most single operations are close enough to being the same speed that it's not going to matter in most cases. But a mechanism like for (( i=1; i<=1000000; i++ )) is many operations as you can visually see. It is also many more operations per loop than you get from for i in $(seq 1 1000000). And that may not be obvious to you, which is why doing tests like this is valuable.

Demos

# show that seq is fast
$ time (seq 1 1000000 | wc)
 1000000 1000000 6888894

real    0m0.227s
user    0m0.239s
sys     0m0.008s

# show that {..} is fast
$ time (echo {1..1000000} | wc)
       1 1000000 6888896

real    0m1.778s
user    0m1.735s
sys     0m0.072s

# Show that for loops (even with a : noop) are slow
$ time (for i in {1..1000000} ; do :; done | wc)
       0       0       0

real    0m3.642s
user    0m3.582s
sys 0m0.057s

# show that echo is slow
$ time (for i in {1..1000000} ; do echo $i; done | wc)
 1000000 1000000 6888896

real    0m7.480s
user    0m6.803s
sys     0m2.580s

$ time (for i in $(seq 1 1000000) ; do echo $i; done | wc)
 1000000 1000000 6888894

real    0m7.029s
user    0m6.335s
sys     0m2.666s

# show that C-style for loops are slower
$ time (for (( i=1; i<=1000000; i++ )) ; do echo $i; done | wc)
 1000000 1000000 6888896

real    0m12.391s
user    0m11.069s
sys     0m3.437s

# show that arithmetic expansion is even slower
$ time (i=1; e=1000000; while [ $i -le $e ]; do echo $i; i=$(($i+1)); done | wc)
 1000000 1000000 6888896

real    0m19.696s
user    0m18.017s
sys     0m3.806s

$ time (i=1; e=1000000; while [ $i -le $e ]; do echo $i; ((i=i+1)); done | wc)
 1000000 1000000 6888896

real    0m18.629s
user    0m16.843s
sys     0m3.936s

$ time (i=1; e=1000000; while [ $i -le $e ]; do echo $((i++)); done | wc)
 1000000 1000000 6888896

real    0m17.012s
user    0m15.319s
sys     0m3.906s

# even a noop is slow
$ time (i=1; e=1000000; while [ $((i++)) -le $e ]; do :; done | wc)
       0       0       0

real    0m12.679s
user    0m11.658s
sys 0m1.004s

Answered   2023-09-20 20:56:39

  • Nice! Don't agree with your summary though. Looks to me like $(seq) is about the same speed as {a..b}. Also, each operation takes about the same time, so adds about 4μs to each iteration of the loop for me. Here an operation is an echo in the body, an arithmetic compare, an increment, etc. Is any of this surprising? Who cares how long it takes the loop paraphernalia to do its job—the runtime is likely to be dominated by the contents of the loop. - anyone
  • @bobbogo you are right, it really is about operation count. I updated my answer to reflect this. Many calls we make actually perform more operations than we might expect. I narrowed this down from a list of about 50 tests I ran. I expected that my research was too nerdy even for this crowd. As always, I suggest prioritizing your coding efforts like so: Make it shorter; Make it readable; Make it faster; Make it portable. Often #1 causes #3. Don't waste your time on #4 until you must. - anyone
  • That's an interesting exercise, though the initial question is about using a variable-countrol iteration, which for example {..} doesn'T allow. - anyone
  • I never knew for {i..n} ! That's so cool, every language should have this. - anyone

If you need it prefix than you might like this

 for ((i=7;i<=12;i++)); do echo `printf "%2.0d\n" $i |sed "s/ /0/"`;done

that will yield

07
08
09
10
11
12

Answered   2023-09-20 20:56:39

  • Wouldn't printf "%02d\n" $i be easier than printf "%2.0d\n" $i |sed "s/ /0/"? - anyone

If you're on BSD / OS X you can use jot instead of seq:

for i in $(jot $END); do echo $i; done

Answered   2023-09-20 20:56:39

  • macOS has seq - anyone
  • The seq command first appeared in Plan 9 from Bell Labs. A seq command appeared in NetBSD 3.0, and ported to FreeBSD 9.0. This command was based on the command of the same name in Plan 9 from Bell Labs and the GNU core utilities. The GNU seq command first appeared in the 1.13 shell utilities release. - anyone

This works fine in bash:

END=5
i=1 ; while [[ $i -le $END ]] ; do
    echo $i
    ((i = i + 1))
done

Answered   2023-09-20 20:56:39

  • echo $((i++)) works and combines it onto one line. - anyone
  • This has unneeded bash extensions. A POSIX version: stackoverflow.com/a/31365662/895245 - anyone
  • @Ciro, since the question specifically states bash, and has a bash tag, I think you'll probably find that bash 'extensions' are more than okay :-) - anyone
  • @paxdiablo I don't mean it's not correct, but why not be portable when we can ;-) - anyone
  • In bash, we can simply do while [[ i++ -le "$END" ]]; do to do the (post-) increment in the test - anyone

There are many ways to do this, however the ones I prefer is given below

Using seq

Synopsis from man seq

$ seq [-w] [-f format] [-s string] [-t string] [first [incr]] last

Syntax

Full command
seq first incr last

  • first is starting number in the sequence [is optional, by default:1]
  • incr is increment [is optional, by default:1]
  • last is the last number in the sequence

Example:

$ seq 1 2 10
1 3 5 7 9

Only with first and last:

$ seq 1 5
1 2 3 4 5

Only with last:

$ seq 5
1 2 3 4 5

Using {first..last..incr}

Here first and last are mandatory and incr is optional

Using just first and last

$ echo {1..5}
1 2 3 4 5

Using incr

$ echo {1..10..2}
1 3 5 7 9

You can use this even for characters like below

$ echo {a..z}
a b c d e f g h i j k l m n o p q r s t u v w x y z

Answered   2023-09-20 20:56:39

I know this question is about bash, but - just for the record - ksh93 is smarter and implements it as expected:

$ ksh -c 'i=5; for x in {1..$i}; do echo "$x"; done'
1
2
3
4
5
$ ksh -c 'echo $KSH_VERSION'
Version JM 93u+ 2012-02-29

$ bash -c 'i=5; for x in {1..$i}; do echo "$x"; done'
{1..5}

Answered   2023-09-20 20:56:39

This is another way:

end=5
for i in $(bash -c "echo {1..${end}}"); do echo $i; done

Answered   2023-09-20 20:56:39

  • This has the overhead of spawning another shell. - anyone
  • Actually, this is extra terrible because it spawns 2 shells when 1 would suffice. - anyone

If you want to stay as close as possible to the brace-expression syntax, try out the range function from bash-tricks' range.bash.

For example, all of the following will do the exact same thing as echo {1..10}:

source range.bash
one=1
ten=10

range {$one..$ten}
range $one $ten
range {1..$ten}
range {1..10}

It tries to support the native bash syntax with as few "gotchas" as possible: not only are variables supported, but the often-undesirable behavior of invalid ranges being supplied as strings (e.g. for i in {1..a}; do echo $i; done) is prevented as well.

The other answers will work in most cases, but they all have at least one of the following drawbacks:

  • Many of them use subshells, which can harm performance and may not be possible on some systems.
  • Many of them rely on external programs. Even seq is a binary which must be installed to be used, must be loaded by bash, and must contain the program you expect, for it to work in this case. Ubiquitous or not, that's a lot more to rely on than just the Bash language itself.
  • Solutions that do use only native Bash functionality, like @ephemient's, will not work on alphabetic ranges, like {a..z}; brace expansion will. The question was about ranges of numbers, though, so this is a quibble.
  • Most of them aren't visually similar to the {1..10} brace-expanded range syntax, so programs that use both may be a tiny bit harder to read.
  • @bobbogo's answer uses some of the familiar syntax, but does something unexpected if the $END variable is not a valid range "bookend" for the other side of the range. If END=a, for example, an error will not occur and the verbatim value {1..a} will be echoed. This is the default behavior of Bash, as well--it is just often unexpected.

Disclaimer: I am the author of the linked code.

Answered   2023-09-20 20:56:39

These are all nice but seq is supposedly deprecated and most only work with numeric ranges.

If you enclose your for loop in double quotes, the start and end variables will be dereferenced when you echo the string, and you can ship the string right back to BASH for execution. $i needs to be escaped with \'s so it is NOT evaluated before being sent to the subshell.

RANGE_START=a
RANGE_END=z
echo -e "for i in {$RANGE_START..$RANGE_END}; do echo \\${i}; done" | bash

This output can also be assigned to a variable:

VAR=`echo -e "for i in {$RANGE_START..$RANGE_END}; do echo \\${i}; done" | bash`

The only "overhead" this should generate should be the second instance of bash so it should be suitable for intensive operations.

Answered   2023-09-20 20:56:39

Replace {} with (( )):

tmpstart=0;
tmpend=4;

for (( i=$tmpstart; i<=$tmpend; i++ )) ; do 
echo $i ;
done

Yields:

0
1
2
3
4

Answered   2023-09-20 20:56:39

If you're doing shell commands and you (like I) have a fetish for pipelining, this one is good:

seq 1 $END | xargs -I {} echo {}

Answered   2023-09-20 20:56:39

if you don't wanna use 'seq' or 'eval' or jot or arithmetic expansion format eg. for ((i=1;i<=END;i++)), or other loops eg. while, and you don't wanna 'printf' and happy to 'echo' only, then this simple workaround might fit your budget:

a=1; b=5; d='for i in {'$a'..'$b'}; do echo -n "$i"; done;' echo "$d" | bash

PS: My bash doesn't have 'seq' command anyway.

Tested on Mac OSX 10.6.8, Bash 3.2.48

Answered   2023-09-20 20:56:39

This works in Bash and Korn, also can go from higher to lower numbers. Probably not fastest or prettiest but works well enough. Handles negatives too.

function num_range {
   # Return a range of whole numbers from beginning value to ending value.
   # >>> num_range start end
   # start: Whole number to start with.
   # end: Whole number to end with.
   typeset s e v
   s=${1}
   e=${2}
   if (( ${e} >= ${s} )); then
      v=${s}
      while (( ${v} <= ${e} )); do
         echo ${v}
         ((v=v+1))
      done
   elif (( ${e} < ${s} )); then
      v=${s}
      while (( ${v} >= ${e} )); do
         echo ${v}
         ((v=v-1))
      done
   fi
}

function test_num_range {
   num_range 1 3 | egrep "1|2|3" | assert_lc 3
   num_range 1 3 | head -1 | assert_eq 1
   num_range -1 1 | head -1 | assert_eq "-1"
   num_range 3 1 | egrep "1|2|3" | assert_lc 3
   num_range 3 1 | head -1 | assert_eq 3
   num_range 1 -1 | tail -1 | assert_eq "-1"
}

Answered   2023-09-20 20:56:39