Copy with a progress bar with cpv

It won’t come as a surprise by now that we love shell progress bars.

When we are copying big files it would be nice to have an ETA, progress bar and some feedback, but the cp command doesn’t provide us with this.

Of course, we can use rsync to get some feedback of the process, but if we want a progress bar probably the most simple way is using pv.

pv, or pipe viewer is a very neat shell tool that allows you to monitor a pipe. It is invoked in a way that is similar to the cat command. You can pipe to it and monitor the throughput of the data that goes through the pipe

cat srcfile | pv > dstfile

1	cat srcfile \| pv > dstfile

You can also open a file directly with it, just like cat. This is in fact how you would “copy” a file using pv

pv srcfile > dstfile

1	pv srcfile > dstfile

But that syntax is not very nice to use, plus it is limited to a single file at a time. I wanted to use it with whole directories, recursively so I came up with a little wrapper, with the astoundingly original name cpv.

Usage

Invoke just like cp. No options. Obviously requires that you have pv installed.

cpv file_or_dir_src file_or_dir_dst

1	cpv file_or_dir_src file_or_dir_dst

That’s it. It is really a very simple wrapper but it is handy.

It is also useful when copying from or to network mounts to measure the transfer speed.

Installation

Add the code to your .zshrc or .bashrc. You can do this in one line, but inspect it first so you don’t have to trust me blindly.

wget https://raw.githubusercontent.com/nachoparker/cpv/master/cpv.sh -O - >> ~/.zshrc  # ~/.bashrc

1	wget https://raw.githubusercontent.com/nachoparker/cpv/master/cpv.sh -O - >> ~/.zshrc # ~/.bashrc

More cool stuff with pv

You can do much more that copying. You can get data about just about anything that uses files or pipes.

You can for instance monitor that netcat transfer

pv srcfile | nc -w 1 somewhere.com 3000

1	pv srcfile \| nc -w 1 somewhere.com 3000

Or the creation of that tarball

tar -cf - dir | pv > tarball.tar

1	tar -cf - dir \| pv > tarball.tar

Or a docker export

docker export minidebian_container | pv > minidebian_export.tar

1	docker export minidebian_container \| pv > minidebian_export.tar

Or that ssh pipe

tar -cfz - dir | pv | ssh pi@raspi "cat > ~/dir.tar.gz"

1	tar -cfz - dir \| pv \| ssh pi@raspi "cat > ~/dir.tar.gz"

Or that database dump

mysqldump -u root --single-transaction nextcloud | pv > db-backup.sql

1	mysqldump -u root --single-transaction nextcloud \| pv > db-backup.sql

You can flash a SD card with progress (even though we finally have dd status=progress )

pv NextCloudPi.img > /dev/sdc; sync

1	pv NextCloudPi.img > /dev/sdc; sync

Zero a drive

pv < /dev/zero > /dev/sda

1	pv < /dev/zero > /dev/sda

It can even track open file descriptors, so you can monitor the operation afterwards the process started

mysqldump -u root --single-transaction nextcloud > db-backup.sql
pv -d $(pidof mysqldump)

1 2	mysqldump -u root --single-transaction nextcloud > db-backup.sql pv -d $(pidof mysqldump)

Code

#!/bin/bash
# pv wrapper that imitates cp usage
#
# Copyleft 2017 by Ignacio Nunez Hernanz <nacho _a_t_ ownyourbits _d_o_t_ com>
# GPL licensed (see end of file) * Use at your own risk!
#
# Usage:
#   cpv file_or_dir_src file_or_dir_dst
#
# More details at ownyourbits.com

function cpv()
{
  local DST=${@: -1}                    # last element
  local SRC=( ${@: 1 : $# - 1} )        # array with rest of elements

  # checks
  type pv &>/dev/null || { echo "install pv first"; return 1; }
  [ $# -lt 2  ]       && { echo "too few args"    ; return 1; }

  # special invocation
  function cpv_rename()
  {
    local SRC="$1"
    local DST="$2"
    local DSTDIR="$( dirname "$DST" )"

    # checks
    if   [ $# -ne 2     ]; then echo "too few args"          ; return 1; fi
    if ! [ -e "$SRC"    ]; then echo "$SRC doesn't exist"    ; return 1; fi
    if   [ -d "$SRC"    ]; then echo "$SRC is a dir"         ; return 1; fi
    if ! [ -d "$DSTDIR" ]; then echo "$DSTDIR does not exist"; return 1; fi

    # actual copy
    echo -e "\n$SRC 🡺  $DST"
    pv   "$SRC" >"$DST"
  }

  # special case for cpv_rename()
  if ! [ -d "$DST" ]; then cpv_rename "$@"; return $?; fi;

  # more checks
  for src in "${SRC[@]}"; do 
    local dst="$DST/$( basename "$src" )"
    if ! [ -e "$src" ]; then echo "$src doesn't exist" ; return 1;
    elif [ -e "$dst" ]; then echo "$dst already exists"; return 1; fi
  done

  # actual copy
  for src in "${SRC[@]}"; do 
    if ! [ -d "$src" ]; then 
      local dst="$DST/$( basename "$src" )"
      echo -e "\n$src 🡺  $dst"
      pv "$src" > "$dst"
    else 
      local dir="$DST/$( basename "$src" )"
      mkdir "$dir" || continue
      local srcs=( $src/* )
      cpv "${srcs[@]}" "$dir";
    fi
  done
  unset cpv_rename
}

# License
#
# This script is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This script is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this script; if not, write to the
# Free Software Foundation, Inc., 59 Temple Place, Suite 330,
# Boston, MA  02111-1307  USA

#!/bin/bash

# pv wrapper that imitates cp usage

# Copyleft 2017 by Ignacio Nunez Hernanz <nacho _a_t_ ownyourbits _d_o_t_ com>

# GPL licensed (see end of file) * Use at your own risk!

# Usage:

# cpv file_or_dir_src file_or_dir_dst

# More details at ownyourbits.com

function cpv()

{

local DST=${@: -1} # last element

local SRC=( ${@: 1 : $# - 1} ) # array with rest of elements

# checks

type pv &>/dev/null || { echo "install pv first"; return 1; }

[ $# -lt 2 ] && { echo "too few args" ; return 1; }

# special invocation

function cpv_rename()

{

local SRC="$1"

local DST="$2"

local DSTDIR="$( dirname "$DST" )"

# checks

if [ $# -ne 2 ]; then echo "too few args" ; return 1; fi

if ! [ -e "$SRC" ]; then echo "$SRC doesn't exist" ; return 1; fi

if [ -d "$SRC" ]; then echo "$SRC is a dir" ; return 1; fi

if ! [ -d "$DSTDIR" ]; then echo "$DSTDIR does not exist"; return 1; fi

# actual copy

echo -e "\n$SRC 🡺 $DST"

pv "$SRC" >"$DST"

}

# special case for cpv_rename()

if ! [ -d "$DST" ]; then cpv_rename "$@"; return $?; fi;

# more checks

for src in "${SRC[@]}"; do

local dst="$DST/$( basename "$src" )"

if ! [ -e "$src" ]; then echo "$src doesn't exist" ; return 1;

elif [ -e "$dst" ]; then echo "$dst already exists"; return 1; fi

done

# actual copy

for src in "${SRC[@]}"; do

if ! [ -d "$src" ]; then

local dst="$DST/$( basename "$src" )"

echo -e "\n$src 🡺 $dst"

pv "$src" > "$dst"

else

local dir="$DST/$( basename "$src" )"

mkdir "$dir" || continue

local srcs=( $src/* )

cpv "${srcs[@]}" "$dir";

done

unset cpv_rename

}

# License

# This script is free software; you can redistribute it and/or modify it

# under the terms of the GNU General Public License as published by

# the Free Software Foundation; either version 2 of the License, or

# (at your option) any later version.

# This script is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the

# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License

# along with this script; if not, write to the

# Free Software Foundation, Inc., 59 Temple Place, Suite 330,

# Boston, MA 02111-1307 USA

Check out other shell articles

5 Comments on “Copy with a progress bar with cpv”

Sarg says:

October 4, 2017 at 11:57 am

Or just use rsync –info=progress2

1. nachoparker says:
  
  October 4, 2017 at 2:56 pm
  
  does that provide a progress bar? not last time I checked
  
An says:

January 25, 2018 at 8:31 am

Hello nachoparker,
Which distro os linux you are using?

I like GUI desktop , may you help me sent the name ?

Thanks

1. nachoparker says:
  
  February 7, 2018 at 3:58 pm
  
  Hi, I use Arch Linux with plain openbox… quite minimal stuff really
  
Pingback: Safely flash SD card images with ddsh – Own your bits

Own your bits