How to sort human readable du output

I originally ripped my movies to my hard drive in divx/avi, but I have redone them in h264/mp4 and I now wish to compare the spaces they take. Easy enough -- du -ch *.mp4 and du -ch *.avi. But I found this to be a bit unsatisfactory. I then wanted to sort the output of all the files. No problem again -- du | sort -rn. But what if I want it to be human-readable? Ah, there's the problem -- 741M is sorted as bigger than 1.4G because sort doesn't know the difference. So I thought to myself, "What if I could take the output as blocks, do the sort, and then transform the blocks into human readable format?" Voila! The following statement was born:


du -cm *.mp4 | sort -n | awk '{ printf "%s GB -",$1/1000; for (i=2;i<=NF+1;i++) printf "%s%s",i==NF+1 ? ORS : FS,$i }'


Note the use of the -m flag on du to convert it to megabytes first -- this means I only have to divide by 1000 in the awk script. Despite the obvious advantage of being able to sort by human-readable format, it also has the advantage of keeping it more accurate -- a "du -h" would display 1.4G where my method displays 1.359. Essentially I take the output in megabytes, do an ascending sort, and then use awk to manipulate the numeric field and display the rest of the fields as is. Also, I originally used 'print' for the first 'printf' but this automatically appended a '\n' so I switched it to a printf without a '\n'. When $NF+1 is evaluated in the for loop, ORS (output record separator, '\n' by default) is output, but when $2-$NF are evaluated, FS (field separator, [space] by default) is output.


UPDATE:

As a recursive variant, this appears to work well:


du -cam . | grep .mp4$ | sort -n | awk '{ printf "%s GB -",$1/1000; for (i=2;i<=NF+1;i++) printf "%s%s",i==NF+1 ? ORS : FS,$i }'


Add all unknown files to subversion repository

I installed a new plugin into my rails app which created a whole bunch of new files. I went through them editing them into the plugin worked correctly with my existing app's schematics and then decided I wanted to add the files into my repository, but I really didn't want to type 'svn add file' 50 times, so I came up with this:


svn status | grep ^? | sed -r -e 's/^\?\s+//' | xargs svn add


This finds all the files in the current directory that svn doesn't know about, strips off the beginning question mark and spaces, and adds them to svn one at a time. Perfect.

UPDATE:

I just discovered there's a rake task that does this for you. Simply type "rake add". For a listing of all the default rake tasks, just type "rake --tasks".

UPDATE2:

The 'rake add' task I mentioned in the last update actually isn't a default task. When I configured my system for subversion, I read some blogs on how to configure it to exclude log/ and tmp/, etc. I copied the code I found into a rake task and forgot about it. It turns out I copied a command for adding new files to subversion and it looks almost identical to the one I came up with. That being said, to make the above work as a rake task, include it in your lib/tasks folder. For instance, in a file called lib/tasks/svn.rake:


namespace :svn do
desc "Add new files to subversion repository"
task :add_new_files do
system "svn status | grep ^? | sed -r -e 's/^\?\s+//' | xargs svn add"
end
end


Then simply call 'rake svn:add_new_files' to run it.

Rsync over SSH issues

I tried to rsync a DVD rip from laptop (OS X) to my desktop (Ubuntu) only to get a weird error telling me there was a 'corrupted MAC on input' almost instantly. The command I used was this:

rsync -av /path/to/movie me@mydesktop:/new/path/to/movie

I googled around for a bit and mostly found that people said this error was due to bad RAM. This didn't make sense to me though as rsync always worked just fine through my wireless connection. In this case, though, I went with ethernet since it would be much faster for transferring 4.5G. I also read that a bad nic would cause this, so I thought maybe my ethernet port was bad since I rarely use it. As it turns out, the problem went away as soon as I added the compression switch ( -z ) to rsync. Perhaps I was trying to pipe more data through than it could handle?

Progress viewer for local cp

As I mentioned I have been moving around large files and I thought it would be cool to have a progress bar for local copies in a Bash shell. After a bit of digging around, I found out about a program called pv, which stands for Pipe Viewer. Basically you can stick pv in between commands to see how much and how fast data is passing through. For a local cp, it's as simple as this:

pv /path/to/file > /path/to/copied/file

For a little more complexity, you can make a function out of it, and place it in ~/.bashrc to be sourced automatically:

progcp() {

  if [ -d "$1" ]; then
    echo "Input is a directory.  Exiting."
    return
  fi

  if [ -d "$2" ]; then
    OUTFILE="$2/"`basename "$1"`
  else
    OUTFILE="$2"
  fi

  pv "$1" > "$OUTFILE"
}

I'm interested to know how to add normal command-line switches to a Bash function, so you could add flags like -r (recursive), or some of pv's switches, but this function works well for a single file local copy, such as a movie or an iso. Just call it like this:

progcp /path/to/file /path/to/output/file/or/directory

Finding duplicate filenames in a Bash shell

Several years ago, when DivX was the big thing, I ripped my DVD library to DivX/mp3 .avi's, but I've realized these are severely lacking as I can now crank up the bitrate with H.264/AAC .mp4's for high quality videos. Since disk space keeps getting cheaper and cheaper, I no longer even care how big the files get. Anyway, the point is, I've been ripping/encoding these movies as fast as possible into the same directory that all my other videos reside in, resulting in many duplicate files where one ends in .avi and the other ends in .mp4. I thought to myself that there must be a way to filter these out and after some fiddling I managed to produce a one-liner to do it. Here's what I came up with, but I'd love to know if anyone else has come up with a more efficient solution.

find . -type f -print0 | xargs -0 -I{} basename {} .avi | tr '\n' '\0' | xargs -0 -I{} basename {} .mp4 | sort | uniq -d

If the 'basename' command could take multiple arguments and remove multiple suffixes simultaneously, that would remove both 'xargs' command and the 'tr' command, but AFAIK it doesn't. Anybody know if it's possible to force xargs to do print0 like find does so I could eliminate the tr comman? Oh, and the sort isn't really necessary, but it was helpful as I was composing my pipes.

First Blog Post

Well, as the title says, this is my first blog post. I created it mostly to share random snippets of useful information regarding programming and Linux, though occasionally I might slip in some sports talk as I play Ultimate Frisbee and am an avid fan of football. Who knows how often I'll update this, but I hope the usefulness of the posts I create live on. Enjoy!