Sharing some bash-script-fu: Re-installing Flash dependencies

Posted by hkwint on Mar 30, 2011 11:49 AM EDT
LXer Linux News; By Hans Kwint 		Mail this story
Print this story

LXer feature 30-Mar-11

Lately Flash became very unstable and crashed both Firefox and Opera time and time again. I had the latest version, so I thought the problem might be outdated dependencies. When finding all the dependencies, I used some of my limited bash-scripting skills, this time also using my first 'intermediate' regular expression. Nothing fancy but on the other hand I thought I might share them with anyone interested; maybe it's of any help to anyone! Because some of the commands may also be useful for doing other work while dealing with file and text streams.

The commands to be used in this article include ls, awk, sed, cut, ldd, grep, the Gentoo-specific commands for dealing with the packages 'equery' and 'emerge', or using the Debian apt-file command as an alternative.

Of course this doesn't mean you would have to use the command line to 'update Linux', in Ubuntu and most other distrobutions you could simply use a graphic interface to update all your packages. In Gentoo you could do 'emerge -e world" and after four days reach the same result. So basically I use scripting because I like fiddling, it's educating and in this particular sense it's quicker than the alternative.

Please note I'm using the semicolon ; to distinguish between different lines of code. Instead of using the semicolon, you also could start a new command on the next line.

Finding and rebuilding Flash dependencies goes in several steps:

1) Locate plugin



First we locate the plugin. Usually, it's in ~/.mozilla/plugins. 
$ ls -alF ~/.mozilla/plugins


Here is the output line we're looking for:

lrwxrwxrwx 1 kwint wheel 41 Mar 6 14:49 libflashplayer.so -> /opt/Adobe/flash-player/libflashplayer.so


We use the '-l' switch to list where the links are pointing to. The -a and -F switch aren't needed actually, but typing '-alF' is one of my automatisms when using ls. So, on my system, it's a link to the file in /opt. From this file, we want to find out its dependencies. We do this by using 'ldd' in step 2.

2) Listing dependent library files



$ ldd /opt/Adobe/flash-player/libflashplayer.so | awk '{print $3}' | sed -n "2,$p" > dependencies.log


We use the redirect-sign > (greater than) to save the output in a file called 'dependencies.log' to use later on.

We can see what's in dependencies.log by means of opening it with a text editor or using '$cat dependencies.log'. Without using the the part after the first pipe (| awk...), ldd on my system would show this:

       linux-gate.so.1 =>  (0xffffe000)
        libX11.so.6 => /usr/lib/libX11.so.6 (0xb6a4e000)
        libXext.so.6 => /usr/lib/libXext.so.6 (0xb6a3c000)
        libXt.so.6 => /usr/lib/libXt.so.6 (0xb69d7000)
        libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0xb691d000)
        libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0xb68e5000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xb68cb000)
        librt.so.1 => /lib/librt.so.1 (0xb68c2000)
...


And a few lines more. Those are the library files the flash plugin depends on.

We're only interested in the 'file' column which shows the path as well. So we use the awk-command to only print the third column.

Instead of using awk, it would also have been possible to use 'cut' instead of "awk" between the two pipe | characters:

$ ldd /opt/Adobe/flash-player/libflashplayer.so | cut -d " " -f 3 | sed -n "2,$p" > dependencies.log


This says, using the " " (space) as column delimiter, 'cut' the third column of the input, meaning "disregard the other columns".

Using the 'awk' version of the commands, output looks like this:

(0xffffe000)
/usr/lib/libX11.so.6
/usr/lib/libXext.so.6
/usr/lib/libXt.so.6
/usr/lib/libfreetype.so.6
/usr/lib/libfontconfig.so.1
/lib/libpthread.so.0


The first line when adding the awk command shows "(0xffffe000)" but that's something we're not interested in. Of course, 'cut' prevents the first line from showing the hex value, but I thought stripping the first line may be an interesting additional excercise, so we go with the 'awk' variant. To 'strip' the first line, I use sed. Maybe it's not the best way, but it's convenient for me.

However we're not really 'stripping' a line, instead I just print "all other lines". So we print line two until the last line. By default sed prints any output , and then when you give 'sed' the print command it prints the lines again. So 'sed p' would print input lines twice, once because it's sed's default, and once because the 'p' (print) command tells it to do so.

We prevent sed from copying any input line to the output by using the '-n' option. 2,$ is the address range sed acts upon. $ means the last line of the file. So, "2,$ p" prints anything except the first line. Please note the space between the $ and p, without the space this command will not work! It took me some time to figure this out. Still, there's one empty line in my output. To 'filter out' the empty line, I could have used grep:

$ ldd /opt/Adobe/flash-player/libflashplayer.so | awk '{print $3}' | sed -n "2,$p" | grep  -v "^$" > dependencies.log


What the grep-part does, is "filtering lines which start (the ^ character matches the start of the line) and then immediately terminates (the $ character matches the end of the line)". So using "grep ^$" would print empty lines. But we want the opposite, only the non-empty lines. To make grep filter 'in reverse', we supply the -v switch, and then we're done processing text.

3) Searching which packages correspond to the library files



Now here comes the 'fancy' stuff: We're going to find out what packages these files belong too!

We do this - again - by using some redirects.

$ while read a; do sudo equery b ${a} >> packages.log; done < dependencies.log


What this does is 'reading' the variable "a", one line at a time, from the file 'dependencies.log' which is redirected to the 'read' command at the end of the line using the < character. Note I have to use "sudo" because only root is allowed to use package managing commands on my system. "equery" is a python script made for gentoo, and the 'b' option means 'belongs'. So it finds out to which package a certain file on my system belongs. ${a} substitutes the line which 'read' received from dependencies.log, and "while" is a loop. So while there are lines coming from the redirect and dependencies.log, the command tries to find the package the file belongs to. It then 'adds' the output to the file packages.log, using the << operator.

This is a python script, so it takes several minutes to complete. Besides waiting, there's nothing else we can do!

We could add a countdown counter, using the total number of lines and the line number the script is currently working on.

$ i=$( wc -l dependencies.log | cut -d " " -f 1); while read a; do echo ${i}; sudo equery b ${a} >> packages.log; let i-=1;done < dependencies.log


wc -l counts the number of lines in the file dependencies.log, the cut command strips the filename which wc seems to output by default - but we only need the number so we don't want it. And the for loop executes starting at the number of lines (58 on my system), counting back (i-- does this) until i is equal to 1.

Fancy, don't you think?

When on a Debian derived distribution, you might install apt-file. After running "apt-file update" you might use "apt-file search" instead of "equery b". apt-file returns the packages in another format though, so when using apt-file you can continue re-installing the packages without the scripting-fu below.

Now my packages.log looks something like this:

x11-libs/libX11-1.4.1
x11-libs/libXext-1.2.0
x11-libs/libXt-1.0.9
media-libs/freetype-2.4.3-r2
media-libs/fontconfig-2.8.0
sys-libs/glibc-2.11.3
sys-libs/glibc-2.11.3
x11-libs/gtk+-2.22.1-r1
x11-libs/gtk+-2.22.1-r1


Several library files belonged to the same package, but we don't want to install them twice - of course. Moreover, maybe the listed package versions are not the newest ones, so we want to disregard version numbers. How do we achieve this?

I decided I only wanted the package names, liek 'glibc, freetype, libXt, gtk+". So, I'm going to fix this with a regular expression. First we have to decide what we're looking for, which means contemplating the format of the list given.

Basically, the format is "category-subcategory/package-version". We want to extract "package" only. Ditching "category-subcategory" is easy as shown above: We could use the 'cut' command or 'awk' with a slash / as the field separator. Then, we're left with "package-version". You might think "easy" I do another 'field ditching' using awk and using the dash as field separator', but that's too easy! Because there are packages like "qt-core-4.7.2". When using the first dash as field separator, the result will try to install 'qt' instead of 'qt-core'. Basically we have to search for a dash followed by a number and strip that part. We're going to use a simple regular expression to do so and instead of 'stripping' we're replacing the part which we do not want with the "empty string", meaning we're exchanging some characters for 'vacuum'.

So here we go:

4) Fixing the package list



$ awk -F / '{print $2}' packages.log|sed 's/-[0-9]+.*//'|sort|uniq > sorted_packages.log


We use awk with the field separator slash / and print the second column of the input-file packages.log. Then, we pipe | the output to sed. Sed receives the substitute command "s". On its turn, that command searches for a dash, followed by one or more "+" numbers. The plus sign means "one or more occurrences of what's before it ( 0-9, so "numbers" ), but please don't forget the backslash before the "+" sign, because it needs the escape character! Took me over an hour to figure that one out. Whatever is after the dash/number combination, we also want to delete. So if there's an occurrence of "-4.7.2-bladibla" we also want to delete ".7.2.bladibla". We do this by after the dash/number combination, matching anything which follows. The dot . matches any character, and the asterisk * says: "What's before it zero or more times". Zo, dot asterisk .* matches anything. We replace by the empty string, the empty string is "between the slashes" //.

Then we want to eliminate 'doubles'. Unix has an excellent command for this called "uniq", but it only works if it's input is alphabetically sorted! To do this, we use the "sort" command. Output is redirected to a file called "sorted_output.log".

We're done! So let's start installing.

5) Installing



We can do this using the same while/read loop shown before:

$ while read a; do sudo emerge -O ${a}; done < sorted_packages.log


Here we add the -O-switch meaning no-dependencies to emerge to make sure it doesn't install the same dependencies more than one time. For example ldd lists gcc as a dependency, but gcc may also be a dependency of qt-core. Please note, on a system which is not updated first (using emerge -DNu world or equivalent) this may pose the risk of updating a package but not the dependencies it relies upon.

That's it for now, I hope you learned something and if not at least you enjoyed!

  Nav
» Read more about: Story Type: LXer Features, Tutorial; Groups: Community, Debian, Gentoo, Mozilla

« Return to the newswire homepage

Subject Topic Starter Replies Views Last Post
Thank you kenholmz 2 2,680 Mar 30, 2011 6:04 PM

You cannot post until you login.