Calling a bash function with find

[TimO]

This one is annoying me, but I'm not sure that there's an easy solution.  I'm trying to call a function in a bash script from a find command in the same script eg:

Code: [Select]

#!/bin/bash

function xxx
{
        echo $1
}

find . -exec xxx {} \;
Obviously this is a trivial example, and more easily achieved with other scripts.  The actual function is more complex and moves a file between servers, doing an MD5 check before deleting the original.  It's useful to have everything in the one script (rather than just call the function as a script) since there are some globals which I would otherwise have to pass across using something like environment variables.

There are ways to solve this, it's just irritating me that there must be a way to get find to call a function...?

[bobajobrob]

Stick the function in a different script and tell find to run that. You could separate the function into new file and use "source" on it if you don't want to duplicate it.

[Greenbank]

Code: [Select]

#!/bin/bash

function xxx
{
        echo $1
}

for i in `find .` do
        xxx $i
done


[TimO]

Stick the function in a different script and tell find to run that. You could separate the function into new file and use "source" on it if you don't want to duplicate it.

I know I can do it that way, I'll also have to pass some globals out as environment variables, but I was just trying to think of a way to do this in the script, since there ought to be some way to do it.  More annoying that it won't let me do this than anything else!

Code: [Select]

#!/bin/bash

function xxx
{
        echo $1
}

for i in `find .` do
        xxx $i
done


I don't think this will work, since the number of files in my find is actually huge, and I think the for will barf on it.

[Maladict]

You're trying to apply a function to each file that matches the find expression; why not do something like:

#!/usr/local/bin/bash
for i in $( find . -print ); do
    echo item: $i
done

[TimO]

Arse, I'm sure I originally had a similar loop with a find statement, and it barfed on a large directory, but I just created a directory with 60000 files in it, and the loop works fine...

[Maladict]

It works for at least 23,917 items.  Bear in mind it doesn't cope too well with spaces in filenames (spaces in filenames are of course evil).

[TimO]

On reflection I may be misremembering it failing because it was just using wildcard expansion rather than a find statement, which of course does go castors up very easily with only a few files. Doh!

There are no spaces in the filenames, they are suitably cryptic names like this:

C1_CP_FGM_5VPS__20071022_083109_20071024_173843_V01.cef.gz

None of your friendly filenames here!

[philip]

There are ways to solve this, it's just irritating me that there must be a way to get find to call a function...?

find knows nothing about bash so it cannot exec a bash function.  You need to exec bash itself, having first exported the function so that it is visible in subshells:

Code: [Select]

#!/bin/bash

function xxx ()
{
  echo $1
}

export -f xxx

find . -maxdepth 1 -exec bash -c "xxx {}" \;

[TimO]

The script seems to be working, so I'm off home for my tea now.  It'll be a while finishing, since it needs to transfer 14GB over a not terribly fast link!

Cheers all for putting me right.

[TimO]

There are ways to solve this, it's just irritating me that there must be a way to get find to call a function...?

find knows nothing about bash so it cannot exec a bash function.  You need to exec bash itself, having first exported the function so that it is visible in subshells:

Code: [Select]

#!/bin/bash

function xxx ()
{
  echo $1
}

export -f xxx

find . -maxdepth 1 -exec bash -c "xxx {}" \;

Smacks head.  I knew there was a way to do it!  You win.

[Maladict]

Would never occur to me to do that but I rarely write bash scripts.

Probably rather less efficient to exec bash once per item?

[Greenbank]

Probably rather less efficient to exec bash once per item?

Yes. horribly inefficient. If you're doing that then you may aswell have foo.sh contain:-

Code: [Select]

#!/bin/bash

function xxx ()
{
  echo $1
}

and do:-

find . -maxdepth 1 | sed -e 's/^/xxx /' >> foo.sh ; ./foo.sh

[TimO]

Efficiency wouldn't impact this too much, the function has calls to SSH a few times to copy a file (which can be anything up to 100MBytes) and do a remote MD5 on the copied file.  At the moment there are about 30 seconds between each function call.

[David Martin]

Arse, I'm sure I originally had a similar loop with a find statement, and it barfed on a large directory, but I just created a directory with 60000 files in it, and the loop works fine...

It works on a half million files as well. The loop streams the results from the backticks rather than loading them into a single variable (or some such).

..d

[philip]

It works on a half million files as well. The loop streams the results from the backticks rather than loading them into a single variable (or some such).

I'm skeptical, is that documented? If I try

Code: [Select]

$ for i in `seq 1234567`;do true;donethe bash process first grows to over 100MB and then appears to process the list; I think the entire output of the backticks gets held in memory. Now 100MB is not usually a problem on a modern machine, but the exec approach avoids the issue.

find . -maxdepth 1 | sed -e 's/^/xxx /' >> foo.sh ; ./foo.sh

That also avoids any memory issue, but now you have to consider the security implications of using a temporary file.

My advice in this situation would be to use the form that you find easiest to understand/maintain and only worry about efficiency if it proves to be too slow.

[Greenbank]

find . -maxdepth 1 | sed -e 's/^/xxx /' >> foo.sh ; ./foo.sh

That also avoids any memory issue, but now you have to consider the security implications of using a temporary file.

find . -maxdepth 1 | sed -e 's/^/xxx /' >> foo.sh ; ./foo.sh & sleep 5; rm foo.sh

[philip]

find . -maxdepth 1 | sed -e 's/^/xxx /' >> foo.sh ; ./foo.sh

That also avoids any memory issue, but now you have to consider the security implications of using a temporary file.

find . -maxdepth 1 | sed -e 's/^/xxx /' >> foo.sh ; ./foo.sh & sleep 5; rm foo.sh

Am I missing a joke?  You need to ensure that the temporary file gets written into a directory where it cannot be modified or deleted.  Now that might be straightforward but experience shows that programmers often get it wrong.

For example: Melanie the malicious developer moves or deletes foo.sh while the find is running (that's fine the file is already open so find will continue to write to it) and replaces it with a new foo.sh containing whatever arbitrary commands she wants you to execute. When the find completes the hostile foo.sh gets executed.

[Greenbank]

Am I missing a joke?

I'd assumed you were complaining about the leftover foo.sh file, and/or that could also be read whilst it is still being run.

You need to ensure that the temporary file gets written into a directory where it cannot be modified or deleted.

[lager powered rubbish deleted]

For example: Melanie the malicious developer moves or deletes foo.sh while the find is running (that's fine the file is already open so find will continue to write to it) and replaces it with a new foo.sh containing whatever arbitrary commands she wants you to execute. When the find completes the hostile foo.sh gets executed.

How does that differ from Melanie removing/replacing the self-contained script that you are running in the first place?

[philip]

How does that differ from Melanie removing/replacing the self-contained script that you are running in the first place?

That is another thing one has to consider.  A script will often be installed in a person's home directory or a system directory and that is usually safe.  Temporary files often get written somewhere else (the directory where the script is run, or $TMPDIR) and that makes them more vulnerable.  Avoiding problems may be trivial or it may require a lot of care. Look at the security notices for your favourite Linux distribution and you will see that insecure temporary file handling crops up repeatedly; experience shows that programmers get it wrong in all sorts of programs.

[Greenbank]

How does that differ from Melanie removing/replacing the self-contained script that you are running in the first place?

That is another thing one has to consider.  A script will often be installed in a person's home directory or a system directory and that is usually safe.  Temporary files often get written somewhere else (the directory where the script is run, or $TMPDIR) and that makes them more vulnerable.

Valid points, but irrelevant since my suggested commandline doesn't use temporary files. It is a strange UNIX straw man you are constructing here.

[philip]

Valid points, but irrelevant since my suggested commandline doesn't use temporary files. It is a strange UNIX straw man you are constructing here.

Huh?  One of us is having difficulty following this discussion, is it me or you?  If I try to backtrack through the thread I think we are discussing a command using foo.sh as a temporary file; the 9:32, 11:20, 11:25 and 11:44 messages all contain such a command. I accept that your very first suggestion, at 8:31, doesn't use temporary files but I never brought up security as a consideration for that command.

[Greenbank]

Huh?  One of us is having difficulty following this discussion, is it me or you?  If I try to backtrack through the thread I think we are discussing a command using foo.sh as a temporary file;

You've assumed foo.sh is a temporary script. It was my invented name of the OP's script (because this wasn't specified by the OP) that contains the function xxx that the OP is wanting to use. No new files (temporary or otherwise) need to be created.

If the OP's script is in a location where it can be modified/removed/replaced by someone else then that's a problem, but it affects all of the proposed solutions, not just mine, as they are all as susceptible to modification/replacement.

And yes, with that solution you've got to clear it out if you want to run it again on another set of files, but that's not exactly tricky. And it was only a suggestion in reply to the point about forking/execing bash for each file.

If you're going to go down the security route then I'd concentrate more on the binaries that are specified within these scripts without full paths which opens the door for PATH and alias style attacks.