[ILUG] Slightly Off Topic : Concurrent Access to a Text File in a bash script - can I enforce thread safety

[Oisin Kim]

Thanks for the responses, I promise I'll update all with benchmarking
results for Pádraig's inode sorting suggestion.

I'll have a good read of the links Pádraig gave too and give me 2c on it.

Thanks all,

Cheers,
Oisin


On 7/13/07, Pádraig Brady <P at draigbrady.com> wrote:
>
> Efficiently checksumming files is something I've thought a bit about¹
> The biggest bottleneck I've found is disk head seeking, so to
> minimise that, the handiest thing I've found is to sort by inode
> (sorting by path is nearly as efficient). 1 modern CPU should be more
> than enough to checksum data as fast as most disks can throw at it.
>
> Also you do not want the overhead of starting a cksum process per file.
>
> As a first pass can you compare the running speed of the following:
>
> find . -maxdepth 1 -type f -printf "%i\t%f\n" |
> sort -k1,1n |
> cut -f2 |
> tr '\n' '\0' |
> xargs -r0 cksum
>
> Now for multiple spindles it would be worth having multiple
> checksum processes (especially if you have multiple CPUs).
> So to answer your original question, how do you syncronize
> writes to a single file in this case?
>
> Well when you open a file with O_APPEND set (as the shell
> does when you `>> file`), on each write, the file offset
> private to each process is automatically set to the current
> file size. All you have to worry about is that cksum does
> not write a partial line the whole way to the kernel
> before it scheduled. I think this is OK, but I leave it
> as an exercise for the reader to verify there are
> no issue with buffering²
>
> ¹ http://www.pixelbeat.org/fslint/
> ² http://www.pixelbeat.org/programming/stdio_buffering/
>