up vote 3 down vote favorite

I want to resize a bunch of images (around 1 million) using imageMagick convert function. I am using Ubuntu 16.04

This is taking longer than I thought it would (been running about 5 hours - I can give it another 5, but then I want to stop it)

My question is: If I stop the program, will I have to worry about corrupt images? Or will the program be stopped in a safe way?

This is my script:

find . -name "*.JPEG" | xargs -I {} convert {} -resize "256^>" {}

(Basically I am resizing the imagenet dataset)

Would pressing Ctrl-C corrupt some images?

share|improve this question
1  
Since you're overwriting the original image, you should use mogrify instead of convert. – Yay295 4 hours ago
up vote 3 down vote accepted

In theory, no, it shouldn't result in corrupted images. From memory, I think imagemagick works with temporary files, which are renamed to the chosen destination file once processing is done.

BTW converting images the way you did is a one-way operation, I guess you've noticed that, hence your question, I guess.

To find which image was converted last, list files in chronological order, newer files first. If there's a noticeable difference in file size you'll know easily where to start from next time. Otherwise you'll have to retrieve image dimensions using imagemagick's identify.

Also note that for one million images, if each image is processed in one second that would still require about 280 hours to complete! I hope you have a fast, powerful machine...

EDIT: If you want to show some progress, here's what I can think of, based on Martin's answer:

mkdir -p thumbs; \
COUNT=$(find -name "*.JPEG" | wc -l); \
find -name "*.JPEG" | while read IMG; do \
    printf "\n$(( ++i )) / $COUNT\n" && \
    [ -s "thumbs/${IMG%.JPEG}-small.jpg" ] || \
    convert "$IMG" -resize "256^>" "thumbs/${IMG%.JPEG}-small.jpg"; \
done

What this does:

  1. create the thumbnail directory...
  2. count the total number of .JPEG images...
  3. for each image...
  4. print the current image file index vs total...
  5. if the image has not yet been converted...
  6. make a thumbnail of it.

Assuming all the images to resize are in the same directory, I've used a subdirectory to save the thumbnail images to avoid cluttering the main directory with the corresponding thumbnails. Otherwise you can remove the first line and the thumbs/ subdirectory from the 5th and 6th lines.

If images are spread into subdirectories:

COUNT=$(find -name "*.JPEG" | wc -l); \
find -name "*.JPEG" | while read IMG; do \
    printf "\n$(( ++i )) / $COUNT\n" && \
    [ -s "${IMG%.JPEG}-small.jpg" ] || \
    convert "$IMG" -resize "256^>" "${IMG%.JPEG}-small.jpg"; \
done

Note that this script accounts for spaces in file names as read stops at a new line character.

share|improve this answer
    
Thank you very much! This is what I was hoping for :) It's not my machine, it's a server my university gave me, so it's reasonably powerful. Not sure if it is that fast though :) Could you give me an example of how to check progress? I am not an expert of bash :/ – Ant 10 hours ago
    
@Ant I have updated my answer accordingly. – Nasha 9 hours ago
    
Thank you very much, that's helpful :) What I meant though was how to check the current status of the job - I guess I should look through all the images and see how many have been resized. Do you think this will be fast enough to be usable? Do you have ideas on how best to achieve it? – Ant 8 hours ago
    
I hope you have a fast, powerful machine... you would also have to use parallel or equivalent, because convert seems to run on a single core when it processes a file. – xenoid 8 hours ago
    
@xenoid Yes, I think you're right. I usually use GPUs for this kind of conversion and it's very fast (well under 1 hour). I did not think that with that command I was using a single core of the CPU, which is understandably much slower – Ant 8 hours ago
up vote 3 down vote

It is not a good idea to combine find and xargs in this way. Filenames with spaces and other characters that have special meaning on the command line can mess things up. It is better to use

find . -name '.JPEG' -print -exec convert '{}' -resize "256^>" '{}-small.jpg' \;

In your command it seems that you overwrite the images, is that correct? The problem is that convert is not necessarily an atomic operation. It would be much better if you had done this:

shopt -s extglob
for i in **/*.jpg; do
  convert "$i" -resize "256^>" tmp.jpg
  mv tmp.jpg "$i"
done

The mv command most likely is atomic. If you interrupt this anywhere, the only damage could be a stray tmp.jpg somewhere.

If you press CtrlC now, you will likey cancel a convert command. If that catches the interrupt signal, it might finish writing the image, but it may as well stop writing.

You might be able to move the whole directory structure somewhere else, if you are lucky, the writing will finish but any new convert calls will fail.

Why don't you try to check how far the process went? find does not give ordered output, so you would have to scan the sizes of the images in your directories and check how many are already resizes.

share|improve this answer
    
Thank you for the very informative answer! I will keep your suggestions in mind next time :) For the progress checking, could you suggest a way to do this? I have never used bash so I'm not really sure how I would go to implement it – Ant 9 hours ago
    
Or find -print0 and xargs -0 (from memory) – abligh 5 hours ago
up vote 2 down vote

As I understand it (and I am damned sure enough people will chip in to put me right if there is any error) Ctrl+C sends SIGINT to bash and to any scripts and child processes/threads running.

Bash breaks the script at the next point at which it regains any control (like the next iteration of a loop, or when a call is made through a pipe), other processes are notified of the signal and exit according to their own strategy, if they have one.

Whether there is any real risk of data loss depends on how gracefully convert catches and handles the SIGINT. A well written piece of code (and convert has been around for a while) would handle this gracefully and exit without damage.

ps -eT | grep -i convert

should show you how many threads are running and how many images are at 'risk', though given that the write cycle is considerably shorter than the processing time for each image, you would be unlucky to damage more than one or two images even if convert simply dropped everything when the signal landed.

share|improve this answer
    
Thank you for your very nice answer! Very clear :) – Ant 9 hours ago
1  
@Ant to be honest I hadn't considered the exact process Ctrl+C precipitated until you asked the question. The answer was the result of half an hour's reading and I consider myself slightly better educated. Thank you for asking the question. – bu5hman 9 hours ago

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.