"gsutil rm" command using STDIN

Vote count:

0

I use gsutil in a Linux environment for managing files in GCS. I enjoy being able to use the command

gsutil -m cp -I gs://...

preceded by some other command to pass the STDIN to gsutil for uploading files; in doing so, I can maintain a local list of files that have been uploaded or generate specific patterns to upload and hand them off.

I would like to be able to do a similar command like

gsutil -m rm -I gs://...

to scrub files similarly. Presently, I build a big list of files to remove and run it with the following code:

while read line
do
gsutil rm gs://...
done < "$myfile.txt"

This is extraordinarily slow compared to the multithreaded "gsutil -m rm..." command, and enabling the -m flag has no effect when you have to process files one at a time from a list. I also experimented with just running

gsutil -m rm gs://.../* # remove everything
<my command> | gsutil -m cp -I gs://.../ # put back the pieces that I want

but this involves recopying a lot of a data and wastes a lot of time; the data is already there and just needs to have some removed. Any thoughts would be appreciated. Also, I don't have a lot of flexibility on either end with renaming files; otherwise, a quick rename before uploading would handle all of this.

asked 2 mins ago

user3461099

1