ls -l *.mkv *.avi *.mp4 The above command will give me errors if I happen to have no .avi files or no .mkv files. How can I write a command to give a list of all video files and not give an error if some don't exist? Preferably in bash, but I'm willing to try another shell if necessary. As an aside I'm not actually after running ls, the below is what I'm really trying to do: for n in *.mkv *.avi *.mp4 ; do -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On 17/09/2013, at 8:06 PM, Russell Coker <russell@coker.com.au> wrote:
How can I write a command to give a list of all video files and not give an error if some don't exist?
There are a few options here to try: http://stackoverflow.com/questions/1133698/find-name-pattern-that-matches-mu... Cheers, Avi
On 2013-09-17 20:06, Russell Coker wrote:
ls -l *.mkv *.avi *.mp4
The above command will give me errors if I happen to have no .avi files or no .mkv files.
How can I write a command to give a list of all video files and not give an error if some don't exist? Preferably in bash, but I'm willing to try another shell if necessary.
As an aside I'm not actually after running ls, the below is what I'm really trying to do:
for n in *.mkv *.avi *.mp4 ; do
You want nullglob: mattcen@isis:tmp$ touch a.mkv b.mp4 mattcen@isis:tmp$ ls *.mkv *.mp4 *.avi ls: cannot access *.avi: No such file or directory a.mkv b.mp4 mattcen@isis:tmp$ shopt -s nullglob mattcen@isis:tmp$ ls *.mkv *.mp4 *.avi a.mkv b.mp4 -- Regards, Matthew Cengia
Matthew Cengia <mattcen@gmail.com> wrote:
mattcen@isis:tmp$ shopt -s nullglob mattcen@isis:tmp$ ls *.mkv *.mp4 *.avi a.mkv b.mp4
Excellent. Note that the find command also returns no errors if nothing matches the search criteria. There's an option to prevent it from recursively descending subdirectories, which my quick search of the manual page didn't locate but which I'm confident exists.
On Tue, 17 Sep 2013, Jason White <jason@jasonjgw.net> wrote:
Matthew Cengia <mattcen@gmail.com> wrote:
mattcen@isis:tmp$ shopt -s nullglob mattcen@isis:tmp$ ls *.mkv *.mp4 *.avi a.mkv b.mp4
Excellent.
Yes, Matthew's suggestion works really well for interactive use. I couldn't work out how to make it run from a Makefile though.
Note that the find command also returns no errors if nothing matches the search criteria. There's an option to prevent it from recursively descending subdirectories, which my quick search of the manual page didn't locate but which I'm confident exists.
find . -maxdepth 1 -name "*.mkv" -o -name "*.avi" -o -name "*.mp4" Good idea, the above does what I need. Thanks for the suggestion. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On 17/09/13 20:42, Russell Coker wrote:
Yes, Matthew's suggestion works really well for interactive use. I couldn't work out how to make it run from a Makefile though.
Make initiates a new subshell for each command line, so putting the shopt on the same line as the command with the wildcards will fix this. Glenn -- sks-keyservers.net 0x6d656d65
On 2013-09-17 20:42, Russell Coker wrote:
On Tue, 17 Sep 2013, Jason White <jason@jasonjgw.net> wrote:
Matthew Cengia <mattcen@gmail.com> wrote:
mattcen@isis:tmp$ shopt -s nullglob mattcen@isis:tmp$ ls *.mkv *.mp4 *.avi a.mkv b.mp4
Excellent.
Yes, Matthew's suggestion works really well for interactive use. I couldn't work out how to make it run from a Makefile though.
GNU makefiles have a magical $(wildcard *.avi) function that I don't fully understand, but which may do what you want if you do some research into it. -- Regards, Matthew Cengia
On 17 September 2013 22:28, Matthew Cengia <mattcen@gmail.com> wrote:
On 2013-09-17 20:42, Russell Coker wrote:
On Tue, 17 Sep 2013, Jason White <jason@jasonjgw.net> wrote:
Matthew Cengia <mattcen@gmail.com> wrote:
mattcen@isis:tmp$ shopt -s nullglob mattcen@isis:tmp$ ls *.mkv *.mp4 *.avi a.mkv b.mp4
Excellent.
Yes, Matthew's suggestion works really well for interactive use. I couldn't work out how to make it run from a Makefile though.
GNU makefiles have a magical $(wildcard *.avi) function that I don't fully understand, but which may do what you want if you do some research into it.
Gnu makefile syntax might look like this if in the body of the makefile: found_files := $(wildcard *.mkv) $(wildcard *.mp4) $(wildcard *.avi) ifdef found_files some_command $(found_files) endif but this will break if the filenames contain whitespace, because make intrinsically treats whitespace as a separator and there's no way to avoid that using $(wildcard), or when using the make $(shell) function. So if the filenames might contain whitespace, don't even consider using these make functions, filenames with whitespace must be handled entirely in shell commands invoked by make, without ever trying to store them in make variables.
On 17 September 2013 23:36, David <bouncingcats@gmail.com> wrote:
found_files := $(wildcard *.mkv) $(wildcard *.mp4) $(wildcard *.avi) ifdef found_files some_command $(found_files) endif
After sending I realised this could work for filenames with whitespace, provided it is part of a make rule: found_files := $(wildcard *.mkv) $(wildcard *.mp4) $(wildcard *.avi) ifdef found_files <tab>some_command *.mkv *.mp4 *.avi endif It uses $(wildcard) to detect if any matching files exist, and then uses the shell to expand the wildcards in the command line. You might also want nullglob eg found_files := $(wildcard *.mkv) $(wildcard *.mp4) $(wildcard *.avi) ifdef found_files <tab>shopt -s nullglob ; some_command *.mkv *.mp4 *.avi endif but nullglob is not in Bourne shell so that might require telling make to use SHELL := bash The above will fail if the filename is entirely whitespace :) The easiest way is to avoid whitespace in filenames :)
On 2013-09-17 23:52, David wrote: [...]
After sending I realised this could work for filenames with whitespace, provided it is part of a make rule:
found_files := $(wildcard *.mkv) $(wildcard *.mp4) $(wildcard *.avi)
I'm pretty sure this is also valid: found_files := $(wildcard *.mkv *.mp4 *.avi) -- Regards, Matthew Cengia
On 18 September 2013 00:03, Matthew Cengia <mattcen@gmail.com> wrote:
On 2013-09-17 23:52, David wrote: [...]
After sending I realised this could work for filenames with whitespace, provided it is part of a make rule:
found_files := $(wildcard *.mkv) $(wildcard *.mp4) $(wildcard *.avi)
I'm pretty sure this is also valid:
found_files := $(wildcard *.mkv *.mp4 *.avi)
Yes! You are correct, I just tested and confirmed. Also, while various good points have been made in this thread, it's hard to be confident to give optimal assistance until the question is better defined. It would help to know the reason why a makefile is involved. It is hard to be precisely helpful and give a tested solution when the precise problem is unknown. Tricky aspects are: 1) Filename whitespace. 2) Shell glob options. 3) Makefile syntax. If filename whitespace is not an issue, either #2 or #3 should be sufficient to solve the problem. But if it is, I imagine 'find' would be a solution, it avoids the other issues provided one doesn't try to read its results into the makefile. But then, why use a makefile?
On Wed, Sep 18, 2013 at 12:39:45AM +1000, David wrote:
But then, why use a makefile?
because make is a good tool to use whenever you need to automate creating or updating one or more files that are dependent on other files being created or changed. it's not just for programming, it's a general purpose tool you can use to define relationships between particular files, or between *types* of files, and have certain actions taken based on changes in files. The most common use is to create object or executable files whenever one or more of the source files has changed or a new source file created and, *just as importantly*, do nothing if nothing has changed. another common use is to generate config files. e.g. automation of an apache include file to define virtual hosts with a script to do the work, a simple text file containing details (like domain name, ip address, directory, etc) and a Makefile to say that the include file is dependant on the script and the text file. running make will update the include file if either changes, or do nothing if neither has changed. the point is that the simple text file contains *only* the variable details in a very simple, easily edited and easily parsed format (e.g. one entry per line, tab or comma or colon delimited. or a .ini file). the script contains a template to generate the actual "<VirtualHost ...> ... </VirtualHost>" config fragment for each vhost. this same simple text file is also used by other scripts - e.g. to generate a web page listing all the virtual hosts with clickable links, to run a web log analyser on every vhost, or a link checker like webcheck, or a search engine like htdig. the simple text file is an easily parsed authoritative single-source of information about every vhost on the system. theoretically, you could parse the same information out of the apache config file itself, but it's much harder to do that. i've used this technique repeatedly over the years to automate many repetitive tasks (incl. apache config, mailing lists, bind zone files, virtual mail domains, MTA junk maps, spamassassin rules, bulk-creation of the current semester's student accounts on a HPC cluster, and more). anything that would otherwise be a copy/paste/edit job. automating it makes it simpler, faster, reduces typing, and eliminates most sources of human error. of course, this kind of automation doesn't require Make. you could do the same things entirely in shell or perl or python or whatever - but you'd have to implement your own method of detecting if any source files had changed or been created, and possibly your own language for defining relationship rules. craig -- craig sanders <cas@taz.net.au>
Craig Sanders <cas@taz.net.au> writes:
On Wed, Sep 18, 2013 at 12:39:45AM +1000, David wrote:
But then, why use a makefile?
because make is a good tool to use whenever you need to automate creating or updating one or more files that are dependent on other files being created or changed.
it's not just for programming, it's a general purpose tool you can use to define relationships between particular files, or between *types* of files, and have certain actions taken based on changes in files.
Once you're sitting down with a stiff drink, look at http://cyber.com.au/~twb/.bin/twb-get
of course, this kind of automation doesn't require Make. you could do the same things entirely in shell or perl or python or whatever - but you'd have to implement your own method of detecting if any source files had changed or been created, and possibly your own language for defining relationship rules.
Or just not care that you might be doing unnecessary work. :-) There is something to be said for the simplicity of "cc *.c".
On Wed, 18 Sep 2013, "Trent W. Buck" <trentbuck@gmail.com> wrote:
Or just not care that you might be doing unnecessary work. :-) There is something to be said for the simplicity of "cc *.c".
In this case converting a movie from one video format to another can take an hour, running such a conversion on all my movies and youtube videos would probably take both CPU cores of my server for several days. I really don't want to do it needlessly. In regard to David's question, files with spaces cause needless pain, I'll just rename them. I'll investigate using the wildcard directive in the Makefile. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Russell Coker wrote:
On Wed, 18 Sep 2013, "Trent W. Buck" <trentbuck@gmail.com> wrote:
Or just not care that you might be doing unnecessary work. :-) There is something to be said for the simplicity of "cc *.c".
In this case converting a movie from one video format to another can take an hour, running such a conversion on all my movies and youtube videos would probably take both CPU cores of my server for several days. I really don't want to do it needlessly.
Ah, sorry, I lost track of the original problem description.
On Thu, Sep 19, 2013 at 01:39:05PM +1000, Russell Coker wrote:
On Wed, 18 Sep 2013, "Trent W. Buck" <trentbuck@gmail.com> wrote:
Or just not care that you might be doing unnecessary work. :-) There is something to be said for the simplicity of "cc *.c".
In this case converting a movie from one video format to another can take an hour, running such a conversion on all my movies and youtube videos would probably take both CPU cores of my server for several days. I really don't want to do it needlessly.
i figure you know this but for other readers, i'll point out that another way to cater for this without using a Makefile is to move the original file somewhere else after it has been processed. e.g. i've used a ./originals/ or similarly-named subdirectory and mv the original file into it after processing. also makes it easy to delete them all later. alternatively, you can check to see if the target file already exists before doing the conversion (you'd have to manually delete the targets if you wanted to re-run the conversion with different options) craig -- craig sanders <cas@taz.net.au>
On 20/09/13 09:15, Rick Moen wrote:
Quoting Russell Coker (russell@coker.com.au):
In regard to David's question, files with spaces cause needless pain, I'll just rename them.
It's also useful to darwinate the people who create them.
That’s solving the problem at the wrong layer. As long as keyboards have space bars, people will attempt to create files with spaces in their filenames. If filenames are unacceptable, we should design our kernels and filesystems not to accept them. Good luck with that. (tm) Or, we could all just not be dicks, accept the fact that some filenames have spaces (because, shock horror, people *like* spaces), and deal with it like proper engineers should. Yeah, some problems are hard. I get that. That doesn’t impact on what should/shouldn’t be solved.
On 20.09.13 14:05, Jeremy Visser wrote:
Or, we could all just not be dicks, accept the fact that some filenames have spaces (because, shock horror, people *like* spaces), and deal with it like proper engineers should.
IME, when underscore is the *nix idiom for filenames_with_spaces, even the M$ fans can be amicably accommodated. (Just tell 'em it saves wear on their space bars.) It's just like taking your boots off before coming into the house avoids dog faeces on the carpet. That's a lot less effort than trying to deal with the pollution after it has occurred. So just naturalise any foreign filename. (Maybe running a rename on ~/Downloads/ would cover Russell's case.) Admittedly, having over more than a quarter of a century used¹ only unix variants, any solution I offer will naturally be unixversal. YMMV. Erik ¹ And my software teams were blessed with *nix servers all that time too, though many members used the other OS on their machines. A project often had thousands of files, with never a space character in them. -- Tecoma's Macca's-striking flash mob: " http://www.youtube.com/watch?v=H7-0T1vbnWE Stop fat food joint opposite Tecoma preschool: www.change.org Taking democracy (98,000 signatures) from Australia to Chicago: http://www.abc.net.au/news/2013-09-17/tecoma-residents-take-fight-against-mc...
On 20.09.13 19:18, Trent W. Buck wrote:
Erik Christiansen writes:
¹ And my software teams were blessed with *nix servers all that time too, though many members used the other OS on their machines.
VMS?
OK, once (in the early days) there was a third OS involved - one team member was putting Mac documentation files onto the *nix server via NFS. All those "resource" files, one for each data file, stuffed into a parallel ".resource" directory at each directory level, were something that surfaced when I ran an audit script. Rather than running around trying to catch every kind of cruft, I chose to keep it out, because there's enough productive work to do already. That said, I have once or twice used -print0 with find and xargs, but only when dealing with foreign material. And tabbing for filename completion in bash will escape the space characters, to allow the command line to work ... but that just confirms how alien such filenames are to a space delimited command line. The couple of curiously emotional posts on this thread, do seem to reveal a lack of scripting (and possibly even command line) experience, because having to quote "filenames with real space characters in them", in order to avoid the CLI barfing, tends to clarify that they are alien. The CLI cannot deal with them, and it is only our protective work-arounds which allow them to be tunnelled in a script. Those of us old enough to remember the days when "no spaces in filenames" was accepted *nix practice, remember them with fondness, but s/ /_/g will do me. Erik -- Whenever people agree with me I always feel I must be wrong. - Oscar Wilde
Erik Christiansen <dvalin@internode.on.net> wrote:
On 20.09.13 19:18, Trent W. Buck wrote:
Erik Christiansen writes:
¹ And my software teams were blessed with *nix servers all that time too, though many members used the other OS on their machines.
VMS?
I hope so, and preferably not a system running that kernel which some VMS developers later contributed to.
The couple of curiously emotional posts on this thread, do seem to reveal a lack of scripting (and possibly even command line) experience, because having to quote "filenames with real space characters in them", in order to avoid the CLI barfing, tends to clarify that they are alien. The CLI cannot deal with them, and it is only our protective work-arounds which allow them to be tunnelled in a script.
That's why I keep such file names off my machines, and I don't bother with the work-arounds in local scripts because the presence of any file name containing spaces on any of my machines is a problem to be fixed by appropriate renaming.
On Fri, 20 Sep 2013 08:24:38 PM Erik Christiansen wrote:
Rather than running around trying to catch every kind of cruft, I chose to keep it out, because there's enough productive work to do already.
I suspect the issue is that we are having a discussion between people who have sole control over their machines and people who run systems with external users and have to be careful to catch whatever odd (but legal) filenames they can come up with. ...and yes, that includes newlines in filenames. cheers! Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC This email may come with a PGP signature as a file. Do not panic. For more info see: http://en.wikipedia.org/wiki/OpenPGP
On Fri, Sep 20, 2013 at 07:12:30PM +1000, Erik Christiansen wrote:
On 20.09.13 14:05, Jeremy Visser wrote:
Or, we could all just not be dicks, accept the fact that some filenames have spaces (because, shock horror, people *like* spaces), and deal with it like proper engineers should.
IME, when underscore is the *nix idiom for filenames_with_spaces, even
underscores are worse than spaces. they require use of the shift key. personally, i use '-' and don't really care what other people use. craig -- craig sanders <cas@taz.net.au> BOFH excuse #212: Of course it doesn't work. We've performed a software upgrade.
On 18/09/2013 11:20 AM, "Trent W. Buck" <trentbuck@gmail.com> wrote:
Once you're sitting down with a stiff drink, look at http://cyber.com.au/~twb/.bin/twb-get
I should've heeded your advice on the stiff drink _before_ reading that, as I most certainly need one now; that is both horrible, and simultaneously brilliant! Well played sir.
On 18 September 2013 00:39, David <bouncingcats@gmail.com> wrote:
If filename whitespace is not an issue, either #2 or #3 should be sufficient to solve the problem. But if it is, I imagine 'find' would be a solution, it avoids the other issues provided one doesn't try to read its results into the makefile. But then, why use a makefile?
Responses upthread indicate I failed to express myself clearly. Where I wrote "But then, why use a makefile", it was not intended to convey "I have no clue what makefiles are for, please tell me", because I know 'make' very well. It was intended apply in the previous context of "But if it is [...] then why use a makefile". Sorry for not writing more clearly. Here is a rewrite of what I was trying to say there: Even if 'find' is used to find filenames containing whitespace, there is *no* way to use them as 'make' targets or prerequisites. Hence "why use a makefile"; ie. there's no point in trying to use 'make' with such files. Though I suppose it is possible that what 'find' finds could be written into a file that was managed by 'make'. But I will shut up now until Russell provides more info on exactly what he needs, or tells us that he has solved it himself.
Matthew Cengia <mattcen@gmail.com> writes:
On 2013-09-17 20:42, Russell Coker wrote:
On Tue, 17 Sep 2013, Jason White <jason@jasonjgw.net> wrote:
Matthew Cengia <mattcen@gmail.com> wrote:
mattcen@isis:tmp$ shopt -s nullglob mattcen@isis:tmp$ ls *.mkv *.mp4 *.avi a.mkv b.mp4
Excellent.
Yes, Matthew's suggestion works really well for interactive use. I couldn't work out how to make it run from a Makefile though.
nullglob is a bashism. gmake defaults to SHELL=/bin/sh.
GNU makefiles have a magical $(wildcard *.avi) function that I don't fully understand, but which may do what you want if you do some research into it.
RTFM; gmake $(wildcard) always behave like nullglob is on.
On 2013-09-17 20:26, Jason White wrote:
Matthew Cengia <mattcen@gmail.com> wrote:
mattcen@isis:tmp$ shopt -s nullglob mattcen@isis:tmp$ ls *.mkv *.mp4 *.avi a.mkv b.mp4
Excellent. Note that the find command also returns no errors if nothing matches the search criteria. There's an option to prevent it from recursively descending subdirectories, which my quick search of the manual page didn't locate but which I'm confident exists.
find . -maxdepth 1 -name '*.avi' -- Regards, Matthew Cengia
On Tue, September 17, 2013 8:06 pm, Russell Coker wrote:
ls -l *.mkv *.avi *.mp4
The above command will give me errors if I happen to have no .avi files or no .mkv files.
How can I write a command to give a list of all video files and not give an error if some don't exist? Preferably in bash, but I'm willing to try another shell if necessary.
You could always redirect the error message to /dev/null e.g., lev@racoon:~/Desktop$ ls *.mkv *.avi *.mp4 *.xxx 2> /dev/null 1.mkv 2.mkv 3.avi 4.avi 5.mp4 6.mp4 -- Lev Lafayette, BA (Hons), GCertPM, MBA mobile: 0432 255 208 RFC 1855 Netiquette Guidelines http://www.ietf.org/rfc/rfc1855.txt
On 2013-09-17 20:35, Lev Lafayette wrote: [...]
You could always redirect the error message to /dev/null e.g.,
lev@racoon:~/Desktop$ ls *.mkv *.avi *.mp4 *.xxx 2> /dev/null 1.mkv 2.mkv 3.avi 4.avi 5.mp4 6.mp4
Assuming you're happy dealing with a non-zero error code from ls: mattcen@adam:tmp$ ls *.c *.x ls: cannot access *.x: No such file or directory dh.c x.c mattcen@adam:tmp$ echo $? 2 -- Regards, Matthew Cengia
On Tue, Sep 17, 2013 at 08:35:16PM +1000, Lev Lafayette wrote:
On Tue, September 17, 2013 8:06 pm, Russell Coker wrote:
ls -l *.mkv *.avi *.mp4 ...
You could always redirect the error message to /dev/null e.g.,
lev@racoon:~/Desktop$ ls *.mkv *.avi *.mp4 *.xxx 2> /dev/null 1.mkv 2.mkv 3.avi 4.avi 5.mp4 6.mp4
Not if you're using globs directly though: > for n in *.mkv; do echo $n; done *.mkv Karl
participants (16)
-
Avi Miller -
Chris Samuel -
Craig Sanders -
David -
Erik Christiansen -
Glenn McIntosh -
Jason White -
Jeremy Visser -
Joel W Shea -
Karl Billeter -
Lev Lafayette -
Matthew Cengia -
Rick Moen -
Russell Coker -
Trent W. Buck -
trentbuck＠gmail.com