13 Dec
2013
13 Dec
'13
7:26 a.m.
git beginner here, so forgive me.
As part of change management of a couple hundred servers, we do a regular
'git add .' on a central repository of half a million files, followed by a
bit of munging, then a 'git commit -a -m ...' (rhel5, so 'git -a' is
'git
-A' elsewhere).
'git add .' is very very slow at finding the additions and modifications
(we don't care about the '-u' deletions at this stage, because of the
future munging required) and staging them. I suspect it's actually
staging every single file rather than just changes.
'git status -s' on a freshly changed repository prior to doing any git add
is really quite quick (no, it's not a cold or undersized cache issue), and
finds all additions, deletions and modifications. We could simply pipe
that rather small output to 'git add' and it would be much much quicker at
staging them (er, I think; but also, a bit of a kludge). Any known reason
why git-add would appear to be recursing through the entire tree staging
even unchanged files, and not just acting on the changed files that
git-status obviously can find very quickly? Any missing bit of git magic
I could be applying?
--
Tim Connors
13 Dec
13 Dec
7:47 a.m.
Try this?
git add -i
u
1-$max
a
1-$max
q
git commit -m ...
(where $max is the maximum number printed in the list it just gave you prior)
On 13 December 2013 18:26, Tim Connors <tconnors(a)rather.puzzling.org> wrote:
git beginner here, so forgive me.
As part of change management of a couple hundred servers, we do a regular
'git add .' on a central repository of half a million files, followed by a
bit of munging, then a 'git commit -a -m ...' (rhel5, so 'git -a' is
'git
-A' elsewhere).
'git add .' is very very slow at finding the additions and modifications
(we don't care about the '-u' deletions at this stage, because of the
future munging required) and staging them. I suspect it's actually
staging every single file rather than just changes.
'git status -s' on a freshly changed repository prior to doing any git add
is really quite quick (no, it's not a cold or undersized cache issue), and
finds all additions, deletions and modifications. We could simply pipe
that rather small output to 'git add' and it would be much much quicker at
staging them (er, I think; but also, a bit of a kludge). Any known reason
why git-add would appear to be recursing through the entire tree staging
even unchanged files, and not just acting on the changed files that
git-status obviously can find very quickly? Any missing bit of git magic
I could be applying?
--
Tim Connors
_______________________________________________
luv-main mailing list
luv-main(a)luv.asn.au
http://lists.luv.asn.au/listinfo/luv-main
--
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world
2:29 p.m.
On 2013-12-13 18:26, Tim Connors wrote:
git beginner here, so forgive me.
As part of change management of a couple hundred servers, we do a regular
'git add .' on a central repository of half a million files, followed by a
bit of munging, then a 'git commit -a -m ...' (rhel5, so 'git -a' is
'git
-A' elsewhere).
'git add .' is very very slow at finding the additions and modifications
(we don't care about the '-u' deletions at this stage, because of the
future munging required) and staging them. I suspect it's actually
staging every single file rather than just changes.
'git status -s' on a freshly changed repository prior to doing any git add
is really quite quick (no, it's not a cold or undersized cache issue), and
finds all additions, deletions and modifications. We could simply pipe
that rather small output to 'git add' and it would be much much quicker at
staging them (er, I think; but also, a bit of a kludge). Any known reason
why git-add would appear to be recursing through the entire tree staging
even unchanged files, and not just acting on the changed files that
git-status obviously can find very quickly? Any missing bit of git magic
I could be applying?
Hi Tim,
My first bet would be write load vs read load. git add has to not only
check the hashes of each file, but for each new file, after it hashes
it, it has to add it to its object database. git status, however, needs
to do zero writing.
I find particularly when using git over NFS, that the add is far slower
than the status.
You could always do an strace to find out what it's doing in more
detail.
Keep in mind that by its very nature, git would not stage unchanged
files, because it would hash the file, determine immediately via hash
table that the hash already existed in the object store, and not bother
to store it again. It *does* have to go through the entire process of
*calculating* the hash for each file every time though, as far as I
know.
Hope this helps.
--
Regards,
Matthew Cengia
10:37 p.m.
Matthew Cengia <mattcen(a)gmail.com> wrote:
Keep in mind that by its very nature, git would not
stage unchanged
files, because it would hash the file, determine immediately via hash
table that the hash already existed in the object store, and not bother
to store it again. It *does* have to go through the entire process of
*calculating* the hash for each file every time though, as far as I
know.
does it exclude unchanged files based on the last modification time? If so, it
only needs to look at the directory entries. It already has the date/time of
the commit relative to which changes are to be calculated.
11:39 p.m.
On 2013-12-14 09:37, Jason White wrote:
Matthew Cengia <mattcen(a)gmail.com> wrote:
No, it does not check the datestamp, and nor should it; git doesn't
track the modification dates of individulal files, it only cares about
the date of commits, and heavy-weight tags. If it checked the date, the
below example would allow me to trick git, and the user, into thinking
the repo was unchanged, when in fact I'd changed the entire contents of
file 'bar'. When it comes to files, git only cares about one thing: The
sha1sum of the file. Elegant in its simplicity, even if it does take a
bit longer to read data in order to verify its integrity.
mattcen@owen:tmp$ git init
Initialized empty Git repository in /tmp/tmp/.git/
mattcen@owen:tmp(master)$ echo foo > bar
mattcen@owen:tmp(master)$ git add bar
mattcen@owen:tmp(master)$ git commit -m "initial commit"
[master (root-commit) bd218a4] initial commit
1 file changed, 1 insertion(+)
create mode 100644 bar
mattcen@owen:tmp(master)$ cp -a bar baz
mattcen@owen:tmp(master)$ echo quz > bar
mattcen@owen:tmp(master)$ touch -r baz bar
mattcen@owen:tmp(master)$ ls -l --time-style=full-iso
total 8
-rw-rw-r-- 1 mattcen mattcen 4 2013-12-14 10:21:07.000000000 +1100 bar
-rw-rw-r-- 1 mattcen mattcen 4 2013-12-14 10:21:07.000000000 +1100 baz
mattcen@owen:tmp(master)$ git status
# On branch master
# Changes not staged for commit:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working
directory)
#
# modified: bar
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# baz
no changes added to commit (use "git add" and/or "git commit -a")
--
Regards,
Matthew Cengia
Keep in mind that by its very nature, git would
not stage unchanged
files, because it would hash the file, determine immediately via hash
table that the hash already existed in the object store, and not bother
to store it again. It *does* have to go through the entire process of
*calculating* the hash for each file every time though, as far as I
know.
does it exclude unchanged files based on the last modification time? If so, it
only needs to look at the directory entries. It already has the date/time of
the commit relative to which changes are to be calculated.
14 Dec
14 Dec
4:43 a.m.
Matthew Cengia <mattcen(a)gmail.com> wrote:
On 2013-12-14 09:37, Jason White wrote:
Yes, of course it's possible to change the date stamp of files and avoid the
detection of modifications, when date stamps are relied on. However, it's
obviously more efficient to check date stamps than to compute hashes of file
contents, users generally don't change the former, and a version control
system isn't a security tool designed to resist unexpected user behaviour. As
I remember, cvs and svn both use date stamps. It's interesting that Git has
opted for a different trade-off in this case, a reasonable decision to make,
arguably, but one which has its efficiency costs.
does it exclude unchanged files based on the last
modification time? If so, it
only needs to look at the directory entries. It already has the date/time of
the commit relative to which changes are to be calculated.
No, it does not check the datestamp, and nor should it; git doesn't
track the modification dates of individulal files, it only cares about
the date of commits, and heavy-weight tags. If it checked the date, the
below example would allow me to trick git, and the user, into thinking
the repo was unchanged, when in fact I'd changed the entire contents of
file 'bar'. When it comes to files, git only cares about one thing: The
sha1sum of the file. Elegant in its simplicity, even if it does take a
bit longer to read data in order to verify its integrity.
6:02 a.m.
On 2013-12-14 15:43, Jason White wrote:
[...]
contents, users generally don't change the former,
and a version control
system isn't a security tool designed to resist unexpected user behaviour. As
I disagree. Assuming you trust SHA1 (which is getting a bit long in the
tooth), Git has end-to-end security: You can sign a tag with a GPG key
and that tag points to a commit which can't be modified without changing
its hash. The hash of the commit is dependent on all previous commits,
so you can't change any of the previous commits either, without
invalidating the signature. If it were possible for a user to clone a
git repo and then have somebody edit a file in the working tree while
maintaining the datestamp, git should be able to detect that, otherwise
the entire security model breaks down.
--
Regards,
Matthew Cengia
16 Dec
16 Dec
1:02 a.m.
Matthew Cengia <mattcen(a)gmail.com> writes:
On 2013-12-14 15:43, Jason White wrote:
[...]
See also https://en.wikipedia.org/wiki/Monotone_(software)
contents, users generally don't change the
former, and a version
control system isn't a security tool designed to resist unexpected
user behaviour. As
I disagree. Assuming you trust SHA1 (which is getting a bit long in
the tooth), Git has end-to-end security: You can sign a tag with a GPG
key and that tag points to a commit which can't be modified without
changing its hash. The hash of the commit is dependent on all previous
commits, so you can't change any of the previous commits either,
without invalidating the signature. If it were possible for a user to
clone a git repo and then have somebody edit a file in the working
tree while maintaining the datestamp, git should be able to detect
that, otherwise the entire security model breaks down. 
12:53 a.m.
On Sat, Dec 14, 2013 at 03:43:53PM +1100, Jason White wrote:
...
Yes, of course it's possible to change the date
stamp of files and avoid the
detection of modifications, when date stamps are relied on. However, it's
obviously more efficient to check date stamps than to compute hashes of file
contents, users generally don't change the former, and a version control
system isn't a security tool designed to resist unexpected user behaviour. As
I remember, cvs and svn both use date stamps. It's interesting that Git has
opted for a different trade-off in this case, a reasonable decision to make,
arguably, but one which has its efficiency costs.
Don't forget, git is distributed. Neither cvs nor svn is by nature.
Karl
3122
days inactive
3125
days old
8 comments
6 participants
participants (6)
-
Jason White -
Karl Billeter -
Matthew Cengia -
Tim Connors -
Toby Corkindale -
trentbuck＠gmail.com