- luv-main - luv.asn.au

Re: btrfs :(
by Peter Ross 10 Oct '14

10 Oct '14

From: "James Harper" >> > Rsync integration >> > >> > Not claimed - no patches yet - Not in kernel yet >> > >> > Now that we have code to efficiently find newly updated files, we need >> to >> > tie it into tools such as rsync and dirvish. (For bonus points, we can >> > even allow rsync to use btrfs's builtin checksums and, when a file has >> > changed, tell rsync _which blocks_ inside that file have changed. >> Would >> > need to work with the rsync developers on that one.) >> > >> > Update rsync to preserve NOCOW file status. >> >> Means: Make rsync work like btrfs send/receive;-) and put filesystem >> specific code in it. > > I'm just testing out some of the deduplication stuff in btrfs, and was > actually a little shocked to find it calculating the hashes itself. btrfs > already has checksums, and if nothing else it could have used them to > trivially reject blocks that are different before calculating a stronger > hash. There is talk about exposing the btrfs checksums to userspace, but > of course that puts contraints on further development as they now have to > consider userspace compatibility. ZFS has feature flags you can test. >> I am not sure whether this is a great idea. >> >> Most of the time you will have the same filesystem on both ends. Then >> you >> can use zfs/btrfs etc. tools. Or rsync if it's not a COW system. >> >> It is more "code polluting" than it's worth I think. > > Maybe. Depends on the speedup. In a lot of cases, the above optimisations > would speed up the processing that rsync has to do, but if 90% of the time > taken in your rsync was actually moving data then you're never going to > get anymore than 10% faster. For LAN links though, I normally just use -W > for rsync because computing changes just adds overhead (I mean you have to > read the file at both ends anyway, and unless your disk can pull data > faster than 1GByte/second you're not going to saturate your 10GBit /second > link so don't bother computing changes. If you got the change computation > "for free", then it's a big win. Yes, that's what you get from btrfs/zfs send/receive:-) You only need rsync if you have to deal with a different filesystem at the other end and then it may help you only in a few percent of the cases. If both cases have a 10% likelihood you start this "layer blurring" coding for 1% of useful cases. >> And if btrfs send/receive isn't stable there is a good chance to >> implement an unstable rsync as well. >> > > (I think) Russell was supposing that there weren't many bugs reported for > send/receive because not many people were using it. I'm not sure how we > got from there to "send/receive isn't stable". >From Chris Samuel today: > Watch out with 3.17 then, there are early reports that it's broken btrfs > send. That may need a bit more evaluation, maybe. Overall I hear a lot of "btrfs is having this or that issues" which I never followed up properly. But it makes me wary. I am using ZFS on FreeBSD for at least 3 years in production now (and upgrade the kernel regularly when needed) without any issues. Besides of early problems with nearly full ZFS and memory which were known. On both fronts there seems to be improvement. E.g. I have 92% full ZFS volumes which are still performing - there was a 80% "warning" in the past, and I run ZFS on 3GB and 4GB RAM boxes and do not see issues. Regards Peter

10 45

Re: The rise or fall of systemd
by Peter Ross 10 Oct '14

10 Oct '14

Hi all, to start with Douglas Adams: "There is a theory which states that if ever anyone discovers what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this already happened." I have this sometimes in my mind if I look at Linux related development. Especially in the area of desktops, btw. Everybody knows that Unix is around for so long because it suffers the "good enough" syndrome. What are the killer features that make systemd so much appealing to change? "Making a boot a bit faster" isn't good enough for such an intrusive change, I would think. What else does it do better? Why do I need a journald (mandatory) if I have already a bunch of syslog services developed over many years? Are there binary logs which are hard to read if things go really really wrong? (That really would be a show stopper for me.) What exactly is the network demon used for? Does it work nicely with approaches as OS virtualisation (e.g. Linux containers with own network stack) I have simply problems to understand motivation and design. It seems to be more than just init. Which "other parts" will it take over? Does it "eat into he kernel" finally and makes Linux a microkernel? Or does it replace traditional userland applications (e.g. syslog) and makes the systemd equivalents mandatory even sometimes unintended? (Let's say: the modern Linux desktop is Gnome, Gnome packages require systemd, systemd includes journald)? BTW: Making parts of a system less exchangeable does not only hurts porting, it creates "single point of failures" if there is a problem. Think of bash recently. You cannot replace it if your scripts have a lot of "bashisms" in it. If you do not have a fix you are stuffed. If FreeBSDs sh is vulnerable I can simply replace it with bash (e.g.) If rsyslogd is vulnerable and there is no fix: Just replace it with syslogd. If systemd requires journald and there is an issue: good luck! Not having a choice makes a system less modular and flexible. This looks very much "Un-Unix-like" to me. Obviously Solaris SMF is there for a while. Being a commercial product it just arrived and was not discussed widely in public. Admins looked, were confused and .. just went on with it. For me it is between "A bit annoying" and "This works quite well"! I did not do too much "cool stuff" to appreciate it. So I do not have a definite well-informed opinion about it. I used Ubuntu for a few years with upstart and was mildly annoyed. But I did not explore it too much. It was just my desktop and it's mostly web browser and terminal multiplexer. No point in getting religious about it. It is a different story if I have to deploy and administrate Unix/Linux servers. To be honest, if I can I choose FreeBSD. If I need commercial support it more or less dictates Red Hat so Debian will not be my choice. Well, maybe I "find" somewhere Debian servers to look after. But it dd not happen to me for more than ten years now I think. But it is more or less likely that I have to deal with systemd in the future, Debian or Red Hat. What are the shortcomings of the existing GNU/Linux framework so it has to be replaced by systemd? I see a lot of cons (Maybe some only because I am not well-informed? Please correct me!) and not many pros. Regards Peter

5 6

systemd vs inittab
by James Harper 10 Oct '14

10 Oct '14

Some real world feedback... just now I am trying to fix a bug in my dual tuner video card where using both tuners at once causes problems. One of my diagnostics caused a crash I wanted to capture so I wanted to set up serial console (via IPMI LAN). Having added console=ttyS0 to the Linux command line I started googling to see what I needed to do to get a login prompt. Pre-systemd it would involve changing inittab which I always hate doing because the next update that comes along might also want to modify inittab. It turns out that what I had to do under systemd is nothing. It automatically starts a login prompt on any terminal designated as system console. Now that may not be what everyone wants, and I don't know if turning it off is easy or not, but it's what I wanted, so I'm happy. James

1 0

Re: [luv-talk] All your base are belong to us
by Peter Ross 10 Oct '14

10 Oct '14

Peter Ross wrote: > From: "Steve Roylance" <roylance(a)corplink.com.au> >> >> from G+ this morning >> -------------------------------- >> Lennart Poettering >> Shared publicly - 04:39 >> >> German language source code comments are not acceptable in the +systemd >> source tree, according to our coding style conventions. In preparation >> for merging LibreOffice into systemd I have thus started translating a >> few of their source code comments from German into English. >> -------------------------------- > > Maybe we all got it wrong ans systemd is a distribution slightly more > religiously motivated than others > > and not an init replacement. The idea behind process supervision is not a bad one as such: --- http://en.wikipedia.org/wiki/Process_supervision Benefits[1] compared to traditional process launchers and system boot mechanisms, like System V init, include: - Ability to restart services which have failed - The fact that it does not require the use of "pidfiles" - Clean process state - Reliable logging, because the master process can capture the stdout/stderr of the service process and route it to a log - Faster (concurrent) and ability to start up and stop --- They all make sense. The init process has some tools in place to aide it but it is true: You rely on the quality of init scripts and started tools. E.g. the ability to log to syslog. Here from http://0pointer.de/blog/projects/the-biggest-myths.html --- Myth: systemd is a feature creep. Well, systemd certainly covers more ground that it used to. It's not just an init system anymore, but the basic userspace building block to build an OS from, but we carefully make sure to keep most of the features optional. You can turn a lot off at compile time, and even more at runtime. Thus you can choose freely how much feature creeping you want. --- The question here is: Why do they make some functions part of the package in the first place? It reminds me of some consultants recently: "You want our software on Linux? Okay, choose Oracle as a DB. We can help you with the licensing etc." After signing the contract: "No, do not use Linux. We rather like it all with MS SQL Server because we are used to it. So, you have two very different server systems then? Well, just replace all your Linux/Unix machines with MS Windows Server and you are fine." The approach fosters the belief that you may have a hard time to choose in the future because the people responsible for a critical part of the software may have a "my way or the highway" attitude at heart which makes it difficult to deal with. Regards Peter

1 0

Re: The rise or fall of systemd
by Peter Ross 10 Oct '14

10 Oct '14

From: "Brian May" <brian(a)microcomaustralia.com.au> > On 10 October 2014 10:31, Peter Ross < Petros.Listig(a)fdrive.com.au > > wrote: > What else does it do better? > It *can* replace annoying init.d scripts, that are deceptively hard to > write, often are very buggy and suffer from various race conditions. FreeBSD's /etc/rc.subr? https://svnweb.freebsd.org/base/stable/10/etc/rc.subr?revision=272874 > Yes, sure, well written init.d scripts wouldn't have these problems, > however the fact is these problems do get into distributions, and often go > unnoticed by the responsible maintainers. "It works for me!" -- It looks as most FreeBSD ports maintainers using this when adapting the software. It's definitely used in the base system. Regards Peter

2 1

multi monitor hardware / graphics card advice
by h 10 Oct '14

10 Oct '14

Hi, I am sure I have seen a post on (a) newsgroup about this lately, but am unable to find it. Please excuse me posting what I think I should be able to google - but cann't. I run a desktop opensuse / KDE system and I want to replace my existing dual monitor system with a quad monitor system. I am looking for information on the linux support for graphics card setups which support 4+ monitors. I am resigned to using either nvidia or radeon with proprietary drivers, so would appreciate any advice regarding hardware from anyone who has done this sort of thing recently. -- With thanks, H

2 1

RE: systemd (was btrfs :()
by James Harper 10 Oct '14

10 Oct '14

> > BTW: I am happy to be convinced about systemd advantages:-) I also would > like to know how it fits in containers, e.g. if you run them for hosting. > FWIW, the first time I had to care about it was a few days ago when I upgraded one of my mythtv clients and it didn't start mythtv automatically anymore (well... it didn't start X anymore). It seems systemd just ignores whatever is in inittab, which is where I had put my xinit or startx or whatever it was. It was clunky and never felt really clean but it seemed to work. Originally I had wanted to set it up with a gdm autologin but autologin was broken. A bit of googling and there were plenty of examples of how to do it in systemd, and it seems so much nicer. Turn on box and mythtv starts, without having to fake a login or anything. Other than that, I haven't had to care. James

3 2

Re: btrfs :(
by Peter Ross 09 Oct '14

09 Oct '14

From: "Russell Coker" <russell(a)coker.com.au> > <andrew.mcglashan(a)affinityvision.com.au> > wrote: >> Did you even read what was already posted? " ... and I run ZFS on 3GB >> and 4GB RAM boxes ...." -- that's fine, but how much storage. My smallest ZFS server box with 3GB of RAM has 350 GB in disk space (a mirror). But it desnot run a fileserver. Icinga is running on it (which will write a bit but not hammering) and some "smaller" apps (DHCP and DNS server etc.) I have a 2 GB RAM laptop with PC-BSD and ZFS. It works. (It's not the greatest laptop ever, though. I just wanted to try PC-BSD) > I've had RAM problems on a system with 4G of RAM running not much apart > from Samba. The "big" samba server is on a 16GB RAM server (sharing it with a few other apaches etc.) > I don't think that zfsonlinux has changed much since then. > > ZFS just manages memory differently from anything that was developed for > Linux. Changing that would require significant code changes which would > be another potential source of stability problems. Yes. ARC is a bit of a "parasite". Think of a storage controller grabbing a few GigaByte of RAM;-) BTW: I am happy to be convinced about systemd advantages:-) I also would like to know how it fits in containers, e.g. if you run them for hosting. The German magazine iX had an article about Linux containers recently. It compared them with FreeBSD jails too. Amongst others, it was a bit elusive about "some security concerns" but did not elaborate further. FreeBSD jail seems to be quite small in "coding footprint" which makes an audit not that difficult. Not having /proc and /sys helps too. The devfs rules are easy to understand and minimize the risks. I wonder whether someone can add something to the Linux container coding/admin/security side. Regards Peter

1 0

OSDC 2014 - Griffith Uni GC Campus 4-6 Nov - high time to get your ticket!
by Arjen Lentz 07 Oct '14

07 Oct '14

Hi all With less than a month before the Open Source Developers' Conference 2014 begins, it's important to book your ticket now! If you can't make it to Auckland for LCA in January (and even if you can), this is fabulous extra opportunity! At around $300, full OSDC conference tickets cost less than most single IT training days, and provide so much more! Approx 50 awesome speakers, lots of like-minded similarly awesome people to exchange ideas with, tutorials to really learn something new in-depth with experts right there, great food including the ever fabulous conference dinner. There are tutorials/workshops on Inkscape, OpenStack, Docker, and Web Security, and session topics from setting up monitoring in your own home, bare bones ARM hacking, robotics, to 3D printing, and language-related talks on Python, PHP, Perl, Go, Haskell, Node/JS, Ruby/Rails, Arduino, some opensource-in-business topics, and so much more! What new magic will you learn during these days? There will even be a special BlueHackers plenary by a counsellor. Basically, you can't afford to not be there! For even more info, see http://2014.osdc.com.au/ To register, you can use the following link for the special ticket price: https://ti.to/osdc/2014goldcoast?release_id=zkbbeff1zjc If you're a student, use the regular registration page and pay only $180 Regards, Arjen & the OSDC 2014 team.

1 0

RE: btrfs :(
by James Harper 07 Oct '14

07 Oct '14

I found I was subscribed to luv-main with a slightly incorrect address (different to what I was emailing as)... anyway... > > parent transid verify failed on 699477344256 wanted 11799 found 12458 > > My btrfs lasted about a week before this happened. > > I'm googling around to see what my options are... I have a btrfs send backup > of my main subvolume but would lose a bunch of mythtv recordings that I'd > rather keep. > > Any suggestions appreciated! > All fixed. I restored from the btrfs send backup, and before I did that I copied my mythtv recordings off, and nothing appears to have been lost (although they could all be corrupt... copying them back now). The mysql database appears intact too, even though it would have been active right up until it crashed. The send/receive functions of btrfs are awesome. I have never seen an easier way of recovering before. The send/receive is well documented, but not the actual recovery when things go bad. I ended up re-creating a completely new btrfs volume and receiving the backup into that, which means the UUID's are different etc which is only a bit of a nuisance. I still don't know exactly what triggered the failure. My best guess is bcache, which was documented as not playing nicely with btrfs late last year (http://thread.gmane.org/gmane.comp.file-systems.btrfs/31018) although I'm not seeing any of those particular errors. I was using bcache in writeback mode, but switched back to writethrough for now. James

3 7