- luv-main - example.com

swapping fields
by Russell Coker 12 Feb '12

12 Feb '12

I have a file with lines like "a,b,c" which I want to be "c,b,a". I tried using "cut -f 3,2,1 -d," but it didn't do what I wanted. It seems that you can specify the desired fields on the cut command line any way you like but it will sort them to be in numeric order. Any ideas other than a sed command with positional parameters? -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

2 1

RAID-1 synchronisation
by Russell Coker 12 Feb '12

12 Feb '12

I run a bunch of servers with Linux software RAID-1. I use bitmaps on all of them because the ongoing overhead (*) of bitmaps is better than the occasional overhead of a full resync. Recently one of my servers suddenly decided to do a complete RAID-1 resync for no apparent reason. Other servers with the same versions of all software (Debian/Squeeze with all updates) didn't do it. The server in question did crash a few times recently (**). Is a server crash likely to result in an entire RAID resync even when bitmaps are used? Does anyone have any advice other than throwing the server in the bin? (*) I really doubt that the overhead is as bad as some people claim. I plan to test it but haven't had time so far. (**) Currently dmesg output includes the following: [87347.834590] NETDEV WATCHDOG: eth0 (r8169): transmit queue 0 timed out [87347.844958] BUG: soft lockup - CPU#0 stuck for 94s! [swapper:0] I'm not sure if this is related to the crashes. I suspected a problem with eth1 and turned off TSO etc which seems to have helped. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

8 23

swap over iSCSI
by Tim Connors 12 Feb '12

12 Feb '12

I presume swap over iSCSI and NFS are still prone to deadlock? I've seen no followup to this thread back in 2007: http://kerneltrap.org/Linux/Swap_Over_NFS other than this thread not finding a definitive answer: http://groups.google.com/group/open-iscsi/browse_thread/thread/bdff45207969… Can anyone trace whether these patches and iscsi patches ever made it upstream? It'd be a pity to waste all the memory in my new NAS. Swap to something essentially seekless would be quite nice. -- Tim Connors

2 1

Re: state of ZFS and btrfs
by Richard Archer 12 Feb '12

12 Feb '12

On 11/02/12 4:26 PM, Daniel Pittman wrote: > Total, actually. Although that was US dollars, and only cheap > consumer grade non-ECC memory. With ZFS able to parity-check data on disk for consistency and detect/repair bit errors, is it necessary to use ECC memory for a dedicated ZFS file store? ...Richard.

3 2

Install debian on Acer Aspire 1 D257?
by Craig Sanders 12 Feb '12

12 Feb '12

I'm installing debian on an Acer Aspire 1 D257 notebook for someone, and the installer won't recognise the hard disk. all i get from the installer is "No disk drive was detected. If you know the name of the driver needed by your disk drive, you can select it from the list." followed by a selectable list of mostly-ancient drivers, all irrelevant. i've tried the squeeze, wheezy, and sid installers (all pretty much the same, except for minor variations in the kernel*)...and tried both amd64 and i386 for each (the CPU is a 64-bit quad-core Atom N570). anyone have any ideas? lspci says: # lspci 00:00.0 Host bridge: Intel Corporation N10 Family DMI Bridge (rev 02) 00:02.0 VGA compatible controller: Intel Corporation N10 Family Integrated Graphics Controller (rev 02) 00:02.1 Display controller: Intel Corporation N10 Family Integrated Graphics Controller (rev 02) 00:1b.0 Audio device: Intel Corporation N10/ICH 7 Family High Definition Audio Controller (rev 02) 00:1c.0 PCI bridge: Intel Corporation N10/ICH 7 Family PCI Express Port 1 (rev 02) 00:1c.1 PCI bridge: Intel Corporation N10/ICH 7 Family PCI Express Port 2 (rev 02) 00:1c.2 PCI bridge: Intel Corporation N10/ICH 7 Family PCI Express Port 3 (rev 02) 00:1d.0 USB Controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #1 (rev 02) 00:1d.1 USB Controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #2 (rev 02) 00:1d.3 USB Controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #4 (rev 02) 00:1d.7 USB Controller: Intel Corporation N10/ICH 7 Family USB2 EHCI Controller (rev 02) 00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev e2) 00:1f.0 ISA bridge: Intel Corporation NM10 Family LPC Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation N10/ICH7 Family SATA AHCI Controller (rev 02) 00:1f.3 SMBus: Intel Corporation N10/ICH 7 Family SMBus Controller (rev 02) 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8101E/RTL8102E PCI Express Fast Ethernet controller (rev 05) 02:00.0 Network controller: Atheros Communications Inc. Device 0032 (rev 01) 03:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 5209 (rev 01) the Intel ICH7 should be well-supported. here's the syslog from when i run the Detect Disks menu option: Feb 10 09:17:18 main-menu[4292]: INFO: Menu item 'disk-detect' selected Feb 10 09:17:19 net/hw-detect.hotplug: Detected hotpluggable network interface eth0 Feb 10 09:17:19 kernel: [ 407.101140] libata version 3.00 loaded. Feb 10 09:17:19 kernel: [ 407.116877] ahci 0000:00:1f.2: version 3.0 Feb 10 09:17:19 kernel: [ 407.116918] ahci 0000:00:1f.2: PCI INT B -> GSI 17 (level, low) -> IRQ 17 Feb 10 09:17:19 kernel: [ 407.117004] alloc irq_desc for 28 on node -1 Feb 10 09:17:19 kernel: [ 407.117011] alloc kstat_irqs on node -1 Feb 10 09:17:19 kernel: [ 407.117038] ahci 0000:00:1f.2: irq 28 for MSI/MSI-X Feb 10 09:17:19 kernel: [ 407.117116] ahci: SSS flag set, parallel bus scan disabled Feb 10 09:17:19 kernel: [ 407.117172] ahci 0000:00:1f.2: AHCI 0001.0100 32 slots 4 ports 3 Gbps 0x3 impl SATA mode Feb 10 09:17:19 kernel: [ 407.117184] ahci 0000:00:1f.2: flags: 64bit ncq ilck stag pm led clo pmp pio slum part Feb 10 09:17:19 kernel: [ 407.117196] ahci 0000:00:1f.2: setting latency timer to 64 Feb 10 09:17:19 kernel: [ 407.117714] scsi0 : ahci Feb 10 09:17:19 kernel: [ 407.118134] scsi1 : ahci Feb 10 09:17:19 kernel: [ 407.118390] scsi2 : ahci Feb 10 09:17:19 kernel: [ 407.118661] scsi3 : ahci Feb 10 09:17:19 kernel: [ 407.119536] ata1: SATA max UDMA/133 abar m1024@0x56204000 port 0x56204100 irq 28 Feb 10 09:17:19 kernel: [ 407.119550] ata2: SATA max UDMA/133 abar m1024@0x56204000 port 0x56204180 irq 28 Feb 10 09:17:19 kernel: [ 407.119557] ata3: DUMMY Feb 10 09:17:19 kernel: [ 407.119562] ata4: DUMMY Feb 10 09:17:19 net/hw-detect.hotplug: Detected hotpluggable network interface lo Feb 10 09:17:19 kernel: [ 407.600033] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Feb 10 09:17:19 kernel: [ 407.601999] ACPI Warning for \_SB_.PCI0.SATA.PRT0._GTF: Return type mismatch - found Integer, expected Buffer (20090903/nspredef-1006) Feb 10 09:17:19 kernel: [ 407.602016] ata1.00: _GTF unexpected object type 0x1 Feb 10 09:17:19 kernel: [ 407.602351] ata1.00: ATA-8: WDC WD3200BPVT-22JJ5T0, 01.01A01, max UDMA/133 Feb 10 09:17:19 kernel: [ 407.602365] ata1.00: 625142448 sectors, multi 16: LBA48 NCQ (depth 31/32), AA Feb 10 09:17:19 kernel: [ 407.604433] ata1.00: _GTF unexpected object type 0x1 Feb 10 09:17:19 kernel: [ 407.604796] ata1.00: configured for UDMA/133 Feb 10 09:17:19 kernel: [ 407.620280] scsi 0:0:0:0: Direct-Access ATA WDC WD3200BPVT-2 01.0 PQ: 0 ANSI: 5 Feb 10 09:17:19 kernel: [ 407.940034] ata2: SATA link down (SStatus 0 SControl 300) Feb 10 09:17:20 kernel: [ 407.965473] sd_mod: Unknown symbol scsi_verify_blk_ioctl Feb 10 09:17:20 kernel: [ 407.967957] sd_mod: Unknown symbol scsi_cmd_blk_ioctl Feb 10 09:17:20 hw-detect: Loading PCMCIA bridge driver module: i82365 Feb 10 09:17:20 hw-detect: FATAL: Module i82365 not found. Feb 10 09:17:21 check-missing-firmware: no missing firmware in /dev/.udev/firmware-missing those Unknown Symbol and/or the ACPI Warning lines are probably where the problem is. googling for the ACPI warning got me a page that suggested adding "acpi_apic_instance=2" to the kernel cmd line, but that made no difference. I'm beginning to think i'm going to have to re-learn the PITA procedure to make a custom kernel/initrd for d-i so i can boot with a 3.0.x or 3.2.x kernel. it's been a few years since i did it, but i remember that it was a tediously convoluted process that involved shaving far too many yaks (including a full d-i build when all i want is a suitable kernel+initrd to place on my tftp server). (*) for instance, the latest squeeze kernel/initrd has removed the firmware for the Realtek RTL8101E/RTL8102E NIC. hooray, i just love screwing around with USB sticks and windows-like hunts for missing drivers. I also love PXE-booting into an installer and then finding that a NIC that IPXE has no problem with won't work with the debian kernel. The sid and wheezy installers still have the firmware...boo for nasty un-PC kernels. maybe i should try Linux Mint. craig -- craig sanders <cas(a)taz.net.au>

3 11

Will this ruling have implications for secure booting?
by Allan Duncan 11 Feb '12

11 Feb '12

A win in France for a Lenovo customer who didn't want bundled software on a laptop he wanted to run linux on appears to have ramifications for all of the EU. http://no.more.racketware.info/news/hardware-software-bundling-crumbles-fra… I wonder if that might extend to the boot process as well.

9 12

RAID magic
by Russell Coker 11 Feb '12

11 Feb '12

Below is a magic entry I wrote to support Linux software RAID components. Does anyone here have a big-endian system that they can run software RAID on to test this? I suspect that the solution is to duplicate the section and change every "lelong" to a "belong". Also note that I want to recognise big endian devices on little endian systems and vice-versa. I'll file a Debian bug report about this as soon as I think I have big-endian sorted out. # Magic local data for file(1) command. # Insert here your local magic data. Format is described in magic(5). 4096 lelong 0xa92b4efc Linux Software RAID >4100 lelong x version %d >4112 belong x UUID=%8x: >4116 belong x \b%8x: >4120 belong x \b%8x: >4124 belong x \b%8x >4128 string x name=%s >4168 lelong x level=%d >4188 lelong x disks=%d -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

1 0

Re: state of ZFS and btrfs
by Craig Sanders 11 Feb '12

11 Feb '12

On Fri, Feb 10, 2012 at 06:53:23PM -0800, Daniel Pittman wrote: > > it's in ZFS. the catch is that it takes enormous amounts of > > memory....about 1GB RAM per TB of disk space IIRC to store the hashes > > for each block. I suspect the same catch would apply to other > > implementations. > > FreeBSD report that as ... optimistic, in the real world: quite probably. my figure of 1GB/TB was no better than a guess based on something half-remembered. > There are some resources that suggest that one needs 2GB per TB of > storage with deduplication [i] (in fact this is a misinterpretation of > the text). In practice with FreeBSD, based on empirical testing and > additional reading, it's closer to 5GB per TB. ouch. > I am inclined to take their comment seriously, given reports from the > few people I know who did try that. (OTOH, 64GB of RAM is US $379 for > four consumer grade sticks, $369 total, or $369 per 16GB stick? I thought 8GB & 16GB sticks were still stupidly expensive. > so you could realistically use that for a 12TB pool in hardware that > is reasonably affordable.) OTOH, there's probably lots better uses for that RAM on the same system. de-duping is, IMO, one of those things that sounds nice but that there's little practical use for. in most cases, it's cheaper and more effective to just add more disk than to add enough RAM to support it. craig -- craig sanders <cas(a)taz.net.au> BOFH excuse #40: not enough memory, go get system upgrade

1 0

Re: state of ZFS and btrfs
by Russell Coker 11 Feb '12

11 Feb '12

On Sat, 11 Feb 2012, Daniel Pittman <daniel(a)rimspace.net> wrote: > There are some resources that suggest that one needs 2GB per TB of > storage with deduplication [i] (in fact this is a misinterpretation of > the text). In practice with FreeBSD, based on empirical testing and > additional reading, it's closer to 5GB per TB. > > http://wiki.freebsd.org/ZFSTuningGuide#Deduplication > > I am inclined to take their comment seriously, given reports from the > few people I know who did try that. (OTOH, 64GB of RAM is US $379 for > four consumer grade sticks, so you could realistically use that for a > 12TB pool in hardware that is reasonably affordable.) You could use 64G of RAM for deduplication on 12TB of disk, that would save some amount of storage and also increase cache efficiency. But you could use the 64G of RAM for caching and just buy some extra disk. If you have a large number of DomUs then they probably aren't loaded up with a full KDE or GNOME environment. A basic DomU can have as little as 400M of data (when excluding /var) - based on inspecting one of my servers. So if I was to use deduplication for DomUs then I might need more than 24,000 to fill the 12TB of storage before deduplication. There are other ways that deduplication could really do some good, such as when a large MS-Word document is attached to an email that is sent to everyone in a company. Of course you would probably have to remove the Delivered-To: header to get a good result with that (I'm guessing that ZFS deduplication only works with block aligned data). But there are other solutions to that problem like using my maildir-bulletin program or some other delivery agent that is specific to the case of mass local mail-outs. Given modern storage prices I find it difficult to imagine a situation where deduplication does some real good but where it can't be done better at the application level. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

1 0

Re: RAID-1 synchronisation
by Russell Coker 11 Feb '12

11 Feb '12

On Sun, 5 Feb 2012, Daniel Pittman <daniel(a)rimspace.net> wrote: > Yes, it does good. For something like RAID-5 that has enough > information to detect which device is wrong you can find and correct > problems with the system. With a RAID-1 you can know that a device is > returning bad data. You mean RAID-6. RAID-5 is no better than RAID-1 when it comes to determining which one of the disks is returning corrupt data. On Sun, 5 Feb 2012, Daniel Pittman <daniel(a)rimspace.net> wrote: > Yes. This is the substantial advantage that BTRFS and ZFS have over > device-level RAID and LVM. The stronger checksums are useful because > they can be checked inline with lower I/O cost, and the intimate > knowledge of allocated space means you can check more quickly. Both > very attractive features. https://btrfs.wiki.kernel.org/ The above URL lists mirroring as "Additional features in development, I don't think I can use this on serious servers for a while. https://btrfs.wiki.kernel.org/articles/u/s/i/Using_Btrfs_with_Multiple_Devi… mkfs.btrfs in Debian/Unstable apparently supports RAID (the above URL has background information), so I will test it out. On Sun, 5 Feb 2012, James Harper <james.harper(a)bendigoit.com.au> wrote: > One thing it does do for you is 'touch' unused blocks, and finding that > those are bad now rather than later is better IMO. Also, verifying > consistency and finding that you have a silent corruption problem early > can only be a good thing. This is especially important for RAID5 without > battery backed write cache as it can detect the RAID5 write-hole > (http://en.wikipedia.org/wiki/RAID_5_write_hole) Maybe write-intent > bitmaps get around this these days though? The only difference between the RAID-5 write hole and the issue of mismatched RAID-1 devices is what happens on a device failer. It's a lot worse for RAID-5 so that's a good reason for doing such checks on a RAID-5 array, or just using RAID-6. RAID-6 allows detecting and correcting the situation where one disk has corrupt data and all disks are working and it allows detecting the situation where the disks aren't consistent if there has been a single disk failure. Just don't use RAID-5. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

8 15