On Sat, 3 Jan 2015 04:32:54 PM David Zuccaro wrote:

Everything, click on directories for example. Funny thing is that it seems OK at the moment -- other times it is unusable. I have been monitoring CPU usage in top and there is nothing untoward going on there.

I'd suggest it's worth monitoring with latencytop to see if anything shows up there. -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

On 3 January 2015 at 21:24, Russell Coker <russell@coker.com.au> wrote:

I think that everyone here should host their own mail server.

You say that, yet apparently even you, an expert, can't get it right. (All mail from you is still ending up in my spam filter because the DKIM and DMARC checks fail.) Given the complexities around SPF, DKIM, DMARC, DNSSEC, DomainKeys, and in the other direction, relay control, bayesian filters, storage, searching and backup, maybe it's best that you don't recommend people host their own mail server? There's a lot of time to sink into Doing It Right that could be better spent on other problems. On the other hand, yeah, sure, it's a good way to learn a bit about some of the underlying fundamentals and flaws of SMTP. IMHO, Toby

On 12/01/15 18:24, Anders Holmstrom wrote:

On Mon, Jan 12, 2015 at 3:40 AM, Toby Corkindale <toby@dryft.net <mailto:toby@dryft.net>> wrote:

On 3 January 2015 at 21:24, Russell Coker <russell@coker.com.au <mailto:russell@coker.com.au>> wrote: > I think that everyone here should host their own mail server.

You say that, yet apparently even you, an expert, can't get it right. (All mail from you is still ending up in my spam filter because the DKIM and DMARC checks fail.)

[...] +1 it's a a bit tiresome having to check spam for the threads to make sense.

Curiously, I've got it the other way around. I wasn't looking through luv mail, but this was in my junk-review folder due to an unclear classification. Working with clients like non-profits, sending quite a lot of mail, I find I'm recommending services like Sendgrid and Mandrill for managing outgoing mail. I note that Mandrill runs a free service for up to 12K emails a month, which would do just fine for most people running personal mail servers. Regards, Andrew McNaughton

Hi all, A couple years ago I =finally= discovered the most likely cause of this. And you will notice that it happens at the start of every new year for some of you. It does for me. It also happens if you export/import large email collections on ThunderBird. Soon after I recover from my New Year celebrations, I "tidy up" my ThunderBird email folders. e.g. Create a 2014 Sent folder and a 2014 InBox folder, etc. Then I move every InBox and Sent message from the previous year into their new folders. (Note: I don't formally archive. I like them immediately handy.) If you're like me, you save everything, so we're talking about thousands of emails being moved at once. This automatically triggers a thorough (?) re-indexing of all messages in all folders. On my system it took nearly 100% of CPU utilisation, and it ran for many hours. Shutting down ThunderBird and restarting only made the process begin again from scratch. To make matters worse: Here in Australia early January happens to be the hottest days of the year, and my CPU temperature sensors were scaring me quite a bit. I never found a way to "throttle down" a process to ease demand on the CPU. So I found myself shutting down my computer until sunset; putting it in the coolest room of the house; taking the cover off; pointing a room fan at the motherboard; and praying I wouldn't cook it overnight. Good luck. Carl Bayswater, Victoria On 05/01/15 20:49, Piers Rowan wrote:

On 03/01/15 13:09, David Zuccaro wrote:

...

I have an i5-2450M CPU @ 2.50GHz with 4Gb of memory running linux mint 17.

As the title suggests I'm finding Thunderbird to be very sluggish so I would welcome suggestions to help boost performance. (actually evolution was better in this regard)

I have moved about 6 years worth of emails into the archive directory -- I presume exporting these out of Thunderbird into a searchable format (which one?) would help matters.

I've used TB extensively over the years and supported it in the workplace. One of this issues that can happen is that if you are using IMAP and you are moving large datasets around then the index files need to be rebuild. Often this is a problem because you have the folders opened on other devices / other users.

The net effect is that each starts polling and kills the other's indexing process and starts its own. Because you have such an extensive "backup" of emails the process can also take too long and time out (or you restart TB / PC to "fix" things).

Mail clients are not databases and many people (including me) treat them as such. When they stop working like mail clients in my experiences it is because you have made a change and it takes time to fake being a database again as it pulls itself together.

I recommend limiting your mailbox quota and using your CRM to pull in your email for searching / attachments so you have the data if you need it.

Cheers

P _______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main

On Tue, 6 Jan 2015, Craig Sanders <cas@taz.net.au> wrote:

On Tue, Jan 06, 2015 at 10:07:59AM +1100, Carl Turney wrote:

...

I never found a way to "throttle down" a process to ease demand on the CPU.

the 'renice' command can be used to change the priority of a process.

e.g. to make a process use only 'idle' cpu time, you need to find its process id (PID) and then:

renice -n 19 $PID

That will work if the issue is multiple processes competing for CPU time. But for Carl's case of a single process using 100% CPU time and overheating the machine it won't work. kill -stop will suspend a process and kill -cont will continue it. It wouldn't be difficult to write a script that runs kill -stop and kill -cont in a loop. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

On 6/01/2015 11:39 AM, Carl Turney wrote:

On 06/01/15 11:09, Russell Coker wrote:

...

kill -stop will suspend a process and kill -cont will continue it. It wouldn't be difficult to write a script that runs kill -stop and kill -cont in a loop.

Ah. I grasp the underlying concept, and know a bit about ps and kill.

If only I weren't so very damned rusty on other Linux bash commands, and writing scripts with loops and conditionals.

It's really quite simple and a great solution. ps -fe|grep someprocess You'll get the process id (PID) in the second column. Then you could set up a couple of aliases to make it simple. alias stop_tb='kill -STOP nnnn' alias start_tb='kill -CONT nnn' Now, so long as the PID doesn't change, then you can simply run the commands quickly as desired. stop_tb start_tb Cheers A.

On Tue, 6 Jan 2015, Andrew McGlashan <andrew.mcglashan@affinityvision.com.au> wrote:

alias stop_tb='kill -STOP nnnn' alias start_tb='kill -CONT nnn'

Now, so long as the PID doesn't change, then you can simply run the commands quickly as desired.

while true ; do stop_tb sleep 1 start_tb sleep 1 done -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

On Tue, 6 Jan 2015 11:09:14 AM Russell Coker wrote:

But for Carl's case of a single process using 100% CPU time and overheating the machine it won't work.

This sounds like a job for control groups (cgroups) - you can use the cpu.shares setting to restrict how much of a share of CPUs processes in that cgroup can get. The Arch Linux documentation says: https://wiki.archlinux.org/index.php/cgroups # Similarly you can change the CPU priority ("shares") of this group. # By default all groups have 1024 shares. A group with 100 shares # will get a ~10% portion of the CPU time: Worth a shot! All the best, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

On Tue, 6 Jan 2015, Trent W. Buck wrote:

Carl Turney <carl@boms.com.au> writes:

...

I never found a way to "throttle down" a process to ease demand on the CPU.

nice ionice -c3 thunderbird

Set the CPU governor(s) to ondemand or conservative, and set "don't speed up for niced loads".

Untested; YMMV &c.

Look at /sys/class/thermal/cooling_device?/cur_state max_state, etc too. Some of the cooling devices are CPU governors that insert a fake idle process at various percentages of 100% "CPU". Ie, they steal cycles from a process that is trying to hog the CPU, but don't heat the processor in the process[1] (some turn on more fans, or maybe make higher sleep states). [1] World record for different meanings of process in a sentence? -- Tim Connors

On 06/01/15 10:11, Russell Coker wrote:

If the problem is indexing on the IMAP server then you could try configuring the IMAP server to use different indexing or to use a different IMAP server program. Dovecot seems to have some reasonable options for indexing that don't waste much CPU time.

This also depends on the nature of your back end storage - like NFS for example when you can get time out's for read lock. The issue isn't so much the dovecot can be configured to do x with y resources it has to do with the fact that things are generally in production and have web mail, AV, mailman, SMTP, procmail, SA, etc all working together in real time on a single node. Fixing/messing with one function of the server can impact the others. My recommendations are don't have endless email archives (none of us are really that important) and expect to wait when you start moving relatively large chunks of plain text files around that are split are arbitrary lines aka mail spool files. Cheers P

Russell Coker <russell@coker.com.au> writes:

If you want decent performance with IMAP then just don't use NFS. The write pattern of mail stores is a poor match for the way NFS works and the large number of files doesn't work too well for read caching.

Nitpick: NFS doesn't play nice with lots of small files, i.e. maildir and MH storage formats. An IMAP server need not use those formats (e.g. domino).

On 13 January 2015 at 03:12, Robin Humble <rjh+luv@cita.utoronto.ca> wrote: [snip]

so how about SSDs? assuming 100k iops & 500 MB/s, that works out at a "small" file size of maybe ~5kB or a bit less, which is impressive. however small random write i/o is the absolute worst thing you can do to these things, so be sure to buy a good one (ie. intel only IMHO).

I've been kinda unimpressed by Intel's SSDs. The high-end ones do OK, but.. only OK. And they're expensive. Other brands have enormous random-write performance now, eg. 90,000 IOPS for 4kb random writes on the current-gen Samsungs, that are also cheap and have long durability.

mh also tends towards zillions of files in a dir and some fs's don't deal with that well. XFS used to handle 100k+ file/dir a lot better than ext[34]. dunno if it still does. many files in a dir is not a great idea with any fs.

ext3/4 added indexes to directories years and years ago, and so have been fine with very large directories since then. That said, it's still not a great idea, and I've seen some systems crippled due to having thousands upon thousands of files in single directories. (GlusterFS comes to mind as one that was particularly epic-fail)

the old adage "a filesystem is not a database" springs to mind...

Indeed, that's the crux of it. And this is why your sensible mail programs end up putting your emails into SQLite or LevelDB instead of antique mail files or maildirs. :) (I'd go leveldb over sqlite myself, but with Solr or similar on the side for searching, but we seem to mostly see SQLite used on its own so far) -T

On 14 January 2015 at 11:01, Andrew McGlashan <andrew.mcglashan@affinityvision.com.au> wrote:

On 13/01/2015 3:12 AM, Robin Humble wrote:

...

(TL;DR - put your email on a SSD)

Not with TB on a client machine.... file fragmentation doesn't last with TB files; it's next to useless trying to do that. What that means in a nutshell is that far too many files get re-written over and over again and that /could/ be too much for SSD, although SSD is far more durable these days -- subjecting it to TB though is probably asking for trouble.

Oh come on, that's FUD. SSDs are way more durable than that! http://techreport.com/review/24841/introducing-the-ssd-endurance-experiment http://techreport.com/review/27436/the-ssd-endurance-experiment-two-freaking... Those guys have been hammering a bunch of SSDs *continuously* - like a burn-in test - FOR A YEAR AND A BIT SO FAR - and still haven't killed all the drives. As of the last count, they were up to 2 petabytes of writes on the Samsung 840 Pro and it's still going strong. The earliest drive to drop out was an Intel SSD, at 750 TB of writes, (but that's apparently because they're programmed to go into read-only mode at exactly that amount, rather than risk sudden failure or data loss later). My point here being -- even though it's more about number of block writes rather than total data written, you've still got a massive number of writes you can do to an SSD before it "wears out". Working a handful of mail files during the day is NOT going to be a realistic problem. Your drive will last a thousand years. Toby

On Wed, 14 Jan 2015 03:49:01 PM Andrew McGlashan wrote:

In any case, I have seen TB be absolutely awful with files on at least NTFS .... but I still use it, I won't use it with an SSD though at this stage, but I might one day.

I've been running TB on 2 work laptops with SSDs for the last 5 years without any issues. All the best, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

On Wed, 14 Jan 2015 09:57:30 PM Mike wrote:

maybe the question is "which distro and what does that distro do about updates?"

I'm using Kubuntu 14.04.x which I believe tracks Firefox and Thunderbird updates pretty well. -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

On 15/01/2015 8:32 AM, Andrew McGlashan wrote:

If anyone has a greater store in their TB, I would be surprised.

On the server with ext4 fs, this is my from my Maildir folder: # time find . -type f -name '1*' |wc -l 1835027 real 1m26.659s user 0m9.117s sys 0m28.942s So, heading towards 2 million emails. Cheers A.

On 15/01/2015 11:25 AM, Trent W. Buck wrote:

Andrew McGlashan writes:

...

On 15/01/2015 8:32 AM, Andrew McGlashan wrote:

...

If anyone has a greater store in their TB, I would be surprised.

On the server with ext4 fs, this is my from my Maildir folder:

# time find . -type f -name '1*' |wc -l 1835027

real 1m26.659s user 0m9.117s sys 0m28.942s

So, heading towards 2 million emails.

OK, gloves off ;-P

Haha.

# /usr/bin/time --portability nice ionice -c3 find -O3 /var/mail/mailsec/shared-r[ow] -type f -name 1\* -printf x | wc -c real 73.32 user 2.98 sys 10.03

Okay.... is that one person's archive or more people? You win if it is one person, I win if it is more ;-) And I do delete log [email] files after 90 days by cron jobs and enough spam rubbish gets deleted too. TB gave me serious troubles about a year ago, it was then that I stopped downloading every email to the client machine. Cheers A.

Andrew McGlashan writes:

Debian lists alone in one aggregate folder has nearly 250,000 messages, those are collected from all groups I've subscribed over the years. Each group then has it's own folder as well (so lots of duplication).

I use & recommend gmane for public MLs. WFM, YMMV. (This is when I learn that tbird can't do NNTP...)

On 16/01/2015 1:51 PM, Toby Corkindale wrote:

You know, what you said doesn't really make sense. You acknowledge that SSDs offer significant benefits to your workload, and you seem to acknowledge that SSDs are actually completely fine for durability of your workload.. So why are you still saying you wouldn't use one?

Well ..... yes SSD are very good, but given what I've seen with TB, I believe it would still be a premature end of life -- still it might be good for quite a while.

Why are you not worried about your spinning rust drives wearing out? After all, those bearings and motors won't last forever, and all those rapid head seeks to deal with random i/o must put increased wear and tear on them.

The spinning rust, so to speak, doesn't have write limitations that today's SSD units do. A /better/ class of SSDs will (if they come to market) self heal the oxide layer that is damaged from regular writing. That technology does super heating of the oxide layer to restore it back in to near new condition [1] -- at least that's my rough understanding of the tech involved. If we get those drives, then it might be a case of replacing them for greater capacity and/or even faster speeds as it sounds like they'll last forever. Looks like "racetrack" [2] memory isn't going to happen, but this Crossbar RRAM tech [3] is not so far away. Besides spinning disks still have far greater capacity today and with 8TB drives on the market now and 20TB ones coming later.... [1] http://www.extremetech.com/computing/142096-self-healing-self-heating-flash-... [2] http://en.wikipedia.org/wiki/Racetrack_memory [3] http://www.pcmag.com/article2/0,2817,2422734,00.asp Cheers A.

My two-penneth into this : The reference to bearing failures might be OK - however I think most HDD use voice coils for the head movement these days.. Mike ( are we going to see you guys at the LUV BBQ ? ) On 16/01/15 14:55, Toby Corkindale wrote:

On 16 January 2015 at 14:25, Andrew McGlashan <andrew.mcglashan@affinityvision.com.au> wrote:

...

On 16/01/2015 1:51 PM, Toby Corkindale wrote:

...

You know, what you said doesn't really make sense. You acknowledge that SSDs offer significant benefits to your workload, and you seem to acknowledge that SSDs are actually completely fine for durability of your workload.. So why are you still saying you wouldn't use one? Well ..... yes SSD are very good, but given what I've seen with TB, I believe it would still be a premature end of life -- still it might be good for quite a while. If by "quite a while" you mean "If I received a million emails every day, my drive would still possibly last more than the rest of my life" :)

Anyway, never mind. It's your choice. I'm just trying to point out that once you check the numbers, this technology doesn't need to be scary to you. And once it's going, it's more reliable than your hard drives that can suffer stepper motor or bearing failures due to all their moving parts.

-Toby _______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main

Maildir was designed to work over NFS but I don't think that anyone cared much about performance. Mail servers tend to either be small enough that performance is never a problem or big enough that NFS just isn't viable. Dovecot and other IMAP servers should work together. Dovecot is designed to regenerate indexes when necessary. I don't think performance would suffer much given that a common use case is having a program other than Dovecot doing delivery. On January 12, 2015 10:35:19 PM GMT+13:00, Andrew McN <andrew@mcnaughty.com> wrote:

On 06/01/15 18:02, Russell Coker wrote:

...

If you want decent performance with IMAP then just don't use NFS. The write pattern of mail stores is a poor match for the way NFS works and the large number of files doesn't work too well for read caching.

Wasn't playing nice with NFS the main motivation for Maildir's development? I think the issue was mostly with locking rather than performance, but still...

...

...

The issue isn't so much the dovecot can be configured to do x with y resources it has to do with the fact that things are generally in production and have web mail, AV, mailman, SMTP, procmail, SA, etc all working together in real time on a single node. Fixing/messing with one function of the server can impact the others.

If you have a running mail server it's not too difficult to change the process that serves POP/IMAP without changing the rest.

You can even have 2 programs serving POP/IMAP on different ports or IP addresses until you are happy that the new one does everything correctly.

Hmm. You wouldn't expect other mail daemons to know about Dovecot's indexing systems. It might work correctly, but would at least have a performance impact. _______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main

-- Sent from my Samsung Galaxy Note 3 with K-9 Mail.