Network traffic log analysis

Hello LUVers I have been experiencing an increase of about 5x factor in my outgoing network traffic for a while -at least last week - I cannot explain that change in activity,configuration etc... (It does peak to much higher level at a specific time, but I have worked out why that was so.. a overgrown backup file that was being transfered daily) So I need to find out what that traffic is. What is a recommended log analysers to help me quickly here? Want it to tell me how much data is being transfered by which protocol Or any recommended procedure to identify that traffic? Any ideas please? Its a Ubuntu 10.04 server, running Apache, Postfix and related stuff (Spamassassin, Amavisd etc) (I had varnish on it, but I turned it off) Cheers Daniel.

Wireshark FTW! One way is to run Snort on the server and use WS to analyse the output. mike http://mikelindner.wordpress.com On 19/11/12 15:36, DanyJ wrote:
Hello LUVers
I have been experiencing an increase of about 5x factor in my outgoing network traffic for a while -at least last week - I cannot explain that change in activity,configuration etc... (It does peak to much higher level at a specific time, but I have worked out why that was so.. a overgrown backup file that was being transfered daily)
So I need to find out what that traffic is.
What is a recommended log analysers to help me quickly here? Want it to tell me how much data is being transfered by which protocol
Or any recommended procedure to identify that traffic?
Any ideas please?
Its a Ubuntu 10.04 server, running Apache, Postfix and related stuff (Spamassassin, Amavisd etc) (I had varnish on it, but I turned it off)
Cheers Daniel. _______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main

Michael Lindner <michael@tropyx.com> wrote:
Wireshark FTW! One way is to run Snort on the server and use WS to analyse the output.
Or tshark if you want to run it on the server, using a read filter (the -R option, see wireshark-filter)5) for details) to filter out traffic that you regard as legitimate. Even just running it for a minute or so and looking at the output might be enough to identify the offending protocol. I'm not the right person to ask if you need help with the filter language; I've only used it occasionally with relatively simple filter expressions.

Jason White <jason@jasonjgw.net> writes:
Michael Lindner <michael@tropyx.com> wrote:
Wireshark FTW! One way is to run Snort on the server and use WS to analyse the output.
Or tshark if you want to run it on the server, using a read filter (the -R option, see wireshark-filter)5) for details) to filter out traffic that you regard as legitimate.
tcpdump has fewer dependencies than tshark. IME best practice is to put tcpdump on your router, run tcpdump -wfoo.pcap (or to a stream, if you have no disk), and then analyze that elsewhere with tshark/wireshark. I cannot comment on snort; I haven't tried it.
Even just running it for a minute or so and looking at the output might be enough to identify the offending protocol.
+1. Last time this happened to me, I added a couple of "and not port 22 and not host me" type constraints and immediately went "hey, $brother, are you talking to random-asshole.ru and other-asshole.cn deliberately?" (His machine was doubtless full of viruses and botnets and suchlike. Or maybe just bittorrent.)
I'm not the right person to ask if you need help with the filter language; I've only used it occasionally with relatively simple filter expressions.
There are actually two -- the precapture language uses "dst foo", "port foo" and "tcp" and connects them "and"; the postcapture one looks more like C: "ip.dest==1.2.3.4 && tcp.port=17". I can usually guess the former, but I always forget the latter. Wireshark is helpful in this regard (at least to sighted people, sorry Jason) because it has a little GUI query builder thingy to remind you it's tcp.port, not ip.port, and && not &.

On 20/11/12 09:35, Trent W. Buck wrote:
IME best practice is to put tcpdump on your router, run tcpdump -wfoo.pcap
You want to add -s0 to that if you want to capture the whole packet (tshark does that automatically). cheers, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

On Tue, 20 Nov 2012 12:20:43 +1100 Chris Samuel <chris@csamuel.org> wrote: I should have said in initial post :. It is a hosted Xen VM - not a physical server. and no X, ie: all gui option are out.. Daniel.
On 20/11/12 09:35, Trent W. Buck wrote:
IME best practice is to put tcpdump on your router, run tcpdump -wfoo.pcap
You want to add -s0 to that if you want to capture the whole packet (tshark does that automatically).
cheers, Chris

On 20 November 2012 12:58, DanyJ <dan062@yahoo.com.au> wrote:
It is a hosted Xen VM - not a physical server. and no X, ie: all gui option are out..
That is why you capture the packets on the server and copy the file to another computer that is running X. (or you can run ssh with X forwarding also) -- Brian May <brian@microcomaustralia.com.au>

On 20/11/12 12:58, DanyJ wrote:
It is a hosted Xen VM - not a physical server. and no X, ie: all gui option are out..
Neither tshark not tcpdump require a GUI. -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

On 20/11/12 13:11, Chris Samuel wrote:
Neither tshark not tcpdump require a GUI.
s/not/nor/ -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

Chris Samuel <chris@csamuel.org> writes:
On 20/11/12 09:35, Trent W. Buck wrote:
IME best practice is to put tcpdump on your router, run tcpdump -wfoo.pcap
You want to add -s0 to that if you want to capture the whole packet (tshark does that automatically).
Ah, yes. The manpage says the default is the max spanlen, but experimentation on Debian builds shows it is lying. I assume this is because Debian change the default without fixing the manpage. I should file a bug about that...

On 19 November 2012 15:36, DanyJ <dan062@yahoo.com.au> wrote:
Hello LUVers
I have been experiencing an increase of about 5x factor in my outgoing network traffic for a while -at least last week - I cannot explain that change in activity,configuration etc... (It does peak to much higher level at a specific time, but I have worked out why that was so.. a overgrown backup file that was being transfered daily)
So I need to find out what that traffic is.
What is a recommended log analysers to help me quickly here? Want it to tell me how much data is being transfered by which protocol
Or any recommended procedure to identify that traffic?
Any ideas please?
Its a Ubuntu 10.04 server, running Apache, Postfix and related stuff (Spamassassin, Amavisd etc) (I had varnish on it, but I turned it off)
Cheers Daniel. _______________________________________________ luv-main mailing
For a quick hit on the top ten you can try iotop package (in debian at least) .... Depends: python2.7 | python2.6, python (>= 2.6.6-7~), python (<< 2.8) Description-en: simple top-like I/O monitor iotop does for I/O usage what top(1) does for CPU usage. It watches I/O usage information output by the Linux kernel and displays a table of current I/O usage by processes on the system. It is handy for answering the question "Why is the disk churning so much?". . iotop can only run under a Linux 2.6.20 or later kernel built with the CONFIG_TASKSTATS, CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING and CONFIG_VM_EVENT_COUNTERS build config options on. Homepage: http://guichaz.free.fr/iotop/

I think iotop is more for disk io than network. For live per-process network monitoring I like the nethogs program.

On 19 November 2012 18:18, Andrew Spiers <andrew@andrewspiers.net> wrote:
I think iotop is more for disk io than network. For live per-process network monitoring I like the nethogs program.
Sorry - I got confused - jnetop Description-en: View hosts/ports taking up the most network traffic jNettop captures traffic coming across the host it is running on and displays streams sorted by bandwidth they use. Result is a nice listing of communication on network by host and port, how many bytes went through this transport and the bandwidth it is consuming. .... It does require all your traffic goes through your PC Andrew

Andrew Worsley <amworsley@gmail.com> wrote:
Description-en: View hosts/ports taking up the most network traffic jNettop captures traffic coming across the host it is running on and displays streams sorted by bandwidth they use. Result is a nice listing of communication on network by host and port, how many bytes went through this transport and the bandwidth it is consuming. ....
That's the best solution so far in this thread.
It does require all your traffic goes through your PC
It should be possible to install it on an OpenWRT or similar router however, and it may already have been packaged for that environment.

Jason White <jason@jasonjgw.net> wrote:
It should be possible to install it on an OpenWRT or similar router however, and it may already have been packaged for that environment.
Actually, no, it wouldn't be useful on a router - it seems to require that the processes it identifies originate the packets. It doesn't do packet capture as far as I can tell from the description.

For a variety of reasons I'd still go with snort + wireshark. On 19/11/12 18:52, Jason White wrote:
Andrew Worsley <amworsley@gmail.com> wrote:
Description-en: View hosts/ports taking up the most network traffic jNettop captures traffic coming across the host it is running on and displays streams sorted by bandwidth they use. Result is a nice listing of communication on network by host and port, how many bytes went through this transport and the bandwidth it is consuming. .... That's the best solution so far in this thread. It does require all your traffic goes through your PC It should be possible to install it on an OpenWRT or similar router however, and it may already have been packaged for that environment.
_______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main

On 19/11/12 17:57, Andrew Worsley wrote:
For a quick hit on the top ten you can try iotop package (in debian at least)
I think you mean iftop.. :-) -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

Hello I have run tshark and downloaded the capture file. I am reading the capture file using wireshark I see a lot of the following showing: 18 1.047790 111.67.22.211 111.67.23.255 DB-LSP-DISC 153 Dropbox LAN sync Discovery Protocol What are these? There is not Dropbox on the server. Any one can help please? Cheers Daniel.
Hello LUVers
I have been experiencing an increase of about 5x factor in my outgoing network traffic for a while -at least last week - I cannot explain that change in activity,configuration etc... (It does peak to much higher level at a specific time, but I have worked out why that was so.. a overgrown backup file that was being transfered daily)
So I need to find out what that traffic is.
What is a recommended log analysers to help me quickly here? Want it to tell me how much data is being transfered by which protocol
Or any recommended procedure to identify that traffic?
Any ideas please?
Its a Ubuntu 10.04 server, running Apache, Postfix and related stuff (Spamassassin, Amavisd etc) (I had varnish on it, but I turned it off)
Cheers Daniel. _______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main

DanyJ <dan062@yahoo.com.au> writes:
I have run tshark and downloaded the capture file. I am reading the capture file using wireshark. I see a lot of the following showing:
18 1.047790 111.67.22.211 111.67.23.255 DB-LSP-DISC 153 Dropbox LAN sync Discovery Protocol
What are these? There is not Dropbox on the server. Any one can help please?
From context, I'm guessing that dropbox spams the LAN to find other dropbox users. Thus, someone else on your LAN is running dropbox.
IIRC dropbox showed up in my squid logs attempting to do crazy made-up HTTP verbs, too. I get the impression they're pretty cowboy.

On 2012-11-20 16:15, Trent W. Buck wrote:
DanyJ <dan062@yahoo.com.au> writes:
I have run tshark and downloaded the capture file. I am reading the capture file using wireshark. I see a lot of the following showing:
18 1.047790 111.67.22.211 111.67.23.255 DB-LSP-DISC 153 Dropbox LAN sync Discovery Protocol
What are these? There is not Dropbox on the server. Any one can help please?
From context, I'm guessing that dropbox spams the LAN to find other dropbox users. Thus, someone else on your LAN is running dropbox.
IIRC dropbox showed up in my squid logs attempting to do crazy made-up HTTP verbs, too. I get the impression they're pretty cowboy.
I don't know exactly how it works, but yes, somehow, Dropbox on individual desktops tries to communicate with other dropbox instances on the LAN so that if I have Dropbox on my laptop and my desktop, those two machines can sync directly with each other rather than go out to the internet and back in. -- Regards, Matthew Cengia

Hi All Thanks for all help. I think I worked out what the problem was with tshark and wireshark. For info: I would appear that the increase in traffic was due to a broken dovecot-imap-thunderbird 16.0.2 interaction. When left open, thunderbird was synching up to several 100's Mb of email from server to client hourly - it had been set to synch every 10 mins - but instead of only checking new headers it was downloading every email message. It was set to locally cache messages. Plus other legimate traffic it was adding up to ~3 Gb of upload traffic daily - which was way more than I had expected. There are related bugs reported on the thunderbird website. I had recenly upgraded t/bird, so its a regression. There are posts relating to dovecot and t/bird. I am not sure if it also applies to other imap servers or only dovecot-t/bird combination. Seems to have been fixed in latest T/bird 17 beta which I am using now. Cheers Daniel. On Tue, 20 Nov 2012 15:57:42 +1100 DanyJ <dan062@yahoo.com.au> wrote:
Hello
I have run tshark and downloaded the capture file. I am reading the capture file using wireshark
I see a lot of the following showing:
18 1.047790 111.67.22.211 111.67.23.255 DB-LSP-DISC 153 Dropbox LAN sync Discovery Protocol
What are these? There is not Dropbox on the server. Any one can help please?
Cheers Daniel.
Hello LUVers
I have been experiencing an increase of about 5x factor in my outgoing network traffic for a while -at least last week - I cannot explain that change in activity,configuration etc... (It does peak to much higher level at a specific time, but I have worked out why that was so.. a overgrown backup file that was being transfered daily)
So I need to find out what that traffic is.
What is a recommended log analysers to help me quickly here? Want it to tell me how much data is being transfered by which protocol
Or any recommended procedure to identify that traffic?
Any ideas please?
Its a Ubuntu 10.04 server, running Apache, Postfix and related stuff (Spamassassin, Amavisd etc) (I had varnish on it, but I turned it off)
Cheers Daniel. _______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
_______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
participants (9)
-
Andrew Spiers
-
Andrew Worsley
-
Brian May
-
Chris Samuel
-
DanyJ
-
Jason White
-
Matthew Cengia
-
Michael Lindner
-
trentbuck@gmail.com