[luv-main] help urgently needed tomorrow when I buy new hardware
Dear Linux users, Firstly, Sheila has asked me to apologise because, due to interruptions, she is still working on an article about the FOSS and GNU Linux Open Day in Victoria, As I am writing, Sheila is working feverishly on that article and says it should be published soon on http://candobetter.net . My existing hardware configuration and/or the Linux distributions I have tried to install have got the better of me. See APPENDIX 1. For this reason and because I want a PC with sufficient grunt to be able to run a java application (which will run the order of 100-200 or more threads ( http://www.informatik.uni-trier.de/~ley/db/conf/pdpta/pdpta2004-3.html#Sinna...). It seems that a 64 bit processor will be needed as a 32 bit processor cannot address more than 4 Gig of memory (see http://developer.amd.com/Pages/1130200686.aspx http://www.intel.com/content/www/us/en/architecture-and-technology/microarch... ). I intend to go to a hardware dealer (possibly MSY in Cheltenham tomorrow) so i need advice as to what will be a hardware configuration that: 1. has sufficient power to run the multi-threaded Java application mentioned above; and 2. will mostly allow at least one Linux distribution to be installed and quickly without causing me anywhere the number of problems I have faced for at least the last two weeks (see APPENDIX 1). I would also be interested in what Linux Distribution others would recommend. I have had serious problems with Debian and Saboyan and have found Linux Mint and Ubuntu wanting (see APPENDIX 1). With the latter I two, I simply could not get the Apache server to log its rewriting of URLs with the Rewrite module. If you aren't able to help with an e-mail before 10.30AM tommorow, could you please consider texting me on 0412 319669 so I can ring you back and get advice from you after I have left home? Thanks, James Sinnamon APPENDIX 1 : Problems I experienced trying to create a Linux installation on which I could develop an test Drupal web-site It seems as if I won't be able to install the Linux configuration I need to update and test the Drupal content-managed site I help administer ( http://candobetter.net) I just could not get Apache to write logs of its rewriting of URLs on either my Ubuntu or Linux Mint distributions and have not yet got the advice I need from a Linuxquestions forum ( http://www.linuxquestions.org/questions/linux-server-73/how-to-log-apache-mo...). I tried installing Debian Linux, then Samboyan Linux. Debian just may have worked but it came with almost no Apache configuration files. I would have had to write everything myself or copy them from elsewhere, So I tried to install Samboyan Linux. Samboyan promised to work "out of the box" but after it spent many hours downloading over 1 Gig of files, most programs failed to work. Some programs simply failed to appear after I selected them from the menu, I wish I had persevered with Debian. What subsequently bothered me about Debian was that when I told it to re-boot it simply re-booted straight back to Debian! it somehow skipped over both BIOS and the Boot sector boot menu. I could only see BIOS or the boot sector meu if I hit the reset button. What had it done to BIOS and the boot sector to make that happen? It seems that Linux Mint also tampers with the boot sector making some earlier Linux installations inaccessible. I found that the Linux distributions I used were unable to cope with my having two hard disk drives. They created boot sectors which contradicted each other on both Hard disks. Also, confusingly more than one Linux distribution was written to the same partition. One partition seemed to have the exact same Linux distribution written to it twice.
On Sun, 16 Oct 2011, James Sinnamon <james.sinnamon@gmail.com> wrote:
amonT04). It seems that a 64 bit processor will be needed as a 32 bit processor cannot address more than 4 Gig of memory (see
A 32bit processor can address up to 64G of RAM with PAE. A 32bit process can't address more than 4G of RAM.
1. has sufficient power to run the multi-threaded Java application mentioned above; and
There probably aren't many people here who are going to read a research paper on short notice just to give you hardware advice. I will note however that I was running multi-threaded Java programs on servers 10 years ago when a P3 was a fast CPU and 2G of RAM was a lot. You don't necessarily need a lot of CPU power to do such things. But it really depends on what the application does. Presumably the research paper will have some data on the CPU requirements, why don't you summarise that in your next message to the list?
2. will mostly allow at least one Linux distribution to be installed and quickly without causing me anywhere the number of problems I have faced for at least the last two weeks (see APPENDIX 1).
I think that pretty much every desktop and server system will allow all the major Linux distributions to be installed without much effort.
I would also be interested in what Linux Distribution others would recommend. I have had serious problems with Debian and Saboyan and have found Linux Mint and Ubuntu wanting (see APPENDIX 1). With the latter I two, I simply could not get the Apache server to log its rewriting of URLs with the Rewrite module.
Actually it sounds like you have Drupal problems.
If you aren't able to help with an e-mail before 10.30AM tommorow, could you please consider texting me on 0412 319669 so I can ring you back and get advice from you after I have left home?
There probably aren't many people here who provide that service without a support contract. In the past one of my clients needed some advice and bug fixes in a hurry and offered $100US by Paypal for every issue that was solved. $100US was well accepted for such things. You might consider making a similar offer.
What subsequently bothered me about Debian was that when I told it to re-boot it simply re-booted straight back to Debian! it somehow skipped over both BIOS and the Boot sector boot menu. I could only see BIOS or the boot sector meu if I hit the reset button.
That's a BIOS issue. If the BIOS is configured to just boot without displaying a menu then there's nothing the OS boot loader (in this case GRUB) can do about it.
What had it done to BIOS and the boot sector to make that happen? It seems that Linux Mint also tampers with the boot sector making some earlier Linux installations inaccessible.
It's pretty much a default action to install a boot loader on the primary boot device as part of the OS install. You have to go out of your way to get a different result.
I found that the Linux distributions I used were unable to cope with my having two hard disk drives. They created boot sectors which contradicted each other on both Hard disks. Also, confusingly more than one Linux distribution was written to the same partition. One partition seemed to have the exact same Linux distribution written to it twice.
That sounds like a bug. The expected result is that the primary disk will be used by default and installing on the secondary disk may be an option. You could just unplug the secondary disk for the duration of the install process if you are worried about such things. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Russell Coker wrote:
I will note however that I was running multi-threaded Java programs on servers 10 years ago when a P3 was a fast CPU and 2G of RAM was a lot. You don't necessarily need a lot of CPU power to do such things. But it really depends on what the application does.
And how badly it is written, of course :-/
On 16/10/11 23:53, James Sinnamon wrote:
My existing hardware configuration and/or the Linux distributions I have tried to install have got the better of me. See APPENDIX 1. For this reason and because I want a PC with sufficient grunt to be able to run a java application (which will run the order of 100-200 or more threads (http://www.informatik.uni-trier.de/~ley/db/conf/pdpta/pdpta2004-3.html#Sinna...). It seems that a 64 bit processor will be needed as a 32 bit processor cannot address more than 4 Gig of memory (see http://developer.amd.com/Pages/1130200686.aspx http://www.intel.com/content/www/us/en/architecture-and-technology/microarch...).
I couldn't find a way to read the paper you linked to at uni-trier.de. It appears to be your paper, if I'm not mistaken, so perhaps you can provide a link to the actual article for us here? Secondly, I note that the paper was published in 2004. If that is the case, then surely *any* modern hardware will suffice - I mean, even if it took a powerful system to run the software in 2004, then you should be able to manage with desktop-grade gear easily now. A high-end CPU in 2004 was a 3.6 GHz Pentium 4, which had a single core; now you can get four or six core CPUs that run at that speed -- yet actually get twice as much done per cycle thanks to also doubling the memory bus speed and assorted other advances. (I'm looking at the Intel Core i5 2500, and the AMD Phenom II 1100T, both worth about two hundred dollars)
I intend to go to a hardware dealer (possibly MSY in Cheltenham tomorrow) so i need advice as to what will be a hardware configuration that:
1. has sufficient power to run the multi-threaded Java application mentioned above; and 2. will mostly allow at least one Linux distribution to be installed and quickly without causing me anywhere the number of problems I have faced for at least the last two weeks (see APPENDIX 1).
I'm a bit confused actually - if you wrote the paper, and presumably the java app it describes, then why are you unfamiliar with its hardware requirements?
I would also be interested in what Linux Distribution others would recommend. I have had serious problems with Debian and Saboyan and have found Linux Mint and Ubuntu wanting (see APPENDIX 1). With the latter I two, I simply could not get the Apache server to log its rewriting of URLs with the Rewrite module.
I quite like Ubuntu Server Edition myself. Reading your appendix below, it seems like your problems stem from your unfamiliarity with Apache configuration, rather than the particular Linux distribution you are using. -Toby [appendix snipped]
Toby Corkindale <toby.corkindale@strategicdata.com.au> wrote:
Secondly, I note that the paper was published in 2004. If that is the case, then surely *any* modern hardware will suffice - I mean, even if it took a powerful system to run the software in 2004, then you should be able to manage with desktop-grade gear easily now.
I would expect kernel changes made since 2004 to improve performance as well.
On Sun, 16 Oct 2011 11:53:26 PM James Sinnamon wrote:
sufficient grunt to be able to run a java application (which will run the order of 100-200 or more threads)
The issue isn't the number of threads, but what those threads will be doing - if only a handful of them are going to be active at any one time then a single modern processor is likely to be more than enough but if a lot of them are going to be CPU bound then you'll be wanting to aim for a box with as many cores as you can get. As for RAM - well Java is not known for its light memory limits, there's one Java code that's run at VLSCI that can fail if it gets less than 30GB or 60GB of RAM, but we believe that's because it's doing stupid things with its file IO. :-( cheers, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC This email may come with a PGP signature as a file. Do not panic. For more info see: http://en.wikipedia.org/wiki/OpenPGP
On Mon, 17 Oct 2011, Chris Samuel <chris@csamuel.org> wrote:
As for RAM - well Java is not known for its light memory limits, there's one Java code that's run at VLSCI that can fail if it gets less than 30GB or 60GB of RAM, but we believe that's because it's doing stupid things with its file IO. :-(
There are also Java implementations on smart cards with RAM measured in kilobytes. The RAM use is determined by what the application does and also by the garbage collection system. Often people tune their JVM to not do GC very often to save CPU time at the cost of using more RAM. One thing that's worth noting is that in many environments Java code is written by people who don't know much about how computers work at a low level (they are the people who can't write a C program that doesn't SEGV regularly), so they tend not to be efficient with memory use. I think that Java gets a bad reputation because of this. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On 17/10/11 23:25, Russell Coker wrote:
On Mon, 17 Oct 2011, Chris Samuel<chris@csamuel.org> wrote:
As for RAM - well Java is not known for its light memory limits, there's one Java code that's run at VLSCI that can fail if it gets less than 30GB or 60GB of RAM, but we believe that's because it's doing stupid things with its file IO. :-(
There are also Java implementations on smart cards with RAM measured in kilobytes.
The RAM use is determined by what the application does and also by the garbage collection system. Often people tune their JVM to not do GC very often to save CPU time at the cost of using more RAM.
One thing that's worth noting is that in many environments Java code is written by people who don't know much about how computers work at a low level (they are the people who can't write a C program that doesn't SEGV regularly), so they tend not to be efficient with memory use. I think that Java gets a bad reputation because of this.
Agreed. For instance, my mobile phone runs what is essentially a tuned version of Java (well, Dalvik via Apache Harmony), and it runs many simultaneous processes, quite fast, with each process using only a dozen or so megabytes of RAM. Likewise, I did some experimentation with OpenJDK, and was able to run some small java or scala apps in a dozen or so megabytes of RAM as well. However, the default java implementation on 64bit machines assumes that it can grab a gigabyte or so up front, and generally does so! :/ You can set some environment variables in your .bashrc to fix that though. Or, just turn linux's memory overcommit feature up to infinite, although I don't particularly like that method. -Toby
On Tue, Oct 18, 2011 at 10:43 AM, Jason White <jason@jasonjgw.net> wrote:
Toby Corkindale <toby.corkindale@strategicdata.com.au> wrote:
However, the default java implementation on 64bit machines assumes that it can grab a gigabyte or so up front, and generally does so! :/
I hope it wouldn't allocate a gigabyte just to run "hello world".
Thanks all for your responses. I have not responded sooner because I spent yesterday going to MST wher I spent almost $900 on the components to build new Intel 64-bit PC. (In retrospect, I think I should have just ordered a second hand PC from EBay. I am sure I could have got considerably more bang for buck there, but I am now committed to the course I have taken. ) Yes, it's my own Honours Year project of 1999 I want to implement. I think it still has some potential for improvement, even after all this time. If I don't manage to get it to run and am not able to modify it, I won't be that bothered, it it would be nice if I could. No, I don't remember the hardware requirements of 'Multipi'. It is over 10 years since I wrote it. The intervening time and my diffuse axonal brain damage has made my memory of it not as clear as it once was. Anyway, I would still like to be able to use a computer that is a bit closer to the leading edge than 32-bit processors. Thanks, James
On 18/10/11 10:43, Jason White wrote:
Toby Corkindale<toby.corkindale@strategicdata.com.au> wrote:
However, the default java implementation on 64bit machines assumes that it can grab a gigabyte or so up front, and generally does so! :/
I hope it wouldn't allocate a gigabyte just to run "hello world".
On my machine (w/6GB of RAM and a couple of GB of swap) java just allocated 1855M (virtual) of RAM to say "hello world" and sleep for 10 seconds. Of that, only 26M was resident, and about 9M of that was shared. However, if I pass the -Xmx2M flag, indicating I want it to only use 2M of memory, it'll still run just fine. However it still allocated 373M virtual, and 26M resident. So there's some overhead for the actual java system that's always allocated. But still.. by default allocating nearly 2GB?! Even with some degree of overcommit enabled, that's still crazy! I ran these tests with OpenJDK 6 and OpenJDK 7. Versions: OpenJDK Runtime Environment (IcedTea6 1.11pre) (6b23~pre10-0ubuntu5) OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode) OpenJDK Runtime Environment (IcedTea7 2.0pre (7~b147-2.0~pre6-1ubuntu1) OpenJDK 64-Bit Server VM (build 21.0-b17, mixed mode) Toby
On 18/10/11 10:43, Jason White wrote:
Toby Corkindale<toby.corkindale@strategicdata.com.au> wrote:
However, the default java implementation on 64bit machines assumes that it can grab a gigabyte or so up front, and generally does so!
:/
I hope it wouldn't allocate a gigabyte just to run "hello world".
On my machine (w/6GB of RAM and a couple of GB of swap) java just allocated 1855M (virtual) of RAM to say "hello world" and sleep for 10 seconds.
Can you post the output you parsed to come up with those figures? It's easy to look at a 'virtual memory usage' figure and jump to conclusions about memory usage but it's much more complicated than that. James
On 19/10/11 10:39, James Harper wrote:
On 18/10/11 10:43, Jason White wrote:
Toby Corkindale<toby.corkindale@strategicdata.com.au> wrote:
However, the default java implementation on 64bit machines assumes that it can grab a gigabyte or so up front, and generally does so!
:/
I hope it wouldn't allocate a gigabyte just to run "hello world".
On my machine (w/6GB of RAM and a couple of GB of swap) java just allocated 1855M (virtual) of RAM to say "hello world" and sleep for 10 seconds.
Can you post the output you parsed to come up with those figures?
It's easy to look at a 'virtual memory usage' figure and jump to conclusions about memory usage but it's much more complicated than that.
I was just looking at the output from 'top' - VIRT, RSS and SHR. I know it's more complicated than just that, but it's a good way to get a quick reading and I wasn't looking to spend more than a minute or so on this :) (I posted all three figures in my original post to the list.) It's worth noting that even though the figure is listed as "virtual", that if you try running a couple of instances up on a virtual machine with limited memory, then the virtual machine runs out of memory and things fail to work. (Tested on OpenVZ) So.. if you can't run more than two or three "hello world" programs at once on a machine with a couple of gig of ram.. it does appear like java really is allocating all that memory. Toby
participants (7)
-
Chris Samuel -
James Harper -
James Sinnamon -
Jason White -
Russell Coker -
Toby Corkindale -
Trent W. Buck