Use several computers to do the calculation?

Topic: Use several computers to do the calculation? (Read 8700 times) previous topic - next topic

Use several computers to do the calculation?

June 05, 2012, 03:07:34 AM

Hi,

In my lab we haven't dedicated server to do XCMS processing, so it can be really long (depending on number of sample, parameters used....) or even impossible on our PCs.
I was wondering if it's possible to use several computers at the time (connected to our network) to do the calculations?

If it's impossible, could you comment on the characteristics needed on a dedicated server ( one process at the same time / 70 samples max / centwave / obiwarp...) to be fine?

Many thanks in advance for any help,

Cheers,
Sylvain.

Re: Use several computers to do the calculation?

Reply #1 – June 05, 2012, 02:26:58 PM

Hi,

an important question upfront is unfortunately still,
whether the computers in the Lab are windows or Unix/Linux.
I doubt that the efforts to get distributed jobs on windows
to run reliably will be worth it.

But a beefy PC should be able to do it just fine,
look for e.g. Intel 6core and at least 12GB RAM (or 16GB),
they should sell between €1000,- (no-name)
and €2000 (brands like HP or Fujitsu). With such a machine
you can use nSlaves=6 or even =12 (Hyperthreading),
and it should be able to finish 70 samples over lunchtime
or an extended coffee. YMMV.

Yours,
Steffen

Re: Use several computers to do the calculation?

Reply #2 – June 05, 2012, 03:03:50 PM

Did I understand that you have only about 70 samples?

My workstation is just a measly Core2duo 3GHz; But after I added 16GB of RAM I can without issue process 500 samples (6mins each); though granted it takes about a day.
The key to be able to process the samples is memory; if you don't have enough the speed will drop to nothing.
The key to getting a good speed is as Steffen said a fast computer with several cores.

I would go for the cheap solution first and add RAM (something like 16GB) to the machine you have. RAM is very cheap at the moment. A very fast computer not so much! It really depends on your budget and needs.

And as Steffen said use nSlaves if you have several cores.

Jan.

Re: Use several computers to do the calculation?

Reply #3 – June 05, 2012, 03:53:46 PM

You could process the samples on many different computers and then combine the xcmsSet objects back together on a single computer. However, this single computer will need to have the RAM to hold the final xcmsSet object (shouldn't be much for 70 samples). This will only work for the peak detection side of things. After that i.e. the grouping, retention time alignment you will need to process on the same computer.

To combine the xcmsSets from different computers make sure all the parameters are the same and then use something like the following code:

Code: [Select]

## computer 1:
xs1<-xcmsSet()
save(xs1, file="xs1.Rda")

## computer 2:
xs2<-xcmsSet()
## transfer xs1 to computer 2
load("xs1.Rda")
xset<-c(xs1,xs2)
xset

Re: Use several computers to do the calculation?

Reply #4 – June 06, 2012, 12:56:57 AM

Hi,
First of all, thanks for the fast replies!
the nSlaves parameters helped a lot

.

Thanks again,

Sylvain.