How to evaluate different software? XCMS and Progenesis QI

Topic: How to evaluate different software? XCMS and Progenesis QI (Read 5035 times) previous topic - next topic

How to evaluate different software? XCMS and Progenesis QI

April 18, 2016, 10:29:07 AM

Dear all:

I believe that many people encounter such kind of question, XCMS and Progenesis QI, two kind of softwares who is better or more accuracy? Now I have a project, it contains many cell samples, the samples were analyzed in waters synapt G2 Si and the raw data was converted into the netCDF format by databridge software. Then the CDF files were processing by XCMS and QI, Finally, the table list had a large difference. the table list of XCMS is about 17000 features and the QI only is about 7000 features. So I am not sure who is better ? Maybe many nosie features go into the XCMS table list or maybe the QI lost the important features? Base on the example, I hope we may discuss the data processing workflow , how to ensure that the real difference existing in the raw data could be dig accurately by our data processing step ? If necessary, I can upload my R scripts command for everybody check. Thank you

Bset regards!

Bingpeng Yan

Re: How to evaluate different software? XCMS and Progenesis

Reply #1 – April 20, 2016, 12:43:31 PM

Hi,

software evaluation is always a tricky beast.

* You can ask three mass spectrometrists to evaluate the 7000+17000 features. Not going to happen.
* You can create a special evaluation experiment, e.g. http://pubs.acs.org/doi/abs/10.1021/ac301482k and evaluate which software has better results
* You can create a special evaluation experiment, e.g. http://bmcbioinformatics.biomedcentral. ... 2105-9-504 where the design allows to automatically designate a ground truth, and then make the task harder for the software, while the same ground truth should be detected.
* You can try to import the Progenesis output into an xcmsSet, and compare the quality score from the IPO package http://bmcbioinformatics.biomedcentral. ... 562-8#CR15. This has the benefit that you can do it on your existing data.

If you measure a well designed benchmark dataset, consider submitting it to http://www.ebi.ac.uk/metabolights .

Yours,
Steffen

Re: How to evaluate different software? XCMS and Progenesis

Reply #2 – April 21, 2016, 07:17:29 AM

Dear Steffen:

Thank you for your answer in details, you provide many useful information and suggestion. Actually, the XCMS optimized experiments was finished, and I will try other methods you provided to compare. The final aim is find real biological information from raw data, so thank you for you help again, I hope could consult with you profession questions in the future.

Best regards

Bingpeng Yan