shapeworks-users

Text archives Help


Re: [shapeworks-users] unstable shapeworks results


Chronological Thread 
  • From: Shireen Elhabian <shireen@sci.utah.edu>
  • To: shapeworks-users@sci.utah.edu
  • Subject: Re: [shapeworks-users] unstable shapeworks results
  • Date: Thu, 18 Feb 2016 13:44:43 -0700
Yes Alan, this is the fix I did to make sure that the random generators are starting from the same seed each time ....

Shireen

On 02/18/2016 12:50 PM, Alan Morris wrote:
There are some random number generators in there as well.  When the particles split, they go in a random direction to separate themselves.  If you truly need identical results from the same runs, you will need to add a couple of lines of code to seed them with a fixed value.  I've done this before to debug various issues.

Alan

On Thu, Feb 18, 2016 at 12:36 PM, Shireen Elhabian <shireen@sci.utah.edu> wrote:

Openmp needs to be disabled to get identical results  ...

minor comments on the parameters: (1) you might want to increase the iterations per split to 1000 to give it more chance to spread over the shape. (2) starting regularization seem to be little bit large, you might want to try 10 or 100 (it sort of depend on the data set) (3) you might also want to crank up the optimization iterations to couple of thousands to make sure that you are converging to a stable shape model (i.e. not exiting too early from the optimization process)  ...


On 02/18/2016 12:21 PM, Murat Maga wrote:

Unfortunately, I didn’t get identical results using the commit you provided with openmp.

 

Here is what I did: First executed the groom and run, move the output*wpts to a new folder and execute the run one more time. Attached are the parameters in xml.

 

Is there anything I am missing?

 

ShapeWorksGroom /home/murat/SW/1/shapeworks.groom.xml center  isolate hole_fill antialias fastmarching blur

ShapeWorksRun /home/murat/SW/1/shapeworks.run.xml

 

Groom.xml

<antialias_iterations>100</antialias_iterations>

<background>0</background>

<foreground>2</foreground>

<transform_file>/home/murat/SW/1//shapeworks.transform</transform_file>

<inputs>/home/murat/SW/1/def_152-11_to_atlas.nii.gz_inv.nii.gz

/home/murat/SW/1/def_152-10_to_atlas.nii.gz_inv.nii.gz

/home/murat/SW/1/def_152-15_to_atlas.nii.gz_inv.nii.gz

/home/murat/SW/1/def_152-14_to_atlas.nii.gz_inv.nii.gz

/home/murat/SW/1/def_152-18_to_atlas.nii.gz_inv.nii.gz

/home/murat/SW/1/def_152-13_to_atlas.nii.gz_inv.nii.gz

/home/murat/SW/1/def_152-17_to_atlas.nii.gz_inv.nii.gz

/home/murat/SW/1/def_152-19_to_atlas.nii.gz_inv.nii.gz

/home/murat/SW/1/def_152-12_to_atlas.nii.gz_inv.nii.gz</inputs>

<outputs>/home/murat/SW/1/def_152-11_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-10_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-15_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-14_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-18_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-13_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-17_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-19_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-12_to_atlas_DT.nii.gz_inv_DT.nii.gz

</outputs>

 

Run.xml

<?xml version="1.0" encoding="UTF-8"?>

<iterations_per_split>100</iterations_per_split>

<starting_regularization>1000</starting_regularization>

<ending_regularization>0.1</ending_regularization>

<number_of_particles>1024</number_of_particles>

<optimization_iterations>1000</optimization_iterations>

<procrustes_scaling>0</procrustes_scaling>

<output_points_prefix>/home/murat/SW/1//output_points</output_points_prefix>

<output_transform_file>/home/murat/SW/1//output_transform_file</output_transform_file>

<inputs>/home/murat/SW/1/def_152-11_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-10_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-15_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-14_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-18_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-13_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-17_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-19_to_atlas_DT.nii.gz_inv_DT.nii.gz

/home/murat/SW/1/def_152-12_to_atlas_DT.nii.gz_inv_DT.nii.gz

</inputs>

 

 

 

From: shapeworks-users-request@sci.utah.edu [mailto:shapeworks-users-request@sci.utah.edu] On Behalf Of Murat Maga
Sent: Thursday, February 18, 2016 10:07 AM
To: Shireen Elhabian; shapeworks-users@sci.utah.edu
Subject: RE: [shapeworks-users] unstable shapeworks results

 

Hi Shireen,

 

Thanks a bunch. We are testing it right now, and I will let you know.

M

 

From: Shireen Elhabian [mailto:shireen@sci.utah.edu]
Sent: Thursday, February 18, 2016 9:20 AM
To: shapeworks-users@sci.utah.edu; Murat Maga
Subject: Re: [shapeworks-users] unstable shapeworks results

 


Hi Murat,

Please clone the latest source code of shapeworks (https://github.com/SCIInstitute/shapeworks) ... I have fixed this issue ...you should be able now to get repeatable results with the same set of shapes and parameters and with disabling openmp ... you might want to try this first on a small subset of shape just to make sure that it actually works ...

best regards
Shireen

On 02/17/2016 10:28 PM, Murat Maga wrote:

Hi Shireen,

 

There are few things going on I think.

 

First off using the provided SW studio windows binary we get repeatable results. But that’s not going to cut it for us, because I have 500 samples. I don’t have windows box that has the necessary amount of memory to run all the samples. At about 200 samples, it is maxing out on my 32GB RAM.

 

Therefore downloaded the latest linux commandline tool (from the website links) and compiled it. We did not get repeatable results with or without openMP enabled. When I execute the script on the same samples with using the same parameter set back to back, I get different results. I am not so concerned that the exact correspondence is not achieved. What I am concerned is if you rank them by their relative position to their respective mean (from closest to farthest) based on procrustes distance, ranking is not identical in those runs. That is quite concerning, at least to me.

 

So there is something we are missing in Linux causing the inconsistency, but I can’t quite figure it out. I even exported the groom and optimize xml files I used on windows and try on Linux and that didn’t make any difference.

 

M

 

 

 

From: shapeworks-users-request@sci.utah.edu [mailto:shapeworks-users-request@sci.utah.edu] On Behalf Of Shireen Elhabian
Sent: Wednesday, February 17, 2016 6:09 PM
To: shapeworks-users@sci.utah.edu
Subject: Re: [shapeworks-users] unstable shapeworks results

 


Hi,

1 . In order to get repeatable results, i.e same exact correspondence model, you might want to disable openmp at the expense of longer computation time ... nonetheless, with openmp, you still get similar correspondence model in terms of representation, yet not the same exact ordering and positional information.

2. points in wpts are already procrustes aligned and can directly be fed to PCA. we are working on an improved version of SW and will add the exportation of the eigenmodes and scores ... thanks for bringing this up  ... for the time being, please refer to ComputeModeLengths.cxx for utilities that would compute pca modes and scores ...


hope this helps ...

best regards
Shireen

On 02/17/2016 12:07 PM, Murat Maga wrote:

Hi,

 

I want to follow up on this. We want to use the PC scores from the shapeworks analysis as input for our quantitative trait mapping to find genomic intervals associated with molar morphology and size. We successfully used this approach using traditional landmarks, but I definitely see value using SW. However, there are couple concerns:

 

1.      We are not getting repeatable results (as Ryan described below) with identical datasets and parameter xmls. Parameter space is too big to randomly experiment on our own. We can use suggestions on what to focus and what might be driving this randomness.

2.      Although I can visualize the modes of variation using SW Studio, I can’t get to the individual eigenvectors or the PC scores. To overcome this, I am feeding the wpts files into a Procrustes analysis and doing alignment and PCA, which is kind of silly, since SW does it already. It would be good to expose the PC scores an eigenvectors.

 

Best wishes,

Murat

 

 

A. Murat Maga, PhD

Assistant Professor

UW Dept. of Pediatrics

Division of Craniofacial Medicine

&

Member

Seattle Children’s Research Institute

Center for Developmental Biology and Regenerative Medicine

 

 

 

 

From: shapeworks-users-request@sci.utah.edu [mailto:shapeworks-users-request@sci.utah.edu] On Behalf Of Young, Ryan
Sent: Tuesday, February 16, 2016 3:14 PM
To: shapeworks-users@sci.utah.edu
Subject: [shapeworks-users] unstable shapeworks results

 

Hello,

I have been using shapeworks to landmark six different mouse molars.  Shapeworks was run twice on each molar, with identical parameters and volumes.  The results are surprisingly different.  Regressing the distance to the mean shape of each the landmarks configuration between runs, 5-10 samples are drastically different.  See attached plot.  I was wondering if I am running the samples with enough iterations?  I am going to experiment with increasing the number of iteration to see if the results stabilize, but would greatly appreciate any suggestions.  The top of the groom.xml and run.xml files are below.

groom.xml

<?xml version="1.0" encoding="UTF-8"?>
<blur_sigma>0.5</blur_sigma>
<antialias_iterations>50</antialias_iterations>
<background>0</background>
<foreground>2</foreground>
<transform_file>/cm/shared/data/n2_molars/shapeworks/label1//shapeworks.transform</transform_file>
<inputs>/cm/shared/data/n2_molars/shapeworks/label1/def_152-11_to_atlas.nii.gz_inv.nii.gz

run.xml
<?xml version="1.0" encoding="UTF-8"?>
<iterations_per_split>100</iterations_per_split>
<starting_regularization>1000</starting_regularization>
<ending_regularization>0.1</ending_regularization>
<number_of_particles>1024</number_of_particles>
<optimization_iterations>1000</optimization_iterations>
<output_points_prefix>/cm/shared/data/n2_molars/shapeworks/label1//output_points</output_points_prefix>
<output_transform_file>/cm/shared/data/n2_molars/shapeworks/label1//output_transform_file</output_transform_file>
<inputs>/

CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information protected by law. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

 

 





Archive powered by MHonArc 2.6.18.

Top of page