Documentation for Preprocessor

scdataloader.preprocess.Preprocessor

Prepare data into training, valid and test split. Normalize raw expression values, binning or using other transform into the preset model input format.

Initializes the preprocessor and configures the workflow steps.

Your dataset should contain at least the following obs: - organism_ontology_term_id with the ontology id of the organism of your anndata - gene names in the var.index field of your anndata that map to the ensembl_gene nomenclature or the hugo gene symbols nomenclature (if the later, set is_symbol to True)

Parameters:
  • filter_gene_by_counts (int or bool, default: False ) –

    Determines whether to filter genes by counts. If int, filters genes with counts. Defaults to False.

  • filter_cell_by_counts (int or bool, default: False ) –

    Determines whether to filter cells by counts. If int, filters cells with counts. Defaults to False.

  • normalize_sum (float or bool, default: 10000.0 ) –

    Determines whether to normalize the total counts of each cell to a specific value. Defaults to 1e4.

  • n_hvg_for_postp (int or bool, default: 0 ) –

    Determines whether to subset to highly variable genes for the PCA. Defaults to False.

  • use_layer (str, default: None ) –

    The layer to use for preprocessing. Defaults to None.

  • is_symbol (bool, default: False ) –

    Whether genes are provided as symbols instead of Ensembl IDs. Defaults to False.

  • hvg_flavor (str, default: 'seurat_v3' ) –

    Specifies the flavor of highly variable genes selection. See :func:scanpy.pp.highly_variable_genes for more details. Defaults to "seurat_v3".

  • binning (int, default: None ) –

    Determines whether to bin the data into discrete values of number of bins provided.

  • result_binned_key (str, default: 'X_binned' ) –

    Specifies the key of :class:~anndata.AnnData to store the binned data. Defaults to "X_binned".

  • length_normalize (bool, default: False ) –

    Determines whether to length normalize the data. Defaults to False.

  • force_preprocess (bool, default: False ) –

    Determines whether to bypass the check of raw counts. Defaults to False.

  • min_dataset_size (int, default: 100 ) –

    The minimum size required for a dataset to be kept. Defaults to 100.

  • min_valid_genes_id (int, default: 10000 ) –

    The minimum number of valid genes to keep a dataset. Defaults to 10_000.

  • min_nnz_genes (int, default: 200 ) –

    The minimum number of non-zero genes to keep a cell. Defaults to 200.

  • maxdropamount (int, default: 50 ) –

    The maximum amount of dropped cells per dataset. (2 for 50% drop, 3 for 33% drop, etc.) Defaults to 2.

  • madoutlier (int, default: 5 ) –

    The maximum absolute deviation of the outlier samples. Defaults to 5.

  • pct_mt_outlier (int, default: 8 ) –

    The maximum percentage of mitochondrial genes outlier. Defaults to 8.

  • batch_keys (List[str], default: ['assay_ontology_term_id', 'self_reported_ethnicity_ontology_term_id', 'sex_ontology_term_id', 'donor_id', 'suspension_type'] ) –

    The keys of :class:~anndata.AnnData.obs to use for batch information. This arg is used in the highly variable gene selection step.

  • skip_validate (bool, default: True ) –

    Determines whether to skip the validation step. Defaults to False.

  • additional_preprocess (Callable, default: None ) –

    Additional preprocessing function. Defaults to None.

  • additional_postprocess (Callable, default: None ) –

    Additional postprocessing function. Defaults to None.

  • do_postp (bool, default: True ) –

    Whether to perform postprocessing. Defaults to True.

  • organisms (List[str], default: ['NCBITaxon:9606', 'NCBITaxon:10090'] ) –

    List of organisms to support. Defaults to ["NCBITaxon:9606", "NCBITaxon:10090"].

  • use_raw (bool, default: True ) –

    Whether to use raw counts. Defaults to True.

  • keepdata (bool, default: False ) –

    Determines whether to keep the data in the AnnData object. Defaults to False.

  • drop_non_primary (bool, default: False ) –

    Determines whether to drop non-primary cells. Defaults to False.

Source code in scdataloader/preprocess.py
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
def __init__(
    self,
    filter_gene_by_counts: Union[int, bool] = False,
    filter_cell_by_counts: Union[int, bool] = False,
    normalize_sum: float = 1e4,
    n_hvg_for_postp: int = 0,
    use_layer: Optional[str] = None,
    is_symbol: bool = False,
    hvg_flavor: str = "seurat_v3",
    binning: Optional[int] = None,
    result_binned_key: str = "X_binned",
    length_normalize: bool = False,
    force_preprocess: bool = False,
    min_dataset_size: int = 100,
    min_valid_genes_id: int = 10_000,
    min_nnz_genes: int = 200,
    maxdropamount: int = 50,
    madoutlier: int = 5,
    pct_mt_outlier: int = 8,
    batch_keys: List[str] = [
        "assay_ontology_term_id",
        "self_reported_ethnicity_ontology_term_id",
        "sex_ontology_term_id",
        "donor_id",
        "suspension_type",
    ],
    skip_validate: bool = True,
    additional_preprocess: Optional[Callable[[AnnData], AnnData]] = None,
    additional_postprocess: Optional[Callable[[AnnData], AnnData]] = None,
    do_postp: bool = True,
    organisms: List[str] = ["NCBITaxon:9606", "NCBITaxon:10090"],
    use_raw: bool = True,
    keepdata: bool = False,
    drop_non_primary: bool = False,
) -> None:
    """
    Initializes the preprocessor and configures the workflow steps.

    Your dataset should contain at least the following obs:
    - `organism_ontology_term_id` with the ontology id of the organism of your anndata
    - gene names in the `var.index` field of your anndata that map to the ensembl_gene nomenclature
    or the hugo gene symbols nomenclature (if the later, set `is_symbol` to True)

    Args:
        filter_gene_by_counts (int or bool, optional): Determines whether to filter genes by counts.
            If int, filters genes with counts. Defaults to False.
        filter_cell_by_counts (int or bool, optional): Determines whether to filter cells by counts.
            If int, filters cells with counts. Defaults to False.
        normalize_sum (float or bool, optional): Determines whether to normalize the total counts of each cell to a specific value.
            Defaults to 1e4.
        n_hvg_for_postp (int or bool, optional): Determines whether to subset to highly variable genes for the PCA.
            Defaults to False.
        use_layer (str, optional): The layer to use for preprocessing.
            Defaults to None.
        is_symbol (bool, optional): Whether genes are provided as symbols instead of Ensembl IDs.
            Defaults to False.
        hvg_flavor (str, optional): Specifies the flavor of highly variable genes selection.
            See :func:`scanpy.pp.highly_variable_genes` for more details. Defaults to "seurat_v3".
        binning (int, optional): Determines whether to bin the data into discrete values of number of bins provided.
        result_binned_key (str, optional): Specifies the key of :class:`~anndata.AnnData` to store the binned data.
            Defaults to "X_binned".
        length_normalize (bool, optional): Determines whether to length normalize the data.
            Defaults to False.
        force_preprocess (bool, optional): Determines whether to bypass the check of raw counts.
            Defaults to False.
        min_dataset_size (int, optional): The minimum size required for a dataset to be kept.
            Defaults to 100.
        min_valid_genes_id (int, optional): The minimum number of valid genes to keep a dataset.
            Defaults to 10_000.
        min_nnz_genes (int, optional): The minimum number of non-zero genes to keep a cell.
            Defaults to 200.
        maxdropamount (int, optional): The maximum amount of dropped cells per dataset. (2 for 50% drop, 3 for 33% drop, etc.)
            Defaults to 2.
        madoutlier (int, optional): The maximum absolute deviation of the outlier samples.
            Defaults to 5.
        pct_mt_outlier (int, optional): The maximum percentage of mitochondrial genes outlier.
            Defaults to 8.
        batch_keys (List[str], optional): The keys of :class:`~anndata.AnnData.obs` to use for batch information.
            This arg is used in the highly variable gene selection step.
        skip_validate (bool, optional): Determines whether to skip the validation step.
            Defaults to False.
        additional_preprocess (Callable, optional): Additional preprocessing function.
            Defaults to None.
        additional_postprocess (Callable, optional): Additional postprocessing function.
            Defaults to None.
        do_postp (bool, optional): Whether to perform postprocessing.
            Defaults to True.
        organisms (List[str], optional): List of organisms to support.
            Defaults to ["NCBITaxon:9606", "NCBITaxon:10090"].
        use_raw (bool, optional): Whether to use raw counts.
            Defaults to True.
        keepdata (bool, optional): Determines whether to keep the data in the AnnData object.
            Defaults to False.
        drop_non_primary (bool, optional): Determines whether to drop non-primary cells.
            Defaults to False.
    """
    self.filter_gene_by_counts = filter_gene_by_counts
    self.filter_cell_by_counts = filter_cell_by_counts
    self.normalize_sum = normalize_sum
    self.hvg_flavor = hvg_flavor
    self.binning = binning
    self.organisms = organisms
    self.result_binned_key = result_binned_key
    self.additional_preprocess = additional_preprocess
    self.additional_postprocess = additional_postprocess
    self.force_preprocess = force_preprocess
    self.min_dataset_size = min_dataset_size
    self.min_valid_genes_id = min_valid_genes_id
    self.min_nnz_genes = min_nnz_genes
    self.maxdropamount = maxdropamount
    self.drop_non_primary = drop_non_primary
    self.madoutlier = madoutlier
    self.n_hvg_for_postp = n_hvg_for_postp
    self.pct_mt_outlier = pct_mt_outlier
    self.batch_keys = batch_keys
    self.length_normalize = length_normalize
    self.skip_validate = skip_validate
    self.use_layer = use_layer
    self.is_symbol = is_symbol
    self.do_postp = do_postp
    self.use_raw = use_raw
    self.keepdata = keepdata

scdataloader.preprocess.LaminPreprocessor

Bases: Preprocessor

Methods:

Name Description
__call__

Process data with format controlling different input value wrapping.

Source code in scdataloader/preprocess.py
474
475
476
477
478
479
480
481
482
483
484
485
486
487
def __init__(
    self,
    *args,
    cache: bool = True,
    keep_files: bool = True,
    force_lamin_cache: bool = False,
    assays_to_drop: List[str] = ["EFO:0008939"],
    **kwargs,
):
    super().__init__(*args, **kwargs)
    self.cache = cache
    self.keep_files = keep_files
    self.force_lamin_cache = force_lamin_cache
    self.assays_to_drop = assays_to_drop

__call__

Process data with format controlling different input value wrapping.

Includes support for categorical binned style, fixed-sum normalized counts, log1p fixed-sum normalized counts, etc.

Parameters:
  • data (Union[Collection, AnnData], default: None ) –

    The AnnData object or Collection to preprocess.

  • name (str, default: 'preprocessed dataset' ) –

    Name for the preprocessed dataset. Defaults to "preprocessed dataset".

  • description (str, default: 'preprocessed dataset using scprint' ) –

    Description for the preprocessed dataset. Defaults to "preprocessed dataset using scprint".

  • start_at (int, default: 0 ) –

    Starting index for resuming preprocessing. Defaults to 0.

  • version (str, default: '2' ) –

    Version string for the dataset. Defaults to "2".

Source code in scdataloader/preprocess.py
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
def __call__(
    self,
    data: Union[ln.Collection, AnnData] = None,
    name: str = "preprocessed dataset",
    description: str = "preprocessed dataset using scprint",
    start_at: int = 0,
    version: str = "2",
):
    """
    Process data with format controlling different input value wrapping.

    Includes support for categorical binned style, fixed-sum normalized counts,
    log1p fixed-sum normalized counts, etc.

    Args:
        data (Union[ln.Collection, AnnData]): The AnnData object or Collection to preprocess.
        name (str, optional): Name for the preprocessed dataset. Defaults to "preprocessed dataset".
        description (str, optional): Description for the preprocessed dataset.
            Defaults to "preprocessed dataset using scprint".
        start_at (int, optional): Starting index for resuming preprocessing.
            Defaults to 0.
        version (str, optional): Version string for the dataset.
            Defaults to "2".
    """
    files = []
    all_ready_processed_keys = set()
    if self.cache:
        for i in ln.Artifact.filter(description=description):
            all_ready_processed_keys.add(i.stem_uid)
    if isinstance(data, AnnData):
        return super().__call__(data)
    elif isinstance(data, ln.Collection):
        for i, file in enumerate(data.artifacts.all()[start_at:]):
            # use the counts matrix
            i = i + start_at
            print(i)
            if file.stem_uid in all_ready_processed_keys:
                print(f"{file.stem_uid} is already processed... not preprocessing")
                continue
            print(file)
            if self.force_lamin_cache:
                path = cache_path(file)
                backed = read_h5ad(path, backed="r")
            else:
                # file.cache()
                backed = file.open()

            if "is_primary_data" in backed.obs.columns:
                if backed.obs.is_primary_data.sum() == 0:
                    print(f"{file.key} only contains non primary cells.. dropping")
                    # Save the stem_uid to a file to avoid loading it again
                    with open("nonprimary.txt", "a") as f:
                        f.write(f"{file.stem_uid}\n")
                    continue
            else:
                print("Warning: couldn't check unicity from is_primary_data column")
            if backed.obs.assay_ontology_term_id[0] in self.assays_to_drop:
                print(f"{file.key} is in the assay drop list.. dropping")
                continue
            if backed.shape[1] < 1000:
                print(
                    f"{file.key} only contains less than 1000 genes and is likely not scRNAseq... dropping"
                )
                continue
            if file.size <= MAXFILESIZE:
                adata = backed.to_memory()
                print(adata)
            else:
                badata = backed
                print(badata)
            try:
                if file.size > MAXFILESIZE:
                    print(
                        f"dividing the dataset as it is too large: {file.size // 1_000_000_000}Gb"
                    )
                    num_blocks = int(np.ceil(file.size / (MAXFILESIZE / 2)))
                    block_size = int(
                        (np.ceil(badata.shape[0] / 30_000) * 30_000) // num_blocks
                    )
                    print(
                        "num blocks ",
                        num_blocks,
                        "block size ",
                        block_size,
                        "total elements ",
                        badata.shape[0],
                    )
                    for j in range(num_blocks):
                        start_index = j * block_size
                        end_index = min((j + 1) * block_size, badata.shape[0])
                        block = badata[start_index:end_index]
                        block = block.to_memory()
                        print(block)
                        block = super().__call__(
                            block,
                            dataset_id=file.stem_uid + "_p" + str(j),
                        )
                        saved = False
                        while not saved:
                            try:
                                myfile = ln.Artifact.from_anndata(
                                    block,
                                    description=description
                                    + " n"
                                    + str(i)
                                    + " p"
                                    + str(j)
                                    + " ( revises file "
                                    + str(file.stem_uid)
                                    + " )",
                                    version=version,
                                )
                                myfile.save()
                                saved = True
                            except OperationalError:
                                print(
                                    "Database locked, waiting 30 seconds and retrying..."
                                )
                                time.sleep(10)
                        if self.keep_files:
                            files.append(myfile)
                            del block
                        else:
                            del myfile
                            del block
                else:
                    adata = super().__call__(adata, dataset_id=file.stem_uid)
                    saved = False
                    while not saved:
                        try:
                            myfile = ln.Artifact.from_anndata(
                                adata,
                                # revises=file,
                                description=description + " p" + str(i),
                                version=version,
                            )
                            myfile.save()
                            saved = True
                        except OperationalError:
                            print(
                                "Database locked, waiting 10 seconds and retrying..."
                            )
                            time.sleep(10)
                    if self.keep_files:
                        files.append(myfile)
                        del adata
                    else:
                        del myfile
                        del adata

            except ValueError as v:
                if v.args[0].startswith("we cannot work with this organism"):
                    print(v)
                    continue
                else:
                    raise v
            except Exception as e:
                if e.args[0].startswith("Dataset dropped due to"):
                    print(e)
                    continue
                else:
                    raise e
            gc.collect()
            # issues with KLlggfw6I6lvmbqiZm46
        if self.keep_files:
            # Reconstruct collection using keys
            dataset = ln.Collection(
                [ln.Artifact.filter(key=k).one() for k in files],
                name=name,
                description=description,
            )
            dataset.save()
            return dataset
        else:
            return
    else:
        raise ValueError("Please provide either anndata or ln.Collection")

scdataloader.preprocess.additional_preprocess

Source code in scdataloader/preprocess.py
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
def additional_preprocess(adata):
    adata.obs = adata.obs.replace(
        {
            "self_reported_ethnicity_ontology_term_id": {
                "multiethnic": "unknown",
                "American": "unknown",
                "Jewish Israeli": "unknown",
                "na": "unknown",
            }
        }
    )  # multi ethnic will have to get renamed
    adata.obs["cell_culture"] = "False"
    # if cell_type contains the word "(cell culture)" then it is a cell culture and we mark it as so and remove this from the cell type
    loc = adata.obs["cell_type_ontology_term_id"].str.contains(
        "(cell culture)", regex=False
    )
    if loc.sum() > 0:
        adata.obs["cell_type_ontology_term_id"] = adata.obs[
            "cell_type_ontology_term_id"
        ].astype(str)
        adata.obs.loc[loc, "cell_culture"] = "True"
        adata.obs.loc[loc, "cell_type_ontology_term_id"] = adata.obs.loc[
            loc, "cell_type_ontology_term_id"
        ].str.replace(" (cell culture)", "")

    loc = adata.obs["tissue_ontology_term_id"].str.contains(
        "(cell culture)", regex=False
    )
    if loc.sum() > 0:
        adata.obs.loc[loc, "cell_culture"] = "True"
        adata.obs["tissue_ontology_term_id"] = adata.obs[
            "tissue_ontology_term_id"
        ].astype(str)
        adata.obs.loc[loc, "tissue_ontology_term_id"] = adata.obs.loc[
            loc, "tissue_ontology_term_id"
        ].str.replace(" (cell culture)", "")

    loc = adata.obs["tissue_ontology_term_id"].str.contains("(organoid)", regex=False)
    if loc.sum() > 0:
        adata.obs.loc[loc, "cell_culture"] = "True"
        adata.obs["tissue_ontology_term_id"] = adata.obs[
            "tissue_ontology_term_id"
        ].astype(str)
        adata.obs.loc[loc, "tissue_ontology_term_id"] = adata.obs.loc[
            loc, "tissue_ontology_term_id"
        ].str.replace(" (organoid)", "")

    loc = adata.obs["tissue_ontology_term_id"].str.contains("CL:", regex=False)
    if loc.sum() > 0:
        adata.obs["tissue_ontology_term_id"] = adata.obs[
            "tissue_ontology_term_id"
        ].astype(str)
        adata.obs.loc[loc, "tissue_ontology_term_id"] = "unknown"
    return adata

scdataloader.preprocess.additional_postprocess

Source code in scdataloader/preprocess.py
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
def additional_postprocess(adata):
    # import palantir

    # define the "up to" 10 neighbors for each cells and add to obs
    # compute neighbors
    # need to be connectivities and same labels [cell type, assay, dataset, disease]
    # define the "neighbor" up to 10(N) cells and add to obs
    # define the "next time point" up to 5(M) cells and add to obs  # step 1: filter genes
    # if len(adata.obs["batches"].unique()) > 1:
    #    sc.external.pp.harmony_integrate(adata, key="batches")
    #    sc.pp.neighbors(adata, use_rep="X_pca_harmony")
    # else:
    print("starting post processing")
    sc.pp.neighbors(adata, use_rep="X_pca")
    sc.tl.leiden(adata, key_added="leiden_2", resolution=2.0)
    sc.tl.leiden(adata, key_added="leiden_1", resolution=1.0)
    sc.tl.leiden(adata, key_added="leiden_0.5", resolution=0.5)
    sc.tl.umap(adata)
    mid = adata.uns["dataset_id"] if "dataset_id" in adata.uns else "unknown_id"
    sc.pl.umap(
        adata,
        ncols=1,
        color=["cell_type", "batches"],
        save="_" + mid + ".png",
    )
    COL = "cell_type_ontology_term_id"
    NEWOBS = "clust_cell_type"
    MINCELLS = 10
    MAXSIM = 0.94
    from collections import Counter

    import bionty as bt

    from .config import MAIN_HUMAN_MOUSE_DEV_STAGE_MAP

    remap_stages = {u: k for k, v in MAIN_HUMAN_MOUSE_DEV_STAGE_MAP.items() for u in v}

    adata.obs[NEWOBS] = (
        adata.obs[COL].astype(str) + "_" + adata.obs["leiden_1"].astype(str)
    )
    coun = Counter(adata.obs[NEWOBS])
    relab = {}
    for i in adata.obs[COL].unique():
        num = 0
        for n, c in sorted(coun.items(), key=lambda x: x[1], reverse=True):
            if i in n:
                if c < MINCELLS or num == 0:
                    relab[n] = i
                else:
                    relab[n] = i + "_" + str(num)
                num += 1

    adata.obs[NEWOBS] = adata.obs[NEWOBS].map(relab)

    cluster_means = pd.DataFrame(
        np.array(
            [
                adata.X[adata.obs[NEWOBS] == i].mean(axis=0)
                for i in adata.obs[NEWOBS].unique()
            ]
        )[:, 0, :],
        index=adata.obs[NEWOBS].unique(),
    )

    # Calculate correlation matrix between clusters
    cluster_similarity = cluster_means.T.corr()
    cluster_similarity.values[np.tril_indices(len(cluster_similarity), -1)] = 0

    # Get pairs with similarity > 0.95
    high_sim_pairs = []
    for i in range(len(cluster_similarity)):
        for j in range(i + 1, len(cluster_similarity)):
            if (
                cluster_similarity.iloc[i, j] > MAXSIM
                and cluster_similarity.columns[i].split("_")[0]
                == cluster_similarity.columns[j].split("_")[0]
            ):
                high_sim_pairs.append(
                    (
                        cluster_similarity.index[i],
                        cluster_similarity.columns[j],
                    )
                )
    # Create mapping for merging similar clusters
    merge_mapping = {}
    for pair in high_sim_pairs:
        if pair[0] not in merge_mapping:
            merge_mapping[pair[1]] = pair[0]
        else:
            merge_mapping[pair[1]] = merge_mapping[pair[0]]

    # Apply merging
    adata.obs[NEWOBS] = adata.obs[NEWOBS].map(merge_mapping).fillna(adata.obs[NEWOBS])
    adata.obs[NEWOBS] = adata.obs[NEWOBS].astype(str)
    coun = Counter(adata.obs[NEWOBS]).most_common()
    merge_mapping = {}
    for i in adata.obs[COL].unique():
        num = 0
        for j, c in coun:
            if i in j:
                merge_mapping[j] = i + "_" + str(num) if num > 0 else i
                num += 1
    adata.obs[NEWOBS] = adata.obs[NEWOBS].map(merge_mapping).fillna(adata.obs[NEWOBS])

    stages = adata.obs["development_stage_ontology_term_id"].unique()
    if adata.obs.organism_ontology_term_id.unique() == ["NCBITaxon:9606"]:
        relabel = {i: i for i in stages}
        for stage in stages:
            if stage in MAIN_HUMAN_MOUSE_DEV_STAGE_MAP.keys():
                continue
            stage_obj = bt.DevelopmentalStage.filter(ontology_id=stage).first()
            parents = set([i.ontology_id for i in stage_obj.parents.filter()])
            parents = parents - set(
                [
                    "HsapDv:0010000",
                    "HsapDv:0000227",
                ]
            )
            if len(parents) > 0:
                for p in parents:
                    if p in MAIN_HUMAN_MOUSE_DEV_STAGE_MAP:
                        relabel[stage] = p
        adata.obs["age_group"] = adata.obs["development_stage_ontology_term_id"].map(
            relabel
        )
        for stage in adata.obs["age_group"].unique():
            if stage in remap_stages.keys():
                adata.obs["age_group"] = adata.obs["age_group"].map(
                    lambda x: remap_stages[x] if x == stage else x
                )
    elif adata.obs.organism_ontology_term_id.unique() == ["NCBITaxon:10090"]:
        rename_mapping = {
            k: v for v, j in MAIN_HUMAN_MOUSE_DEV_STAGE_MAP.items() for k in j
        }
        relabel = {i: "unknown" for i in stages}
        for stage in stages:
            if stage in rename_mapping:
                relabel[stage] = rename_mapping[stage]
        adata.obs["age_group"] = adata.obs["development_stage_ontology_term_id"].map(
            relabel
        )
    else:
        # raise ValueError("organism not supported")
        print("organism not supported for age labels")
    # palantir.utils.run_diffusion_maps(adata, n_components=20)
    # palantir.utils.determine_multiscale_space(adata)
    # terminal_states = palantir.utils.find_terminal_states(
    #    adata,
    #    celltypes=adata.obs.cell_type_ontology_term_id.unique(),
    #    celltype_column="cell_type_ontology_term_id",
    # )
    # sc.tl.diffmap(adata)
    # adata.obs["heat_diff"] = 1
    # for terminal_state in terminal_states.index.tolist():
    #    adata.uns["iroot"] = np.where(adata.obs.index == terminal_state)[0][0]
    #    sc.tl.dpt(adata)
    #    adata.obs["heat_diff"] = np.minimum(
    #        adata.obs["heat_diff"], adata.obs["dpt_pseudotime"]
    #    )
    return adata