Analyzing databases in parallel using the Python API

The parallel_runners module contains the core analysis pipelines used to identify and compare neuronal ensembles.

These functions can be executed exclusively from Python scripts.

Each parallel runner implements a complete analysis workflow for a set of databases, including:

  • Input validation

  • Algorithm selection

  • Execution

  • Result formatting

See also

Read the documentation for the parallel runners over databases Parallel Runners

Example parallel run

 1import numpy as np
 2
 3databases = {}
 4neurons = 100
 5timepoints = 3600
 6orientations = 4
 7directions = 8
 8behaviors = 2
 9orientations_labels = ["0", "45", "90", "135"]
10directions_labels = ["0", "45", "90", "135", "180", "225", "270", "315"]
11behaviors_labels = ["speed", "pupil"]
12
13for database_id in ["MiceA", "MiceB", "MiceC"]:
14   # The binary activity matrix for the neurons
15   spikes_matrix = np.random.rand(neurons, timepoints)
16   spikes_matrix = (spikes_matrix > 0.9).astype(np.int_)
17
18   # A binary matrix for the orientations presentation
19   orientations_matrix = np.random.rand(orientations, timepoints)
20   orientations_matrix = (orientations_matrix > 0.75).astype(np.int_)
21
22   # A binary matrix for the directions presentation
23   directions_matrix = np.random.rand(directions, timepoints)
24   directions_matrix = (directions_matrix > 0.75).astype(np.int_)
25
26   # A continuous matrix for the behaviors
27   behaviors_matrix = np.random.rand(behaviors, timepoints)
28
29   databases[database_id] = {
30         "spikes": spikes_matrix,
31         "orientations": orientations_matrix,
32         "directions": directions_matrix,
33         "behaviors": behaviors_matrix,
34         "orientations_labels": orientations_labels,
35         "directions_labels": directions_labels,
36         "behaviors_labels": behaviors_labels
37   }

Define the parameters to use for each database and add a fallback if needed.

 1sessions_parameters = {
 2   "default": {
 3      "analysis": "svd",
 4      "data_names": {"data_neuronal_activity": "spikes"},
 5      "parameters": {
 6            "pks": 5,
 7            "scut": 0.4,
 8            "hcut": 0.4,
 9            "state_cut": 6,
10            "csi_start": 0.01,
11            "csi_step": 0.01,
12            "csi_end": 0.1,
13            "tf_idf_norm": True,
14            "parallel_processing": False,
15            "fixed_ens_cant": 0,
16      },
17      "evaluate_similarity": True,
18      "similarity_elements": ["orientations", "directions"],
19   },
20   "MiceB": {
21         "analysis": "ica",
22         "data_names": {"data_neuronal_activity": "spikes"},
23         "parameters": {
24             "threshold_method": "MarcenkoPastur",
25             "permutations_percentile": 95.0,
26             "number_of_permutations": 20,
27             "min_ensembles_cant": 4,
28             "max_ensembles_cant": 10,
29             "patterns_method": "ICA",
30             "number_of_iterations": 1000,
31             "threshold_for_p_value": 1.90,
32         },
33         "evaluate_similarity": True,
34         "similarity_elements": ["orientations", "behaviors"],
35   },
36}

These parameters indicates that the databases with ID MiceA and MiceC will be analyzed with the SVD algorithm, while the database MiceB will be analyzed with the algorithm ICA.

For MiceB the similarity will be computed between the activity of the identified ensembles and the presentation of orientations and the behaviors. For MiceA and MiceC the similarities will be computed for the presentation of stimuli only.

Important

In the example, we are using the variables "orientations" and "directions" for the similarities comparisons, as stated in the variable sessions_parameters["default"]["similarity_elements"].

It’s important for clarity to also include variables with the labels for each one. The runner will append the string "_labels" at the end of the name of the variable for similarity and will look for that variable in the databases[database_id] dictionary. That’s why here I included the variables databases[database_id]["orientations_labels"] and databases[database_id]["directions_labels"].

If no label is provided the index of each element will be used as label.

See also

Check the Algorithms configuration file and Python API Usage for a better understanding of the data and parameters needed for each algorithm.

Important

When using parallel execution of the algorithms do not to use the parallel processing options of any algorithm, like SVD, X2P or PCA.

Then call the parallel runner with:

1from encore.parallel_runners.sessions import run_parallel_sessions
2
3workers = 4 # This depends on your computer's available resources
4
5results = run_parallel_sessions(
6   data=databases,
7   parameters=sessions_parameters,
8   max_workers_cant=workers,
9)

Note

Each worker will create a new python instance with a new MATLAB instance each.

Results structure

The results of this analysis will produce a python directory with the following fields:

  1results = {
  2   "info": {
  3      "analyzer": "ENCORE Parallel Sessions API", # str, Identifier of this analysis
  4      "date": "250826_143355",            # str, with the date, formatted DDMMYY_HHMMSS
  5      "ENCORE_version": "3.0.0"           # str, showing ENCORE version used for the analysis
  6   },
  7
  8   "parameters": {
  9      # The parameters passed to the parallel analysis, a copy of the input
 10      # In this example it is:
 11      "default": {
 12         "analysis": "svd",
 13         "data_names": {"data_neuronal_activity": "spikes"},
 14         "parameters": {
 15               "pks": 5,
 16               "scut": 0.4,
 17               "hcut": 0.4,
 18               # ...
 19         },
 20         "evaluate_similarity": True,
 21         "similarity_elements": ["orientations", "directions"],
 22      },
 23      "MiceB": {
 24         "analysis": "ica",
 25         # ...
 26      },
 27   },
 28
 29   "parameters_used": {
 30      # Each key is the name of each database analyzed, with the parameters used for that one
 31      # In this case MiceA and MiceB have the same parameters because those used the default parameter
 32      "MiceA": {
 33         "analysis": "svd",
 34         "data_names": {"data_neuronal_activity": "spikes"},
 35         # ...
 36         "similarity_elements": ["orientations", "directions"],
 37      },
 38      "MiceB": {
 39         "analysis": "ica",
 40         "data_names": {"data_neuronal_activity": "spikes"},
 41         # ...
 42         "similarity_elements": ["orientations", "behaviors"],
 43      },
 44      "MiceC": {
 45         "analysis": "svd",
 46         "data_names": {"data_neuronal_activity": "spikes"},
 47         # ...
 48         "similarity_elements": ["orientations", "directions"],
 49      },
 50   },
 51
 52   "results": {
 53      # The most important one, each key is the name of each database used
 54      "MiceA": {
 55         "success": True, # bool, Shows if the analysis terminated successfully
 56
 57         "results": {
 58            # The minimal results of the algorithm
 59            "ensembles_cant": 8, # int, number of ensembles identified
 60            "neus_in_ens": np.ndarray,  # 2D binary numpy array showing what neurons belong to each ensemble
 61                                       # with shape (N, E) for N neurons and E identified ensembles
 62            "timecourse": np.ndarray,   # 2D binary array showing the activity of each ensemble in time
 63                                       # with shape (T, E) for T timepoints and E ensembles.
 64         },
 65
 66         "similarity": {
 67            # Contains a pair of similarity_metric: similarity_matrix
 68            # The keys are the names of the metric used
 69            # The matrix are square matrix, each side with the number of ensembles identified
 70            # plus the number of elements in the similarity variables
 71            # In this case (8 ensembles) + (4 orientations) + (8 directions)
 72            "Correlation": np.ndarray,
 73            "Cosine": np.ndarray,
 74            "Euclidean": np.ndarray, # The euclidean distance
 75            "Jaccard": np.ndarray, # The Jaccard index
 76         },
 77
 78         # The labels for the elements in the similarity matrices, in this case
 79         # ["ens 0", "ens 1", ..., "ens 8", "ori 0", ..., "ori 135", "dir 0", ..., "dir 315"]
 80         "similarity_labels": np.ndarray,
 81      },
 82      "MiceB": {
 83         "success": True,
 84         "results": {
 85            "ensembles_cant": 6,
 86            "neus_in_ens": np.ndarray,
 87            "timecourse": np.ndarray,
 88         },
 89         "similarity": {
 90            # In this case (6 ensembles) + (4 orientations) + (2 behaviors)
 91            "Correlation": np.ndarray,
 92            "Cosine": np.ndarray,
 93            "Euclidean": np.ndarray, # The euclidean distance
 94            "Jaccard": np.ndarray, # The Jaccard index
 95         },
 96
 97         # ["ens 0", ..., "ens 6", "ori 0", ..., "ori 135", "beh speed", "beh pupil"]
 98         "similarity_labels": np.ndarray,
 99      },
100      "MiceC": {
101         "success": True,
102         # ...
103      }
104   }
105}

Saving results example

You can easily save the results of that analysis using the built in function save_data_to_hdf5_file, check Save data.

1from encore.data.save_data import save_data_to_hdf5_file
2
3result_path = folder_path / "parallel_databases_results.h5"
4save_data_to_hdf5_file(result_path, results)

Tip

Use a graphical HDF5 visualizer for a quick glance on the results.