Adding External Standards to MIDRC Standards

---

This notebook will be used to add standards that are external to MIDRC's dictionary. The added standard can then be used to process data provided by the user.

Getting started

---

Setting up directory

#Set directory and expansion path
import wget
import sys, os, webbrowser

home_dir = ""  #set home directory here (eg. /Users/username)
demo_dir = f"{home_dir}/Desktop"

os.chdir(demo_dir)

Import libraries

import gen3
import pandas as pd
import requests

from gen3.submission import Gen3Submission
from gen3.auth import Gen3Auth
from gen3.index import Gen3Index
from IPython.display import display
from gen3.query import Gen3Query
from io import StringIO

Set up local variables and define Gen3 classes

The location of your MIDRC credential should be downloaded from https://data.midrc.org/identity by clicking the "Create API key" button and saving the credentials.json locally to your downloads folder.

api = "https://data.midrc.org"
cred = f"{home_dir}/Downloads/midrc-credentials.json"
auth = Gen3Auth(api, refresh_file=cred)
sub = Gen3Submission(api, auth)
query = Gen3Query(auth)

Now we can set the file path of the external standard we want to merge with MIDRC's data dictionary.

#Loading in external standard
# external_standard_file = f"{home_dir}/Downloads/external_standards_file.csv" #change to file name of external standard you want to merge with MIDRC's standards
# df_ext = pd.read_csv(external_standard_file)

#The above lines can be used to import your own standard you wish to merge.
#Sample data will be read from the string below for tutorial purposes.
#If you wish to work with your own standard, simply un-comment the above lines and comment out the string below.

sample_data = """
Node,Property,Property Type,Property Enumeration,Property Required,Property Description,Property Term,Property Term URL,Property Term Standard,NCIt Preferred Name,NCIt Definition,NCIt Synonyms & Abbreviations,NCIt Synonyms & Abbreviations Source
Case,age_at_index,number,n/a,no,"The study participant's age, in years, at the index event. The index event is determined by the data submitter and used as an anchor date for all temporal variables. Note that an age of 0 indicates a participant who is younger than 1 year old. For participants with ages greater than 89 years, please use the property 'age_at_index_gt89'. This property is included to ensure personal privacy protection in accordance with the HIPAA Safe Harbor Method. More information can be found here https://www.hhs.gov/hipaa/for-professionals/special-topics/de-identification/index.html.",C181702,https://ncit.nci.nih.gov/ncitbrowser/ConceptReport.jsp?dictionary=NCI_Thesaurus&ns=ncit&code=C181702,NCIt,Age in Years at Index Date,Age at the in years at the index date.,"age at index, Age in Years at Index Date, Age in Years at Index Date, age_at_index","GDC, NCIt, , GDC"
Case,covid19_positive,enum,"Yes, No, Indeterminate, Not Reported",no,"An indicator of whether the patient has ever had a positive COVID-19 test or been diagnosed with one of the following ICD-10 COVID-19 conditions: COVID-19 (U07. 1), Influenza due to unidentified influenza virus with other manifestations (J11.8), Post COVID-19 condition (U09.9), Myalgic encephalomyelitis/chronic fatigue syndrome (G93.32), or Sequelae of other specified infectious and parasitic diseases (B94.8). For more information about a patient's specific COVID-19 diagnosis, details can be found under Condition.",C171614,https://ncit.nci.nih.gov/ncitbrowser/ConceptReport.jsp?dictionary=NCI_Thesaurus&version=24.07e&ns=ncit&code=C171614,NCIt,Positive History of COVID-19,A finding indicating that an individual has a history of COVID-19.,"Known History of COVID-19, Positive History of COVID-19, Positive History of COVID-19, Positive History of COVID-19, Positive History of COVID-19 Disease","NCI, CTRP, NCI, , NCI"
Case,ethnicity,enum,"Hispanic or Latino, Not Hispanic or Latino, Not Reported",no,A social group characterized by a distinctive social and cultural tradition that is maintained from generation to generation. Members share a common history and origin and a sense of identification with the group. They have similar and distinctive features in their lifestyle habits and shared experiences. They often have a common genetic heritage which may be reflected in their experience of health and disease.,C16564,https://ncit.nci.nih.gov/ncitbrowser/ConceptReport.jsp?dictionary=NCI_Thesaurus&ns=ncit&code=C16564,NCIt,Ethnic Group,A social group characterized by a distinctive social and cultural tradition that is maintained from generation to generation. Members share a common history and origin and a sense of identification with the group. They have similar and distinctive features in their lifestyle habits and shared experiences. They often have a common genetic heritage which may be reflected in their experience of health and disease.,"Ethnic Group, Ethnic Group, Ethnic Group, Ethnic Origin, Ethnic Origins, ethnicity, Ethnicity, ethnicity, ethnicity, Ethnicity, ETHNICITY, ETHNICITY, ETHNICITY, ETHNICITY, ETHNICITY, Ethnicity, Patient Reported Ethnicity","NCI, NICHD, , NCI, NCI, CDISC-GLOSS, CTDC, CTDC, GDC, NCI, PCDC, PCDC, PCDC, PCDC, PCDC, PCDC, SeroNet, OORO"
""" #comment out if using personal standard
df_ext = pd.read_csv(StringIO(sample_data)) #comment out if using personal standard

Generate a copy of the MIDRC data dictionary

We will now pull down a copy of the MIDRC data dictionary so we can have a local copy to work with.

#Viewing list of endpoints
headers = {"Authorization": f"Bearer {auth._access_token}"}

url = f"{api}/api/v0/submission/_dictionary"
resp = requests.get(url, headers=headers)

dictionary = resp.json()

dictionary

Now we can display the data dictionary for a preliminary view before we begin the merge process.

#Viewing data dictionary
url = f"{api}/api/v0/submission/_dictionary/_all"
resp = requests.get(url, headers=headers)

#Will error out if incomplete values are generated
resp.raise_for_status()

dictionary = resp.json()

rows = []

#Flatten dictionary
for node_name, node_data in dictionary.items():
    if not isinstance(node_data, dict):
        continue

    props = node_data.get("properties", {})
    required = node_data.get("required", [])
    links = node_data.get("links", [])

    #Build link map
    link_map = {}
    for link in links or []:
        targets = link.get("target_type", [])
        if isinstance(targets, list):
            for target in targets:
                link_map[link.get("name")] = target

    for prop_name, prop_data in props.items():
        if not isinstance(prop_data, dict):
            continue

        prop_type = prop_data.get("type")
        if isinstance(prop_type, list):
            prop_type = ", ".join(prop_type)

        prop_enum = prop_data.get("enum")
        if isinstance(prop_enum, list):
            prop_enum = ", ".join(prop_enum)

        row = {
            "node": node_name,
            "property": prop_name,
            "type": prop_type or "",
            "description": prop_data.get("description", ""),
            "required": prop_name in required if isinstance(required, list) else False,
            "enum": prop_enum or "",
            "link_to": link_map.get(prop_name, None)
        }

        rows.append(row)

df = pd.DataFrame(rows)
df.head()

Generating flat file of standards

Now that we have a local copy of the data dictionary we will flatten it so it is easier to work with. This is done because the MIDRC data dictionary is by default organized by nodes.

rows = []

for node_name, node_data in dictionary.items():
    if not isinstance(node_data, dict):
        continue

    props = node_data.get("properties", {})
    required = node_data.get("required", [])
    links = node_data.get("links", [])

    link_map = {}
    for link in links or []:
        targets = link.get("target_type", [])
        if isinstance(targets, list):
            for target in targets:
                link_map[link.get("name")] = target

    for prop_name, prop_data in props.items():
        if not isinstance(prop_data, dict):
            continue

        prop_type = prop_data.get("type")
        if isinstance(prop_type, list):
            prop_type = ", ".join(prop_type)

        prop_enum = prop_data.get("enum")
        if isinstance(prop_enum, list):
            prop_enum = ", ".join(prop_enum)

        #Including standards here
        term_def = prop_data.get("termDef", {})

        term_id = term_def.get("cde_id") if isinstance(term_def, dict) else None
        term_url = term_def.get("term_url") if isinstance(term_def, dict) else None
        term_source = term_def.get("source") if isinstance(term_def, dict) else None

        row = {
            "node": node_name,
            "property": prop_name,
            "type": prop_type or "",
            "description": prop_data.get("description", ""),
            "required": prop_name in required if isinstance(required, list) else False,
            "enum": prop_enum or "",
            "link_to": link_map.get(prop_name, None),
            "term_id": term_id,
            "term_url": term_url,
            "term_source": term_source
        }

        rows.append(row)

df = pd.DataFrame(rows)

Now we can take a look at the flat-file version of the data dictionary that we will be working with. This table shows us the name of the property, the node it is associated with, the enumeration of the property, and a description of the property.

#Creating short-list of standards
exclude = ["data_release", "metaschema", "root"] #Removes system nodes, these are irrelevant to the MIDRC standards.

df[~df["node"].isin(exclude)]
standards_df = df[
    (df["enum"] != "") |
    (df["term_id"].notna()) |
    (df["term_url"].notna())
].copy()

standards_df = standards_df[
    ["node", "property", "enum", "description"] #"term_id", "term_url", "term_source" will be generated later
]

standards_df.sort_values(["node", "property"]).head(20)

Now we can save a .csv version of the flat-file dictionary.

#Save file
standards_df.to_csv("MIDRC_flatfile_dict.csv", index=False)

Adding external standards

---

Here we will begin incorportaing the external standards into MIDRC's data dictionary. The resulting product will be a table that has both MIDRC's properties and the properties from an external standard.

Load in and format standards

df_MIDRC = pd.read_csv("MIDRC_flatfile_dict.csv")

#Normalize column names
df_MIDRC.columns = df_MIDRC.columns.str.lower().str.strip()
df_ext.columns = df_ext.columns.str.lower().str.strip()

#Align columns names
df_MIDRC["node"] = df_MIDRC["node"].str.lower().str.strip()
df_MIDRC["property"] = df_MIDRC["property"].str.lower().str.strip()

df_ext["node"] = df_ext["node"].str.lower().str.strip()
df_ext["property"] = df_ext["property"].str.lower().str.strip()

In the above cells, we called the flat-file version of the MIDRC data dictionary and normalized/aligned the columns and column names in the flat-file and the external standard so they could be combined without issue.

#Select desired columns from external standard
df_ext_subset = df_ext[[
    "node",
    "property",
    "property type",
    "property enumeration",
    "property description",
    "property term",
    "property term url"
]]
df_ext_subset.head()

#Rename external standard columns (if needed)
df_ext_subset = df_ext_subset.rename(columns={
    "property description": "description",
    "property type": "type",
    "property enumeration": "enum"
})
df_ext_subset.head()

Now that we have selected the columns we want and renamed the external standard columns to match what is seen in the MIDRC file, we can begin the merge the two.

Merge the standards

Labels will be added so we can keep track of which standards were native to MIDRC and which were from the external standard post merge.

#Adding labels for demo purposes
df_MIDRC["source"] = "MIDRC"
df_ext_subset["source"] = "NCIt"

#Combine and view merged standards
df_combined = pd.concat([df_MIDRC, df_ext_subset], ignore_index=True)
df_combined.loc[~df_combined['property term'].isna()]

The table above shows the standards that were merged into the MIDRC data dictionary. This can be noted under the 'source' column (standards that were already in MIDRC would be labled as 'MIDRC' under the source column). We can see here that all of the external standards have been added to the table!