Connections to national-level external data

Overview

SCALES's vision of an open justice data ecosystem encompasses not only accessible, queryable records from primary data providers like courts and law enforcement agencies, but also secondary resources that provide additional context and research value. Our interest in linking external datasources also stems from our participation in the NSF's Open Knowledge Network (OKN) project, whose mission is the interweaving of data across subject-matter domains in order to better understand the overarching systems and processes at work in those domains.

As of this writing, we have integrated two national-level datasets into the SCALES knowledge graph. The first is the 2018–22 set of annual extract files from the FBI's National Incident-Based Reporting System (NIBRS), maintained by the Inter-university Consortium for Political and Social Research (ICPSR); the second is the set of identifiers from the SEC's Electronic Data Gathering, Analysis, and Retrieval (EDGAR) system that correspond to publicly traded companies appearing as litigants in our PACER dataset, as well as some additional information about those companies.

Data features

NIBRS extract files

• Size: These files are quite large, given that they aggregate data reported to NIBRS by state and local law enforcement agencies across the whole country; the five years we incorporated into our dataset totaled 21.7m records in the arrestee files and 41.4m in the incident files, from which we generated a set of 404m triples.
• Shape and origin: The ICPSR offers straightforward tabular data, divided into six overlapping files according to the kind of record occupying each row; we found that the Arrestee and Incident files contained the vast majority of interesting data, and we discarded the rest. We also pulled in a Law Enforcement Agency Identifiers (LEAIC) crosswalk file to translate originating agency identifiers (ORIs) to human-readable agency names.
• Notable coverage:
  • Geographical data, in the form of Federal Information Processing Standard (FIPS) state/county codes for reporting agencies
  • Three-digit UCR offense codes for all applicable arrests
  • NIBRS drug codes for drug-related arrests

EDGAR company data

• Size: At 24.8k triples, corresponding to 7.1k SCALES entities matched to SEC companies (100 of which also matched to SUDOKN companies), this dataset is relatively small, albeit connected to our much larger PACER dataset.
• Shape and origin: We matched companies from our dataset of PACER federal court dockets to companies listed with the SEC using simple strict matching rules, queried an SEC API with the Central Index Keys (CIKs) of those companies, and pulled back JSON data containing various company info/filings/classifications, which we stored as a flat CSV before graph import.
• Notable coverage:
  • CIK numbers for all companies in the dataset
  • Four-digit Standard Industrial Classification (SIC) codes for most CIK-matched companies
  • Six-digit North American Industry Classification System (NAICS) codes, wherever the SIC codes unambiguously translate to NAICS

Use cases

NIBRS extract files

• Given another dataset that can interface with NIBRS (such as SCALES state-court case files, which contain geo/UCR data), we can use the federal measures to better understand that dataset, e.g. by comparing agency report rates for certain UCR codes with the rate at which that code is prosecuted or sentenced.
• The sheer size of the NIBRS dataset can provide some insight into the volume and nature of the activities of NIBRS-reporting law-enforcement agencies. For instance, given a UCR offense code, a consumer of the SCALES graph can ask (via MCP, raw SPARQL, etc) which agencies are reporting that code the most, or which cities/counties.
• Because primary data from law enforcement agencies tends to take a shape similar to the NIBRS data, as we discovered via our Atlanta PD arrests dataset, we can quantify the rate of data-practices drift between the FBI and local agencies by measuring the rate of disagreement between the NIBRS data and any corresponding local data.

EDGAR company data

• Our integration of SIC/NAICS codes into the SCALES graph means that any data connected to those codes can be aggregated in any way that SIC/NAICS allow. For example, a researcher interested in companies sued for health violations in federal court could break down various PACER measures (e.g. length, speed, outcome) according to the SIC/NAICS code of the defendant company.
• Because knowledge graphs have many industrial applications (supply-chain management, resource allocation, infrastructure maintenance, etc), we expect that our EDGAR dataset shares entities with existing corporate knowledge graphs; one such graph is Arizona State University's Supply and Demand Open Knowledge Network (SUDOKN), which is a part of the NSF OKN and is focused on small-to-medium-size manufacturers. Our desire to create a SCALES-SUDOKN interface played into our decision to add EDGAR triples to the SCALES graph, and the connection between these two knowledge networks yields some interesting potential comparisons, e.g. How often are manufacturers in certain cities/states involved in lawsuits? How litigious are companies with certain manufacturing capacities?

Caveats

NIBRS extract files

• Some agencies in the ICPSR dataset appear to generate a separate arrest number for each arrestee even when multiple arrestees were arrested at once (based on the fact that those agencies didn't record any multi-arrestee arrest numbers), so in the incident-arrest-arrestee hierarchy, the arrest/arrestee distinction is only accurate insofar as the underlying agencies are accurately making that distinction, meaning e.g. we may overcount arrests if we take the agencies at their word that "new arrest number" implies "new real-world arrest." Relatedly, some agencies seem to use the arrestee sequence number as an ordinal arrestee index across a set of arrestee-specific arrest numbers rather than within a multi-arrestee arrest number, but this is less consequential given that the arrestee indices are still unique.

• Many values in the "arrest transaction number" column are missing from the ICPSR data; specifically, the value "-8" appears in that column for 46.7% of the test subset of the 2022 arrests file and 66.2% of the first of three such columns in the '22 incidents subset. This may be an ostensible safeguard against de-anonymization (based on the comment "NA LT 3 values" in the codebook) or a result of bad record-keeping (based on the fact that the second and third such columns in the incidents subset are almost entirely "-8," suggesting that someone is using that value as a generic missing-data code). Regardless, by adding incident-related triples from the arrestee files rather than the incident files when an arrestee is de-identified from their incident, we're able to ameliorate this issue and avoid miscounts.

EDGAR company data

• At present, the confidence-score model for the SCALES-SUDOKN crosswalk is a Fellegi-Sunter-style function with weights trained on 50 human-validated true/false matches, but it's simple enough that it's essentially a decision tree whose branching points involve boolean variables pertaining to match exactness, string length, number of tokens, uniqueness in the name set, and word commonness. As a result, our crosswalk likely contains a handful of false positives. (While we would have liked to produce a larger validation set, any substantive attempt at ground-truth annotations for these company matches tends to become a prohibitively time-consuming process, wherein a human searched for addresses/aliases/etc in Recap documents on the one hand and contact info from the company website on the other.)