Changelog
0.11.0
- The packages are published on PyPI and npm (
pip install linked-codebook,npm install linked-codebook) from a tag, by the Release workflow, with the same files thatreleases/keeps for pinning by checksum. npm packages carry provenance. - The JSON-LD context is published as a file for each version (
context-<version>.jsonld, immutable once released) and for the current sources (context.jsonld), next to the test page.buildstill embeds it. - The documentation is a VitePress site at
/docs/of the test page. New page: Language detectors (which detector to use withinfer, and snippets). The TypeScript and Python pages list every exported type and name (Cell,Condition,Data,Detector,EvaluateOptions,FoundLanguage,Inferred,Options,Result,StatVariable,Tables). The README shows the commands by task and mentions REDCap, tables, conditions, missingness and language. The rationale drops the comparison of earlier projects and gains the language protocol and visions. - The keys that no vocabulary defined (
missingRange,codeBookClosed,appliesWhen,minimumValue,maximumValue,columnGroup,columnFormat,identifying) now have IRIs, so they stay in the RDF when the document is loaded as JSON-LD. Before, JSON-LD dropped them: skip rules, missing ranges and limits could not be queried.minimumValue,maximumValueand thelowandhighof a missing range arecsvw:minInclusiveandcsvw:maxInclusive(CSVW puts these facets on a datatype description; here they sit on the column, ascsvw:datatypealready does), andcolumnFormatiscsvw:format.appliesWhen,codeBookClosed,columnGroup,identifyingandmissingRangeare slots of this project's own data model,lc:applies_when,lc:closed,lc:group,lc:identifyingandlc:missing_range(lcishttps://bennokr.github.io/linked-codebook/profile#, the namespace ofspec/profile.yaml, which the test page now serves asprofile.yaml).make pinchecks the slots against the model and the others against their vocabularies. - The prefix of the DSV ontology in the context and the documents is now
dsvo, becausedsvis also the preferred prefix of the Data Specification Vocabulary (https://w3id.org/dsv#). The IRIs are the same. - A float
NaNis blank in conditions and inmissingness(a data frame holds a missing number asNaN). JSON cannot containNaN, so the TypeScript library and the conformance cases are unchanged.
0.10.0
- Added the slot
language(language of the text in a free-text column, a lower-case BCP 47 tag; JSON-LDdct:language). Validation checks the tag against the pattern in the model; the validators of both libraries now apply apatternrule. - Added language detection as a protocol, not a detector:
infer_language/inferLanguagetake a function from a text to a language tag and decide when it is asked and when its answers are accepted (section 7.1 of the specification, limits inspec/language.yaml, cases inspec/cases/language.json, run by both libraries).infertakes an optional detector and then addslanguageto text columns. - Added
is_identifier_name/isIdentifierName: the identifier-name test of the inference rule, so applications can drop their own copies. - The statement budgets rise by the language protocol and the pattern check (Python 1060 to 1086, TypeScript 948 to 972).
0.9.0
- Added the slots
format(how dates or numbers are written, in the notation of the source; Frictionlessformat),minimumandmaximum(limits of a numeric column),group(the form of a REDCap field, or any named group of columns) andidentifying(true if the values identify a person or a record). JSON-LD keysminimumValue,maximumValue,columnGroup,identifying; Frictionlessconstraints.minimumandmaximum;groupandidentifyinginx-linked-codebook. from-redcapreads Form Name (group), Identifier? (identifying) and numeric Text Validation Min and Max.- Added tables:
from-tables,to-tablesandcheck-tables, withfromTables,toTables,tableWarningsand the Python equivalents. A set of sheets (Columns, Values, Mappings, Dataset) is read and written with the headers of the model.closedis only onColumns; missing codes only onValues. Workbook files are opened by the application; see section 12 of the specification. from-redcapreads CSV as Python'scsvmodule does (a quote inside an unquoted field is text; text after a closing quote continues the field). The TypeScript reader used to reject a PhenX dictionary that Python read; with the change, both libraries print identical output for all 985 PhenX REDCap dictionaries (35,541 columns).- Conditions are evaluated as REDCap does by default (
blank: value): a blank is the empty value, so[a] <> 1is true whenais blank.blank: unknowngives the SQL reading (every comparison with a blank is unknown). The REDCap behaviour comes from university guides and a community knowledge base, not from REDCap's own documentation. - Fixed: the tables module had a function called
require, which broke the package when a tool compiled it to CommonJS (ts-jest did). A test now guards against it. - Added conditions:
parse-conditionandconditions(parse_condition,parseCondition,evaluate_condition,evaluateCondition). One syntax reads REDCap branching logic and a subset of SQLWHEREinto a JsonLogic-shaped tree and evaluates it in three values (true, false, unknown). 2,649 of the 2,650 different rules of the PhenX REDCap dictionaries parse, identically in both libraries. - Added
missingness: for a profile and data, how values divide into present values, declared missing codes and blanks, and the blanks into skipped by design, missing although applicable, and unknown. - Added
from-readstat: the variables of an SPSS, Stata or SAS file (as ReadStat-based readers deliver them) become columns, with variable and value labels, measure, format, and user-defined and extended missing values. - The statement budgets rise by the size of the tables module (Python 563 to 1060, TypeScript 499 to 948; conditions, missingness and statistical files are about 300 statements in each library).
0.8.0
- Added the command
from-redcap,fromRedcapandfrom_redcap: a REDCap data dictionary (CSV) becomes a profile. Dropdown, radio, yes/no, true/false and checkbox fields get closed value lists; text validation types give datatypes; the rules are inspec/redcap.yaml. - Added the slot
applies_when: the condition under which a column has a value, as the source writes it (REDCap branching logic). Written to JSON-LD asappliesWhen.
0.7.0
- Added the slot
source: the IRI of a web page that documents the column. It is written asdct:sourceon the variable concept, and read from there or from the column property. - For ordinal columns, the code concepts carry
schema:position, and readers order the codes by it, because RDF has no order. Closed ordinal lists also set the FrictionlesscategoriesOrdered.
0.6.2
- Validation of JSON-LD skips keys whose term is null or a keyword such as
@nest, as JSON-LD does, instead of failing.
0.6.1
- Python:
rowsread from JSON-LD is an integer, not a float.
0.6.0
- Added the slot
closed: the values of a column are all the codes it may hold. Only closed lists become Frictionlesscategories; written to JSON-LD ascodeBookClosed. - Added the slot
observed: codes found in the data with their counts, separate from the declaredvalues.inferfills it for categorical columns, andobservecomputes it for one column. - Validation reports observed codes outside a closed list.
0.5.0
- Added the option
explicitof reading, which keeps variable IRIs that the library would write itself.
0.4.0
- Added the slots
rowsandcompleteness, which are written as DSV summary statistics. - Integers are checked in profiles.
0.3.1
- Times may end in
UTC.
0.3.0
- Added the option
identifierof the inference.
0.2.1
- Rows of SSSOM files are also matched to columns by the end of the subject IRI.
0.2.0
- Added the Frictionless Table Schema projection and the command
to-frictionless. - Added the option
limitsof the inference,unmatchedSssomandisNumeric. - Written documents no longer contain empty objects.
0.1.0
First version of the specification, the TypeScript library, the Python library, the conformance suite and the test page.