Credential Engine Registry Search API Handbook
The Credential Registry Search API provides a way to programmatically search the Credential Registry and return top-level resources based on data within those resources, including their connections to other resources. This guide explains how to use the Search API.
To use the Search API, you will need:
- A Credential Engine account approved to use the Search API, and that account's API key
- A basic understanding of JSON-LD, RDF, and Linked Data
- An intermediate understanding of the Credential Transparency Description Language family of Schemas
- The ability to create JSON documents and make HTTP POST requests to the Search API itself, as described by this Handbook
- The ability to interpret and use the JSON documents returned by the API
Use Cases and Alternative Consuming Methods to Consider
The Search API is intended to cover a wide variety of use cases involving queries to the Registry. However, there are some cases where alternative approaches will yield better results.
Getting Real-Time Data
The Search API is well-suited to retrieving real-time data from the Registry. The search index is kept up to date with the Registry itself, and is typically current within a few minutes of new records being published. This kind of usage is the primary purpose of the Search API.
Following Links in the Data
While it is possible to find many resources by searching for them, the resources in the Registry are highly linked together. These links are leveraged by the Search API to build its index, but you can leverage these links directly. For example, you can follow the links in a Credential record to find all of the Competencies it requires. Following the links in the resources directly does not require use of the Search API, nor does it require an API key or even a Credential Engine account. This process is also typically instantaneous, as no query is necessary to return a record that is linked to from another record. For these reasons, it is worth considering whether your particular use case can be satisfied by simply following the links in the records themselves.
In some cases, a hybrid approach is best. For example, using a query to find a list of credentials, and then following the links in those credentials to find their associated competencies.
It's also worth keeping in mind that many properties in CTDL are inverse of each other (for example, "offers" and "offeredBy"). The data in the records in the registry is dependent on the publishers, so in some cases one side or the other of these two-way connections may not be complete or available in the records. For these cases, it is advisable to use the Search API to traverse these connections in both directions and ensure that you retrieve the full set of data.
For more information about how these records link together, including ways to manipulate the URIs to load even more information, see the CTID page.
Scraping the Registry or Downloading Records in Bulk
The Search API is not intended to enable scraping of the Registry to download large quantities of records in bulk. An example of this usage includes doing a search for all resources of a given type and paging through the results to download thousands of records.
If your use case requires this kind of data, you should instead use one of these options for Bulk Download for Offline Use.
Credential Engine monitors for use of the Search API for mass downloading of resources, and will contact you if this behavior is detected.
CTDL JSON Query Structure
The Search API uses a unique query structure built from plain JSON that leverages the actual properties and structure of CTDL and the related schemas. This structure is intended to enable highly flexible queries that allow finding resources based on properties deep within the records or even based on connections to other records. This section explains various features of this query structure.
The Search API expects a JSON object as an input, which serves as a simple wrapper to carry paging and sorting information, as well as the CTDL query itself. This section focuses specifically on the CTDL query inside of this wrapper.
Filter By Resource Type
@type property to filter results to a given CTDL type or list of types.
Filter By Text Properties
The Search API supports a number of features for querying properties that have text-based values, including plain strings, URIs, language maps, and dates.
By default, all string-based fields allow for case-insensitive partial matches. For language map fields (such as
ceterms:description, the Search API abstracts away the intricacies of the language map itself, allowing you to query these fields as if their values were plain strings.
It is possible to force more specific kinds of matching by using a JSON object with
search:matchType properties as shown:
Valid values for
search:exactMatch- Will only match if the string is an exact match (case-insensitive) to the provided value
search:contains- Will only match if the string contains an exact match (case-insensitive) to the provided value
search:startsWith- Will only match if the string starts with the provided value
search:endsWith- Will only match if the string ends with the provided value
Matching CTIDs and URIs
In addition to the support for partial matches described above, the Search API also supports matches based on resource URIs and CTIDs. The Search API's index is optimized to work faster if you provide a full URI where expected, but a CTID in a URI property will also work.
Filtering by Boolean Properties
There are a handful of boolean properties in CTDL. The default value for these properties is
null, so you only need to use them if you are looking for an explicitly
false value. Otherwise, these properties will be ignored when filtering.
Filtering by Numeric Properties
The Search API supports querying for numeric properties, both directly, or as a range by searching for an array with exactly two values where the first value is the minimum boundary, and the second value is the maximum boundary (both the upper and lower bounds are inclusive):
Range queries require an array with exactly two values, and only work with properties whose values are numeric (otherwise they will be treated like any normal array of potential options - in other words, an array of three numbers will only match results that contain at least one of those numbers):
Range queries also may also be unbounded at one end or the other - just pass
null to remove the associated boundary:
Filtering by Date Properties
As with numeric properties, the Search API supports querying for date properties both directly, or as a range (both upper and lower bounds are inclusive; see above for more details). Note that you will need to use the ISO 8601 date format when writing dates:
Date queries can also be unbounded at one end or the other; just pass an empty string to remove the associated boundary:
Filtering by Concepts
There are many areas of CTDL where controlled values (Concept Schemes) are used. Concepts from these schemes may be referenced either directly or via Credential Alignment Object, depending on the property in question.
For querying properties that use Credential Alignment Object as their value, see the Filtering by Credential Alignment Objects section below.
Filtering by Current Data
The Search API allows for querying all resources in the Registry, including those which are no longer considered to be in use or available. This is intentional, as it allows the lookup of older credentials which may still be valid even if they are no longer offered.
There are three primary properties (and their associated concept schemes) for describing the status of a resource:
- For Credentials, use the
ceterms:credentialStatusTypeproperty with terms from the
- For Competency Frameworks, Concept Schemes, and Progression Models, use the
ceasn:publicationStatusTypeproperty with the
- For many other top-level resources in CTDL, use the
ceterms:lifeCycleStatusTypeproperty with the
Filtering by Any Value or No Value for a Property
In some cases, you only need to know if a resource has a given property, regardless of the value of that property. For these cases, use the special token
search:anyValue as the value for the property in question:
Similarly, the special token
search:noValue can be used to find records which do not have any value for the property in question:
To filter resources based on data that is nested within them, simply use a nested JSON object:
This nesting matches the structure of the CTDL records, and may occur several times for deeply-nested data:
Note that while it's possible to explicitly indicate the
@type(s) of the nested objects, it often isn't necessary, particularly when the property referencing them only ever points to instances of a single class.
For more examples, see the Examples section.
Objects and Arrays: Logical AND vs Logical OR
Often, you will want to find resources which have certain values for multiple properties. For example, you may want all Certificates with "nurse" in their name. Or, you may want results that match any value from a list of potential values, such as credentials with "developer" or "programmer" in their keywords. The Search API allows for such cases by way of the structure of the JSON itself, and a few special tokens to allow further control over how the data is queried.
By default, all properties in a JSON object are treated as being logically AND-ed together:
By contrast, all tokens in a JSON array are treated as being logically OR-ed together:
There are cases where you may want to override these behaviors. This can be done via the special property
search:operator and the special token values,
Note that for arrays, the
search:operator property and its
search:andTerms value exist alongside the array rather than within it, all of which is wrapped in an otherwise empty JSON object:
For more examples, see the Examples section.
Controlling Filter Evaluation via Term Groups
When overriding logical AND-ing and OR-ing, it is often necessary to group sets of terms together. For example, you may want all resources where the type is a
ceterms:Certificate regardless, but also where either the name or the description may contain "nurse". This kind of query can be done with the special property
search:termGroup, which can be thought of as adding parenthesis around a set of terms within a query to force a specific order of operations and logical AND or OR within that group.
Note that despite using a nested JSON object,
search:termGroup does not trigger traversal across or into an object in the data; it operates on the object itself.
For more examples, see the Examples section.
The most powerful feature of linked data is, of course, the linking between resources. The Search API is designed to allow you to perform such traversal by structuring your query as though all of the data is nested, even across multiple resources:
In some cases, it is necessary to traverse the graph connections in reverse. This can be done just like normal graph traversal, except that you will prefix the reverse properties with
These forward and reverse connections can be mixed as needed in a deeply nested query. For more examples, see the Examples section.
Filtering by Credential Alignment Objects
Credential Alignment Object provides a way to link to another resource, such as a competency or concept, and/or to provide a name and description of the intended data, even when no such resource exists. While this is convenient for publishing data, it must be taken into account when querying the data. Aside from needing to be aware of where and when the schema uses a Credential Alignment Object, the actual structure of the query works as it would with any other nesting and traversal:
For more examples of queries that flow through Credential Alignment Object, see the examples section below.
The Search API also enables some special tokens for filters based on record metadata. The sections below describe these features.
Filtering by Record Owner and Record Publisher
In some cases, it is necessary to filter records to just those that are owned or published by a specific organization. Note that ownership and publishing, in this case, refers to ownership and/or publishing of the record itself, not necessarily of the resource that the record describes.
To filter records to a specific owner or publisher, use the CTID of the organization along with the special
Filtering by Record Created Date and Record Updated Date
It is also possible to filter records based on when the record was published to the Registry and/or when that record was most recently updated. This is not to be confused with sorting by these dates. As with other date properties, these can be queried based on a date range where both the upper and lower bounds of the range are inclusive.
Note: When a record is initially created, both its
search:recordCreated and its
search:recordUpdated dates are set to the same value. Subsequent updates to the record will only update the value of
Filtering by Primary and Secondary Record Source
In most cases, a record in the Registry comes from a primary source (that is, the intellectual property owner of that resource). These records may be published in one of three ways:
- The record was directly published by the primary source itself.
- The record was published by a third party with the explicit authorization of the primary source itself, but otherwise originated from that primary source (this is commonly called "third-party publishing").
- The record was published by a third party with overriding authority, such as a state agency, to publish for that source without needing its permission (this is commonly called "trusted third-party publishing").
However, there are special rare cases where an organization needs to publish data for which it is not a primary source, but where the organization's authority does not rise to the level of trusted third-party publisher. For these cases, Credential Engine may authorize such an organization to perform "secondary source" publishing. A record published by a secondary source is just like any other record, except that its metadata indicates that it is a secondary source record.
By default, the Search API will include primary and secondary source records in its results. You can filter primary or secondary records in or out using the
search:resourcePublishType property, with a value of either
Filtering by Subclasses
The CTDL schema has a number of class hierarchies. The most obvious example of these are the various subclasses of Credential, which form a tree of subclasses. The Search API allows you to explicitly select one or more of these subclasses of credential, but there are cases where you are looking for any kind of credential, regardless of its specific subclass. To avoid needing to keep track of (and write queries with) every subclass of Credential, you can use the following value for
This also works for any other classes that have subclasses. As with the Certificate example above, results will include the class you specify, as well as subclasses of that class.
The CTDL Types List page provides a useful reference for the various class/subclass hierarchies in CTDL.
The CTDL family of schemas provides a variety of ways to describe information. It also provides a variety of places for that information to be stored. When searching for information in the Registry via the Search API, you will need to consider all of the places in CTDL where that information may have been expressed, and write your queries to check all of them. Some common examples are below.
Owns vs Offers vs Owned By vs Offered By vs Creator vs Publisher vs Record Owned By vs Record Published By
There are a number of properties in the CTDL family of schemas for connecting resources to the organizations that provide them. However, the notion of "providing" a resource encompasses several more granular notions, each of which have their own specific properties. Additionally, there is a distinction between the records in the Registry and the resources those records describe.
ceterms:owns- Points from an organization to a resource, and indicates that the organization is the intellectual property owner of that resource
ceterms:offers- Points from an organization to a resource, and indicates that the organization offers that resource, typically to end users
ceterms:ownedBy- Points from a resource to an organization, and indicates that the organization is the intellectual property owner of that resource
ceterms:offeredBy- Points from a resource to an organization, and indicates that the organization offers that resource, typically to end users
ceasn:creator- Points from a resource to an organization, and indicates that the organization created that resource
ceasn:publisher- Points from a resource to an organization, and indicates that the organization makes that resource available, typically to end users
search:recordOwnedBy- Special Search API property (see above) indicating the organization that owns the record in the registry (regardless of the owner of the resource)
search:recordPublishedBy- Special Search API property (see above) indicating the organization that published the record in the registry (regardless of who makes that resource available to others)
Compounding this, data may be present for some of these properties, but not others (data is always present for the two special search API properties). This is due to differences in which properties certain organizations choose to use, when/how often they update their data, and rules preventing publishers from modifying data they do not own. This leads to subtle inconsistencies in the connections between records that need to be accounted for when using the Search API.
Some common examples include:
- A credential may indicate that it is "owned by" an organization, but not have any "offered by" data even when it is also offered by that organization
- A credential may indicate that it is "offered by" an organization, but that organization's record may not indicate that it "offers" that credential
- A competency framework may only indicate that its "creator" is a particular organization, even when that organization is also its publisher
Typically, the safest approach is to rely on the connections that exist on the resource and point to the organization. This kind of data is usually part of the minimum requirements for publishing the resource, so it is more likely to be present.
Quality Assurance is expressed in CTDL via a series of properties, each with an inverse:
ceterms:accreditspoints from the Organization doing the accrediting to the resource being accredited. Its inverse is
ceterms:accreditedBywhich points from the resource being accredited to the Organization that does the accrediting.
ceterms:approvespoints from the Organization doing the approving to the resource being approved. Its inverse is
ceterms:approvedBywhich points from the resource being approved to the Organization that does the approving.
ceterms:recognizespoints from the Organization doing the recognizing to the resource being recognized. Its inverse is
ceterms:recognizedBywhich points from the resource being recognized to the Organization that does the recognizing.
ceterms:regulatespoints from the Organization doing the regulating to the resource being regulated. Its inverse is
ceterms:regulatedBywhich points from the resource being regulated to the Organization that does the regulating.
The nature of the data published to the Registry means that the above connections will not always be present in both directions. For example, a government agency may publish its own Organization record to the Registry, but not provide a list of credentials that it accredits. Publishers of the data for those credentials, however, may provide such data. In this case, the
ceterms:accreditedBy connections will exist, but the
ceterms:accredits connections will not. Therefore, the lack of connections flowing both ways does not necessarily imply that a false claim of quality assurance is being made, but if this data is important to your use case, you should reach out to the publishers in question to verify the information.
Additionally, some Organizations perform quality assurance on other Organizations, or on the programs those Organizations offer, rather than on the Credentials themselves. This must also be taken into account when building your queries.
A common use case is to look for credentials that are available in a particular area, such as a state. There are several ways that the physical location where a resource is offered can be communicated in CTDL, each of which will need to be accounted for when looking for resources based on location. Which ways are in use for a given record depends on the which properties the publisher of the data chose to use, but will fall into one or more of these categories:
- On the resource itself, generally via
- Described in detail for the resource itself via
- Indirectly via a related resource, such as the required assessments or learning opportunities for a credential (also communicated using
- Implicitly via the address(es) of the organization(s) that offer(s) the resource, via
- Implicitly via the jurisdiction(s) of the organization(s) that offer(s) the resource, via
When querying for locations, you will need to leverage the structure of the
ceterms:Place class, which contains various familiar properties for describing an address at varying levels of granularity.
Address matching can be somewhat awkward when there is more than one way to write an address. For example, in the US, the
ceterms:addressRegion may be published with either the full name of a state, or its two-character postal shorthand. Both kinds of address need to be accounted for in order to avoid missing valid resources, while also avoiding false positives where the two-character string is a substring of a full state's name (for example, Indiana's abbreviation, "IN", can be found in "North Carolina". To handle this, you will need to look both for the full name of the state and its exact abbreviation via
search:exactMatch, as shown in the examples below.
You should also be aware that many resources may (also) be offered online. This would be indicated via use of
Combining the above and making use of a Term Group, we can look for Certificates based on address even when we don't know exactly where the address data is published:
Credential Data Describing Related Resources
Some properties of credentials are intended to provide shortcuts for publishers of the data, to avoid needing to publish records for related data, such as assessments or learning opportunities. These properties will also need to be accounted for when designing your queries. For example:
ceterms:assessmentDeliveryTypeexists on credentials, but actually describes the
ceterms:deliveryTypeof a related assessment (often where there is no actual assessment record).
ceterms:learningDeliveryTypeexists on credentials, but actually describes the
ceterms:deliveryTypeof a related learning opportunity (often where there is no actual learning opportunity record).
- Several properties exist on credentials, but typically actually apply to the program, course, or assessment that leads to the credential, rather than the credential itself. Some examples include:
The results returned by the Search API always consist of the entire Registry record for each result, for the current "page" of results (in addition to a count of total results). There is no need to specify particular properties to return.
The following sections describe different aspects of how data is returned from the Search API.
Note that the following sections show the complete HTTP POST body object that would be sent to the Search API, whereas the examples above focus on the CTDL portion of the query.
The Search API will only return results consisting of "top level" classes. Top level classes are those which have a
CTID property. The following is a list of classes with CTIDs:
Paging: Skip and Take
Pagination is handled via the
Take properties of the root of the query:
Skipproperty indicates how many results (not how many pages) to skip, and uses a 0-based index.
Takeproperty indicates how many of the remaining results (after skipping) to include in the response. It has a maximum limit of 100 resources per page.
You can control the sort order of results via the
Sort property of the root of the query:
Sort property accepts one of the following values:
search:relevance- This will return results based on how closely their text-based properties match any text-based properties in your query. This is the default order if your query contains any text matching properties.
search:recordUpdated- This will return results based on how recently the record was updated, with the most recently updated records coming first. This is the default order if your query does not contain any text matching properties.
search:recordCreated- This will return results based on how recently the record was created, with the most recently created records coming first.
- You can also sort based on any alphanumeric property in the root level of the search results. For example, use
ceterms:nameto order results alphabetically (A-Z) by name.
Note: Do not confuse the usage of
search:recordUpdated here as sort orders in the root of the query with the usage of those properties as filters in the body of the query as shown here.
To reverse the sort order, prefix the value with
^. For example, to sort records by
ceterms:name in reverse (Z-A), use
Additional metadata about the search results are also available as part of the response to a query. This metadata is encapsulated in an array of JSON objects separate from the results themselves, where each object in the array goes with one of the results. Such metadata includes:
ResourceURI- The URI of the search result that this metadata object describes
RecordCreated- The date and time that this Record was created in the Registry
RecordUpdated- The date and time that this Record was most recently updated in the Registry
RecordOwnedBy- The CTID of the organization that owns this Registry record (but does not necessarily own the resource described by the record)
RecordPublishedBy- The CTID of the organization that published this Registry record (but does not necessarily publish the resource described by the record)
RecordPublishType- Indicates whether this Record is considered a primary or secondary source record.
You can include the metadata in the response by using
"IncludeResultsMetadata": true in the root of your query:
This will be included in the
ResultsMetadata property of the response object:
Some resources contain additional objects in their JSON-LD
@graph. These objects range from stub references to things which do not have full registry records (RDF Blank Nodes) to resources that exist as top level resources, but which are also closely tied to some other resource, such as:
- The Competencies for a Competency Framework
- The Concepts for a Concept Scheme
- The Pathway Components for a Pathway
You can include the data from the
@graphs for the results by using
"IncludeGraphData": true in the root of your query:
All of the additional data from the results'
@graphs (not including the results themselves) will be returned in the
RelatedItems array of the response:
Description Sets and Other Related Data
A Description Set is a collection of resources directly relevant (and directly or indirectly connected to) some specific resource, typically beyond that resource's
@graph. A Description Set is useful for obtaining a broad depth of data about and logically adjacent to a resource. For example, the Description Set for a credential might include:
- The credential itself
- The organization(s) that own or offer that credential
- Any required assessments or learning opportunities for the credential
- Any competencies related to the credential, either directly (e.g. the credential requires the competencies) and/or indirectly (e.g. the credential requires an assessment which assesses those competencies)
- Other similarly relevant information
The Search API also supports returning the Description Sets for results, in two parts: The Description Set itself, and (optionally) the related resources connected via the linkages described by that Description Set. The Description Set itself is an array of objects that provide the URIs of the related resources for each result, along with a special path string that indicates the chain of properties and classes that connects that result to that list of related resources, including the directionality of those connections. A Description Set looks like this:
The example above features several key aspects of Description Sets. Namely:
ResourceURI- The URI of the search result being described by this Description Set. This is used because Description Sets are returned in a different part of the response from the search results themselves.
RelatedItems- The array of objects describing the paths to or from the search result out to some other resource or set of resources, each of which includes:
Path- Describes the path taken, through a series of properties and classes, between the search result and the items referenced in the
URIsarray. The search result is always assumed to be the leftmost object, and the rightmost object is the type of resource being referenced. Path strings may contain a mix of outbound and inbound connections, represented by
<, respectively. For example:
> ceterms:offeredBy > ceterms:Agentmeans "The search result has a ceterms:offeredBy property that references one or more instances of ceterms:Agent (the superclass for various types of organization), which are linked to in the URIs array".
< ceterms:offers < ceterms:Agentmeans "The search result is referenced by one or more instances of a ceterms:offers property, each of which belongs a ceterms:Agent (the superclass for various types of organization), which are linked to in the URIs array".
TotalURIs- A count of total URIs that belong with the provided path. When the number of URIs and/or related resources returned is limited (see below), this provides a way to know how many URIs would be in the
URIsarray if no limit were applied.
URIs- The list of URIs found on the rightmost end of the path. The number of URIs in the array is subject to the limit described below.
DescriptionSetType property of the outermost layer of the query itself controls what kind of data is returned. It has the following options:
"DescriptionSetType": "Resource"returns just the results themselves. This is the default option, so you do not need to include this property if you only want the resources.
"DescriptionSetType": "Resource_RelatedURIs"returns the results, plus the Description Sets for those results.
"DescriptionSetType": "Resource_RelatedURIs_RelatedData"returns all of the above, plus the related resources (in the
RelatedItemsarray of the response).
Note: Description Sets, when combined with their related resources, may contain a lot of data, and returning them will often take longer than returning just the results. Only request Description Sets if your use case requires them. This is doubly true for returning related data.
You can limit the number of URIs returned for each connection in the Description Set using the
DescriptionSetRelatedURIsLimit property in the root of your query. This will also limit the related resources returned to those whose URIs were included in the response.
Since the related resources for all results are combined in a single
RelatedItems array in the response, there will only be no duplicate instances of a resource in that array. However, since each "page" of results is independent of each other "page", it is possible to return the same related items multiple times if multiple pages of your results have Description Sets which reference the same related resource. In other words, if 3 Credentials on one page of your results reference the same Organization, that Organization will only appear once in the
RelatedItems array. However, if that same Organization is referenced by one or more Credentials on 3 pages of your results, that Organization will be returned once for each page. So again, it is important to only return the related resources if your use case truly needs them.
Combining Graphs and Description Sets
It is possible to include
@graph data in addition to Description Set data (see above). Note that the
IncludeGraphData property ignores the
DescriptionSetRelatedURIsLimit property, making it possible to return an entire
@graph plus a limited Description Set for each result in a single query.
However, it is strongly advised that you only do so if/when truly necessary, in order to conserve resources and ensure the best speed/performance for your users. It is preferable, for example, to return only the resources for a search, and then use a separate page (after the user has clicked on a search result) to query for the
@graph and/or Description Set for a single resource.
Calling the Search API
The Credential Registry supports the Search API on both its Sandbox and Production systems, with the following endpoints:
To call the Search API, make an HTTP POST request to one of the above endpoints, including an
Authorization header with a value of
Bearer followed by a space and then your API key, for example:
Authorization Bearer abcdef12345
The body of the POST request will be the query object, for example:
For more examples, see the Query Helper section below.
The Query Helper is a tool meant to help developers build, test, and experiment with their queries. It provides:
- A list of pre-made query examples
- A simple interface for building basic queries
- A simple query editor for creating fully customizable queries
- An interface for configuring the root layer of the query
- A display of exactly what the root layer of the query looks like (this is what your system will need to generate and send to the Search API in the body of your HTTP POST request)
- Live, real-time results in both a simplified name/description display and raw JSON data formats