Wegtam Search Agent

3/SRVAPI (Service Module HTTP API)

RAW

License

Copyright (c) 2021 the Editor and the Contributors.

This specification is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

Language

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.

Overview

Use Case

The generic use case is that the service module runs as a server process / daemon and provides an API via which searches can be executed. Also other functionality needed to build a frontend for the Wegtam Search Agent SHALL be accessible via the API.

Design of the API

SAPI is designed based upon RESTful design principles.

Network Transports

In this document we use HTTP as reference transport. So far no other transport is intended. If SAPI over other transports is designed then this SHALL be defined by separate RFCs.

Performance Tradeoffs

To make it easier to work with and extend the API it uses a verbose request-reply protocol and JSON documents. This is slower than using some message serialisation format (Avro, Protobuf, etc.) but is trivial to use in comparison.

Technical Specification

Methods and Arguments

All endpoints of the API MUST be queried via the HTTP GET method. This is in accordance with the REST design because we only want to retrieve information not create or update something.

Arguments to an endpoint MUST be passed as URL parameters (like a HTTP query). The arguments are a map of key/value pairs for setting desired parameters for the endpoint functionality. Parameter key names are case-insensitive. Parameter values are case-sensitive instead!

Security

All endpoints are considered public resources and provide no authentication or authorisation. If it is desired to add some security layer then a reverse-proxy SHALL be used to add that.

Furthermore no encryption layer is implemented. Which also indicates if needed a reverse-proxy MUST be used.

Endpoints

In general endpoints are queried using the following URI schema:

{transport}://{host}:{port}/{endpoint}

List engines

The complete URL path to the endpoint is: /engines

The client retrieves a list of all available search engines as follows:

  • The client must know the complete URI to the endpoint.
  • The client sends a GET method to the endpoint URI.
  • The server either returns “200 OK” with a resource document or 4xx or 5xx with an error text.
Example Query
% curl -X GET http://localhost:12345/engines

List profiles

The complete URL path to the endpoint is: /profiles

The client retrieves a list of all available search engines as follows:

  • The client must know the complete URI to the endpoint.
  • The client sends a GET method to the endpoint URI.
  • The server either returns “200 OK” with a resource document or 4xx or 5xx with an error text.
Example Query
% curl -X GET http://localhost:12345/profiles

Search

The complete URL path to the endpoint is: /search

This endpoint has a number of supported parameters:

engine
Specify a search engine name to be used in the search. This option can be given multiple times.
profile
Specify a search profile to be used.
query
The search query which has to be quoted if it contains multiple words.
region
An alpha-2 country code to narrow the search into a specific region or locale if possible.
results
The number of desired results per search engine.

For a valid call the following parameters are REQUIRED:

  • A valid query parameter MUST be given.
  • Either a valid profile MUST be given OR at least one valid engine MUST be given.
Hint

Please note that you MAY combine a profile parameter with several engine parameters. This will result in the usage of all engines defined in the profile plus the additional specified engines.

The client retrieves a list of all available search engines as follows:

  • The client must know the complete URI to the endpoint.
  • The client sends a GET method to the endpoint URI with at least the minimum required parameters.
  • The server either returns “200 OK” with a resource document or 4xx or 5xx with an error text.
Example Query
% curl -X GET http://localhost:12345/search?query=foo&profile=generic&engine=National+Archives+of+the+UK&results=25

Resource Documents

All resource documents except the error messages MUST be returned in the JSON format. Attributes that MAY be empty or missing are annotated with the Option type (e.g. foo: Option[String]).

Note

Any attributes of the type List in the documents MAY be empty!

Search Engine

{
  name: "SearchEngineName",
  capabilities: [
    SearchEngineCapability,
    SearchEngineCapability
  ]
}

Search Profile

{
  name: SearchProfileName,
  engines: [
    SearchEngineName,
    SearchEngineName
  ],
  mode: Option[SearchMode]
}

Search Result

{
  authors: [AuthorName, AuthorName],
  description: Option[DescriptionText],
  imageUrl: Option[Uri],
  published: Option[OffsetDateTime],
  publisher: [PublisherName, PublisherName],
  publishingDetails: [PublishingDetails, PublishingDetails],
  searchEngines: [SearchEngineName, SearchEngineName],
  title: Option[TitleString],
  url: Option[Uri],
  number: SearchResultNumber
}

SearchQuery

{
  query: String,
  region: Option[Alpha2CountryCode],
  results: Integer
}

Search Endpoint Response

The response of the /search endpoint does include the search query that was constructed from the given parameters as well as a possible profile name and a list of search engines that were used in the request. For convenience the list of engines also includes the ones that are implied by the search profile.

Last but not least a list of search result entries is returned as well.

{
  query: SearchQuery,
  profile: Option[SearchProfileName],
  engines: [SearchEngineName, SearchEngineName],
  results: [
    SearchResult,
    SearchResult
  ]
}

Response Status Codes