Getting started with searching

This guide will introduce you to the search API and the search syntax.

Searches are sent as POST requests to the search endpoint, with the search specification as a JSON object in the request body.

Below is an example request to get you started, which will return data in JSON format. Replace yourtoken with your token.

curl "https://api.opoint.com/search/" \
	-H "Authorization: Token yourtoken" \
	--json @search.json

The --json argument in curl implicitly includes the headers content-type: application/json and accept: application/json, you may have to define these manually in your client for the purpose of reproducing this example.

The next sections will walk through how to construct the request body.

Starting out

At its most basic level, a search consists of a search string, with search terms and filters, determining which articles should be considered a match, and a set of parameters, controlling the results, like what parts of the article should be returned, the header, the lede, whether to only include articles published in a given time span, what kind of meta data to add, and many similar options.

Let's start with a minimal example. First, we add a search string to the searchterm field in the request. Based on the search string, each article is either considered a match or not. As a start, we can search for articles containing a single word:

{
  "searchterm": "spotify"
}

This should give back articles with spotify mentioned somewhere in the text.

Being such a general search, there will be many matching articles, and by default only the the most recent content is returned. Generally, however, we also want some control over what to include in the response – maybe we are looking for yesterdays news, and we would like to get back the body text with highlighting information, indicating where a match was found.

These settings, how the matches are interpreted, and what additional data to include, is controlled by a params object.

Modifying our request slightly, we can easily ask for 5 articles, with headers and summaries included in the results:

{
  "searchterm": "spotify",
  "params": {
      "requestedarticles": 5,
      "main": {
          "header": 1,
          "summary": 1
      }
  }
}

And this is the search workflow at the core. From here on, it is about exploring the search functionality more deeply.

Searching based on content

Let's continue by expanding our search a little. A simple extension is to add another word to the search string. The default behaviour is to match articles having all words somewhere in the text, but not necessarily in consecutive order. However, multiple words can be treated as a single phrase by enclosing them in quotes.

spotify music will match articles containing spotify somewhere and music somewhere.
'Michael Jordan' will match articles containing Michael followed directly by Jordan.

The API also supports more advanced string matching, like common wildcards and proximity search. For instance, to search for articles mentioning Trump and Biden close by, we can use the proximity search operator ~:

{
  "searchterm": "'Trump Biden'~5"
}

Note: More details on these search string options are found in the reference.

Directing the search

We can also be more specific about which part of the article we search in.

Each article is divided into a header, a summary – the lede, and a body, along with image captions, if present. With a prefix, the phrase will only be matched against content in a particular section. Multiple sections can be searched by separating them with a comma.

header:spotify will match articles with spotify in the header.
summary:spotify will match articles with spotify in the lede.

Without any prefix, the search is performed against all the associated content, both the header, body, summary and any image captions. Extending our search string to only look in the header, we use:

{
  "searchterm": "header:'Trump Biden'~5"
}

Note: Phrases, wildcards and other search operators can be combined freely with the restrictions.

Filters

Filters give more fine grained control over the matches. There are many different types of filters, helping to narrow the results based on e.g. language, publisher and popularity of the site.

Let's see how these filters can be used in practice. The filters are added to the seach string by first mentioning the type, and then inserting the value following a colon, just as when restricting the search to particular section of the article.

We can search for articles mentioning tax in the header, and restrict the language to english like:

{
  "searchterm": "header:tax lang:en"
}

If we are interested in content from the state of New York, we can add a geo filter:

{
  "searchterm": "header:tax geo:1372"
}

Not all of the available filters have values that directly translate to a human understandable form, like the id supplied to the geo filter above. This is partly to make the filters language agnostic. France is called Frankrike in Norwegian, but it is still the same country. In order to translate from names to ids that the search engine understands, the suggestion server is used.

The suggestion server will not be covered here, but its working and motivation is summarized in a separate guide.

Boolean expressions

The search engine supports boolean operators for more complex search queries. You can add AND, OR and ANDNOT operators between the any of the terms:

{
  "searchterm": "football AND (url:theguardian.com OR url:bbc.com)"
}

Instead of ANDNOT, a hyphen can be added as a prefix. Thus, you can easily search for articles having football in the header, but not Premier league:

{
  "searchterm": "header:football -header:'premier league'"
}

Regular precedence rules apply, such that the following is equivalent:

"searchterm": "a AND b OR c AND d"

"searchterm": "(a AND b) OR (c AND d)"

In addition to these basic operators, the API has a special mode syntax, which fill in operators between terms that miss one. This simplifies specifying multiple AND or OR operators. The ANDMODE makes the following queries equivalent:

"searchterm": "ANDMODE[ a b OR c d ]",

"searchterm": "a AND b OR c AND d",

The ORMODE is similar, and makes the following queries equivalent:

"searchterm": "ORMODE[ a b AND c d ]",

"searchterm": "a OR b AND c OR d",

When using ANDMODE and there are multiple filters of the same type, the filters will not obey the ANDMODE logic, but instead be grouped, and an OR operator will be added in between instead.

The filters are mutually exclusive, such that having multiple filters of the same type in an AND clause will not return any results. Grouping filters like this helps avoid such mistakes.

"searchterm": "ANDMODE[ a lang:no lang:sv ]",

"searchterm": "a AND (lang:no OR lang:sv)",

By default, the API will wrap the search string in ANDMODE, and the filters will be grouped.

Quick start examples

This section provides a few ready-made examples to help you get started with the Opoint Search API quickly. Each example shows a different way to structure your search query using curl.

These examples use POST requests with JSON bodies — the standard way to interact with our Search API. Just replace yourtoken with your actual API token.

Basic search

Search for articles containing the keyword "spotify" anywhere in the article (header, summary, body). This search will return up to 10 articles, including the header, summary, and body. You can customize the results using the params object.

curl --location "https://api.opoint.com/search/" \
--header "Authorization: Token yourtoken" \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '{
    "searchterm": "spotify",
    "params": {
        "requestedarticles": 10,
        "main": {
            "header": 1,
            "summary": 1,
            "text": 1
        }
    }
}'

Search in a specific section

Limit the search to just the header of articles. It's useful when you want to find articles that mention a term prominently in the title, rather than buried in the text.

curl --location "https://api.opoint.com/search/" \
--header "Authorization: Token yourtoken" \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '{
    "searchterm": "header:spotify",
    "params": {
        "requestedarticles": 10,
        "main": {
            "header": 1,
            "summary": 1,
            "text": 1
        }
    }
}'

Search with a Boolean expression

Find articles where both "spotify" and "football" appear, anywhere in the article. You can also use OR, ANDNOT or – operators.

curl --location "https://api.opoint.com/search/" \
--header "Authorization: Token yourtoken" \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '{
    "searchterm": "spotify AND football",
    "params": {
        "requestedarticles": 10,
        "main": {
            "header": 1,
            "summary": 1,
            "text": 1
        }
    }
}'

Search in frequency

Find articles where "spotify" appears at least three times in the full content and also appears in the header. Great for focusing on content with higher relevance.

curl --location "https://api.opoint.com/search/" \
--header "Authorization: Token yourtoken" \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '{
    "searchterm": "spotify[3..] AND header:spotify",
    "params": {
        "requestedarticles": 10,
        "main": {
            "header": 1,
            "summary": 1,
            "text": 1
        }
    }
}'

Search in language

Same as the above, but now limited to English-language articles. lang:en is one of many filters you can use — others include geo, etc.

curl --location "https://api.opoint.com/search/" \
--header "Authorization: Token yourtoken" \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '{
    "searchterm": "spotify[3..] AND header:spotify AND lang:en",
    "params": {
        "requestedarticles": 10,
        "main": {
            "header": 1,
            "summary": 1,
            "text": 1
        }
    }
}'

Search in proximity

Return articles where "spotify" and "football" appear within five words of each other. This is perfect for identifying closely related mentions in the same context.

curl --location "https://api.opoint.com/search/" \
--header "Authorization: Token yourtoken" \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--data '{
    "searchterm": "SPAN/5(spotify, football)",
    "params": {
        "requestedarticles": 10,
        "main": {
            "header": 1,
            "summary": 1,
            "text": 1
        }
    }
}'

Try these examples in Postman

To help you get started quickly, we've created a Postman collection that includes all the example queries from this guide. You can import this collection into Postman to start experimenting with the API right away.

Download Postman Collection

How to use this collection

Download the collection file using the button above
Open Postman and click "Import"
Drag and drop the downloaded file or click "Upload Files" to select it
Once imported, you'll find all the example queries ready to use
Remember to replace "yourtoken" with your actual API token in the collection variables

Where to go next?

The search request reference has an extensive list of search options, covering both the search syntax and the available params.

A suit of examples, covering common use cases like pagination, grouping results based on similarity and getting meta data, can be found on the examples page.

The fields in the response also have a reference page.