Twingly Blog Search API v3

• Introduction
• API Endpoint
  • HTTP options
  • Request Parameters
    • Example query
  • Example Request
  • Response
  • Rate limits
  • Cache
  • Errors
    • 4xx Client errors
    • 5xx Server errors
• Best practices
  • Pagination
• Available data
• Search Language
  • Searching for blog posts using keyword
    • Searching for Chinese, Japanese and Korean characters
    • Limiting search to a specific field
  • Searching for blog posts linking to specific sites
  • Searching for blog posts on a specific domain
  • Searching for blog posts on a specific blog
  • Searching for blog posts written in a specific language
  • Searching for blog posts from a specific location
  • Searching for blog posts tagged with a specific tag
  • Searching for blog posts written by a specific author
  • Searching for a blog post by ID
  • Searching for blog posts within a given time span
  • Sorting the results
    • Sort order
  • Supported languages
  • Supported locations
  • URL normalization
    • Scheme
    • Domains
    • Blogger (blogspot.com)
• Known issues
  • Search timeouts
• Clients
• Changes from previous version
  • Migrating from Search v2
• Documentation changelog
• API changelog

Introduction

Twingly Blog Search API is a commercial XML over HTTP API that enables machine access to Twingly’s blog search index. Currently, the last 12 months of data is searchable through the API. To be able to retrieve data through the API an API key issued by Twingly must be used. The API key then grants access to blog data for one or more languages. If you don’t have access, you can sign up for a free trial.

API Endpoint

A GET request to /search retrieves blog posts which match the specified query. The blog posts are by default returned in date order starting with the newest. At most 1000 hits are returned for a given query.

HTTP options

• Base URL: https://api.twingly.com/blog/search/api/v3/search
• TLS SNI client support required for HTTPS
• HTTP compression available

Request Parameters

Parameter values must be URL encoded.

Note: The length of the query string may not exceed 2048 characters.

Example query

This will search for all posts with tag “fashion” from the blog blogg.veckorevyn.com/fridagrahn, the result will be returned in ascending order by publish date.

tag:fashion blog:blogg.veckorevyn.com/fridagrahn sort:published sort-order:asc

See our search language documentation for definitions.

Example Request

With searchpattern: banan page-size:1

curl -s "https://api.twingly.com/blog/search/api/v3/search?apikey=KEY&q=banan%20page-size:1"

Response

<twinglydata numberOfMatchesReturned="1" secondsElapsed="0.001" numberOfMatchesTotal="114628" incompleteResult="false">
  <post>
    <id>13735921747213240519</id>
    <author>Janetember</author>
    <url>
      http://nouw.com/janetember/oatmeal-carrot-cake-30001633
    </url>
    <title>Oatmeal carrot cake</title>
    <text>
      Jättegott alternativ till frukost när man har en lite längre morgon! En oatmeal carrot cake, på bara fyra ingredienser! Blanda havregryn, kanel, rivna morötter och mosad banan i en bunke. Tillsätt lite vatten om det behövs, den ska vara hyfsat rinnig. Låt stå på 200 grader en kort stund i ugnen! Smakar precis som en morotskaka!
    </text>
    <languageCode>sv</languageCode>
    <locationCode>se</locationCode>
    <coordinates/>
    <links/>
    <tags>
      <tag>Foodporn</tag>
      <tag>Recept i världsklass</tag>
    </tags>
    <images/>
    <indexedAt>2017-05-03T05:04:00Z</indexedAt>
    <publishedAt>2017-05-03T05:03:35Z</publishedAt>
    <reindexedAt>0001-01-01T00:00:00Z</reindexedAt>
    <inlinksCount>0</inlinksCount>
    <blogId>9008121184606352387</blogId>
    <blogName>the mind of jane</blogName>
    <blogUrl>http://nouw.com/janetember</blogUrl>
    <blogRank>1</blogRank>
    <authority>0</authority>
  </post>
</twinglydata>

Elements and attributes:

<twinglydata> is the root element with the following attributes:

• numberOfMatchesReturned - the number of posts in the result, the maximum is 1000
• secondsElapsed - total query time for the search request
• numberOfMatchesTotal - the total number of matches the search rendered
• incompleteResult - true if one or multiple servers were too slow to respond within the maximum allowed query time. This means it may, or may not, exist more results that are not returned. This attribute is currently not implemented, it will always return false but it may change within this API version.

Within the root element, <twinglydata>, there can be zero or more <post> elements.

<post> is the root element for a post and contains the following child elements:

• <id> - post ID (Twingly internal identification)
• <author> - author name
• <url> - URL to the post
• <title> - title of the post, limited to 256 characters
• <text> - content of the post, stripped of HTML. Note that the contents of the element may be large (several kilobytes). If the content is excessively large we may shorten it to ensure the stability of the service
• <languageCode> - ISO language code that represents the language that the post was written in
• <locationCode> - ISO country code representing the location that the blog and/or author(s) resides in
• <coordinates> - geographical coordinates from blog post (rarely available)
  • <latitude> - latitude value
  • <longitude> - longitude value
• <links> - array of all links from the blog post to other resources
  • <link> - normalized URL
• <tags> - blog post tags set by the blogger (also known as categories)
  • <tag> - tag (string)
• <images> - array of images (currently not populated)
  • <image> - URL (currently not populated)
• <indexedAt> - timestamp when the post was saved to our database/index (ISO 8601)
• <publishedAt> - when the post was published, in UTC. If no publication date could be found in the post, the date will be set to when the post was indexed (ISO 8601)
• <reindexedAt> - timestamp when the post last was changed in our database/index (ISO 8601)
• <inlinksCount> - number of links to this post found in other blog posts (only posts that are indexed by Twingly)
• <blogId> - blog ID (Twingly internal identification)
• <blogName> - name of the blog. If the blog has not provided a name we will fall back to the blog’s URL
• <blogUrl> - URL to the blog
• <blogRank> - indicates how influential the blog is, BlogRank documentation
• <authority> - indicates how authorative the blog is, authority documentation

Rate limits

We allow for a reasonable amount of concurrent requests, no hard limits. Using the same key for production, staging, and development is also perfectly fine. If you have a need for running lots of queries in parallell for a prolonged time, we’d appreciate if you told us beforehand.

Cache

Search responses are cached for 5 minutes, meaning that you will need to wait at least 5 minutes to get fresh results for a given search query. The cache key is the digest of the search pattern and the parameters; if in need to circumvent the cache it is possible to, for instance, change the timestamps slightly.

Errors

We strive to respond with the correct HTTP status code and respond with valid XML. But since computers can be tricky at times, you should ensure your client don’t blow up if you get if we give a broken response (please contact us if we do).

The errors from the API looks like this:

<error code="123">
<message>foo</message>
</error>

In the GitHub repository for the [Blog Search Ruby client] developed by Twingly, you can find example responses for most errors documented below.

4xx Client errors

Client errors, most likely your client sending invalid requests but please contact us if you can’t figure it out.

5xx Server errors

Best practices

• There are no restrictions regarding how many keywords can be put into a search query. However, we would recommend not to make queries unnecessarily long since, in our experience, results do not improve by having endlessly long search strings
• Handle transient errors
  • Ensure you retry if there are network or server errors
  • Use a back off strategy (exponential backoff for example) for the retry logic
• Use HTTP compression to receive data faster
• Provide a user agent header that identifies you
• Subscribe to our status site

Pagination

If numberOfMatchesTotal is greater than numberOfMatchesReturned, then you will need to paginate through the result in order to retrieve all posts. The best way to do this is to utilize the start-date and/or end-date , creating a sliding time-based window.

Sort with sort-order:asc sort:published Set start-date: to the published time of the newest returned post and repeat your query. Repeat until numberOfMatchesTotal equals numberOfMatchesReturned. This technique can be seen in the examples for the Search API Ruby client.

For the (unusual) case where your search query yields a numberOfMatchesTotal greater than 1,000 (the default page size), where all posts are published at the same time, you can increase numberOfMatchesReturned by adding the page-size option to your search pattern. This works up until the maximum page-size of 10,000. In the very unlikely event that your query still yields more than 10,000 hits you will need to start adding keywords to the query, or add filters such as lang.

Example search pattern with page-size:

"christmas page-size:5000"

If you need to keep the API response small, it’s possible to paginate through up to 10,000 matches, with the page option added to your search pattern. Note that the maximum value for page is 100.

Example search pattern with page and page-size:

"christmas page:2 page-size:100"

Available data

The search index for the Blog Search API contains blog posts from the last 12 months.

As soon as we get new blog content it will be searchable through the Search API. Most often it takes just a few seconds for data to flow through our system, it may be a few minutes during maintenance though.

See also our page about the ingestion system details and challenges.

Search Language

The Twingly search language provides a powerful set of tools to be used when querying the Blog Search API.

Searching for blog posts using keyword

Search all fields for the given keywords.

# search for posts containing the words twingly and blog
twingly blog

# search for posts containing the phrase "I love blogging"
"I love blogging"

# search for posts containing the words twingly or blog
twingly OR blog

# search for posts containing the words twingly and either blog or blogs
twingly AND (blog OR blogs)

# search for posts containing the word twingly but not the word blog
twingly -blog

Searching for Chinese, Japanese and Korean characters

Chinese, Japanese and Korean (CJK) characters are handled as individual words. To search for multi-character words, the word needs to be quoted as a phrase.

# Search for posts containing 東京 (Tokyo).
"東京"

# Search for posts containing 東 (Eastern) and 京 (Capital).
# For example a post mentioning both "東大" Tokyo University and "京都市" (Kyoto) will match.
東京

Limiting search to a specific field

You can limit the search to specific fields using the following syntax:

# search just the text
fields:text twingly

# search the text and the title
fields:text|title twingly

The supported arguments to fields are:

• text - the contents of the post
• summary - deprecated, use text instead
• title - the title of the post
• url - the URL to the post
• author - the author of the post
• tags - the tags/categories of the post
• links - the links found in the blog post content
• blogname - the name of the blog
• blogurl - the URL to the blog

By default, your search query is matched against all of the fields listed above.

Note that using the fields operator affects the whole query, regardless of parenthesis. For example: (fields:blogname obama) summit would search for “obama” and “summit” only in the blogname field.

Searching for blog posts linking to specific sites

You can search for blog posts that link to specific pages. Note that URLs are normalized in the index, check out the URL normalization section.

# search for posts that link to any page on twingly.com
link:twingly.com

# search for posts that link a specific page on twingly.com
link:twingly.com/ping

# search for posts that link to any page on twingly.com *and* techcrunch.com
link:twingly.com,techcrunch.com

# search for posts that link to any page on twingly.com *or* techcrunch.com
link:twingly.com|techcrunch.com

# search for posts linking to twingly.com but not pingomatic.com
link:twingly.com -link:pingomatic.com

Note that the link: and -link: operators are computationally expensive. Excessive use of them may slow down the query considerably or even result in a query timeout.

Searching for blog posts on a specific domain

You can search for blog posts on a specific domain, including subdomains. Note that URLs are normalized in the index, check out the URL normalization section.

# search for blog posts on twingly.com, including blog.twingly.com
site:twingly.com

# search for blog posts on several domains
site:twingly.com|primelabs.se

# search for blog posts mentioning twingly but not on twingly.com
twingly -site:twingly.com

Note that the site: and -site: operators are computationally expensive. Excessive use of them may slow down the query considerably or even result in a query timeout.

Searching for blog posts on a specific blog

You can search for blog posts on a specific blog. Note that URLs are normalized in the index, check out the URL normalization section.

# search for blog posts on blog.twingly.com
blog:blog.twingly.com

# search for blog posts on multiple blogs
blog:blog.twingly.com|roslingsblogger.blogspot.com

# search for blog posts mentioning twingly but not on blog.twingly.com
twingly -blog:twingly.com

Searching for blog posts written in a specific language

You can search for blog posts written in a specific language.

# search for blog posts written in Swedish
lang:sv

# search for blog posts written in Swedish or Finnish
lang:sv|fi

List of all supported languages.

Searching for blog posts from a specific location

You can search for blog posts from a specific location, note that not every post has a location set.

# search for blog posts from Sweden
location:se

# search for blog posts from in Sweden or Finland
location:se|fi

List of all supported locations.

Searching for blog posts tagged with a specific tag

You can search for blog posts with specific tags.

# search for blog posts with the tag election
tag:election

# search for blog posts with the tags election *and* obama
tag:election,obama

# search for blog posts with the tags election, sport *or* fashion
tag:election|sport|fashion

# search for blog posts with the tag obama but not election
tag:obama -tag:election

Searching for blog posts written by a specific author

# search for blog posts with an author named Isabella
author:isabella

# search for blog posts with an author named Isabella but not "Isabella Wight"
author:isabella -author:wight

Searching for a blog post by ID

You can find a specific blog post by using the post id field in the result.

id:13735921747213240519

Searching for blog posts within a given time span

The default is to search in posts published at any time.

You can search for posts that were published within the last 24 hours.

tspan:24h

The supported arguments to tspan are:

• h - posts published the last hour
• 12h - posts published the last 12 hours
• 24h - posts published the last 24 hours
• w - posts published the last week
• m - posts published the last month
• 3m - posts published the last three months

Please note that these are the only supported arguments, e.g. 3h or 4m won’t work

In addition to the tspan parameter you can perform explicit searches on creation (indexed) time or published time.

Examples:

• Search for posts created (indexed) after 2016-01-16 00:00:00 UTC:

  start-created:2016-01-16T00:00:00
  # or
  start-created:"2016-01-16 00:00:00"
  

• Search for posts created (indexed) between 2015-12-01 02:00:00 UTC and 2015-12-01 03:00:00 UTC:

  start-created:2015-12-01T02:00:00 end-created:2015-12-01T03:00:00
  # or
  start-created:"2015-12-01 02:00:00" end-created:"2015-12-01 03:00:00"
  

• Search for posts published between 2015-12-01 02:00:00 UTC and 2015-12-01 03:00:00 UTC:

  start-date:2015-12-01T02:00:00 end-date:2015-12-01T03:00:00
  # or
  start-date:"2015-12-01 02:00:00" end-date:"2015-12-01 03:00:00"
  

Timestamps are interpreted as UTC and must be supplied without any time zone information (like shown in the examples above).

Note that using tspan in the same query as start-created, end-created, start-date or end-date is not supported and will yield unexpected results.

Sorting the results

You can sort the blog posts in multiple ways.

# sort by published
sort:published

# sort by inlinks
sort:inlinks

The supported arguments to sort are:

• published - sort by published
• created - sort by created (i.e. index time)
• inlinks - sort by inlinks to the post
• twinglyrank - sort by TwinglyRank

Sort order

You can specify the sort order.

# ascending
sort-order:asc

# descending
sort-order:desc

Supported languages

To query a specific language, use the the two-letter ISO 639-1 code.

Supported locations

To query a specific location, use the the two-letter ISO 3166-1 alpha-2 format code.

URL normalization

Since a web resource can have multiple URLs, we apply some normalization to make the index more consistent. However, we do not normalize URLs in link:, blog:, and site: searches automatically. The URL normalization has to be done by the user. Below are some hints on how the URL normalization works.

Scheme

The scheme part is not taken into account, the original scheme is preserved in the normalized URL.

Domains

We add www. for all domains that only consists of a SLD and TLD.

https://twingly.com/ -> https://www.twingly.com/

For domains with a non-www subdomain, we remove www. if present.

https://www.blog.twingly.com/ -> https://blog.twingly.com/

Blogger (blogspot.com)

Google introduced country specific TLDs for Blogger (blogspot.com) in early 2012. The result of this change is that the same blog will have different URLs (TLDs) dependening on which country you’re currently browsing the web. Consequently, all blogs from Blogger are normalized to blogspot.com, such as:

• https://googleblog.blogspot.se/ -> https://googleblog.blogspot.com/

Known issues

• The maximum value for the page: option is 100. Querying for pages beyond 100 will always return the 100th page. Either increase page-size: or paginate using start-date: (recommended) instead.
• It is not possible to perform searches using characters from character sets used in the following languages: Bengali, Gujarati, Hebrew, Hindi, Georgian, Kannada, Malayalam, Marathi, Nepali, Punjabi, Tamil, Telugu or Thai. We are working on solving the problem as soon as possible.
• Since we grant access to blog data per language, some posts in a blog might not be searchable if they are written/classified in a language that the API key doesn’t give access to.
• We do not extract any image URLs as of this moment, so <images> will always be empty.
• Neither do we currently extract coordinates from the blogs, <coordinates> is currently mostly empty, there may be some coordinates in older data though.
• For certain search queries, a zero-sized result is returned rather than a 400 HTTP status. Examples of when this happens:
  • Using a page-size larger than 10,000
  • Using double negatives: foo --bar
• Using query strings longer than 2048 characters will result in a HTML error page.
• Nested grouping yields unexpected results. For example: ((keyword1 and keyword2)) is not interpretated as (keyword1 and keyword2) or keyword1 and keyword2.
• Searching on words certain characters, such as & and ' does not work as expected. For example searching on c'est would effectively be the same as searching for c AND est as the ' character has been ignored at index time. To mitigate this problem, always quote terms containing special characters: "c'est".
• The site: operator takes the full URL into account, and just not the domain component. E.g. site:twingly.com would match a blog post with URL https://some-blog.com/post?url=twingly.com. A mitigation is to use the blog: operator where applicable, as it only searches the “base URL” of a blog. E.g. if a blog is located at https://some-hotel.com/some-blog and has a post at https://some-hotel.com/some-blog/post?url=twingly.com, then blog:twingly.com would not match as the query would only search the https://some-hotel.com/some-blog part.
• Operators (for example tag: and author:), do not support quoted phrases combined with the | and , delimiters. For example, author:"Twingly Twinglysson"|"John Smith" does not work.
  • In the | case, consider making multiple requests, with one phrase per request.
  • For the , case, you may re-write the query into using multiple operators instead, such as author:"Twingly Twinglysson" author:"John Smith".

Search timeouts

As the search result property incompleteResult is not yet implemented, it is hard to determine whether the search query was subject to timeouts or not. However, our current search agent timeout is set to 6 seconds. This means that if the secondsElapsed attribute is greater than 6 seconds, then there’s a good chance that at least one search agent timed out – leading to an incomplete result. If this happens repeatedly, consider rewriting your query to a less expensive form.

Clients

• Ruby client, available on GitHub
• .NET client, available on GitHub
• Java client, available on GitHub
• Node.js client, available on GitHub
• PHP client, available on GitHub
• Python client, available on GitHub

Changes from previous version

We have focused on simplifying the API request and extending the response.

• HTTPS required, it is not possible to request data over HTTP (it will not be redirected)
• Renamed searchpattern to q
• Renamed key to apikey
• Fewer parameters
  • Dropped ts and tsTo
  • Dropped documentlang
  • Dropped xmloutputversion
  • Dropped approved
  • Dropped fields
  • Dropped product

Changes to output XML:

• post attribute contentType removed
• Response field changes:
  • Added id - Post ID
  • Added blogId - Blog ID
  • Added author - Author from blog post
  • Added locationCode - Blog location
  • Added inlinksCount - Number of links found in other blog posts (only posts that are indexed by Twingly)
  • Added reindexedAt - Timestamp when the post last was changed in our database/index
  • Added links - All links from the blog post to other resources
  • Added images - Image URLs from the posts (currently not populated)
  • Added coordinates - Geographical coordinates from blog post (currently not populated)
  • Renamed summary to text - To better emphasize that this is the full text that we can extract
  • Renamed published to publishedAt - Easier to tell this is a timestamp
  • Renamed indexed to indexedAt - Easier to tell this is a timestamp
  • All timestamps changed to include time zone
  • Changed url - Changed to non-normalized value
  • Changed blogUrl - Changed to non-normalized value
  • Removed approved attribute
• Changed to ISO 8601 format for all timestamps (example: 2017-05-01T13:37:00Z)

Migrating from Search v2

• Change key to apikey
• Change searchpattern to q (no changes to the query language)
• Remove deprecated parameters
  • Remove xmloutputversion
  • Remove approved
  • Remove fields
  • Remove product
• Rewrite removed parameters to search pattern
  • Replace ts with pattern: start-date:
  • Replace tsTo with pattern: end-date:
  • Replace documentlang with pattern: lang:
• Update any renamed response fields

Documentation changelog

• 2020-04
  • Added clarification that timestamps in the query should not contain time zone information.
  • Moved the search language documentation from a separate page into this one.
• 2019-01
  • Added information about the maximum length of the query string.
• 2018-08
  • Added known issues about operators and quotes.
  • Added known issue about the site: operator.
• 2018-06
  • Added text concerning search timeouts in the “Known issues” section.
• 2018-04
  • Added the “Scheme” section under “URL normalization”.
  • Noted that links are normalized.
  • Added link to Source discovery and ingestion in the “Available data” section.
• 2018-03
  • Removed links to “public blog search”.
• 2018-02
  • Improved the “Errors” section.
• 2017-11
  • Added text, url, blogurl and links arguments to the fields operator.
• 2017-10
  • Added simpler timestamp examples, no need to quote if the “T” separator is used.
  • Corrected datetime format misinformation in the time filters section, datetimes should not have time zone information when passed to the API.
• 2017-08
  • Added text explaining that page: has a limit of 100.
• 2017-05
  • Improved documentation around coordinates.
  • Added text explaining that the timestamp format has changed to ISO 8601 in v3.
  • Added blogId to the list of response fields that was added in v3.
  • Fixed typo in documentation for migrating from v2 to v3: it was previously stated that tsTo should be replaced with pattern stop-date:, but the correct pattern is end-date:.
  • Corrected the release date of Twingly Blog Search API v3 (2017-04 -> 2017-05).
  • Added Search v3 documentation, move Search v2 documentation to Search API v2.
  • Added documentation for the id operator which can be used for finding a specific blog post.
  • Added information about the behaviour when time filters are used without time zone information.
  • Removed ambiguous examples from the time filters section.
• 2016-10
  • Added a note about how excessive use of site and/or link can result in a timeout.
• 2016-08
  • Added location documentation.
  • Added remarks to supported languages listing.
• 2016-07
  • Added information on how to search for CJK characters.
• 2016-06
  • Added a note making it clear that tspan should not be used together with the other time filters.
• 2016-01
  • Added phrase example to search language documentation.
  • Added search language documentation.

API changelog

• 2019-03
  • In certain cases, a language restricted API key could query on all languages. This has now been corrected.
• 2017-05
  • Release Twingly Blog Search API v3.