Twingly Blog Search API is a commercial XML over HTTP API that enables machine access to Twingly’s blog search index. Currently, the last 12 months of data is searchable through the API. To be able to retrieve data through the API an API key issued by Twingly must be used. The API key then grants access to blog data for one or more languages. If you don’t have access, you can sign up for a free trial.
A GET request to /search retrieves blog posts which match the specified query. The blog posts are by default returned in date order starting with the newest. At most 1000 hits are returned for a given query.
https://api.twingly.com/blog/search/api/v3/searchParameter values must be URL encoded.
| Parameter | Format | Example | Notes |
|---|---|---|---|
apikey |
String | E67EFC65-08A9-4086-BE92-074BBD7F78EA |
Required |
q |
Search query | tag:fashion sort:published |
Required |
format |
String | xml |
Optional, defaults to xml (which is the only allowed value) |
Note: The length of the query string may not exceed 2048 characters.
This will search for all posts with tag “fashion” from the blog blogg.veckorevyn.com/fridagrahn, the result will be returned in ascending order by publish date.
tag:fashion blog:blogg.veckorevyn.com/fridagrahn sort:published sort-order:asc
See our search language documentation for definitions.
With searchpattern: banan page-size:1
curl -s "https://api.twingly.com/blog/search/api/v3/search?apikey=KEY&q=banan%20page-size:1"
<twinglydata numberOfMatchesReturned="1" secondsElapsed="0.001" numberOfMatchesTotal="114628" incompleteResult="false">
<post>
<id>13735921747213240519</id>
<author>Janetember</author>
<url>
http://nouw.com/janetember/oatmeal-carrot-cake-30001633
</url>
<title>Oatmeal carrot cake</title>
<text>
Jättegott alternativ till frukost när man har en lite längre morgon! En oatmeal carrot cake, på bara fyra ingredienser! Blanda havregryn, kanel, rivna morötter och mosad banan i en bunke. Tillsätt lite vatten om det behövs, den ska vara hyfsat rinnig. Låt stå på 200 grader en kort stund i ugnen! Smakar precis som en morotskaka!
</text>
<languageCode>sv</languageCode>
<locationCode>se</locationCode>
<coordinates/>
<links/>
<tags>
<tag>Foodporn</tag>
<tag>Recept i världsklass</tag>
</tags>
<images/>
<indexedAt>2017-05-03T05:04:00Z</indexedAt>
<publishedAt>2017-05-03T05:03:35Z</publishedAt>
<reindexedAt>0001-01-01T00:00:00Z</reindexedAt>
<inlinksCount>0</inlinksCount>
<blogId>9008121184606352387</blogId>
<blogName>the mind of jane</blogName>
<blogUrl>http://nouw.com/janetember</blogUrl>
<blogRank>1</blogRank>
<authority>0</authority>
</post>
</twinglydata>
Elements and attributes:
<twinglydata> is the root element with the following attributes:
numberOfMatchesReturned - the number of posts in the result, the maximum is 1000secondsElapsed - total query time for the search requestnumberOfMatchesTotal - the total number of matches the search renderedincompleteResult - true if one or multiple servers were too slow to respond within the maximum allowed query time. This means it may, or may not, exist more results that are not returned. This attribute is currently not implemented, it will always return false but it may change within this API version.Within the root element, <twinglydata>, there can be zero or more <post> elements.
<post> is the root element for a post and contains the following child elements:
<id> - post ID (Twingly internal identification)<author> - author name<url> - URL to the post<title> - title of the post, limited to 256 characters<text> - content of the post, stripped of HTML. Note that the contents of the element may be large (several kilobytes). If the content is excessively large we may shorten it to ensure the stability of the service<languageCode> - ISO language code that represents the language that the post was written in<locationCode> - ISO country code representing the location that the blog and/or author(s) resides in<coordinates> - geographical coordinates from blog post (rarely available)
<latitude> - latitude value<longitude> - longitude value<links> - array of all links from the blog post to other resources
<link> - normalized URL<tags> - blog post tags set by the blogger (also known as categories)
<tag> - tag (string)<images> - array of images (currently not populated)
<image> - URL (currently not populated)<indexedAt> - timestamp when the post was saved to our database/index (ISO 8601)<publishedAt> - when the post was published, in UTC. If no publication date could be found in the post, the date will be set to when the post was indexed (ISO 8601)<reindexedAt> - timestamp when the post last was changed in our database/index (ISO 8601)<inlinksCount> - number of links to this post found in other blog posts (only posts that are indexed by Twingly)<blogId> - blog ID (Twingly internal identification)<blogName> - name of the blog. If the blog has not provided a name we will fall back to the blog’s URL<blogUrl> - URL to the blog<blogRank> - indicates how influential the blog is, BlogRank documentation<authority> - indicates how authorative the blog is, authority documentationWe allow for a reasonable amount of concurrent requests, no hard limits. Using the same key for production, staging, and development is also perfectly fine. If you have a need for running lots of queries in parallell for a prolonged time, we’d appreciate if you told us beforehand.
Search responses are cached for 5 minutes, meaning that you will need to wait at least 5 minutes to get fresh results for a given search query. The cache key is the digest of the search pattern and the parameters; if in need to circumvent the cache it is possible to, for instance, change the timestamps slightly.
We strive to respond with the correct HTTP status code and respond with valid XML. But since computers can be tricky at times, you should ensure your client don’t blow up if you get if we give a broken response (please contact us if we do).
The errors from the API looks like this:
<error code="123">
<message>foo</message>
</error>
In the GitHub repository for the [Blog Search Ruby client] developed by Twingly, you can find example responses for most errors documented below.
Client errors, most likely your client sending invalid requests but please contact us if you can’t figure it out.
| HTTP status | Error code | Description |
|---|---|---|
| 400 | 40001 | Invalid parameters (see the message for more info) |
| 400 | 40002 | Invalid query (see the message for more info) |
| 400 | 40003 | Invalid query (see the message for more info) |
| 401 | 40101 | Unauthorized |
| 402 | 40201 | Access to language(s) denied (see the message for more info) |
| 404 | 40401 | Not Found |
| HTTP status | Error code | Description |
|---|---|---|
| 500 | 50001 | Internal Server Error. Unexpected conditions were encountered, indicating a server-side bug. |
| 503 | 50301 | Service Unavailable. Retry later. |
If numberOfMatchesTotal is greater than numberOfMatchesReturned, then you will need to paginate through the result in order to retrieve all posts. The best way to do this is to utilize the start-date and/or end-date , creating a sliding time-based window.
Sort with sort-order:asc sort:published Set start-date: to the published time of the newest returned post and repeat your query. Repeat until numberOfMatchesTotal equals numberOfMatchesReturned. This technique can be seen in the examples for the Search API Ruby client.
For the (unusual) case where your search query yields a numberOfMatchesTotal greater than 1,000 (the default page size), where all posts are published at the same time, you can increase numberOfMatchesReturned by adding the page-size option to your search pattern. This works up until the maximum page-size of 10,000. In the very unlikely event that your query still yields more than 10,000 hits you will need to start adding keywords to the query, or add filters such as lang.
Example search pattern with page-size:
"christmas page-size:5000"
If you need to keep the API response small, it’s possible to paginate through up to 10,000 matches, with the page option added to your search pattern. Note that the maximum value for page is 100.
Example search pattern with page and page-size:
"christmas page:2 page-size:100"
The search index for the Blog Search API contains blog posts from the last 12 months.
As soon as we get new blog content it will be searchable through the Search API. Most often it takes just a few seconds for data to flow through our system, it may be a few minutes during maintenance though.
See also our page about the ingestion system details and challenges.
page: option is 100. Querying for pages beyond 100 will always return the 100th page. Either increase page-size: or paginate using start-date: (recommended) instead.Bengali, Gujarati, Hebrew, Hindi, Georgian, Kannada, Malayalam, Marathi, Nepali, Punjabi, Tamil, Telugu or Thai. We are working on solving the problem as soon as possible.<images> will always be empty.<coordinates> is currently mostly empty, there may be some coordinates in older data though.400 HTTP status. Examples of when this happens:
page-size larger than 10,000foo --barAs the search result property incompleteResult is not yet implemented, it is hard to determine whether the search query was subject to timeouts or not. However, our current search agent timeout is set to 6 seconds. This means that if the secondsElapsed attribute is greater than 6 seconds, then there’s a good chance that at least one search agent timed out – leading to an incomplete result. If this happens repeatedly, consider rewriting your query to a less expensive form.
We have focused on simplifying the API request and extending the response.
searchpattern to qkey to apikeyts and tsTodocumentlangxmloutputversionapprovedfieldsproductChanges to output XML:
post attribute contentType removedid - Post IDblogId - Blog IDauthor - Author from blog postlocationCode - Blog locationinlinksCount - Number of links found in other blog posts (only posts that are indexed by Twingly)reindexedAt - Timestamp when the post last was changed in our database/indexlinks - All links from the blog post to other resourcesimages - Image URLs from the posts (currently not populated)coordinates - Geographical coordinates from blog post (currently not populated)summary to text - To better emphasize that this is the full text that we can extractpublished to publishedAt - Easier to tell this is a timestampindexed to indexedAt - Easier to tell this is a timestampurl - Changed to non-normalized valueblogUrl - Changed to non-normalized valueapproved attribute2017-05-01T13:37:00Z)key to apikeysearchpattern to q (no changes to the query language)xmloutputversionapprovedfieldsproductts with pattern: start-date:tsTo with pattern: end-date:documentlang with pattern: lang:links are normalized.page: has a limit of 100.coordinates.v3.blogId to the list of response fields that was added in v3.v2 to v3: it was previously stated that tsTo should be replaced with pattern stop-date:, but the correct pattern is end-date:.2017-04 -> 2017-05).