Data center migration plan

Information about an upcoming data center migration.

TL;DR

  • We are doing a data center migration for our blog data systems
  • Make sure you are subscribed to updates on status.twingly.com
  • There will be some amount of downtime when we promote the new data center to primary

Migration plan

Twingly will be moving servers from one data center (LKP) to a new data center (STH). This data center is mainly used for our blog data services.

As we have enough servers to keep the services running at full capacity during the migration, we don’t expect any extended period of downtime. Even though our systems will work at full speed, the risk of outages is increased during this period.

When we make the logical switch from the old data center to the new data center, there will be a short amount of planned downtime. This will be announced in advance through status.twingly.com.

Affected services:

  • Twingly Blog Search API - Slight delays in new data, shorter outage (hosted in data center)
  • Twingly Blog LiveFeed API - Slight delays in new data, shorter outage (hosted in data center)
  • Twingly Blog Box - Slight delays in new links (hosted outside the data center)

We are doing the migration in three steps.

  1. Setup network environment at new data center
    • New network equipment is purchased and installed
    • This step has already been completed
  2. Physically move about half of the servers
    • Once online in STH, the servers will replicate data from LKP
    • Wait for the data to be in sync
    • Make STH the primary data center by updating DNS records (api.twingly.com)
    • Verify that everything is working as intended
    • Servers in LKP are powered down
  3. Move the rest of the servers, done!

Blog Search in API Dashboard

We have added a new page in our API Dashboard (app.twingly.com) that lets you test the Blog Search API using your web browser.

Just head over to app.twingly.com/search/blog and try it out.

Note that in order to use the Blog Search page, you need an account on app.twingly.com with access to the Blog Search API.

Shutdown plan for legacy Blog Search API and Blog LiveFeed API versions

Earlier this year (2017) we released Blog Search API v3 and Blog LiveFeed API v5 to replace our older API versions. We’re now announcing that the older API versions will be shut down 2018-03-31. Our latest API versions are based on feedback from you, our customers, and includes even more metadata than before. The LiveFeed API is much easier to use and includes backend changes that will improve the service availability.

TL;DR

  • Ensure you have updated to the latest API versions by 2018-03-31
  • Latest versions:
  • The latest API versions include everything the old versions include, but with even more metadata per post.
  • Please ensure your technical team has this information so the migration can be planned in advance.

Deprecated Blog Search API versions (previously known as Analytics API)

URL path Version Released
/analytics/Analytics.ashx?xmloutputversion=1 v1 2007-5
/analytics/Analytics.ashx?xmloutputversion=2 v2 2011-5

Documentation for latest version, including changes and migration steps.

Deprecated Blog LiveFeed API versions

URL path Version Released
/livefeed.asmx v1 2007-04
/livefeed2.asmx v2 2010-03
/livefeed3.asmx v3 2010-06
/livefeed4.asmx v4 2011-05

Documentation for latest version, including changes and migration steps.

FAQ

We don’t know how to migrate, can you help us?

Absolutely, contact our support and we’ll do our best to ensure a smooth migration. But start by reading the release post and API documentation.

How do I know which version I’m using?

Look at the URL, basically if it doesn’t start with /blog/ you’re using a deprecated API.

New API versions released: Blog Search API v3 and Blog LiveFeed API v5

We have released new versions of our Blog APIs. Let me introduce:

Common improvements

Although both Search and LiveFeed have always delivered the same blog posts, either by exposing a search index or a “firehose”, there have been slight differences between the data. With these new versions we have tried to make the APIs more similar and easier to use.

LiveFeed API v5 is now a plain HTTP GET API, just like Search API. No more SOAP.

The post element now includes the same tags for both APIs. It also contains more information about the blog post, than the previous API versions. The new fields added to post are:

  • id - our unique ID of the blog post (new only for Search v3)
  • blogId - our unique ID of the blog
  • author - the author of the blog post
  • locationCode - the location of the blog
  • inlinksCount - number of links found in other blog posts (only posts that are indexed by Twingly)
  • reindexedAt - timestamp when the post last was changed in our database/index
  • links - all links from the blog post to other resources
  • tags - blog post tags (may also be referred to as categories)

We have prepared fields for future data, these are currently not populated but it will be possible to start delivering this data in the future without breaking any implementations:

  • images - image URLs from the posts
  • coordinates - Geographical coordinates from blog post (a few posts do have this information)

Blog LiveFeed API

In addition to the common improvements, the following is now true for LiveFeed API:

  • Instant access to 30 days of data, no delays to accumulate data for backfill
  • Prepared nextTimestamp, no need to calculate the next timestamp
  • Impossible to “get stuck” if there’s data ingestion issues
  • Our Ruby API client has gained support for Blog LiveFeed v5

Blog Search API

Besides the new data that Search API is now delivering, we’ve also made the following improvements.

The .NET client hasn’t yet been updated. If you have the need for a .NET client, don’t hesitate to write about it in the issue. We also gladly accept pull requests.

Update 2017-10-26: The .NET client has now been updated to support v3 too.

Migration

For each API, we have documented the changes from the previous version, and listed the steps required to move to the new version. Please check it out:

These new versions are available for all existing, and new, customers. Please upgrade to take advantage of all the improvements. If you need any assistance, don’t hesitate to contact us!

BlogRank changes

We have made a few changes to our BlogRank algorithm in order to make it easier to distinguish between the blogs at the top of the BlogRank scale.

The previous version of the algorithm had the trait of slowly collecting more and more blogs at the top of the BlogRank scale as our blog index grew larger.

This has now been adjusted, making sure only the few most popular blogs has BlogRank 10. For blogs having a low BlogRank this change won’t be noticeable, but a significant part of the high-ranking blogs will get a lower BlogRank, which can be seen in the graphs below:

BlogRank 2-6 distribution for Swedish blogs before and after the change

Distribution of Swedish blogs with BlogRank 2-6 before and after the change.

BlogRank 6-10 distribution for Swedish blogs before and after the change

Distribution of Swedish blogs with BlogRank 6-10 before and after the change.

For the exact numbers on how much the BlogRank distribution has changed for all Swedish blogs, see the table below:

BlogRank Number of blogs before Number of blogs after
1 101,380 104,262
2 4,295 3,995
3 2,081 1,435
4 1,170 609
5 694 279
6 429 129
7 299 54
8 177 19
9 141 13
10 137 8

For details about BlogRank, authority and top authority see the ranking documentation.