Data center migration complete

Just a quick update on our data center migration.

On 2018-10-17 we started serving production traffic from our new datacenter location.

Even though there are many challenges with moving live infrastructure from one data center to another, we managed to complete the project on time and with only a few hours of, planned, delay on the ingested blog data.

We hope you didn’t notice this change, cheers!

Data center migration plan

Information about an upcoming data center migration.

TL;DR

  • We are doing a data center migration for our blog data systems
  • Make sure you are subscribed to updates on status.twingly.com
  • There will be some amount of downtime when we promote the new data center to primary

Migration plan

Twingly will be moving servers from one data center (LKP) to a new data center (STH). This data center is mainly used for our blog data services.

As we have enough servers to keep the services running at full capacity during the migration, we don’t expect any extended period of downtime. Even though our systems will work at full speed, the risk of outages is increased during this period.

When we make the logical switch from the old data center to the new data center, there will be a short amount of planned downtime. This will be announced in advance through status.twingly.com.

Affected services:

  • Twingly Blog Search API - Slight delays in new data, shorter outage (hosted in data center)
  • Twingly Blog LiveFeed API - Slight delays in new data, shorter outage (hosted in data center)
  • Twingly Blog Box - Slight delays in new links (hosted outside the data center)

We are doing the migration in three steps.

  1. Setup network environment at new data center
    • New network equipment is purchased and installed
    • This step has already been completed
  2. Physically move about half of the servers
    • Once online in STH, the servers will replicate data from LKP
    • Wait for the data to be in sync
    • Make STH the primary data center by updating DNS records (api.twingly.com)
    • Verify that everything is working as intended
    • Servers in LKP are powered down
  3. Move the rest of the servers, done!

Update 2018-11-08: Data center migration complete

Blog Search in API Dashboard

We have added a new page in our API Dashboard (app.twingly.com) that lets you test the Blog Search API using your web browser.

Just head over to app.twingly.com/search/blog and try it out.

Note that in order to use the Blog Search page, you need an account on app.twingly.com with access to the Blog Search API.

Shutdown plan for legacy Blog Search API and Blog LiveFeed API versions

Earlier this year (2017) we released Blog Search API v3 and Blog LiveFeed API v5 to replace our older API versions. We’re now announcing that the older API versions will be shut down 2018-03-31. Our latest API versions are based on feedback from you, our customers, and includes even more metadata than before. The LiveFeed API is much easier to use and includes backend changes that will improve the service availability.

TL;DR

  • Ensure you have updated to the latest API versions by 2018-03-31
  • Latest versions:
  • The latest API versions include everything the old versions include, but with even more metadata per post.
  • Please ensure your technical team has this information so the migration can be planned in advance.

Deprecated Blog Search API versions (previously known as Analytics API)

URL path Version Released
/analytics/Analytics.ashx?xmloutputversion=1 v1 2007-5
/analytics/Analytics.ashx?xmloutputversion=2 v2 2011-5

Documentation for latest version, including changes and migration steps.

Deprecated Blog LiveFeed API versions

URL path Version Released
/livefeed.asmx v1 2007-04
/livefeed2.asmx v2 2010-03
/livefeed3.asmx v3 2010-06
/livefeed4.asmx v4 2011-05

Documentation for latest version, including changes and migration steps.

FAQ

We don’t know how to migrate, can you help us?

Absolutely, contact our support and we’ll do our best to ensure a smooth migration. But start by reading the release post and API documentation.

How do I know which version I’m using?

Look at the URL, basically if it doesn’t start with /blog/ you’re using a deprecated API.

New API versions released: Blog Search API v3 and Blog LiveFeed API v5

We have released new versions of our Blog APIs. Let me introduce:

Common improvements

Although both Search and LiveFeed have always delivered the same blog posts, either by exposing a search index or a “firehose”, there have been slight differences between the data. With these new versions we have tried to make the APIs more similar and easier to use.

LiveFeed API v5 is now a plain HTTP GET API, just like Search API. No more SOAP.

The post element now includes the same tags for both APIs. It also contains more information about the blog post, than the previous API versions. The new fields added to post are:

  • id - our unique ID of the blog post (new only for Search v3)
  • blogId - our unique ID of the blog
  • author - the author of the blog post
  • locationCode - the location of the blog
  • inlinksCount - number of links found in other blog posts (only posts that are indexed by Twingly)
  • reindexedAt - timestamp when the post last was changed in our database/index
  • links - all links from the blog post to other resources
  • tags - blog post tags (may also be referred to as categories)

We have prepared fields for future data, these are currently not populated but it will be possible to start delivering this data in the future without breaking any implementations:

  • images - image URLs from the posts
  • coordinates - Geographical coordinates from blog post (a few posts do have this information)

Blog LiveFeed API

In addition to the common improvements, the following is now true for LiveFeed API:

  • Instant access to 30 days of data, no delays to accumulate data for backfill
  • Prepared nextTimestamp, no need to calculate the next timestamp
  • Impossible to “get stuck” if there’s data ingestion issues
  • Our Ruby API client has gained support for Blog LiveFeed v5

Blog Search API

Besides the new data that Search API is now delivering, we’ve also made the following improvements.

The .NET client hasn’t yet been updated. If you have the need for a .NET client, don’t hesitate to write about it in the issue. We also gladly accept pull requests.

Update 2017-10-26: The .NET client has now been updated to support v3 too.

Migration

For each API, we have documented the changes from the previous version, and listed the steps required to move to the new version. Please check it out:

These new versions are available for all existing, and new, customers. Please upgrade to take advantage of all the improvements. If you need any assistance, don’t hesitate to contact us!