Loading of The Daily Mail into the NLA database failed on Tuesday 4th November 2008. This meant that the distribution of feeds to NLA clients did not contain 1st edition Daily Mail content by the target time of 01:00. Loading and distribution of other titles were unaffected.

Cause:

A momentary connectivity failure between the server running the loading module and the storage device to which loading takes place, caused a single thread of the loading module to loop erroneously.

Solution:

NLA engineers restarted the loading module which resulted in the loading and distribution of all the 1st edition Daily Mail content by 01:15.

Analysis of the loading module's source code has identified areas where modifications can be made to prevent a similar incident in future. These modifications will be scheduled soon.

Certain eClips customers had intermittent access to NLA web and FTP services from 7:35am to 9:00am and from 9:19am to 9:33am on Saturday 18th October 2008.

Cause:

The owners of NLA's London hosting facility were carrying out the first phase of a planned, annual, power-down exercise on Saturday 18th October. This involved disabling one of the two power feeds which supply the NLA infrastructure. The NLA's infrastructure can usually tolerate removal of one power feed as it has a dual-fed, clustered architecture. In this instance, the automated failover of one clustered network component did not complete successfully.

Solution:

The failover process for the affected network component required manual intervention by engineers, who ensured that it completed successfully. The engineers also made some configuration changes to the cluster which should reduce the risk of a similar event occurring in future.

Problem:

At approximately 9:30 this morning an incident occurred which impacted eClips service delivery. The incident was resolved at 10:00. During this incident, clients ability to view eClips content was impacted as the service was intermittently unavailable.

Cause:

The root cause of the incident is still under investigation by NLA engineers, however indications show that when attempting to serve a higher than normal number of requests, the eClips database license checking process became less responsive and is being investigated as a potential area requiring optimization.

Solution:

NLA engineers are now reviewing the eClips core code related to this aspect of the service with the aim of discovering the root cause and optimizing it to prevent reoccurrence.

The NLA engineering team is also preparing to deploy a new database architecture which will be more resilient and scalable. This should also have the benefit of preventing such an incident from occurring.

NLA Service Operations Management

Notifications will be sent before the work begins and as soon as it is complete.

We apologise for any inconvenience this may cause.

The NLA apologises for any impact this may have had on PCA eClips production workflow. The issue has now been resolved and all emails are being routed appropriately.

At approximately 9:20 yesterday an incident occurred which impacted eClips service delivery. The incident was resolved at 10:15. During this incident, clients ability to view eClips content was impacted as the service was intermittently unavailable.

Cause:

The root cause of the incident is still under investigation by NLA engineers, however indications show that when attempting to serve a high number of multi-object requests containing more than the permitted 100 objects, the eClips web application spawned unnecessary additional connections to the eClips database which in turn impacted performance. Yesterday, we experienced a higher than normal number of large multi-object requests.

Solution:

Once the cause of the incident was understood by NLA engineers the service was restarted and returned to normal operation immediately. NLA engineers are now putting in place further monitoring which will provide an earlier warning of the potential reoccurrence of this type of incident. The NLA is also investigating the eClips core code related to this aspect of the service with the aim of discovering the root cause and redeveloping it to prevent reoccurrence.

Separately, the NLA engineering team is preparing to deploy a new database architecture which will be more resilient and scalable. This should also have the benefit of preventing such an incident from occurring.

Further notification will be sent when the work is complete.

We apologise for any inconvenience this may cause.

These failures follow a lengthy period of successful delivery and we are working hard with the publishers in question to ensure normal service is resumed.

The NLA Service Operations team would like to apologise for the inconvenience caused and hopes that the established communication processes have ensured clients are notified appropriately.

If you have any questions or concerns please send a mail to the NLA Service Operations Management Team.

Some content was loaded into the eClips database and distributed to clients later than usual on Tuesday 27^th May 2008.

Cause:

A fault with one of the external Internet links, from an eClips production site in India, resulted in corrupt files being transmitted to London. These corrupt files were not loaded into the eClips database at the first attempt.

Solution:

The problematic link was shut down and traffic from the affected production site was distributed over the remaining external links. The corrupt files were then retransmitted and any outstanding content was loaded and distributed by 5:30am.

Further investigation into the exact nature of the fault is ongoing. The implementation of additional monitoring processes will minimise the time required to react if a similar event should occur in future.

Problem:

The eClips Service has recently experienced two service incidents which have impacted the performance of the service.  The first incident happened Friday 16th and the second occurred Monday 19th.  Both incidents occurred during the early morning which is the peak time for usage of the eClips service. 

Cause: