Avoiding the move of a monolithic database into the cloud

Music labels will send to 7digital not just the audio recordings, but also the data pertaining to the audio, such as the artwork, track listings, performing Artists, release dates, prices, and specific rights to stream or download the music in various territories around the world. Approximately 250,000 tracks per week are received by 7digital and added to its catalogue, which is stored in a database. This process is called ingestion.

At the outset of the project there was a single database that stores catalogue and sales data which is used for multiple, unrelated purposes. This database is written to with new albums sent to 7digital from music labels, along with licensing rules governing who can access the music.

Slow queries cause the web applications which use the database to time out and fail which returns errors to the end users. Changing the database structure would help to resolve the errors and failures, however they would necessitate re-writing nearly every other web application that 7digital own, since the web applications are tightly coupled to the database. The database is very large and uses proprietary, licensed technology that cannot be easily moved to data centres around the world.

By separating the catalogue data from other data in our system, it would become possible to, not only write a much more efficient database schema which failed less often, but to also move this database into a cloud provider’s platform. With a database in cloud we could build part of 7digital’s Web API platform in the same cloud provider’s platform and therefore deploy our platform nearer to our customers in Asia.

Creating a separate catalogue database in London and moving the applications to AWS in Asia might help solve the problems with concurrent reads and writes to the database, but it would not help reduce the latency experienced by customers in Asia.

The key issue becomes one of figuring out how to transport the relevant catalogue data from the ingestion process in London and send this data into an AWS region.

Moving the ingestion process into AWS is far larger piece of work and would not yield produce performance improvements for customers in Asia by itself, so that it was decided not to move it out of the London datacentre in 2015.

Failures of the project

During 2015 we did not complete the final link in the chain, the application which could read messages from the Kafka service and persist their contents into the AWS database. We could not deploy a full fledged version of the London based Gateway API service, as it was too complex. Instead we made a naive implementation of this Gateway using a technology called nginx.

The Catalogue Persister service eventually was abandoned. An instance of Kafka was built in late 2015 and the Catalogue consumer was due to be started early 2016.