ECML-PKDD 2012


Hosting Open Data Platforms in Europe: Infrastructure Choices

An open data platform looks simple from the outside: a catalogue of datasets with download buttons. Behind it sit decisions about servers, storage, networks and location that decide whether the service stays fast under load, remains available when a dataset goes viral and complies with the rules governing public data in Europe.

What an open data platform asks of infrastructure

Open data traffic has an unusual shape. Most of the time it is quiet, then a news story, a research deadline or a hackathon sends a surge of requests for the same files. Datasets can be large, from small tables to multi-gigabyte geospatial archives, and applications call APIs repeatedly rather than downloading once.

Several components share the work. A catalogue, often built on open-source software, holds metadata. A storage layer holds files. An API layer serves queries, and a front end presents it all. Each can be hosted separately, and each can be scaled on its own.

Providers such as Delta.BG, a Bulgarian hosting company operating since 2009, offer building blocks for this kind of design, including cloud virtual servers, dedicated servers and colocation, from data centres in Sofia. The right mix depends on the platform's scale, the skills of the team and whether the service is run by one organization or shared among several.

What European rules expect

Public bodies in the European Union work under the Open Data Directive. The EUR-Lex summary of open data and the reuse of public sector information explains that an implementing act, Regulation (EU) 2023/138, lists specific high-value datasets and sets out the arrangements for their publication and reuse.

The Directive requires high-value datasets to be available free of charge, machine-readable, provided through APIs and, where relevant, as a bulk download. It also requires that dynamic data be made available for re-use immediately after collection, through suitable APIs and, where relevant, as a bulk download.

Those expectations have infrastructure consequences. APIs need to be stable and well documented, bulk downloads need storage and bandwidth, and real-time feeds need servers that stay up. The rules describe outcomes, and platform operators must design the hosting to deliver them.

Costs follow the same logic. Because data is meant to be free for re-use, the platform cannot easily charge heavy users for bandwidth, and operators should plan for sudden demand. Rate limits on anonymous access, authenticated API keys for heavy users, and clear terms of use help to keep the service fair, while caching and a content delivery layer reduce the load on origin servers during a spike.

Cloud VPS or dedicated servers

Choice of server type depends on the workload. Delta's documentation describes its cloud virtual servers as machines with guaranteed resources, built on a cloud platform, with full administrative access, NVMe storage, and the ability to resize. Its cloud is powered by OpenStack, and the service page lists full root access and an all-NVMe infrastructure.

Dedicated servers suit steady, heavy workloads. The company's page states that they provide guaranteed resources and high-performance hardware, and it explains that upgrading takes longer than on cloud servers because someone must physically add resources, while cloud servers can be scaled quickly.

For many platforms, the answer is a mix. Cloud servers handle the catalogue and the API, where demand fluctuates, and a dedicated machine holds large static archives. Start small, measure traffic, and grow where the data shows pressure. Block storage volumes can be attached to cloud servers as archives grow, which avoids moving to a bigger machine just for disk space.

Network, availability and attack resilience

Open platforms attract attention, and not all of it is friendly. Delta states that its cloud is protected against Layer 3 and Layer 4 DDoS attacks on a global Anycast network with 12 Tbps of network edge capacity and more than 20 points of presence, one of them in Sofia. Protection at the network level keeps floods away from the application.

Availability needs a number. The company states a 99.9 percent uptime guarantee on its dedicated servers, in a colocation facility described as Tier 3+, and network connectivity above 300 Gbps with several direct, reserved upstream links. Operators should read such statements against their own service goals.

Application-level resilience matters too. Caching, rate limits for anonymous API use, health checks, and backups protect against ordinary failures. A good design assumes that one component will fail and keeps the rest running. Test restores regularly. Document the recovery steps so that anyone on the team can follow them under pressure.

Location, sustainability and operations

Location matters for law and for latency. Hosting inside the EU keeps data under European jurisdiction, which many public bodies require, and a server in Sofia serves users in southeastern Europe quickly. Platforms with users across the continent can add a content delivery layer for large files.

Delta states that its public cloud is built in Equinix data centres that buy certified green energy from renewable sources, with its Sofia region listed as eu-sof-1. For public bodies with sustainability commitments, that is a relevant point to verify when choosing a host.

Day-to-day operation also counts. Look for a documented API, a marketplace of ready-made application stacks, monitoring, snapshots and clear support channels. The best infrastructure for open data is the kind its operators rarely have to think about. Ask for a trial period, run a realistic load test before launch and keep an exit plan, so that data and configuration can move to another provider if needs change.