kDrive: the technological challenge, as described by its developers

Designing a secure and sovereign collaborative cloud is one of Infomaniak’s most audacious projects. To give birth to kDrive, we committed our teams to three years of research and development and invested eight million francs. The result: a Swiss-made cloud platform devoted to productivity and respectful of privacy.

How was an unlisted, 100% independent company like Infomaniak able to carry out such a vast and complex project? We brought our engineers together to tell us about the development of kDrive and share the details of its unique design.

An independent and strategic technology for Infomaniak

kDrive is a “safe” but, just as crucially, it’s also a product of major importance for the future of Infomaniak. It shows the direction of travel of our offers in our effort to meet the needs of SMEs and the general public. This evolution is all about making our know-how and our cloud infrastructures available to tools that are easy to adopt and use.

If kDrive is so important, it’s because it comes in response to intense, unrelenting demand from our users and the market for a collaborative storage solution that’s privacy-friendly and managed entirely on the European continent. This demand was already apparent in 2012 with the launch of WorkSpace, and it’s been growing steadily since the launch of Swiss Backup, our cloud backup solution. So, kDrive finds us eagerly awaited, and we don’t do things in half measures!

kDrive’s foundations are based on exacting and proven standards

kDrive’s development is the result of close engagement with our customers and partners in Switzerland and Europe. Our engineers set the scene right from the start:

Security, confidentiality, availability

That was the first stage of the rocket. From the outset, the developers conceived kDrive to be a failproof product. Our clients would be storing all their files there, both work-related and private. Data loss or unavailability was not an option then, and is not now. Confidentiality requirements meant implementing systematic encryption and a compartmentalised design. Our engineers therefore had to come up with a robust, scalable and fault-tolerant architecture.

Collaboration, simplicity, evolution

kDrive also had to meet the evolving needs of our users. This, the second stage of the rocket, includes all the thinking and technological choices, ergonomics and productivity tools. kDrive was therefore designed from the outset to be a collaborative and evolutionary platform.

An independent technology, developed and hosted in the heart of Europe

How did we manage to develop a collaborative storage solution that meets all these requirements in such a short space of time?

We took our time

It’s the ultimate goal for a Swiss company, but it’s equally the first key to the project’s success:

“As quickly as possible, but as slowly as necessary”.

Our developers took the time to become intimately acquainted with the strengths and weaknesses of the object storage technology envisaged for kDrive, in real use situations and in the long term. We looked for technological mastery above all.

We learned lessons from our own products

If we’ve been able to develop such a robust and evolutionary service, it’s by using our past experience. Our developers tested the reliability and leveraged the feedback and monitoring of services such as SwissTransfer and Swiss Backup to define the role of each technology deployed in kDrive. We concluded that we needed to develop an intermediate software layer between the customer data and our object storage system.

Experience gained from the first version of WorkSpace, which used to be updated simultaneously for all customers and on a large scale, supported the choice of a much more flexible silo architecture for kDrive.

kDrive’s in-house development

kDrive is a team effort that mobilises a large number of employees across all of Infomaniak’s departments. The initial development phase occupied 20 engineers for more than 12 months. Our developers, systems engineers and production managers explain the main technical aspects of kDrive’s development:

Technologies deployed in the project

Most of the kDrive code is written in php. The databases use MariaDB (a fork of MySQL), in which we’ve had expertise for many years.

The object storage is based on OpenStack SWIFT, an open source technology that also powers SwissTransfer and Swiss Backup.

Middleware developed by our team using Kubernetes governs most of the business logic. It acts as a bridge between data management and the customer’s kDrive application.

The programming languages GO and Python are also used to provide previews of file thumbnails and other peripheral tools. Each technology is used for what it does best and nothing else.

Ensuring data availability, confidentiality and integrity

The kDrive specifications called on our engineers to converge various requirements to achieve an architecture that’s both robust and flexible. That’s the challenge they accepted, notably by developing this middleware, the heart of kDrive.

kDrive is based on strong compartmentalisation. Each drive contains its own database that provides a mapping of customer files to link them to their storage location in OpenStack SWIFT. This linking work is performed by the middleware and allows all the instances (databases, storage, logs, etc.) to remain totally independent of each other. The databases containing the identifiers are separate from those of the file structures, specific to each drive and from the general Infomaniak database.

The advantage of Kubernetes is also its high scalability. It means that all kDrive services can dynamically and automatically resize themselves. Finally, each customer’s content and content structure are systematically encrypted and permanently backed up to at least three locations in several data centres under Infomaniak’s control.

Here’s a simplified illustration of the general kDrive architecture:

Infomaniak’s middleware (illustration: kDrive heart) allows us to make the most of compartmentalisation. It also supplies the intelligence and performance that object storage does not provide natively (managing huge volumes of requests, finding files, creating previews, etc.).

Secure architecture that limits security risks and cyber attacks

kDrive’s 100% independent technology limits its vulnerability compared to other systems. The files are very rarely executed and the silo architecture greatly reduces the potential scope for malware or viruses. In practice, running a virus in kDrive does not lead to anything and is therefore pointless. To our knowledge, the compartmentalisation of the data, its architecture and access render unauthorised exploitation attempts impossible.

kDrive: from design to production launch

The first thing that strikes you when you talk to our engineers is that there’s no boundary between developers and production. Mastering every facet of kDrive is a team effort:

Ensure a robust infrastructure

From machines to teams, reliability criteria dictate the entire kDrive structure and organisation. In addition to load tests, anti-DDOS protection measures (internal, external) and automated maintenance tasks, all databases are permanently backed up. Some backup replicas are also time offset in order to be able, if necessary, to make up for any human error that may have managed to pass the validation and release process.

Enable high scalability

The technologies used (SWIFT and Kubernetes) provide for an indefinite growth of services. But this reasoning also applies in the opposite direction. This is one of the technical strengths of the project. In the event of a malfunction, kDrive automatically redistributes the workload and is able to run with a very small number of machines. In theory, even just one would suffice. Finally, thanks to monitoring, kDrive is able to self-repair in the event of an error so as not to impact the services.

Rapidly integrate new features

KDrive evolutions are added progressively according to the Canary Release model. Thanks to the Next platform, we test each update in real-world production conditions using a panel of users before progressively rolling it out to all users. Our teams systematically and manually cross-check every update of the code, line by line.

A laboratory to improve the performance and efficiency of our technologies

The more responsive the system, the fewer resources the infrastructure consumes. This is the central principle of kDrive’s ecodesign. Developers take the time to write quality code rather than splashing out unnecessarily on the most powerful machines on the market. We therefore take to care to optimise all processes, including background tasks such as thumbnail generation and cancelled mailings. The developers have deployed numerous measuring systems to capture and analyse the metrics. This provides us with a balanced view of the resources and allows an optimal load level for the environment to be maintained.

We favour a commodity hardware approach and a high degree of modularity by using the same types of servers across all services. It means that every machine can be assigned to a new task dynamically. We thereby keep the purchase, and therefore the construction, of new machines (a major source of CO2 emissions) to a minimum. We choose reused machines driven by our virtualisation technology. These machines are a perfect fit with kDrive’s highly fault tolerant architecture, while optimising resources to the maximum.

Remove all technical complexity for the end user

To make the front end as intuitive (UX, UI) as possible, our designers analysed usage behaviours in existing services. They designed kDrive’s ergonomics with instinctive and recognised visual cues to facilitate ease of use. This philosophy of simplification is the basic principle behind all our current and future developments.

kDrive on mobile had to be available from the outset. Our team chose a Nextcloud fork while waiting for the release of our own application. The latter will offer unbeatable fluidity with reduced loading times and will support the addition of new features exclusive to kDrive.

kDrive is a platform for the future

Our developers are moving mountains to make kDrive an open and evolutionary platform. This implies favouring open source technologies, but also making kDrive compatible with protocols such as Webdav. A public API will soon be added to the system that will allow the developers to create synergies between kDrive and their applications.

kDrive is constantly improving

Mobile application

kDrive is evolving fast, and the Nextcloud database is no longer sufficient to support forthcoming innovations and features. Our developers are working on a new 100% in-house kDrive application. It will offer all the functionalities of the online interface plus enhanced scanning, Office document (Word, Excel and PowerPoint) editing, improved sharing, and tablet optimisation.

Zero-knowledge safes

KDrive users will be able to issue their own encryption keys for maximum security without compromise. Integration of these safes will be available with the desktop and mobile applications. This end-to-end encryption will be asymmetric (with one private and one public key). It will also be possible to share and collaborate on safes.

On-demand file synchronisation (smart sync)

It’s becoming increasingly difficult to store all one’s data on laptops, where storage space is often insufficient. Intelligent synchronisation is about to be integrated into desktop applications to facilitate on-demand file downloads. To simplify the process, we’ll be incorporating a wizard to choose which files should be stored locally.

We’re strengthening interaction between our products

As we announced in the 2021 roadmap, we’ll be strengthening links between our products. The photo dimension will also gradually evolve to offer galleries and the ability to import images from Google (the Google Photos free service is soon coming to an end).