tus 1.0 - Changing the future of file uploading

Published on November 16, 2015 by Kevin van Zonneveld

As time progresses, we share ever larger media files from our phones and desktops. More than often, however, complications arise during this process. Whether it is through servers misbehaving or mobile users switching to a WiFi connection, the outcome is the same: ‘upload interrupted’.

This is by itself a negative user experience, but it becomes even worse when it happens in the middle of a 2GB upload on a slow connection. And of course, the longer an upload takes, the more exposed it is to poor connections. A failed upload will then have to be retried from the start, if the user even bothers with it at all.

With media files growing larger and networks remaining fragile, it is clear that we need a better solution to handle uploading.

Resumable Uploads

Even though certain decent network libraries already implement retries, these do not give the option to resume at the point where the upload was previously interrupted and then only transfer the remaining bytes. Furthermore, these network libraries often require additional user input in order to retry failed uploads. If retries were initiated behind the scenes, the user might not even notice he had an interrupted connection, as the total duration of the upload would barely be impacted.

There are also many areas in the world where connections are quite poor. In such places, restarting an upload from the beginning usually means that the user only gets to transmit the first 20% of his file, before the connection breaks again. Thus, resuming an upload exactly where it left off is the only way a large file will ever be uploaded in these cases.

From this, it is apparent that we need resumable uploads and therefore we are proud to present tus, the protocol that aims to offer solutions to all of the problems listed above.

In addition to this, tus has redesigned the process of uploading in such a way that multiple file parts can be sent simultaneously. Using tus not only makes uploading a lot more reliable, but also much faster.

Haven’t other companies already implemented this?

Yes, but we see this as another part of the problem. While there are many implementations that offer resumable uploads, they all solve it in a different way. They are either bound to one language or use case. They are also not as thorough or interoperable as we would like, because they all speak different dialects. Some implementations are supported by others, but that is as far as compatibility goes.

This is not remotely how an open web should work.

Dropbox, Vimeo, Google and AWS all have their own API’s for uploading chunks. Some companies describe their API, but make choices that aren’t suitable for general use and they certainly don’t offer a platform to contribute improvements. If we want to support anything beyond their use case, we are on our own again. This further increases fragmentation and leaves resumable file uploads as a luxury, only to be enjoyed by a few large companies, while it should be the standard method that benefits everyone.

tus

tus is a protocol that is fully community-owned. It has been developed using nothing but GitHub and Markdown, and was fueled by a profound motivation - shared by a few passionate high profile developers - to solve this problem once and for all.

Over the course of two years, we have received suggestions from people working at Vimeo, GitHub, Google, and from the author of ØMQ, as well as from the co-authors of HTTP/1.1 and Node.js.

It has not always been easy to find consensus and at times we hit deadlocks. In the past few months, however, we have made considerable progress and we are now excited to announce a finalized version 1.0. tus is ready for the world to use.

How does it work?

Let’s take a quick look at an example of a simple communication between a client and a server speaking tus 1.0:

# Client:
> POST /files HTTP/1.1
> Host: tus.example.org
> Tus-Resumable: 1.0.0
> Content-Length: 0
> Upload-Length: 100
> Upload-Metadata: filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg==

# Server:
< HTTP/1.1 201 Created
< Location: http://tus.example.org/files/24e533e02ec3bc40c387f1a0e460e216
< Tus-Resumable: 1.0.0

# Client:
> PATCH /files/24e533e02ec3bc40c387f1a0e460e216 HTTP/1.1
> Host: tus.example.org
< Tus-Resumable: 1.0.0
> Content-Type: application/offset+octet-stream
> Content-Length: 30
> Upload-Offset: 0
>
> [first 30 bytes]

# Server:
< HTTP/1.1 204 No Content
< Tus-Resumable: 1.0.0

Since tus is a layer on top of HTTP, it is easy to reason about, inspect, extend and deploy in existing projects as well as infra.

A streamlined core with optional extensions

The core of the protocol is lightweight, yet well documented and it defines the way in which communication should take place. In addition to this, tus offers extensions such as Checksum, Expiration and Concatenation, which support different use cases and are opt-in.

To highlight one of the protocol’s extensions: Concatenation allows you to split a 100GB file in 20 parts of 5GB, upload them in parallel, and uses the tus server to stitch them together again. This makes uploads much faster by utilizing the available bandwidth more efficiently. An extra advantage of this method is that individual chunks are considered regular tus uploads, which benefit from all of tus’ features (retries, checksums, etc.).

Implementations

Our aim is to support all major languages and platforms. Alongside the protocol we also develop many implementations, so that developers can have drop-in self-hosted resumable file uploads at zero cost.

We currently support the following platforms:

Additionally, there are many community provided implementations in languages such as: Ruby, Python and PHP.

Quality implementations of the protocol licensed MIT might be adopted in the tus organization on GitHub, making you an official contributor to the project.

Releasing version 1.0

Today we are proud to announce that we have launched version 1.0 on GitHub by merging the 1.0 pull request.

We consider the protocol to be stable and ready for production!

The full protocol is published online here.

Early Adopters

Early adopters Vimeo and Transloadit have already announced they will use tus 1.0 as their main protocol for uploading.

tus is initiated and funded by Transloadit, but it is community owned and this will never change.

We encourage anybody who deals with file uploads to have have a look at tus and to consider using it for your next release.

Special Thanks

It has taken us the better part of three years and we have had to overcome many obstacles along the way. It is in great part thanks to these people that tus is able to push out a stable release today. A special word of thanks goes out to the talented people that have been instrumental in getting us to where we are today: Felix, Naren and Marius. We couldn’t have done tus without you!

Discuss on Hacker News