Fork me on GitHub

S3 as a Storage Back-End

With their Simple Storage System (S3), Amazon Web Services has built one of the major providers of cloud storage for applications ranging from small side projects to enterprise systems. Since the introduction of flexible storage back-ends for the official tusd server, an integration with S3 has been a much desired feature by our users. We are happy to announce that we are now able to deliver on this request. During the time it took to create this, we had to deal with various peculiarities of Amazon’s service and were able to gain a lot of experience. In this post, we want to focus on the downsides of building a tus server on top of S3 and share some of our recently acquired knowledge with you.

Immutable Objects

We, as the designers of tus, have to admit that the protocol uses a data model which is mostly incompatible with AWS S3. In order to understand this sentence, we need to make a small comparison: In tus, when you want to move a file to a remote location, you first create a new upload resource without pushing any of the file’s data to the server. It is even possible to make this operation before you know the length or size of the entire object that you want to transfer. After this step, you are free to upload the data in chunks of any size. The first chunk could be a few MBs, followed by one that is just 100 bytes and a final upload then contains the remaining GB. While this freedom introduces the need for a flexible server implementation, which is capable of handling chunks of any size, it also lays the foundation for tus’ core feature: resumability of an upload at any given time.

S3, however, does not offer this flexibility: once an object - the length of which must also be known beforehand – has been uploaded to a specific location, you are unable to modify its content without transmitting the entire new file. It is simply not possible to add a chunk to an existing object without having to perform additional operations. It may sound, then, as if the main requirement of the tus protocol is not met by Amazon’s service, but that is not the case. You are certainly able to build a proper server implementation for tus, as long as you are willing to accept certain restrictions. This can, for instance, be seen in the S3 storage back-end for the tusd server.

Multipart Uploads

Amazon has been aware of this limitation and therefore supports an alternative approach called Multipart Uploads:

Multipart upload allows you to upload a single object as a set of parts. Each part is a contiguous portion of the object’s data. You can upload these object parts independently and in any order. If transmission of any part fails, you can retransmit that part without affecting other parts. After all parts of your object are uploaded, Amazon S3 assembles these parts and creates the object.

This approach is very similar to tus’ data model described above and it provides a solid foundation to build an implementation upon. However, development would not be called development if it were as easy as mapping a tus upload one-to-one to a multipart upload. The issue is that Amazon sets certain restrictions, the most notable of which is that the minimum size of a single part is limited to 5MB. The only exception to this rule is the last part, which can be smaller. It should be mentioned here that S3 will not complain when you upload a part that is smaller than 5MB, but only when you attempt to finish the multipart upload that does the actual assembly (it will then present you with the EntityTooSmall error message).

The solution - if you want to call it one - is to only upload parts to S3 that match or exceed the minimum size. The storage back-end for tusd achieves this by writing the body of an incoming PATCH request to a temporary file. Once the upload from the user to our tus server reaches a size of 5MB, we are sure that we have enough data for a single part on S3 and can start moving this chunk to Amazon’s service. If the tus server does not receive enough data - ensuring, of course, that it is not the last part, which is allowed to be smaller - it will simply drop the temporarily stored file and require the user to attempt a resume, in the hope that the connection is then more reliable. A look at the code that powers the implementation described above, may help to understand this.

Regrettably, this approach comes with one noticeable downside for the end user: if an upload or resume is interrupted before at least 5MB has reached the tus server, the sent data will be lost and must be retransmitted. Some may ask why we don’t simply locally store the received chunk of data on the tus server, wait for the user to resume the upload and then, once we have enough data, push it to S3. This is certainly a good question, but that solution only works when you can ensure that the resumed request reaches the same tus server as the previously interrupted request. If you are running more than a single tus instance, a special routing mechanism may be required to achieve this. Another option would be to use a second storage medium, such as a shared volume, but that would also need to handle concurrent access correctly.

If this workaround is not acceptable for your application because you do not want to limit the chunks to 5MB, you may want to reconsider using AWS S3 as a storage back-end, since it simply does not offer the required functionality. However, if you are using an alternative back-end that just exposes an S3-compatible API, it may offer a configuration option to change the minimum size of a single part. Riak CS (Cloud Storage), for example, accepts the enforce_multipart_part_size flag, which can entirely remove this constraint.

S3’s eventual consistency model

Amazon’s engineers wanted to provide a highly available service and were therefore unable to offer guaranteed consistency for every operation. They nevertheless do not hide this important property of S3 and instead describe it extensively in their documentation. The most interesting sentence for us, the implementers of tus servers, is the following one:

Amazon S3 does not currently support object locking. If two PUT requests are simultaneously made to the same key, the request with the latest time stamp wins. If this is an issue, you will need to build an object-locking mechanism into your application.

Locking uploads is an important mechanism to prevent data corruption and tus is not immune to it. Imagine a situation where two clients attempt to resume the same upload at the same offset. If the server simply accepts both requests, the latter one may override the data from the first request, resulting in file corruption or loss. In order to prevent this issue, the server needs to acquire an exclusive lock, e.g. a simple semaphore, on the upload resource before it starts transferring the data and then only release that lock once the data is saved. In this scenario, the server will reject the second request from the client, because a lock cannot be obtained when one is already held.

Implementing a proper locking mechanism is, however, difficult and gets even more complicated if you are working in an environment with multiple distributed servers. In this case, a service should be used that manages distributed locks while at the same time guaranteeing consistency. For example, proven technologies include ZooKeeper or Consul, but not AWS S3 as it does not offer absolute consistency. Since they do promise “read-after-write consistency for PUTS of new objects in your S3 bucket [but only] eventual consistency for overwrite PUTS and DELETES”, this cannot be used to build a distributed lock upon. Therefore, you are recommended to use a third-party system for doing so.

Another option for preventing concurrent uploading is to put the responsibility on the client’s side by saying it is their task to prevent multiple accesses to the same upload resource. While this may work, this approach is not able to guarantee corruption-free uploads since a client still might send two or even more requests at the same time by accident and the server does not prevent that.

Finishing Thoughts

With S3, engineers have an incredibly useful tool for storing data with high availability and scalability. However, it does not present the perfect storage back-end for the tus protocol and requires some workarounds. In the future, we will have a look at other storage system and cloud providers.

tus 1.0 - Changing the future of file uploading

As time progresses, we share ever larger media files from our phones and desktops. More than often, however, complications arise during this process. Whether it is through servers misbehaving or mobile users switching to a WiFi connection, the outcome is the same: ‘upload interrupted’.

This is by itself a negative user experience, but it becomes even worse when it happens in the middle of a 2GB upload on a slow connection. And of course, the longer an upload takes, the more exposed it is to poor connections. A failed upload will then have to be retried from the start, if the user even bothers with it at all.

With media files growing larger and networks remaining fragile, it is clear that we need a better solution to handle uploading.

Resumable Uploads

Even though certain decent network libraries already implement retries, these do not give the option to resume at the point where the upload was previously interrupted and then only transfer the remaining bytes. Furthermore, these network libraries often require additional user input in order to retry failed uploads. If retries were initiated behind the scenes, the user might not even notice he had an interrupted connection, as the total duration of the upload would barely be impacted.

There are also many areas in the world where connections are quite poor. In such places, restarting an upload from the beginning usually means that the user only gets to transmit the first 20% of his file, before the connection breaks again. Thus, resuming an upload exactly where it left off is the only way a large file will ever be uploaded in these cases.

From this, it is apparent that we need resumable uploads and therefore we are proud to present tus, the protocol that aims to offer solutions to all of the problems listed above.

In addition to this, tus has redesigned the process of uploading in such a way that multiple file parts can be sent simultaneously. Using tus not only makes uploading a lot more reliable, but also much faster.

Haven’t other companies already implemented this?

Yes, but we see this as another part of the problem. While there are many implementations that offer resumable uploads, they all solve it in a different way. They are either bound to one language or use case. They are also not as thorough or interoperable as we would like, because they all speak different dialects. Some implementations are supported by others, but that is as far as compatibility goes.

This is not remotely how an open web should work.

Dropbox, Vimeo, Google and AWS all have their own API’s for uploading chunks. Some companies describe their API, but make choices that aren’t suitable for general use and they certainly don’t offer a platform to contribute improvements. If we want to support anything beyond their use case, we are on our own again. This further increases fragmentation and leaves resumable file uploads as a luxury, only to be enjoyed by a few large companies, while it should be the standard method that benefits everyone.

tus

tus is a protocol that is fully community-owned. It has been developed using nothing but GitHub and Markdown, and was fueled by a profound motivation - shared by a few passionate high profile developers - to solve this problem once and for all.

Over the course of two years, we have received suggestions from people working at Vimeo, GitHub, Google, and from the author of ØMQ, as well as from the co-authors of HTTP/1.1 and Node.js.

It has not always been easy to find consensus and at times we hit deadlocks. In the past few months, however, we have made considerable progress and we are now excited to announce a finalized version 1.0. tus is ready for the world to use.

How does it work?

Let’s take a quick look at an example of a simple communication between a client and a server speaking tus 1.0:

# Client:
> POST /files HTTP/1.1
> Host: tus.example.org
> Tus-Resumable: 1.0.0
> Content-Length: 0
> Upload-Length: 100
> Upload-Metadata: filename d29ybGRfZG9taW5hdGlvbl9wbGFuLnBkZg==

# Server:
< HTTP/1.1 201 Created
< Location: http://tus.example.org/files/24e533e02ec3bc40c387f1a0e460e216
< Tus-Resumable: 1.0.0

# Client:
> PATCH /files/24e533e02ec3bc40c387f1a0e460e216 HTTP/1.1
> Host: tus.example.org
< Tus-Resumable: 1.0.0
> Content-Type: application/offset+octet-stream
> Content-Length: 30
> Upload-Offset: 0
>
> [first 30 bytes]

# Server:
< HTTP/1.1 204 No Content
< Tus-Resumable: 1.0.0

Since tus is a layer on top of HTTP, it is easy to reason about, inspect, extend and deploy in existing projects as well as infra.

A streamlined core with optional extensions

The core of the protocol is lightweight, yet well documented and it defines the way in which communication should take place. In addition to this, tus offers extensions such as Checksum, Expiration and Concatenation, which support different use cases and are opt-in.

To highlight one of the protocol’s extensions: Concatenation allows you to split a 100GB file in 20 parts of 5GB, upload them in parallel, and uses the tus server to stitch them together again. This makes uploads much faster by utilizing the available bandwidth more efficiently. An extra advantage of this method is that individual chunks are considered regular tus uploads, which benefit from all of tus’ features (retries, checksums, etc.).

Implementations

Our aim is to support all major languages and platforms. Alongside the protocol we also develop many implementations, so that developers can have drop-in self-hosted resumable file uploads at zero cost.

We currently support the following platforms:

Additionally, there are many community provided implementations in languages such as: Ruby, Python and PHP.

Quality implementations of the protocol licensed MIT might be adopted in the tus organization on GitHub, making you an official contributor to the project.

Releasing version 1.0

Today we are proud to announce that we have launched version 1.0 on GitHub by merging the 1.0 pull request.

We consider the protocol to be stable and ready for production!

The full protocol is published online here.

Early Adopters

Early adopters Vimeo and Transloadit have already announced they will use tus 1.0 as their main protocol for uploading.

tus is initiated and funded by Transloadit, but it is community owned and this will never change.

We encourage anybody who deals with file uploads to have have a look at tus and to consider using it for your next release.

Special Thanks

It has taken us the better part of three years and we have had to overcome many obstacles along the way. It is in great part thanks to these people that tus is able to push out a stable release today. A special word of thanks goes out to the talented people that have been instrumental in getting us to where we are today: Felix, Naren and Marius. We couldn’t have done tus without you!

Discuss on Hacker News

Project Status

Here’s a quick update on the status of the project.

1.0

We’re finalizing 1.0. Our project lead Marius Kleidl has been taking care of integrating a large batch of improvements that were made by Vimeo’s Naren Venkataraman and this team.

Things still left to do:

We’re waiting for a final round of feedback to make sure every last bit is taken care of before cementing everything into our first stable release.

Exposure & Swag

To make sure every developer and interested party had their change to chime in, we’re trying to get some extra exposure for the project. We’ll be covered in the upcoming Changelog and Transloadit is sponsoring conferences to hand out swag and raise tus awareness.

Swag is also for sale at the new shop.tus.io. The shop currently only features two, pretty expensive, items and we’re working with Printful to improve on that. Know that any profit flows back to the project and our books are open to anyone who wants to verify.

Implementations

While 1.0 could potentially still be changed, we don’t expect major changes so our official implementations have already been made 1.0 compatible.

All official projects have been updated to the 1.0 branch already for which a big thank you goes out to Mark R. Masterson and Marius who upgraded all of our implementations.

Marius also added Android and standalone Java implementations as official tus projects, and replaced our jQuery implementation with a standalone tus-js-client.

If you’re interested in building & maintaining new 1.0 implementations, becoming a member of tus core, leave a note here:

Design

The whole site has been given a new layout by Fahad Ibnay Heylaal and a completely new logo was designed by Milan Vuckovic.

We’re currently working on improving other:

Demo

Since the first publication of the tus protocol, our website featured a demo page allowing users to see a tus in action, interactively. In the past this service had some issues with reliability and browser-support. Because of this past, we updated the entire stack used by the demo.

It now uses our newly created tus-js-client and we’ve updated the demo page to feature all the browsers & platforms the client has been tested to work on. The demo page uploads files to a tusd 1.0 server written in Go, that we deploy via the newly created infra-tusd repository.

Infra

Still a work in progress, infra-tusd uses a powerful combination of Ansible and Terraform to spin up fully functioning tus servers with a single command. Every bit (except for the AWS & SSH keys) has been added to the repository and is publicly available.

Please each out if you’d like to help us:

Adoption

Finally, a few big companies have taken an interest in tus. tus will always remain open source and community owned, but we’re excited that we’re on the path to realizing our mission to change how the world does file uploading.

We’ll have more announcements on this soon!

If you (plan to) use tus in production, please comment on this issue and get your company listed on the tus.io website:

Protocol v1.0.0 Prerelease

More than a year ago the last release, 0.2.2 was published. Now the final 1.0 release is just around the corner introducing breaking changes and a lot of new features.

The major changes towards the core include the addition of the TUS-Resumable, TUS-Max-Size, TUS-Extension and TUS-Version headers while making the first one mandatory. All these headers must be returned in the new OPTIONS request in order to enable protocol discovery. In addition the Offset header must not be greater or lower than the current offset for the file.

The biggest changes were made by introducing the Upload-Expiration, Checksum, Stream, Retries, Termination, Merge and Metadata extensions.

After all of this work the protocol is now considered stable and ready for use in production environments. Speaking of implementations the official tus-jquery-client and tusd repositories are currently being updated to support the 1.0 release.

The final 1.0 Release will be published by merging the according pull request on GitHub once these changes are done. Furthermore, last feedback may be submitted there to adjust minor things.

Protocol v0.2.2

This is a minor protocol release:

All patches can be seen here.

Protocol v0.2.1

This is a minor protocol release:

All patches can be seen here.

Protocol v0.2

After releasing our first draft a few weeks ago, we received an incredible amount of feedback and suggestions. Based on this feedback as well as discussing the problems with the IETF HTTPbis Working Group, we identified a few key issues with v0.1 of the protocol:

  • PUT requests are not appropiate for transfering partial resources
  • The Content-Range and Range headers are not meant for resuming an interrupted resource transfer.

After lots of careful thinking, we came up with a new approach that uses:

  • PATCH instead of PUT
  • A new Offset header used by HEAD responses and PATCH requests alike
  • A Final-Length header to provide the final file size to the server

We also split the protocol into a core protocol which takes care of resumability, and nothing else, as well as optional protocol extensions.

The result of this has just been published as v0.2 can be seen on the protocol page. Also included is a new FAQ section which will expanded over time.

We feel that the overall result is a drastic simplification of the problem down to its essence, and we encourage interested developers to implement prototypes.

Our next step is upgrading tusd, the jquery client and the ios client to the new protocol version. Once the protocol has reached a little more maturity, we are also thinking about providing an executable protocol verification tool for implementers.

A protocol for resumable file uploads

tl;dr: We are happy to announce version 0.1 of the tus resumable upload protocol and are interested in your feedback!

With mobile devices becoming the dominant source of user generated media files, reliable file uploading through unreliable mobile networks has become an important issue for anybody interested in content acquisition.

Reliability here means the ability to detect network errors, and resuming an upload without having to start from the beginning. In many scenarios this can mean the difference between a file reaching your application, or the user giving up in frustration.

Ideally, this should be a trivial feature to add. In reality however, there is quite a lack of solutions in this space. Sure, there are a few JavaScript libraries that claim to support resumable uploading, but in reality you will end up spending a lot of time coming up with your own API for it, or implementing a poorly specified one specific to a library. This is incredibly frustrating, especially if you are planning to support this feature on multiple platforms such as HTML5, iOS and Android.

Now, if you’re a big company like Google, you may sit down and create such a protocol for your needs. And in fact, Google has been working on a such a protocol since 2010, for the now defunct Google Gears. The latest incarnation of this are two incompatible protocols for Google Drive and YouTube. But unfortunately both of these protocols rely on a non-standard http status code (308 Resume Incomplete), and are far from being generic enough for general adoption.

This means that smaller companies are currently doomed to invent, implement and maintain their own incompatible protocols and solutions for something that should be a trivial component of a modern application.

We find this unacceptable, so the tus project is a community project that was born in order to level the playing field and make resumable file uploading easy for anybody to implement.

Today we are happy to release version 0.1 of our protocol. Interested developers are encouraged to experiment with it, and we are very interested in any feedback you may have.

Later this week we will also release some initial clients for jQuery and iOS, so make sure to follow this blog and these repositories for future updates!

Hello world

Hey everybody, we’re starting a new blog here to discuss file uploading, and the protocol we are working on. So keep an eye on this space!

Subscribe to this blog via RSS.