tus is an open source project by Transloadit

FAQ

Here are some Frequently Asked Questions and their answers. This document is still a work-in-progress and is being re-written as we speak. If you have any questions or topics you would like to see covered, let us know!

What is tus?

tus is a project aiming to make resumable file uploads easily usable and widely available. The most interesting parts of this project is the protocol specification and the many freely available client and server implementations.

When we say “resumable file uploads”, we refer to the ability that uploads can be interupted at any time and afterwards be resumed from the state where the failure began. This interruption can be accidentially (e.g. a connection drop or a server crash) or voluntarily if the user decides to pause the upload. In traditional uploading implementations your progress would be lost in such situation but tus enables you to recover from these interruptions and continues were the upload was stopped.

On a more technical side, the tus protocol is built upon HTTP/HTTPS to make it available on every platform including browsers, desktop software and mobile application. Furthermore, it also allows us to build upon the massive network of software and best practices that HTTP provides.

When should I use tus?

If you are unsure if tus is a good fit for your use case, here is a list of criteria. If one of these points applies to you, it is very likely that you could benefit from tus:

  • You operate on partly unreliable networks where connection can get easily dropped or a connection may not be available at all for some time, e.g. when using mobile data.
  • You handle large files and want to avoid having to reupload parts of it just because the upload got interrupted (Note: “large” is relative word. A 10MB file can be large if you have an uplink speed of 100KB/s).
  • You want to provide your users with the ability to pause an upload and resume it later (maybe even after a few days).
  • You do not want to rely on proprietary upload solution but instead want to base your work on free and open source projects.

Having that said, there are also a few situations where you might not want to use tus. This is mainly in scenarios when you handle many very small files (e.g. a few KBs) on a slow network and were the overhead of additional HTTP requests would significantly impact the performance. If you want us to assist you in your decision-making, contact us and we are happy to assist you.

How do I install tus and get started?

You’ll need a:

  • server implementation (you can read how to get started with tusd which is the most mature option), and a:
  • client implementation (such as the bare-bones tus-js-client for browsers and Node.js, or if you need a rich UI, Uppy.

tus is a low-level building block and there’s no one right way to do deploy resumable file uploads with it. Still, to get started quickly, for the majority use case, we’d recommend tusd + Uppy. But the ecosystem is reach and you’ll know it if one of our community maintained client/server implementations in Ruby, Python, Elixir, etc, are a better fit to your platform.

Finally, if you don’t care for self-hosting tus servers, Transloadit offers a tus-based upload handling service that can export to the storage of your choice (and can optionally encode or resize files passing through). The setup steps are then to create an account and add an Uppy HTML tag to your website, or check out non-browser client integrations.

How does tus work?

This section covers the more low-level details of how tus performs an upload. If you are using one of our open-source implementations, you don’t have to worry about these details but they may still be helpful to ease debugging and troubleshooting.

A tus upload is broken down into different HTTP requests, where each one has a different purpose:

  • At first, the client sends a POST request to the server to initiate the upload. This upload creation request tells the server basic information about the upload, such as its size or additional metadata. If the server accepts this upload creation request, it will return a successfully response with the Location header set to the upload URL. The upload URL is used to unique identify and reference the newly created upload resource.
  • Once the upload has been create, the client can start to transmit the actual upload content by sending a PATCH request to the upload URL, as returned in the previous POST request. Idealy, this PATCH request should contain as much upload content is possible to reduce the upload duration. The PATCH also must contain the Upload-Offset header which tells the server at which byte-offset the server should write the uploaded data. If the PATCH request successfully transfers the entire upload content, then your upload is done!
  • If the PATCH request got interrupted or failed for another reason, the client can attempt to resume the upload. In order to resume, the client must know how much data the server has received. This information is obtained by sending a HEAD request to the upload URL and inspecting the returned Upload-Offset header. Once the client knows the upload offset, it can send another PATCH request until the upload is completely down.
  • Optionally, if the client wants to delete an upload because it won’t be needed anymore, a DELETE request can be sent to the upload URL. After this, the upload can be cleaned up by the server and resuming the upload is not possible anymore.

How do I integrate tus into my application?

If you want to add tus to your application, you need a server and client component. For a quick start, we recommend to have a look at our list of open-source implementations. The exact setup steps vary for each implementation but the basic scheme is following:

The tus server is usually deployed alongside your main application server in its own process. A proxy is then used to route the requests to either the tus server or your application server. The tus server usually stores the uploads on disk but some servers (such as our reference implementation tusd) are also capable of storing the data directly on different cloud providers, such as AWS S3 or Google Cloud Storage.

Once you have a tus server set up, you can integrate the tus client into your client application. We offer tus clients for a variety of different environements including browsers, Android, iOS, Java and Node.js. If you want to pass application-specific information to your upload, such as the current user’s ID, you can add them as key-value-pairs to the upload’s metadata. This metadata is then also available to the tus server for custom processing.

Finally, most tus servers offer a way to subscribe to specific events, such as if a new upload is created or finished. For example, tusd has a hook system which can be used to execute custom scripts or send HTTP request if a specific event occurs. These can be used to move the completed uploads to another location and provide them to your application for further processing. It’s also possible to access the upload’s metadata set by the tus client in your client application inside these hooks.

How do I scale tus?

The tus protocol has been designed with scaling in mind to handle increasing numbers of connections and uploads. While the exact details vary for different server implementations, the basic concepts are following:

If multiple tus servers have a shared storage system (e.g. a network disk or cloud storage), they can be grouped together and placed behind a load balancer. You only need to ensure that there is no parallel access to a single upload resource from multiple tus servers. The implementations can be achieve this in two ways:

  • If your application ensures that an upload is always accessed by only one client, you can enable sticky sessions for your load balancer. In this case, the HTTP requests for the client will always be routed to the same tus server which can handle the concurrency correctly.
  • Alternatively, you can use a distributed lock (e.g. etcd or Redis) to ensure exclusive access to an upload resource. The lock must then be held as long as a server wants to retrieve or modify an upload resource.

Let’s look at an example to see why this complex setup may be necessary: In our scenario we have a single tus client and two tus servers, A and B, which are positioned behind a load balancer.

  1. The client connects to server A through the load balancer and starts to upload data using a PATCH request.
  2. After some time the connection between the client and the server A is interrupted, for example a mobile phone loses its reception. The important detail here is that the client can know very quickly that the connection has been dropped while the server is unaware of this situation. The server is therefore still waiting for more upload data to be written to the upload resource.
  3. The client now wants to resume the upload but connects to server B through the load balancer now. If no locking would be in place, server B and server A may write data to the same upload resource simultaneously which can lead to corrupt data.

This can be prevent if either sticky sessions were enabled, so the client would have connected to server A again, or if a shared locking mechansim between the servers would be required.

How can I deal with bad HTTP proxies?

If you are dealing with HTTP proxies that strip/modify HTTP headers or can’t handle PATCH requests properly, you should consider using HTTPS which will make it impossible for proxies to modify your requests and use the X-HTTP-Method-Override header which allows you to use POST requests.

If that is not an option for you, please reach out to us, we are open to defining a compatibility protocol extension.

How are pause/resume handled? When should I delete partial uploads?

The tus protocol is built upon the principles of simple pausing and resuming. In order to pause an upload you are allowed to end the current open request. The Server will store the uploaded data as long as no violations against other constraints (e.g. checksums) or internal errors occur. Once you are ready to resume an upload, send a HEAD request to the corresponding upload URL in order to obtain the available offset. After receiving a valid response you can upload more data using PATCH requests. You should keep in mind that the Server may delete an unfinished upload if it is not continued for a longer time period (see Expiration extension).

Before deleting an outstanding upload the Server should give the Client enough time to resolve potential networking issues. Since this duration depends heavily on the underlying application model, the protocol does not contain a specific number, but we recommend one week for a general use case.

How can I get the file name or file type for an upload?

For itself, the tus protocol does not have a direct mechanism to obtain the type or filename of an upload as the specification does not have the principle of a disk-based file, allowing you to upload arbitrary data using tus. However, the wanted functionality can be achieved by utilizing metadata. A Client can attach the file’s name and type to an upload when it’s being created by setting the Upload-Metadata header. On the other side, the Server can read these values and determine the name and type of the upload.

Why is the protocol using custom headers?

We have carefully investigated the use of existing headers such as Range and Content-Range, but unfortunately they are defined in a way that makes them unsuitable for resumable file uploads.

We also considered using existing PATCH payload formats such as multipart/byteranges, but unfortunately the XHR2 FormData interface does not support custom headers for multipart parts, and the send() method does not allow streaming arbitrary data without loading all of it into memory.

That being said, custom headers also allowed us to greatly simplify the Client and Server implementations, so we’re quite happy with them.

Why are you not using the “X-“ prefix for your headers?

The “X-“ prefix for headers has been deprecated, see RFC 6648.

What does “tus” even mean?

Historically it stood for Transloadit Upload Server, later repurposed to: The Upload Server. But really, it’s just tus, now, as its own word. Often written in lower casing as an homage to unix ancestry, and (when it was still an abbreviation) to desparately avoid yelling TUS in all caps. We used to insist on lowercasing, but have gotten more pragmatic from our encounters with journalists, who insist on it being Tus in their articles, and nothing much we can do about it. So we guess, write it however you feel like today :smile: