Downloading¶
About half of the use cases involving a HTTP request are revolving around downloading resources from a remote server, and we want to do it as fast, reliably and memory efficiently as possible. To support all of these requirements, this version of the plugin unifies previously different scenarios. Now, every download is a streamed download, no matter how small or large the downloaded content is. This ensures that setting up a request is easier and a seamingly complex use case has a simple solution too.
By default the plugin alows 1 MiB of buffered data. If the target content is larger, the download will pause indefinitely! If your content is presumably larger than this threshold, you can increase this limit or start consuming the content during download.
How to access downloaded content¶
The plugin offers two main ways to access the downloaded data:
- Shortcuts for the most common scenarios
- DownStream for use-cases where the downloaded content can be processed in chunks
Shortcuts¶
For small and simple use-cases we don't want to deal with streams and buffers and all of that, just an easy way to get what the server sent. For these cases the HTTPresponse class has a few ready to use shortcut properties. These properties still use the download-stream to get the content, but they hide all the boring details. Here's the list of the available shortcut properties:
byte[] Data { get; }
: The raw content bytes downloaded from the server, in case of compressed data, it contains the uncompressed final bytes. Use theData
property if the content is small or the consuming API requires continous memory.Example
string DataAsText { get; }
: This shortcut converts the raw content bytes to an utf-8 string.Example
Texture2D DataAsTexture2D { get; }
: Uses ImageConversion.LoadImage to return with a Texture2D.Example
All of these properties are caching their returned object, calling them repeatedly doesn't do any further allocations.
All of these shortcuts assume that the download completed and called after the request's state became finished!
Streaming¶
With the help of the HTTPResponse
's DownStream
property, we can access parts of the downloaded data while the download is still in progress. DownStream
is non-blocking by default, all of its functions return immediately with or without data, depending on whether there's any data in its buffers. This means, we have to try to take data from the stream every frame, or use a thread with a blocking variant of the stream. In the following topics we'll explore these use-cases, but before we dive into these, we also have to talk about the OnDownloadStarted
event that we will use for both cases:
The OnDownloadStarted
event¶
Streaming doesn't have any differences in the request's setup, but to start processing data as soon as we just can, we can subscribe to the OnDownloadStarted
event. This event is called when the plugin expects the server to send content, just right after, in the next frame, when the DownStream
instance is created. We can use this callback to start our content processing logic.
This event is called only when the response has a status code of 2xx, and it does NOT for redirects, request and server errors (4xx & 5xx status codes). Further reducing the number of edge cases you have to handle and code complexity you have to manage.
Streaming With polling¶
Let's see the code first, and discuss it in detail after:
IEnumerator ParseContent(DownloadContentStream stream)
: We are making a coroutine that can be consumed by Unity, this requires that our function's return value must beIEnumerator
. When we will use we also want to pass aDownloadContentStream
object to the function.DownloadStream
'sIsCompleted
returnstrue
only when the download is completed, and there's no more data buffered in the stream to read.stream.TryTake(out var buffer)
tries to take out a downloaded chunk from the stream and will return withtrue
only if the buffer is a valid one with content. When the stream is empty it returns withfalse
.Don't make any assumptions about data length, it's at least 1 byte and can be different every frame.
BufferPool.Release(buffer);
will release back the buffer to theBufferPool
. It's not mandatory, and must be called only after the buffer is no longer in use!yield return null;
skips to next frame.stream.Dispose();
finally, we have to dispose the stream, releasing any resources held by it.
And now, we can modify the OnDownloadStarted
callback to start the new coroutine:
It's done, we now have a working streaming download!
Streaming With blocking¶
The download itself is always done on a non-Unity main thread, it doesn't affect the running game logic and rendering!
In this section we start over using only the code from the 'The OnDownloadStarted
event' section.
Because DownloadContentStream
is non-blocking, we have to use a different kind of stream, a BlockingDownloadContentStream
! Modify the request setup part, to include the following, highlighted code:
DownloadStreamFactory
is called when the plugin is in need of a new DownloadContentStream
instance for the request. Fortunately, BlockingDownloadContentStream
is a specialized DownloadContentStream
so we can use it instead.First, let's write the function that will consume the downloaded content:
This function's logic is very similar what we wrote in the coroutine version's ParseContent
:
- until the download isn't completed
- repeat the
TryTake(out var buffer)
call- process the data
- release the buffer
- repeat the
- and finally dispose the stream.
That's it!
Now, we just have to use it:
- For threading we will use Task.Run and wait for its completion, so have to modify
OnDownloadStarted
too to add theasync
keyword - await the
Task.Run
call where we callConsumeDownloadStream
too
DownloadContentStream
lifecycle¶
The DownloadContentStream
plays a crucial role in handling streamed downloads within the plugin. Understanding its lifecycle is essential for proper resource management and avoiding potential issues.
Default Behavior¶
By default, the DownloadContentStream
is bound to the lifecycle of the associated request. When the request's callback is invoked, the stream is disposed of along with the request. However, complications can arise if the stream is still in use when the callback occurs.
Detachment Mechanism¶
To address this issue, the plugin employs a detachment mechanism. If a read operation is detected from outside the plugin, the stream becomes detached from the request. This detachment allows the stream's lifetime to extend beyond the request's callback, preventing premature disposal.
Manual Disposal¶
It's important to note that users are responsible for manually disposing of the DownloadContentStream
once they have finished using it. Failure to do so can lead to resource leaks and unexpected behavior.
Best Practices¶
When working with DownloadContentStream
, consider the following best practices:
- Detachment Handling: If you anticipate using the stream beyond the request callback, ensure it becomes detached either by initiating a read operation (
Read
,TryTake
) within theOnDownloadStarted
callback or by settingDownloadContentStream
'sIsDetached
property to true. - Manual Disposal: Always dispose the stream manually once you have finished using it to release any resources it holds. This can be done by calling the
Dispose()
method when you no longer need the stream. - Error Handling: Handle any exceptions that may occur during stream operations gracefully and call
Dispose()
to ensure proper cleanup.
Progress Tracking¶
No matter how DownloadStream
is utilised or even used directly, progress tracking is an independent feature. Subscribing to progress events can be done by setting an event handler for DownloadSettings.OnDownloadProgress
:
The handler method receives the originating HTTPRequest itself, the number of bytes downloaded so far and the total length of the content being downloaded, or -1 if the length cannot be determined.