Download large file with progress

How can I download a file from a web service and save it on a file while getting progress on the download? Could Chunks be used?

Answers

  • edited November 2015 Answer ✓

    Daniel -

    The KinomaJS Handler prototype provides a download function for downloading files with progress. You can wrap Handler.prototype.download like this in a local behavior:

    Handler.Bind("/download", class extends Behavior {
        onComplete(handler, message) {
            if (0 == message.error && 200 == message.status)
                trace("Download complete!\n");
        }
        onInvoke(handler, message) {
            let query = parseQuery(message.query);
            handler.download(new Message(query.url), query.file);
        }
        onProgress(handler, message, offset, size) {
            trace("Downloaded " + offset + " of " + size + "\n");
            application.distribute("onDownloadProgress", offset, size);
        }
    });
    

    Then call the download handler like this from a container:

    let uri = "/download?" + serializeQuery({
        url:"http://path/to/file",
        file:mergeURI(Files.temporaryDirectory, "download")
    });
    container.invoke(new Message(uri));
    

    In the example above, the progress function distributes an onDownloadProgress event that can be implemented by the container's behavior to update a progress indicator. The file is downloaded to the temporary items directory.

    Regards, Brian

  • Follow up question: How can I cancel an active download?

  • Daniel -

    To cancel an active download you can cancel the message that triggered the download:

    let uri = "/download?" + serializeQuery({
        url:"http://path/to/file",
        file:mergeURI(Files.temporaryDirectory, "download")
    });
    this.message = new Message(uri);
    container.invoke(this.message);
    
    // cancel the active download
    this.message.cancel();
    
  • edited November 2015

    Kinoma seems to have some problems requesting objects from AWS S3 using pre-signed URLS. Has anybody seen this before? Here's an example of such URL:

    https://s3.amazonaws.com/dev-p2p-outputs/e983f983b5bf4aceb7c5923d5d0703ce.gcode?response-content-disposition=attachment; filename="test.txt"&X-Amz-Content-Sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=/20151117/us-east-1/s3/aws4_request&X-Amz-Date=20151117T215521Z&X-Amz-SignedHeaders=Host&X-Amz-Expires=300&X-Amz-Signature=0da4d2fd4032e0dbe20ad9fd3b2a8d21f252810e99a93c9b7af41abba7cc5e58

    This URL has been edited and was only signed for 5 minutes so it won't work as such. It's only provided as sample. When copying into the browser (within 5 minute of its creation), the file starts to download with no problems. If sending it using "handler.download", the server responds with

    Http Status: 403

    Code: SignatureDoesNotMatch

    Message: The request signature we calculated does not match the signature you provided. Check your key and signing method.

    I can't think of anything else than Kinoma changing something just before sending. Maybe a problem with encoding/decoding... Or a wrong "host" request header?

    any ideas?

  • Daniel,

    To confirm the URL isn't being truncated for some reason, can you retry this request, but move the "X-Amz-Signature" variable earlier in the request string? I think placing it between the "X-Amz-Content-Sha256" and "X-Amz-Algorithm" variables would produce a meaningful result. If truncation is occurring, I would expect S3 to respond with a different 403 Code corresponding to whatever got cut off.

    Thanks, Will

  • No changes in the response. Same exact error

  • I don't know if this helps but I added an endpoint on my server that responds with a redirect to the signed URL. I tried to call that endpoint instead of trying to call the signed URL directly from Kinoma. The result is the same as indicated above.

  • Daniel,

    The URL you're using has an un-encoded space in it after "disposition=attachment;". Please try replacing that space with "%20" and retesting.

    Let me know if that makes a difference.

    --Will

  • It didn't help. I also removed the whole content-disposition part as it's not needed for this download and it made no difference.

  • edited November 2015

    Daniel,

    There are a number of headers that KinomaJS's HTTP stack doesn't set automatically. It is possible that Amazon is looking for one of these and failing without it.

    In my testing, using a Firefox request as a benchmark, headers not set include:

     Accept-Encoding
     Accept-Language
     Accept
     User-Agent
    

    To set these headers, the Message object has a setRequestHeader() method, as shown below:

     let msg = new Message(uri);
     msg.setRequestHeader("User-Agent", "KinomaJS Application")
     container.invoke(msg); 
    

    Cheers,

    Will

  • edited November 2015

    The only header required (i.e. used in the signature) as far as I know is host which I think might be the culprit, as Kinoma passes it like this: s3.amazonaws.com:443 and I think the server calculated the signature using s3.amazonaws.com

    According to the AWS Signature specs when the port is standard (80 for HTTP and 443 for HTTPS) the port should be omitted.

Sign In or Register to comment.