Upload Large Files Directly to GCS with Dropzone and Signed URL’s

Break up files in Chunks on front end

Dropzone.options.dropzoneForm = {
paramName: "file", //name that will be used to transfer the file
acceptedFiles: "image/png, image/jpeg, image/jpeg, video/mp4",
chunking: true, // Enable chunking
chunkSize: 30000000, // Chunk Size
forceChunking: true
}
  1. GAE does not allow you to have file system access, you can still write files to the /tmp directory but it has a limit of 2GB
  2. Each chunk upload and write was very slow

Direct Upload to GCS using signed URL’s

  • XML : Only this supports signed URL’s so have to use this
  • JSON
from google.cloud.storage._signing import generate_signed_url
API_ACCESS_ENDPOINT = 'https://storage.googleapis.com'
def _get_storage_client():
return storage.Client.from_service_account_json("credentials.json")
def get_signed_url(name, content_type):
client = _get_storage_client()
expiration = datetime.datetime.now()+datetime.timedelta(days=1)
canonical_resource = "/" + settings.CLOUD_STORAGE_BUCKET + "/" + safe_filename(name)
url = generate_signed_url(
client._credentials,
resource=canonical_resource,
api_access_endpoint=API_ACCESS_ENDPOINT,
expiration=expiration,
method="PUT",
content_type=content_type
)
return url
Dropzone.options.dropzoneForm = {
acceptedFiles: "image/png, image/jpeg, image/jpeg, video/mp4",
method: "PUT",
timeout: null, // Dropzone has a default time out of 30sec
// Get Upload Url dynamically
url: function (files) {
console.log(files[0].dynamicUploadUrl);
return files[0].dynamicUploadUrl
},
headers: { // Remove unwanted headers
'Cache-Control': null,
'X-Requested-With': null,
'Accept': null
},
// IMP: Have to make this configuration to send raw files to GCS
sending: function (file, xhr) {
let _send = xhr.send;
xhr.send = function () {
_send.call(xhr, file);
};
},
init: function () {
this.on("success", function (file, response) {
//Inform the server. Uploaded Successfully
});
},
accept: function (file, done) {
//Dynamically set the content type header based on file
// uploaded .
this.options.headers['Content-Type'] = file.type;
//Request Signed URL from backend
//On success, Set url in file object which will be used
//in url function above. Something like
//file.dynamicUploadUrl = signed_url;
//call the dropzone
//done();

}
};
[
{
"maxAgeSeconds": 3600,
"method": [
"GET",
"HEAD",
"PUT"
],
"origin": [
<Your origin whitelist>
],
"responseHeader": [
"X-Requested-With",
"Access-Control-Allow-Origin",
"Content-Type",
]
}
]

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Working with Data and External APIs

Build a Flexible React Icon Component

Understand Your Application’s Data Flow

images/data-flow.png

Using Custom Iterators and Generators

images/iterator.png

Chapter 18 Browsing with Websockets and Erlang

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Prashant Pal

Prashant Pal

More from Medium

Clone and backup with GParted

Using p5.js + Arduino as a prototyping workflow

Poker — Stack Size Synopsis

JSSmallTips — Creating custom matcher assertions with Jest