Future Development
PowerClerk by default has a limit of 5 MB for the maximum attachment size. But that limit can be raised in programs, specific to attachment definitions, up to 50 MB. In order to make up- and download work reliable, resume-able and parallelizable, PowerClerk offers additional API methods for uploads in parts and new versions of the existing APIs that support download with the HTTP standard approach of byte range requests.
In order not to risk impacting existing API methods, the new ones get added under a new base URL (please note v2 instead of v1):
In order not to risk impacting existing API methods, the new ones get added under a new base URL (please note v2 instead of v1):
| For testing (sandboxes): | https://api.cleanpowerdemo.com/PCITrial/services/v2 |
| Production: | https://api.powerclerk.com/services/v2 |
Authentication and API keys work exactly the same as in the V1 API. You can use the new methods either exclusively, or mixing with V1 methods.
The V2 service offers the choice of JSON or XML formats. The caller can control what format is emitted by setting the “Accept” header of a request to “application/xml” to get XML. The default is JSON (which can be explicitly requested by setting the “Accept” header to “application/json”. The XML in the V2 version of the API will not emit or require the use of namespaces (as opposed to V1, which did), but apart from that will at least closely mimic the V1 schema to make porting API integrations as easy as possible.
PowerClerk plans to add new APIs only to the V2 surface and will add V2 versions of all the other existing APIs (that will closely mimic the semantics of the existing APIs, with the additional functionality of offering JSON/namespace-free XML). Once all V1 methods are available in a V2 version, we’ll
- a) Make new API keys only work in the V2 surface (existing users stay grandfathered into V1) and
- b) Work with our existing customers to set a plan on when and how the V1 API surface will be obsoleted – we are keenly aware that this needs a lot of time.
Downloading large attachments
PowerClerk has two APIs for attachment download – GetActiveAttachment (for a particular attachment on a particular form, available to all users with access to that form) and GetAttachmentAsAdmin (doesn’t require a form but is only available to admin roles). In the V1 surface, those pull down the whole file in one go – which fails for attachments approaching 10 MB in size. The implementation of both of those in the V2 API surface supports byte range requests. This means:
- You can issue a request with the HEAD verb as opposed to GET to retrieve the length of the attachment in bytes in the Content-Length response header:
Content-length Response Header - If the length of the attachment approaches 10 MB, you MUST use byte range requests in chunks of less then 10 MB to retrieve the body (you CAN if the length is smaller as well), i.e. issue GET requests with “Range” header to specify which byte range you are asking for:
Content-length Response Header
Note that the response will come back with a 206 “Partial Content” status as opposed to a 200 “OK” status to indicate the GET request contains only the requested part of the attachment.
Param(
[parameter(Mandatory=$true)]
[PSCredential]
$creds,
[parameter(Mandatory=$true)]
$apiKey,
[parameter(Mandatory=$true)]
$programId,
[parameter(Mandatory=$true)]
$projectId,
[parameter(Mandatory=$true)]
$attachmentId,
[parameter(Mandatory=$true)]
$localDownloadPath,
[parameter(Mandatory=$true)]
$baseUrl
)
$authHash = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($creds.UserName + ":" + $creds.GetNetworkCredential().Password))
$url = $baseurl + "/Programs/$programId/Projects/$projectId/Attachments/$attachmentId/Download";
$headers = [System.Collections.Specialized.NameValueCollection]::new()
$headers.Add("X-ApiKey", $apiKey)
$headers.Add("Authorization", "Basic $authHash")
$request = [System.Net.WebRequest]::Create($url)
$request.Headers.Add($headers)
$request.Method = "HEAD" # ask for the length and filename
try {
$response = $request.GetResponse()
$length = $response.Headers["Content-Length"] # informs us how often we need to call
# use filename from content-disposition to save
$contentDisposition = $response.Headers['Content-Disposition']
$fileName = $contentDisposition.Split("=")[1].Replace("`"", "")
$path = Join-Path $localDownloadPath $([System.Uri]::UnescapeDataString($fileName))
$file = [System.IO.FileStream]::new($path, [System.IO.FileMode]::Create)
$bufferSize = 5 * 1024 * 1024; # download in chunks of 5 MB, absolute max is 10 MB incl. headers
for ($start = 0; $start -lt $length; $start = $start + $bufferSize) {
$request = [System.Net.WebRequest]::Create($url)
$request.Method = "GET"
$request.Headers.Add($headers)
$request.AddRange($start, $start + $bufferSize - 1) # set range for the current chunk
$response = $request.GetResponse()
$stream = $response.GetResponseStream()
$stream.CopyTo($file) # append to the existing file
write-host "response with" $response.Headers["Content-Range"]
}
$file.close()
return $path
}
catch [System.Exception] {
write-host $_.Exception
throw $_
}
Uploading large attachments
Uploading large attachments works in three steps:
- Request an upload to be started, using the StartUpload API. The response carries the upload ID to be used in the subsequent calls.
- Make a number of POST requests to the UploadChunk/ API, carrying the upload ID and the chunk number. Every POST request body must be between 5MB and 10 MB in size, only the last chunk can be smaller than 5 MB. The chunk number determines the order of how the chunks are being pieced back together. If a particular chunk upload fails, you can just retry with the same chunk number and that will overwrite any previous content for that chunk.
- Once all chunks are finished uploading, you call the SetProjectAttachmentFromUpload API, which unlike the normal SetProjectAttachment API doesn’t require the content in the body but instead takes the UploadId to gather the chunks, sort them in ascending order of the chunk number they got uploaded to and sets that as the attachment. This completes the upload, no further calls to the UploadChunk API with that UploadId will be taken.
- The maximum elapsed time from the call to StartUpload to SetProjectAttachmentFromUpload can be 24 hours, after that the uploaded chunks are discarded and the UploadId is considered invalid. If you need to explicitly abandon an upload (e.g. because want to discard the previously uploaded chunks explicitly), you can call CancelUpload.
Param(
[parameter(Mandatory = $true)]
[PSCredential]
$creds,
[parameter(Mandatory = $true)]
$apiKey,
[parameter(Mandatory = $true)]
$programId,
[parameter(Mandatory = $true)]
$projectId,
[parameter(Mandatory = $true)]
$formId,
[parameter(Mandatory = $true)]
$attachmentId,
[parameter(Mandatory = $true)]
$filePath,
[parameter(Mandatory = $true)]
$baseUrl
)
$authHash = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($creds.UserName + ":" + $creds.GetNetworkCredential().Password))
$startUrl = $baseurl + "/Programs/$programId/Uploads/Start"
$headers = [System.Collections.Specialized.NameValueCollection]::new()
$headers.Add("X-ApiKey", $apikey)
$headers.Add("Authorization", "Basic $authHash")
$filename = Split-Path $filePath -leaf
$file = [System.IO.FileStream]::new($filePath, [System.IO.FileMode]::Open)
try {
$request = [System.Net.WebRequest]::Create($startUrl)
$request.Method = "POST"
$request.ContentLength = 0;
$request.Headers.Add($headers)
$response = $request.GetResponse()
$sr = new-object System.IO.StreamReader ($response.GetResponseStream())
$uploadIdJson = $sr.ReadToEnd() | ConvertFrom-Json
$uploadId = $uploadIdJson.UploadId
$bufferSize = 5 * 1024 * 1024; # upload chunks must be a minimum of 5 MB
$byteArray = new-object Byte[] $bufferSize
$chunkNumber = 1
for ($start = 0; $start -lt $file.Length; $start = $start + $bufferSize) {
$uploadChunkUrl = $baseurl + "/Programs/$programId/Uploads/$uploadId/$chunkNumber"
write-host $uploadChunkUrl
$request = [System.Net.WebRequest]::Create($uploadChunkUrl)
$request.Method = "POST"
$request.Headers.Add($headers)
$rs = $request.GetRequestStream()
$count = $file.Read($byteArray, 0, $bufferSize)
$rs.Write($byteArray, 0, $count);
$rs.Close();
$response = $request.GetResponse()
write-host "sent chunk $chunkNumber"
$chunkNumber += 1
}
# finish upload and set attachment
$setAttachmentUrl = $baseurl + "/Programs/$programId/Projects/$projectId/Forms/$formId/Attachments/$attachmentId/FromUpload/$uploadId";
$request = [System.Net.WebRequest]::Create($setAttachmentUrl)
$request.Headers.Add($headers)
$request.Headers['Content-Disposition'] = "filename=""$filename"""
$request.ContentType = "application/pdf"
$request.ContentLength = 0;
$request.Method = "POST"
$response = $request.GetResponse()
$sr = new-object System.IO.StreamReader ($response.GetResponseStream())
$attachmentJson = $sr.ReadToEnd() | ConvertFrom-Json
return $attachmentJson.Attachment.AttachmentId
}
catch [System.Exception] {
write-error "Exception: $_"
}
finally {
$file.Close()
}
What’s Next?