Forum Discussion

gmanolache19's avatar
gmanolache19
New Contributor
2 years ago
Solved

Upload File to AWS S3 Bucket Error due to AWS Signature Problem

Currently I'm trying to upload a file to an AWS S3 Bucket but I didn't manage to get it done. Firstly I tried using the REST Request step, in which after I set the credentials for AWS Signature, as a response I get error 400 with message "XAmzContentSHA256Mismatch", so I thought that I can calculate myself the SHA256 hash for the file and set the header for that field myself, but when I do that I get 403 with "SignatureDoesNotMatch" so I gave up on REST Request step (I also tried HTTP Request step without any luck). Secondly I tried to do it using Groovy Script but with that I get "com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target" when calling the PutObject function for S3 object. I tried to set the keystore in both the preferences and directly in the code but the error stays the same.

 

Just to be sure that all policies are set correctly in IAM and that I have credentials that I can use to GET/PUT files into the bucket, I created requests in Postman which works seamlessly, the problem is that I want to automate this upload.

  •  

     

    // This step computes the AWS signature for the upload to S3
    
    import javax.crypto.Mac;
    import java.text.*;
    import javax.crypto.spec.SecretKeySpec;
    import java.security.InvalidKeyException;
    
    def testSuite = testRunner.testCase.testSuite
    //	Credentials !!!
    String accessKey = "<accessKey>"
    String secretKey = "<secretKey>"
    
    //	Path to prefered location for generated file
    String folderName = "<folderPathtoFile>"
    String fileName = "<fileName>"
    
    // Calculate the hash for the file
    File generatedFile = new File(folderName + "/" + fileName)
    String fileHash = generatedFile.bytes.digest('SHA-256')
    
    //	Get the current date and time in ISO8061 format and only the date without time
    // in the EDT timezone
    Calendar cal = Calendar.getInstance();
    DateFormat df = new SimpleDateFormat("yyyyMMdd'T'HHmmss'Z'");
    df.setTimeZone(TimeZone.getTimeZone("EDT"));
    String dateTime = df.format(cal.getTime())
    df = new SimpleDateFormat("yyyyMMdd")
    String onlyDate = df.format(cal.getTime())
    
    //	Get the object key from test suite and bucket name from environment
    String objectKey = "<fileNameOrPathToFileInS3>"
    String bucket = "<bucketName>"
    
    //	Set url and properties needed for calculating AWS signature
    String fileUrl = "/" + bucket + "/" + objectKey
    String location = "us-east-1"
    String service = "s3"
    String httpMethod = "PUT"
    
    //	Headers
    String header1 = "host:s3.amazonaws.com"
    String header2 = "x-amz-content-sha256:" + fileHash
    String header3 = "x-amz-date:" + dateTime
    
    // Compose the canonical request
    String canonicalUri = fileUrl
    String canonicalQueryString = ""
    String canonicalHeaders = header1 + "\n" + header2 + "\n" + header3 + "\n"
    String signedHeaders = header1.split(":")[0] + ";" + header2.split(":")[0] + ";" + header3.split(":")[0]
    String hashedPayload = fileHash
    String canonicalRequest = httpMethod + '\n' + canonicalUri + '\n' + canonicalQueryString + '\n' + canonicalHeaders + '\n' + signedHeaders + '\n' + hashedPayload
    log.info "canonicalRequest "+canonicalRequest
    
    // Compose the string to sign
    String path = onlyDate + "/" + location + "/" + service + "/aws4_request"
    String stringToSign = "AWS4-HMAC-SHA256\n" + dateTime + "\n" + path + "\n" + canonicalRequest.digest('SHA-256')
    log.info "stringToSign "+stringToSign
    
    // Function to encrypt a hash using a key
    def hmac_sha256(byte[] key, String data) {
        try {
            Mac mac = Mac.getInstance("HmacSHA256")
            SecretKeySpec secretKeySpec = new SecretKeySpec(key, "HmacSHA256")
            mac.init(secretKeySpec)
            return mac.doFinal(data.getBytes())
        } catch (InvalidKeyException e) {
            throw new RuntimeException("Invalid key exception while converting to HMac SHA256")
        }
    }
    
    // Calculate the signing key
    byte[] secretKeyB = ("AWS4" + secretKey).bytes
    byte[] dateKey = hmac_sha256(secretKeyB, onlyDate)
    byte[] dateRegionKey = hmac_sha256(dateKey, location)
    byte[] dateRegionServiceKey = hmac_sha256(dateRegionKey, service)
    byte[] signingKey = hmac_sha256(dateRegionServiceKey, "aws4_request")
    //log.info "signingKey "+signingKey.toString()
    
    // Calculate the signature and compose the authorization header
    String signature = hmac_sha256(signingKey, stringToSign).encodeHex()
    String authHeader = "AWS4-HMAC-SHA256 Credential=" + accessKey + "/" + path + ", SignedHeaders=" + signedHeaders + ", Signature=" + signature
    //log.info "authHeader "+authHeader
    
    // Set the headers for upload to S3 request
    testSuite.setPropertyValue("authHeader", authHeader)
    testSuite.setPropertyValue("xamzcontent", fileHash)
    testSuite.setPropertyValue("xamzdate", dateTime)
    
    log.info "Finished computing the signature and added the headers as properties"

     

     

    This is the solution I came up with, so in order to upload a file to S3 firstly you need to generate the AWS Signature based on the file you want to upload with this script, to use it you need to put the keys for AWS and the folder and file path for local file and in S3, for object key you can just put the file name you want to be in S3. After you run this script, 3 properties will be set at test suite level which act as headers that need to be set manually for the PUT Request as follows: authHeader for Authorization, xamzcontent for x-amz-content-sha256 and xamzdate for x-amz-date. I attached a screenshot with how the PUT request should look like, for request resource you need to put it like this /<bucketName>/<fileNameOrPath> also the file needs to be loaded in the Attachments tab of the request.

1 Reply

  •  

     

    // This step computes the AWS signature for the upload to S3
    
    import javax.crypto.Mac;
    import java.text.*;
    import javax.crypto.spec.SecretKeySpec;
    import java.security.InvalidKeyException;
    
    def testSuite = testRunner.testCase.testSuite
    //	Credentials !!!
    String accessKey = "<accessKey>"
    String secretKey = "<secretKey>"
    
    //	Path to prefered location for generated file
    String folderName = "<folderPathtoFile>"
    String fileName = "<fileName>"
    
    // Calculate the hash for the file
    File generatedFile = new File(folderName + "/" + fileName)
    String fileHash = generatedFile.bytes.digest('SHA-256')
    
    //	Get the current date and time in ISO8061 format and only the date without time
    // in the EDT timezone
    Calendar cal = Calendar.getInstance();
    DateFormat df = new SimpleDateFormat("yyyyMMdd'T'HHmmss'Z'");
    df.setTimeZone(TimeZone.getTimeZone("EDT"));
    String dateTime = df.format(cal.getTime())
    df = new SimpleDateFormat("yyyyMMdd")
    String onlyDate = df.format(cal.getTime())
    
    //	Get the object key from test suite and bucket name from environment
    String objectKey = "<fileNameOrPathToFileInS3>"
    String bucket = "<bucketName>"
    
    //	Set url and properties needed for calculating AWS signature
    String fileUrl = "/" + bucket + "/" + objectKey
    String location = "us-east-1"
    String service = "s3"
    String httpMethod = "PUT"
    
    //	Headers
    String header1 = "host:s3.amazonaws.com"
    String header2 = "x-amz-content-sha256:" + fileHash
    String header3 = "x-amz-date:" + dateTime
    
    // Compose the canonical request
    String canonicalUri = fileUrl
    String canonicalQueryString = ""
    String canonicalHeaders = header1 + "\n" + header2 + "\n" + header3 + "\n"
    String signedHeaders = header1.split(":")[0] + ";" + header2.split(":")[0] + ";" + header3.split(":")[0]
    String hashedPayload = fileHash
    String canonicalRequest = httpMethod + '\n' + canonicalUri + '\n' + canonicalQueryString + '\n' + canonicalHeaders + '\n' + signedHeaders + '\n' + hashedPayload
    log.info "canonicalRequest "+canonicalRequest
    
    // Compose the string to sign
    String path = onlyDate + "/" + location + "/" + service + "/aws4_request"
    String stringToSign = "AWS4-HMAC-SHA256\n" + dateTime + "\n" + path + "\n" + canonicalRequest.digest('SHA-256')
    log.info "stringToSign "+stringToSign
    
    // Function to encrypt a hash using a key
    def hmac_sha256(byte[] key, String data) {
        try {
            Mac mac = Mac.getInstance("HmacSHA256")
            SecretKeySpec secretKeySpec = new SecretKeySpec(key, "HmacSHA256")
            mac.init(secretKeySpec)
            return mac.doFinal(data.getBytes())
        } catch (InvalidKeyException e) {
            throw new RuntimeException("Invalid key exception while converting to HMac SHA256")
        }
    }
    
    // Calculate the signing key
    byte[] secretKeyB = ("AWS4" + secretKey).bytes
    byte[] dateKey = hmac_sha256(secretKeyB, onlyDate)
    byte[] dateRegionKey = hmac_sha256(dateKey, location)
    byte[] dateRegionServiceKey = hmac_sha256(dateRegionKey, service)
    byte[] signingKey = hmac_sha256(dateRegionServiceKey, "aws4_request")
    //log.info "signingKey "+signingKey.toString()
    
    // Calculate the signature and compose the authorization header
    String signature = hmac_sha256(signingKey, stringToSign).encodeHex()
    String authHeader = "AWS4-HMAC-SHA256 Credential=" + accessKey + "/" + path + ", SignedHeaders=" + signedHeaders + ", Signature=" + signature
    //log.info "authHeader "+authHeader
    
    // Set the headers for upload to S3 request
    testSuite.setPropertyValue("authHeader", authHeader)
    testSuite.setPropertyValue("xamzcontent", fileHash)
    testSuite.setPropertyValue("xamzdate", dateTime)
    
    log.info "Finished computing the signature and added the headers as properties"

     

     

    This is the solution I came up with, so in order to upload a file to S3 firstly you need to generate the AWS Signature based on the file you want to upload with this script, to use it you need to put the keys for AWS and the folder and file path for local file and in S3, for object key you can just put the file name you want to be in S3. After you run this script, 3 properties will be set at test suite level which act as headers that need to be set manually for the PUT Request as follows: authHeader for Authorization, xamzcontent for x-amz-content-sha256 and xamzdate for x-amz-date. I attached a screenshot with how the PUT request should look like, for request resource you need to put it like this /<bucketName>/<fileNameOrPath> also the file needs to be loaded in the Attachments tab of the request.