S3 CDN Getting Started

Hi –

I’ve been trying to get S3/CloudFront working to stream RTMP without much luck…. At the end of this post http://www.s2member.com/forums/topic/download-links/page/2/ per Jason’s suggestion I tried to have s2member deactivate and reactive Cloudfront.

I even tried the policy change suggestion in this post per Jeremy Pigg. No luck. (Besides, I think that policy change simply makes everything in your bucket readable by the world… But I’m no expert on S3.)

So, I decided to start from scratch. Turned off CloudFront and turned off S3/CDN on s2member. Deleted Cloudfront distributions (which were marked disabled.)

Created a new bucket “iratest”. Put one file in it “copyright.png”.

First checked – put copyright.png in the file s2member-files directory. This URL works fine when logged in to WordPress http://www.DOMAINNAME.org/wordpress/?s2member_file_download=/copyright.png. File downloads okay.

Now taking it one step at a time, setup ONLY s2Member->Download Options-> Amazon S3/CDN Storage Option.
Deleted old S3 access keys and generated NEW key. Loaded both access key & secret key into s2member.

When I go to the URL that was working ( http://www.DOMAINNAME.org/wordpress/?s2member_file_download=/copyright.png) I get this URL
http://iratest.s3-us-west-1.amazonaws.com/?response-cache-control=no-cache%2C+must-revalidate%2C+max-age%3D0%2C+post-check%3D0%2C+pre-check%3D0&response-content-disposition=attachment%3B+filename%3D%22copyright.png%22&response-content-type=image%2Fpng&response-expires=Wed%2C+20+Mar+2013+04%3A22%3A48+GMT&AWSAccessKeyId=AKIAJANGYGY62F2G6JWA&Expires=1364444568&Signature=aLhu1n0RnU%2F0yVUwqGiDKAQ3PFw%3D

and this XML result

This XML file does not appear to have any style information associated with it. The document tree is shown below.

iratest


1000
false

copyright.png
2013-03-27T04:00:00.000Z
"60e316ad8ffdf3af908a4c9969305cb4"
114318


39f82f3961cdb3abd03419f709c2fb348a7fca8e292012d36112ab35becd31fa

support

STANDARD

And for reference http://iratest.s3.amazonaws.com/copyright.png
returns the expected

This XML file does not appear to have any style information associated with it. The document tree is shown below.

<Code>AccessDenied

Access Denied
EAFDE88213BC7563

j6gXESi/4DED3I0VJCpZvOZEfNImFPADNYB+PDg8bUPytoXnhuP4S4OhuD0XWahu

If I change the permissions to Everyone to Open/Download & View, it works of course. But the link above trying to access via s2member/wordpress doesn’t.

What should we check or how do we determine where the problem is?

Thanks,

Ira

PS: For what it is worth using a mod_rewrite like this http://www.DOMAINNAME.org/wordpress//wp-content/plugins/s2member-files/copyright.png doesn’t change the result.

Hi Ira.

So the problem is that instead of the image you get that XML file? I don’t know why that happens, I guess some configuration in Amazon… I’ll ask Jason about it.

About completely wiping out the Amazon integration to reconfigure it, you can try this in case it helps: http://www.s2member.com/kb/reset-the-amazon-s3-cloudfront-integration/

Hi Cristian,

The XML file is an error message say “access denied”. I will try the wipe right now but creating a new bucket should have the same effect.

Yes, kindly ask Jason ASAP. I’m really falling behind in getting this working so any help would be great.

Thanks!

Ira

Hi Cristian,

I tried the wipe of the Amazon integration per the hack provide. Reset fine. And then I did the reset of only the S3 portion. (I didn’t do the CloudfFront portion.) I still get the same error XML. For yucks, I checked it on Firefox (same thing) and on Safari (same thing but no pretty XML formatting).

What’s next? How do we get to the bottom of this quickly? Does Jason want to login to my site or is there a log file that might tell us (either on s2member or Amazon)?

Thanks,

Ira

Thanks for the heads up on this thread :-)

I was unable to reproduce this myself. However, it sounds like this problem could be attributed to permission settings on your Bucket and/or a possible index file problem. Please make sure that your S3 Bucket is not publicly available, and that your S3 Bucket is not setup as a Static Host of any kind.

See also: https://forums.aws.amazon.com/message.jspa?messageID=108471

If problems persist, I would contact AWS about this, to find out why files served through authenticated URLs are resulting in a ListBucketResult set for you. This is not a normal occurrence.

Hi Jason,

I just double checked. My bucket is not readable and I have removed all policies. Only access granted is to me. And the link http://iratest.s3.amazonaws.com/copyright.png results in Access Denied. (As noted above.)

Is there a way to see the URL that s2member is generating either from s2member or in a log somewhere? I’m afraid if I can get support from AWS (without being a paid support user) they will simply say we are generating the authenticated URL wrong.

Thanks,

Ira

Hi Jason,

Via BucketExplorer (www.bucketexplorer.com), I was able to generate this signed URL

http://iratest.s3.amazonaws.com/copyright.png?AWSAccessKeyId=AKIAJANGYGY62F2G6JWA&Expires=1364540393&Signature=o4jDUzWknhf39zkCzXa4Qi6ETs8%3D

which worked just fine. So, I suspect there may be an issue with s2member and how it is generated a signed URL or doing a mod_rewrite.

How do we sort this out?

Thanks,

Ira

Hi Jason & Cristian –

I figured out how to turn on logging on S3. So, looking into the logs here is what I see:

"-" "Bucket Explorer" -
39f82f3961cdb3abd03419f709c2fb348a7fca8e292012d36112ab35becd31fa iratest [29/Mar/2013:06:28:36 +0000] 75.XX.XX.XX 39f82f3961cdb3abd03419f709c2fb348a7fca8e292012d36112ab35becd31fa 4730DE52C4E5B504 REST.GET.OBJECT doof.png "GET /doof.png?AWSAccessKeyId=AKIAJANGYGY62F2G6JWA&Expires=1364799569&Signature=oETy3teoXNthFs2uG%2BcrGmTqkhc%3D HTTP/1.1" 200 - 114318 114318 17 14 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 Safari/537.22" -

39f82f3961cdb3abd03419f709c2fb348a7fca8e292012d36112ab35becd31fa iratest [29/Mar/2013:06:28:38 +0000] 75.XX.XX.XX - D459C46B92C87F83 REST.GET.OBJECT favicon.ico "GET /favicon.ico HTTP/1.1" 403 AccessDenied 231 - 11 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 Safari/537.22" -

39f82f3961cdb3abd03419f709c2fb348a7fca8e292012d36112ab35becd31fa iratest [29/Mar/2013:06:29:23 +0000] 75.XX.XX.XX 39f82f3961cdb3abd03419f709c2fb348a7fca8e292012d36112ab35becd31fa D72F0729EF534CAC 
REST.GET.BUCKET - "GET /?response-cache-control=no-cache%2C+must-revalidate%2C+max-age%3D0%2C+post-check%3D0%2C+pre-check%3D0&response-content-disposition=attachment%3B+filename%3D%22copyright.png%22&response-content-type=image%2Fpng&response-expires=Fri%2C+22+Mar+2013+06%3A28%3A39+GMT&AWSAccessKeyId=AKIAJANGYGY62F2G6JWA&Expires=1364624919&Signature=5JgSLSIFdcuU6p7RuF%2BUmbl6KPI%3D HTTP/1.1" 200 - 3412 - 227 226 "http://www.bitsworkshop.org/wordpress/2013/02/28/session-6-and-at-the-wafer-level-2/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 Safari/537.22" -

(New lines added for readability, IP Address redacted & I *WILL* change my AccessKeys when we are done debugging.)

The first access for doof.png was using a URL generated by Bucket Explorer. (I added a second item with the name doof.png to make it easy to find.)
The middle item appear something random.
The third item is using http://www.MYDOMAIN.org/wordpress/?s2member_file_download=/copyright.png via WordPress / s2Member.

Three items that stand out right off the bat on the s2member access:
1. It is a REST.GET.BUCKET instead of REST.GET.OBJECT like the first one.
2. The file name is not at the beginning of “GET /” but is a parameter buried unlike “GET /doof.png”
3. Whatever response-requires is it expired a week ago (3/22) instead of today (3/29). (I did these tests last night.

How do we track this down on the s2member side?

Thanks,

Ira

PS: Jason, I have the patch you provided in http://www.s2member.com/forums/topic/s2stream-short-code-url-problem/ in place right now. I don’t know if you test this issue with the patch or the “regular” code.

Hi Jason,

On a hunch, I tried removing the patch you provided in http://www.s2member.com/forums/topic/s2stream-short-code-url-problem/. My S3 now appears to be working correctly. (Of course now I will have a problem with the shortcode…) Perhaps, in fixing the shortcode problem the call to S3 got broken?

Would you kindly check into this ASAP? I really need to get things working and it is taking far too long to sort through these issues.

Thanks,

Ira

UPDATE: This patch was formally applied to the maintenance release of s2Member v130404.
See changelog: http://www.s2member.com/changelog/#s2-changes-v130404

Please let us know if the problem continues in the latest release of s2Member®. I just made an attempt to reproduce this against the latest version; and with a sub-directory as well. So far everything seems fine.

1. It is a REST.GET.BUCKET instead of REST.GET.OBJECT like the first one.

s2Member® does not issue a REST call of any kind (other than to generate a link and redirect the customer to that link); which results in a RESTful API use on the part of s2Member; but the final call is established on the Amazon side of things, in the way it handles the URL that we generate.

Based on the output in XML that you supplied earlier, I’m not surprised to see that method being processed by the logs. The question is why was it doing that. It sounds like, the URL was corrupted in some way to me. If that came from the previous PATCH file we supplied, please try the official release of s2Member® and report back.

2. The file name is not at the beginning of “GET /” but is a parameter buried unlike “GET /doof.png”

This is to be expected. s2Member® configures the request beyond that of what many simple services will do.

3. Whatever response-requires is it expired a week ago (3/22) instead of today (3/29). (I did these tests last night.

This is to be expected. The expires value is intentionally set in the past, to prevent browser caching. This is simply the Expires: header value (s2 is following standards for this header). It’s NOT the expiration of the Bucket Object itself; which is configured in a separate component of the query string.

Hi Jason,

Thanks for the follow-up. I just updated to v130404 and it looks like the problem has gone away.

Thanks,

Ira