Scaleway

S3 Object Storage - Generate a AWSv4 Authentication Signature

2018-09-21T00:00:00+00:00

Object Storage - Generate a AWSv4 Authentication Signature

Requests sent to the Object storage API require to include an HTTP Authorization header. Currently the AWS v4 Signature type is supported.

Throughout the Object Storage documentation, v4 signatures are used. When you are using a client library such as aws-cli, s3cmd or s3fs, signatures are automatically generated by the library for you.

To generate the signature, you need to have an Access Key and Secret Key generated in the Credentials section of your management console.

A v4 signature consists of a number of parts. The table below provides more information on each piece of it individually:

AWS4-HMAC-SHA256	Indicates AWS Signature Version 4 (AWS4) and the signing algorithm (HMAC-SHA256).
Credential	Contains your access key and information about the request in the format: `${ACESS_KEY}/${YYYMMDD}/${REGION_SLUG}/s3/aws4_request`
SignedHeaders	A lower-cased list of the names of the request headers used when computing the signature. (e.g. `host;x-amz-acl;x-amz-content-sha256;x-amz-date`)
Signature	A signed hash consisting of a hash of the request body, your secret key, and information about the request (i.e. the canonical request).

The canonical request included in the signature is made up of:

The HTTP request method used.
The path component of the request URI.
The query string parameters included in the request.
The list of request headers and their values, newline separated, lower-cased, and trimmed of whitespace.
The list of header names without their values, sorted alphabetically, lower-cased, and semicolon-separated.
The SHA256 hash of the request body.

This means the following example:

GET /?acl HTTP/1.1

Host: mybucket.s3.ams-nl.scw.cloud
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180914T121549Z

Would be based on the following canonical code:

GET
/
acl=
host:mybucket.s3.ams-nl.scw.cloud
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20180914T121549Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Example Authorization Header

Authorization: AWS4-HMAC-SHA256
Credential=II5JDQBAN3JYM4DNEB6C/20180914/nl-ams/s3/aws4_request,
SignedHeaders=host;x-amz-acl;x-amz-content-sha256;x-amz-date,
Signature=6cab03bef74a80a0441ab7fd33c829a2cdb46bba07e82da518cdb78ac238fda5

Signing Example (Pseudo Code)

canonicalRequest = `
${HTTPMethod}\n
${canonicalURI}\n
${canonicalQueryString}\n
${canonicalHeaders}\n
${signedHeaders}\n
${hashedPayload}
`

stringToSign = "AWS4-HMAC-SHA256" + "\n" +
    date(format=ISO08601) + "\n" +
    date(format=YYYYMMDD) + "/" + ${REGION} + "/" + "s3/aws4_request" + "\n" +
    Hex(SHA256Hash(canonicalRequest))

dateKey = HMAC-SHA256("AWS4" + ${SECRET_KEY}, date(format=YYYYMMDD))
dateRegionKey = HMAC-SHA256(dateKey, ${REGION})
dateRegionServiceKey = HMAC-SHA256(dateRegionKey, "s3")
signingKey = HMAC-SHA256(dateRegionServiceKey, "aws4_request")

signature = Hex(HMAC-SHA256(signingKey, stringToSign))

S3 Object Storage - Object Operations

2018-09-17T00:00:00+00:00

Object Storage - Object Operations

DELETE Object

This operation allows you to remove the null version (if available) of an object and inserts a delete marker, which becomes the latest version of the object.

If there is no no null version, no object is removed.

Example Request

DELETE /sammy.png HTTP/1.1

Host: bucket.s3.nl-ams.scw.cloud
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20170710T194408Z
Authorization: authorization string

Example Response

HTTP/1.1 204 No Content

Date: Mon, 10 Jul 2017 19:44:09 GMT
x-amz-request-id: tx0000000000000027bbc48-005963d908-1268c-ams1
Connection: close

Note: You can also use aws s3api delete-object --bucket BucketName --key ObjectName. If bucket versioning is enabled, the output will contain the version ID of the delete marker.

GET Object

This operation allows you to retrieve objects from Scaleway Object Storage.

Example Request

GET /example.txt HTTP/1.1

Host: bucket.s3.nl-ams.scw.cloud
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180910T190539Z
Authorization: authorization string

Example Response

HTTP/1.1 200 OK

Date: Mon, 10 Sep 2018 19:05:39 GMT
x-amz-request-id: tx00000000000000279f46e-005963d003-1268c-ams1
Content-Type: text/plain
Content-Length: 14
Accept-Ranges: bytes
Last-Modified: Mon, 10 Jul 2017 19:05:09 GMT
Etag: b3a92f49e7ae64acbf6b3e76f2040f5e
Connection: close

Note: You can also use aws s3api get-object --bucket BucketName --key dir/ObjectNameLocation ObjectName. The name of the output file must be the last parameter in the command.

GET Object ACL

This operation allows you to return the access control list (ACL) of an object.

Example Request

GET /myimage.png?acl HTTP/1.1

Host: bucket.s3.nl-ams.scw.cloud.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180910T191224Z
Authorization: authorization string

Example Response

HTTP/1.1 200 OK

Date: Mon, 10 Sep 2018 19:12:24 GMT
x-amz-request-id: tx0000000000000027a42dc-005963d198-1268c-ams1
Content-Type: application/xml
Content-Length: 621
Connection: close

<?xml version="1.0" encoding="UTF-8"?>
  <AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Owner>
      <ID>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</ID>
      <DisplayName>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</DisplayName>
    </Owner>
    <AccessControlList>
      <Grant>
        <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group">
          <URI>http://acs.amazonaws.com/groups/global/AllUsers</URI>
        </Grantee>
        <Permission>READ</Permission>
      </Grant>
      <Grant>
        <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
          <ID>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f6255273</ID>
          <DisplayName>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</DisplayName>
        </Grantee>
        <Permission>FULL_CONTROL</Permission>
      </Grant>
    </AccessControlList>
  </AccessControlPolicy>

Note: You can also use aws s3api get-object-acl --bucket BucketName --key ObjectName.

HEAD Object

The HEAD operation retrieves metadata from an object without returning the object itself. This operation is useful if you’re only interested in an object’s metadata. To use HEAD, you must have READ access to the object.

Example Request

HEAD /my-image.jpg HTTP/1.1
Host: bucket.s3.nl-ams.scw.cloud
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Date: Wed, 28 Oct 2009 22:32:00 GMT
Authorization: authorization string

Example Response

HTTP/1.1 200 OK
x-amz-id-2: ef8yU9AS1ed4OpIszj7UDNEHGran
x-amz-request-id: 318BC8BC143432E5
x-amz-version-id: 3HL4kqtJlcpXroDTDmjVBH40Nrjfkd
Date: Wed, 28 Oct 2009 22:32:00 GMT
Last-Modified: Sun, 1 Jan 2006 12:00:00 GMT
ETag: fba9dede5f27731c9771645a39863328
Content-Length: 434234
Content-Type: text/plain
Connection: close

Note: You can also use aws s3api head-object --bucket BucketName --key ObjectName.

POST Object

This operation allows you to add an object to a specified bucket using HTML forms. POST is an alternate form of PUT that enables browser-based uploads as a way of putting objects in buckets.

Example Request

POST /Neo HTTP/1.1
Content-Length: 4
Host: bucket.s3.nl-ams.scw.cloud
x-amz-content-sha256: 003f0e5fe338b17be8be93fec537764ce199ac50f4e50f2685a753c4cc781747
Date: Wed, 01 Mar  2006 12:00:00 GMT
Authorization: authorization string
Content-Type: text/plain
Expect: the 100-continue HTTP status code

ObjectContent

Example Response

HTTP/1.1 100 Continue
HTTP/1.1 200 OK
x-amz-id-2: LriYPLdmOdAiIfgSm/F1YsViT1LW94/xUQxMsF7xiEb1a0wiIOIxl+zbwZ163pt7
x-amz-request-id: 0A49CE4060975EAC
x-amz-version-id: default
Date: Wed, 12 Oct 2009 17:50:00 GMT
ETag: 1b2cf535f27731c974343645a3985328
Content-Length: 0
Connection: close

PUT Object

This operation allows you to add an object to a bucket.

Example Request

PUT /example.txt HTTP/1.1

Content-Length: 14
Content-Type: text/plain
Host: bucket.s3.nl-ams.scw.cloud
x-amz-content-sha256: 003f0e5fe338b17be8be93fec537764ce199ac50f4e50f2685a753c4cc781747
x-amz-date: 20170710T194605Z
x-amz-meta-s3cmd-attrs:uid:1000/gname:asb/uname:asb/gid:1000/mode:33204/mtime:1499727909/atime:1499727909/md5:fb08934ef619f205f272b0adfd6c018c/ctime:1499713540
x-amz-storage-class: STANDARD
Authorization: authorization string

Example Response

HTTP/1.1 200 OK

Date: Mon, 10 Jul 2017 19:46:06 GMT
x-amz-request-id: tx0000000000000027bd57c-005963d97e-1268c-ams1
Content-Length: 0
Accept-Ranges: bytes
Last-Modified: Mon, 10 Jul 2017 19:05:09 GMT
Etag: fb08934ef619f205f272b0adfd6c018c
Connection: close

Note: You can also use aws s3api put-object --bucket BucketName --key dir-1/ObjectName --body ObjectName.

PUT Object Copy

This operation allows you to create a new object from an existing one.

Example Request

This example copies my-image.jpg into the bucket newbucket, with the key name my-second-image.jpg.

PUT /my-second-image.jpg HTTP/1.1
Host: oldbucket.s3.nl-ams.scw.cloud
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Date: Wed, 28 Oct 2018 22:18:00 GMT
x-amz-copy-source: /bucket/my-image.jpg
Authorization: authorization string

Example Response

HTTP/1.1 200 OK
x-amz-id-2: eftixk72aD6Ap51TnqcoF8eFidJG9Z/2mkiDFu8yU9AS1ed4OpIszj7UDNEHGran
x-amz-request-id: 318BC8BC148832E5
x-amz-copy-source-version-id: 3/L4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY+MTRCxf3vjVBH40Nr8X8gdRQBpUMLUo
x-amz-version-id: QUpfdndhfd8438MNFDN93jdnJFkdmqnh893
Date: Wed, 28 Oct 2018 22:18:00 GMT
Connection: close

<CopyObjectResult>
   <LastModified>2018-10-28T22:18:00</LastModified>
   <ETag>9b2cf535f27731c974343645a3985328</ETag>
 </CopyObjectResult>

Note: You can also use aws s3api copy-object --copy-source bucket-1/test.txt --key test.txt --bucket bucket-2.

PUT Object ACL

This operation allows you to use the acl subressource to set the access control list (ACL) permissions for an object that already exists in a bucket. The --acl subcommand possible values are:

private
public-read
public-read-write
authenticated-read
bucket-owner-read
bucket-owner-full-control

Example Request

PUT /sammy.png?acl HTTP/1.1

content-type:application/xml
Host: bucket.s3.nl-ams.scw.cloud
x-amz-content-sha256:c0bd9ba784be78d4f38bbc1e3b0da2de2e7a8f4ee259b3b840369cf00a78dad2
x-amz-date:20170710T192142Z
Authorization: authorization string

<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Owner>
    <ID>6174283</ID>
  </Owner>
  <AccessControlList>
    <Grant>
      <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
        <ID>6174283</ID>
      </Grantee>
      <Permission>FULL_CONTROL</Permission>
    </Grant>
  </AccessControlList>
</AccessControlPolicy>

Example Response

HTTP/1.1 200 OK

Date: Mon, 10 Jul 2017 19:21:42 GMT
x-amz-request-id: tx0000000000000027aafc9-005963d3c6-1268c-ams1
Content-Type: application/xml
Content-Length: 0
Connection: close

Note: You can also use aws s3api put-object-acl --bucket BucjetName --key file.txt --acl private.

Multipart

Multipart upload allows you to upload a single object as a set of parts. Each part is a contiguous portion of the object’s data. You can upload these object parts independently and in any order. If transmission of any part fails, you can retransmit that part without affecting other parts. After all parts of your object are uploaded, Scaleway Object Storage assembles these parts and creates the object.

Example Request

This operation initiates a multipart upload for the example-object object.

POST /example-object?uploads HTTP/1.1
Host: bucketname.s3.scw.cloud
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Date: Mon, 1 Nov 2018 12:34:56 GMT
Authorization: authorization string

Example Response

HTTP/1.1 200 OK
x-amz-id-2: Uuag1LuByRx9e6j5Onimru9pO4ZVKnJ2Qz7/C1NPcfTWAtRPfTaOFg==
x-amz-request-id: 656c76696e6727732072657175657374
Date:  Mon, 1 Nov 2018 12:34:56 GMT
Content-Length: 197
Connection: keep-alive

<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Bucket>example-bucket</Bucket>
  <Key>example-object</Key>
  <UploadId>VXBsb2FkIElEIGZvciA2aWWpbmcncyBteS1tb3ZpZS5tMnRzIHVwbG9hZA</UploadId>
</InitiateMultipartUploadResult>

Note: You can also use aws s3api create-multipart-upload --bucket my-bucket --key 'multipart/01'.

Generating the Authorization String

You can generate the authorization string, following the authorization string procedure.

Back to the S3 API overview

S3 Object Storage - Bucket Operations

2018-09-14T00:00:00+00:00

Object Storage - Bucket Operations

DELETE Bucket

This operation allows you to delete the bucket mentioned in the command.

Example Request

DELETE / HTTP/1.1

Host: bucketname.s3.nl-ams.scw.cloud
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20170710T181321Z
Authorization: authorization string

This request will delete the bucket named BucketName. Make sure the bucket is empty to avoid loss of data when you delete it.

Example Response

If the operation is successful no output will be returned.

HTTP/1.1 204 No Content
Date: Mon, 10 Jul 2017 18:13:21 GMT
Connection: close

Note: You can also use aws s3api delete-bucket --bucket my-bucket --region us-east-1.

GET Bucket

This operation allows to return some or all (up to 1,000) of the objects in a bucket.

You can use the Example parameters as selection criteria to return a subset of the objects in a bucket.

Example Request

GET / HTTP/1.1

Host: bucketname.s3.nl-ams.scw.cloud
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20170711T183940Z
Authorization: authorization string

Example Response

HTTP/1.1 200 OK

x-amz-request-id: tx000000000000002ba2427-0059651b6d-1268c-ams1
Date: Mon, 10 Jul 2017 17:31:43 GMT
Content-Length: 523
Content-Type: text/plain
Connection: close

<?xml version="1.0" encoding="UTF-8"?>
  <ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Owner>
      <ID>6174283</ID>
      <DisplayName>6174283</DisplayName>
    </Owner>
    <Buckets>
      <Bucket>
        <Name>static-images</Name>
        <CreationDate>2017-06-23T18:37:48.157Z</CreationDate>
      </Bucket>
    <Buckets>
      <Bucket>
        <Name>log-files</Name>
        <CreationDate>2017-06-23T18:37:48.157Z</CreationDate>
      </Bucket>
    </Buckets>
  </ListAllMyBucketsResult>

Note: You can also use aws s3api get-bucket --bucket my-bucket.

GET Bucket ACL

This operation allows you to get the ACL of a bucket.

Example Request

This request triggers the display of the access control list of the bucket BucketName.

GET /?acl HTTP/1.1

Host: bucketname.s3.nl-ams.scw.cloud
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180910T174434Z
Authorization: authorization string

Example Response

HTTP/1.1 200 OK

Date: Mon, 10 Sep 2018 17:44:35 GMT
x-amz-request-id: tx000000000000002764fa6-005963bd03-1268c-ams1
Content-Type: application/xml
Content-Length: 621
Connection: close

<?xml version="1.0" encoding="UTF-8"?>
  <AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Owner>
      <ID>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</ID>
      <DisplayName>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</DisplayName>
    </Owner>
    <AccessControlList>
      <Grant>
        <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group">
          <URI>http://acs.amazonaws.com/groups/global/AllUsers</URI>
        </Grantee>
        <Permission>READ</Permission>
      </Grant>
      <Grant>
        <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
          <ID>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</ID>
          <DisplayName>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</DisplayName>
        </Grantee>
        <Permission>FULL_CONTROL</Permission>
      </Grant>
    </AccessControlList>
  </AccessControlPolicy>

Note: You can also use aws s3api get-bucket-acl --bucket my-bucket.

GET Bucket location

This operation allows you to display the location constraint of a bucket.

Example Request

GET /?location HTTP/1.1

Host: bucketname.s3.nl-ams.scw.cloud
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180908T171432Z
Authorization: authorization string

Example Response

No output is returned.

Note: You can also use aws s3api get-bucket-location --bucket my-bucket.

HEAD Bucket

This operation can be used to determine if a bucket exists and you have permission to access it.

Example Request

HEAD / HTTP/1.1     
Host: bucket.s3.nl-ams.scw.cloud
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Date: date
Authorization: authorization string

Example Response

If the bucket exists and you have access to it, no output is returned. Otherwise, an error message will be shown.

Note: You can also use aws s3api head-bucket --bucket BucketName.

PUT Bucket

This operation allows you to create a new bucket.

Example Request

PUT / HTTP/1.1
Host: bucketname.s3.nl-ams.scw.cloud
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Content-Length: 621
x-amz-acl: private
Date: Wed, 19 Sep 2018 15:21:16 GMT
Authorization: authorization string

Example Response

HTTP/1.1 200 OK
x-amz-id-2: YgIPIfBiKa2bj0KMg95r/0zo3emzU4dzsD4rcKCHQUAdQkf3ShJTOOpXUueF6QKo
x-amz-request-id: 236A8905248E5A01
Date: Wed, 01 Mar  2006 12:00:00 GMT
Location: http://s3.nl-ams.scw.cloud/bucketname
Content-Length: 621
Connection: close

Note: You can also use aws s3 mb s3://bucketname with the command line tools to create a new bucket.

PUT Bucket ACL

This operation allows you to configure the Access Control List of a Bucket.

Example Request

PUT /?acl HTTP/1.1

content-type:application/xml
Host: bucketname.s3.nl-ams.scw.cloud
x-amz-content-sha256: 724483e3830b19d6960345c484fb7904b26e8f2fb34a6c002fa779353b68c8d8
x-amz-date: 20180910T183709Z
Authorization: authorization string

<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Owner>
    <ID>6174283</ID>
  </Owner>
  <AccessControlList>
    <Grant>
      <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
        <ID>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</ID>
      </Grantee>
      <Permission>FULL_CONTROL</Permission>
    </Grant>
    <Grant>
      <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group">
        <URI>http://acs.amazonaws.com/groups/global/AllUsers</URI>
      </Grantee>
      <Permission>READ</Permission>
    </Grant>
  </AccessControlList>
</AccessControlPolicy>

Example Response

No output is returned.

Note: You can also use aws s3api put-bucket-acl --bucket BucketName --acl private.

Generating the Authorization String

You can generate the authorization string, following the authorization string procedure.

Back to the S3 API overview

S3 Object Storage - Common Operations

2018-09-13T00:00:00+00:00

Object Storage - Common Operations

GET Service

The GET Service request returns a list of all buckets owned by the authenticated user that sent sent the request.

The GET Service operation doesn’t list buckets created by other users. It also doesn’t list buckets for anonymous requests.

Example Request

GET / HTTP/1.1
Host: s3.nl-ams.scw.cloud
Date: Wed, 19 Sep 2018 15:00:42 GMT
Authorization: authorization string

Example Response

<?xml version='1.0' encoding='UTF-8'?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Owner>
    <ID>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</ID>
    <DisplayName>a6a05c73-fa53-46a4-9ea1-e53b4f625527:a6a05c73-fa53-46a4-9ea1-e53b4f625527</DisplayName>
  </Owner>
  <Buckets>
    <Bucket>
      <Name>Photos</Name>
      <CreationDate>2018-09-13T13:13:29.000Z</CreationDate>
    </Bucket>
    <Bucket>
      <Name>Documents</Name>
      <CreationDate>2018-09-13T13:06:34.000Z</CreationDate>
    </Bucket>
  </Buckets>
</ListAllMyBucketsResult>

Note: You can also use aws s3 ls with the command line tools to get a listing of your buckets.

Generating the Authorization String

You can generate the authorization string, following the authorization string procedure.

Back to the S3 API overview

S3 Object Storage API

2018-09-13T00:00:00+00:00

S3 API Feature Availability

Note: When you use Object Storage directly with the API, you must generate an Authentication Signature v4 beforehand.

We support currently a subset of S3 and the following operations are available:

Common Operations

✔ = Feature available / ✘ = Feature not yet available

Feature	Status	Remark
GET Service	✔	Returns a list of all buckets

Bucket Operations

✔ = Feature available / ✘ = Feature not yet available

Feature	Status	Remark
DELETE Bucket	✔	Deletes bucket
DELETE Bucket analytics	✘
DELETE Bucket cors	✘
DELETE Bucket encryption	✘
DELETE Bucket inventory	✘
DELETE Bucket lifecycle	✘
DELETE Bucket metrics	✘
DELETE Bucket policy	✘
DELETE Bucket replication	✘
DELETE Bucket tagging	✘
DELETE Bucket website	✘
GET Bucket	✔	Lists objects
GET Bucket accelerate	✘
GET Bucket ACL	✔	Returns the bucket ACL
GET Bucket analytics	✘
GET Bucket cors	✘
GET Bucket encryption	✘
GET Bucket inventory	✘
GET Bucket lifecycle	✘
GET Bucket location	✔	Returns the region where the bucket is
GET Bucket logging	✘
GET Bucket metrics	✘
GET Bucket notification	✘
GET Bucket Object versions	✘
GET Bucket policy	✘
GET Bucket replication	✘
GET Bucket requestPayment	✘
GET Bucket tagging	✘
GET Bucket versioning	✘
GET Bucket website	✘
HEAD Bucket	✔	Checks if the bucket exists
List Bucket Analytics Configurations	✘
List Bucket Inventory Configurations	✘
List Bucket Metrics Configurations	✘
PUT Bucket	✔	Creates bucket
PUT Bucket accelerate	✘
PUT Bucket ACL	✔	Configures the Access Control List of a bucket
PUT Bucket analytics	✘
PUT Bucket cors	✘
PUT Bucket encryption	✘
PUT Bucket inventory	✘
PUT Bucket lifecycle	✘
PUT Bucket inventory	✘
PUT Bucket logging	✘
PUT Bucket notification	✘
PUT Bucket policy	✘
PUT Bucket replication	✘
PUT Bucket requestPayment	✘
PUT Bucket tagging	✘
PUT Bucket versioning	✘
PUT Bucket website	✘

Object Operations

✔ = Feature available / ✘ = Feature not yet available

Feature	Status	Remark
DELETE Multiple Objects	✘
DELETE Object	✔	Deletes an object
DELETE Object tagging	✘
GET Object	✔	Retrieves an object
GET Object ACL	✔	Gets the ACL of an object
GET Object tagging	✘
GET Object torrent	✘
HEAD Object	✔	Gets object metadata
OPTIONS object	✘
POST Object	✔	Adds an object to a bucket by using HTML forms
POST Object restore	✘
PUT Object	✔	Adds an object to a bucket
PUT Object copy	✔	Copies an object
PUT Object ACL	✔	Adds an ACL to an object
PUT Object tagging	✘
SELECT Object Content (Preview)	✘
Multipart	✔	Initiates, aborts, completes, lists, uploads, uploads copy

Backup PostgreSQL databases with Barman

2018-08-29T00:00:00+00:00

Barman Overview

Barman is an open source administration tool for disaster recovery of PostgreSQL servers. It is written in Python and allows to perform remote backups of multiple servers in business critical environments.

The tool creates backups based on the PostgreSQL logs and not by performing a simple SQL dump.

Requirements:

You have an account and are logged into cloud.scaleway.com

You have two servers running a PostgreSQL database on Ubuntu Bionic or Xenial

You have configured your SSH Key

You have sudo privileges or access to the root user.

Installing Barman

Note: This tutorial requires that you have already set up two servers running a PostgreSQL database. They will be called pgsql and pgsql-backup in this tutorial.

As you will backup your databases with Barman, it is recommended to host the application on a different instance than your databases. Ideally in a geographically distant data center. In this tutorial, we install Barman on a second server to migrate the database from one to the other server for backup and redundancy purposes.

Barman is available in the official APT repositories of Ubuntu. Connect to the second server, called pgsql-backup and install the tool:

1 . Update the APT packet cache and upgrade the already installed packages to their latest version:

sudo apt-get update && sudo apt-get upgrade -y

2 . Install Barman:

sudo apt-get install barman

Configuring Barman

In order for Barman to copy data between the two servers, some pre-work is required:

Creating a dedicated PostgreSQL user on `pgsql`

As Barman has been installed on the second server (pgsql-backup), it is required to create a database user for the tool on the first server (pgsql).

1 . Connect on the server pgsql and log into the postgres account:

sudo -i -u postgres

2 . Create the user and enter twice the desired password. When asked if the account should have superuser privileges, type y and press Enter:

createuser --interactive -P barman

3 . Enable the connection to the server from pgsql-backup by editing the file /etc/postgresql/9.5/postgresql.conf. Change the value of listen_addresses to the new server, or use an asterisk to enable the connection from any IP.

listen_addresses = 'localhost, pgsql-backup'

4 . Restart the service

sudo service postgresql restart

Test the connection from pgsql-backup by running a SQL query:

psql -c 'SELECT version()' -U barman -h pgsql postgres

Creating the .pgpass file on pgsql-backup

During the installation of Barman on the server pgsql-backup, a system user account called barman was created.

1 . Set a password for the user:

sudo passwd barman

2 . Enter the user account:

sudo -i -u barman

3 . Create the .pgpass file for the barman user:

echo "pgsql:5432:*:barman:password" >> ~/.pgpass

Note: The data format for the .pgpass file is hostname:port:database:username:password.
If an asterisk is used in any of the first four fields, it will correspond to everything. The username refers to the PostgreSQL user that was created previously, not to the linux user account. You may refer to the official documentation for more information about this file.

The file .pgpass in a user’s home directory can contain passwords to be used if the connection requires a password (and no password has been specified otherwise).

Setting up key-based authentication

To perform backups without manual authentication, generate an SSH key that will be used for authentication:

In order to perform backups without user intervention, we need to set up and copy SSH keys for passwordless authentication. Barman makes use of this method to copy data through rsync.

1 . Connect to the server pgsql-backup, switch to user barman and generate the keys

ssh-keygen -t rsa

2 . Copy the key to the user account postgres on pgsql:

ssh-copy-id postgres@pgsql

The following message will appear if the transfer of the key has been successful:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'postgres@pgsql'"
and check to make sure that only the key(s) you wanted were added.

3 . Barman also requires access to the postgres account on the server pgsql-backup. Copy the key into the directory of the postgres user and test the connection:

ssh-copy-id postgres@localhost
ssh postgres@localhost -C true

4 . Once this is done, log in as postgres user on pgsql and generate an SSH key:

ssh-keygen -t rsa

5 . Copy the generated key to the list of authorized keys of the user barman on pgsql-backup:

ssh-copy-id barman@pgsql-backup

6 . Test the connection to the server:

ssh barman@pgsql-backup -C true

Configuring Barman

The configuration of Barman is done on the server pgsql-backup.

1 . Create the directory /etc/barman.d:

mkdir /etc/barman.d

2 . Open the file /etc/barman.conf and remove the leading semicolon (;) from the following line:

;configuration_files_directory = /etc/barman.d

3 . Enter the directory /etc/barman.d and create a file pgsql.conf with the following content:

[pgsql]
description =  "Old PostgreSQL server"
conninfo = host=10.55.66.77 user=barman dbname=Our_Database
ssh_command = ssh postgres@pgsql
retention_policy = RECOVERY WINDOW OF 2 WEEKS

The information in this file are self-explanatory. The retention_policy defines the time frame during which backups are kept. This can be set to your needs. For more information about this file you may refer to the official documentation.

Configuring PostgreSQL

1 . Connect to pgsql and add this line to the file /etc/postgresql/9.5/main/pg_hba.conf:

host    all             all             10.55.66.88/32         trust

Where 10.55.66.77 should be replaced with the IP of pgsql-backup.

2 . Open the file /etc/postgresql/9.5/main/postgresql.conf and edit the values as shown:

wal_level = archive
archive_mode = on
archive_command = 'rsync -a %p barman@10.55.66.77:/var/lib/barman/pgsql/incoming/%f'

3 . Restart the postgresql service:

sudo systemctl restart postgresql

5 . Check if the connection to the server pgsql works from Barman, by runnning the command barman check pgsql after logging into the barman user account on pgsql-backup:

barman@pgsql-backup:~$ barman check pgsql
Server pgsql:
    PostgreSQL: OK
    archive_mode: OK
    wal_level: OK
    archive_command: OK
    continuous archiving: OK
    directories: OK
    retention policy settings: OK
    backup maximum age: OK (no last_backup_maximum_age provided)
    compression settings: OK
    minimum redundancy requirements: OK (have 0 backups, expected at least 0)
    ssh: OK (PostgreSQL server)
    not in recovery: OK

If all values are OK, you are ready to make your first backup.

Performing Backups

1 . Start a backup with the command barman backup pgsql. You will see an output like the following:

barman@pgsql-backup:~$ barman backup pgsql
Starting backup for server pgsql in /var/lib/barman/pgsql/base/20180911T115925
Backup start at xlog location: 0/2000028 (000000010000000000000002, 00000028)
Copying files.
Copy done.
Asking PostgreSQL server to finalize the backup.
Backup size: 20.7 MiB
Backup end at xlog location: 0/20000C0 (000000010000000000000002, 000000C0)
Backup completed

2 . To view a list of all backups available of pgsql, use the command

barman list-backup pgsql

3 . To view the details about a backup, use the following command:

barman show-backup pgsql backup_id

The backup_id is visible from the backup list and in the form like this example 20180911T115925.

Restoring Backups

It is possible to import the backup in the PostgreSQL server running on pgsql-backup.

1 . Stop the postgresql service:

sudo systemctl stop postgresql

2 . Load the backup into the database with the following command:

barman recover --remote-ssh-command "ssh postgres@localhost" pgsql 20180911T115925 /var/lib/postgresql/9.5/main

You will see an output like the following:

Starting remote restore for server pgsql using backup 20180911T115925
Destination directory: /var/lib/postgresql/9.5/main
Copying the base backup.
Copying required WAL segments.
Generating archive status files
Your PostgreSQL server has been successfully prepared for recovery!

3 . Once the backup has been imported, restart the postgresql service:

sudo systemctl start postgresql

Your database has now successfully migrated to your new database server.

Automatizing Backups

It is possible to run Barman automatically to backup the content of your database regularly. This can be done easily by setting up a cron job on pgsql-backup.

1 . Switch into the barman user:

sudo -i -u barman

2 . Edit the crontab of the user:

crontab -e

3 . To run a backup daily at 3:30, add the following line to the cron tab and save it:

30 03 * * * /usr/bin/barman backup pgsql

How to Configure Hadoop on Ubuntu Xenial

2018-08-29T00:00:00+00:00

Hadoop Overview

Hadoop is an open source framework. It is provided by Apache to process and analyze very huge volume of data. It is written in Java and currently used by Google, Facebook, LinkedIn, Yahoo, Twitter etc. Similar to data residing in a local file system of personal computer system, in Hadoop, data resides in a distributed file system which is called as a Hadoop Distributed File system. Hadoop is suitable for:

Big Data Analysis: Big Data tends to be distributed and unstructured in nature. Hadoop processes the logic (not the actual data) that flows to the computing nodes, less network bandwidth is consumed. This concept is called as data locality concept which helps increase efficiency of Hadoop based applications.
Scalability: Hadoop clusters can easily be scaled to any extent by adding additional cluster nodes
Fault Tolerance: Hadoop ecosystem has a provision to replicate the input data on to other cluster nodes. That way, in the event of a cluster node failure, data processing can still proceed by using data stored on another cluster node.

Hadoop clusters are relatively complex to set up, so the project includes a stand-alone mode which is suitable for learning about Hadoop, performing simple operations, and debugging.

Hadoop is comprised of four main layers:

Hadoop Common is the collection of utilities and libraries that support other Hadoop modules.
HDFS, which stands for Hadoop Distributed File System, is responsible for persisting data to disk.
YARN, short for Yet Another Resource Negotiator, is the “operating system” for HDFS.
MapReduce is the original processing model for Hadoop clusters. It distributes work within the cluster or map, then organizes and reduces the results from the nodes into a response to a query. Many other processing models are available for the 3.x version of Hadoop.

For more information, refer to the official Apache Hadoop documentation.

Requirements:

You have an account and are logged into cloud.scaleway.com

You have configured your SSH Key

You have sudo privileges or access to the root user.

Installing the Default JRE/JDK

1 . Update the package index

apt-get update

2 . Install Java. Specifically, this command will install the Java Runtime Environment (JRE).

apt-get install default-jre

There is another default Java installation called the JDK (Java Development Kit). The JDK is usually only needed if you are going to compile Java programs or if the software that will use Java specifically requires it.

The JDK does contain the JRE, so there are no disadvantages if you install the JDK instead of the JRE, except for the larger file size. You can install the JDK with the following command: apt-get install default-jdk

3 . Once the installation is complete, let’s check the version.

java -version

which returns

OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

Installing Hadoop in Standalone Mode

Visit the Apache Hadoop Releases page to select the most recent stable release. We will install Hadoop version 3.1.1

1 . Copy the link to the release binary.

2 . Use wget to fetch it:

wget http://mirrors.standaloneinstaller.com/apache/hadoop/common/hadoop-3.1.1/hadoop-3.1.1.tar.gz

In order to make sure that the file we downloaded has not been altered, we’ll do a quick check using SHA-256. Return to the releases page, then click and copy the link to the checksum file for the release binary you downloaded:

3 . Copy the link displayed in the browser and use wget on our server to download the file

wget https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-3.1.1/hadoop-3.1.1.tar.gz.mds

4 . Run the verification

shasum -a 256 hadoop-3.1.1.tar.gz

which returns

f837fe260587f71629aad1f4fb6719274e948111dc96ffc5a8e26f27deac5602  hadoop-3.1.1.tar.gz

5 . Compare this value with the SHA-256 value in the .mds file:

cat hadoop-3.1.1.tar.gz.mds

The output of the command we ran against the file we downloaded from the mirror should match the value in the file we downloaded from apache.org.

6 . Use the tar command with the -x flag to extract, -z to uncompress, -v for verbose output, and -f to specify that we’re extracting from a file.

tar -xzvf hadoop-3.1.1.tar.gz

7 . Move the extracted files into /usr/local. Change the version number to match the version you downloaded.

mv hadoop-3.1.1 /usr/local/hadoop

Configuring Hadoop’s Java Home

Hadoop requires that you set the path to Java, either as an environment variable or in the Hadoop configuration file.

The path to Java, /usr/bin/java is a symbolic link to /etc/alternatives/java, which is a symbolic link to the default Java binary.

Use readlink command with the -f option to follow every symbolic link in every part of the path, recursively. Then, use sed to trim bin/java from the output to give us the correct value for JAVA_HOME.

To find the default Java path, launch:

readlink -f /usr/bin/java | sed "s:bin/java::"

which returns

/usr/lib/jvm/java-8-openjdk-amd64/jre/

You can copy this output to set Hadoop’s Java home to this specific version, which ensures that if the default Java changes, this value will not. Or, you can use the readlink command dynamically in the file so that Hadoop will automatically use whatever Java version is set as the system default.

1 . Open hadoop-env.sh:

sudo nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh

Note: With respect to Hadoop, the value of JAVA_HOME in hadoop-env.sh overrides any values that are set in the environment by /etc/profile or in a user’s profile.

2 . Select one of the following options:

Option 1: Static Value

. . .
#export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=//usr/lib/jvm/java-8-openjdk-amd64/jre/
. . .

Option 2: Dynamic Value

. . .
#export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")
 . . .

Running Hadoop

Run Hadoop

/usr/local/hadoop/bin/hadoop

which returns

Usage: hadoop [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
 or    hadoop [OPTIONS] CLASSNAME [CLASSNAME OPTIONS]
  where CLASSNAME is a user-provided Java class

  OPTIONS is none or any of:

buildpaths                       attempt to add class files from build tree
--config dir                     Hadoop config directory
--debug                          turn on shell script debug mode
--help                           usage information
hostnames list[,of,host,names]   hosts to use in slave mode
hosts filename                   list of hosts to use in slave mode
loglevel level                   set the log4j level for this command
workers                          turn on worker mode
...

The help means we’ve successfully configured Hadoop to run in stand-alone mode. To make sure Hadoop is running properly, we will use an example of a MapReduce program.

MapReduce Example

MapReduce is a programming framework that allows us to perform distributed and parallel processing on large data sets in a distributed environment.

MapReduce consists of two distinct tasks – Map and Reduce. As the name MapReduce suggests, reducer phase takes place after mapper phase has been completed.

1 . Create a directory called input in our home directory

mkdir ~/input

2 . Copy Hadoop’s configuration files into it to use those files as our data.

cp /usr/local/hadoop/etc/hadoop/*.xml ~/input

3 . Download Hadoop example

wget  https://jar-download.com/cache_jars/org.apache.hadoop/hadoop-mapreduce-examples/3.0.3/jar_files.zip

4 . Unzip the file

unzip jar_files.zip

5 . Move the jar_files to /usr/local/hadoop/share/hadoop/mapreduce/hadoop/

mv hadoop-mapreduce-examples-3.0.3.jar /usr/local/hadoop/share/hadoop/mapreduce/.

Important: Verify that your /etc/hosts file is properly setup and contains at least the following: 127.0.0.1 localhost <your hostname>

6 . Use the following command to run the MapReduce hadoop-mapreduce-examples program. We’ll invoke its grep program which counts the matches of a literal word or regular expression (e.g., allowed).

/usr/local/hadoop/bin/hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar grep ~/input ~/grep_example 'allowed[.]*'

Results are stored in the output directory and can be checked by running cat on the output directory:

cat ~/grep_example/*

which returns

19  allowed
1   allowed

Installation of PostgreSQL

2018-08-24T00:00:00+00:00

PostgreSQL Overview

PostgreSQL is a mature and advanced open source relational database system. With more than 30 years of active development, the software has earned a strong reputation for reliability, feature robustness, and performance.

It runs on all major operating systems, including Linux, different Unix variants, MacOS X and Windows. This tutorial describes how to run it on an Ubuntu based server.

Requirements:

You have an account and are logged into cloud.scaleway.com

You have an Ubuntu Bionic (18.04) server

You have configured your SSH Key

You have sudo privileges or access to the root user.

Installing PostgreSQL

PostgreSQL provides an APT mirror with the latest builds of the software.

1 . Add the Postgres mirror, update the APT packet cache and install the required packages:

sudo apt-get install wget ca-certificates
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install postgresql-10 postgresql-contrib

Using PostgreSQL Roles and Databases

Postgres uses a concept similar to regular Unix-style accounts, called roles. However Postgres does not distinguish between users and groups and uses the more flexible term role.

By default Postgres is configured to use ident authentication, which means matching a role with a Unix/Linux system account

If a role exists in Postgres a system user with the same name is able to sign in as that role.

During the installation a user called postgres has been created and is used to log into that account having the default Postgres role.

Logging into PostgreSQL from the actual account

You can access the postgres account directly and the PostgreSQL prompt with sudo.

sudo -u postgres psql

To quit the prompt, type:

\q

Logging into PostgreSQL from the postgres Account

It is also possible to login from the postgres account by using an intermediate bash.

Switch into the postgres account:

sudo -i -u postgres

To access the command prompt, type:

psql

To exit the prompt tpye:

\q

Creating New Roles

By default, only the postgres role is configured within the database. Adding of new roles can be done with the command createrole.

You can either create the new role by using sudo from your regular account:

sudo -u postgres createuser --interactive

If you are logged as the postgres user, launch the command as following:

createuser --interactive

You will be asked to enter the name of the new role and some additional questions:

root@pgsql:~# sudo -u postgres createuser --interactive
Enter name of role to add: bill
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) y
Shall the new role be allowed to create more new roles? (y/n) n

More information about the usage of the createuser command is available in the official documentation.

Creating New Databases

The authentication system of Postgres assumes by default that a database exists with the same name as the role .

This means if you have created a role bill in the previous step, Postgres will by attempt by default to connect to a database bill.

From your regular user, you can launch the command with sudo:

sudo -u postgres createdb bill

If you are logged directly as the postgres user, run the following command:

createdb bill

For more information, you may consult the official documentation.

Connecting to a Postgres Prompt with the new Role

The ident based authentication requires a matching Linux user with the same name as the Postgres role and database. If it does not exist yet, create the user with the following command:

sudo adduser bill

If the previous steps have been followed, login now into the database with the following command:

sudo -u bill psql

Creating and Querying Databases

To understand the basic SQL syntax, we create a phonebook that contains a user personal details:

1 . Connect to the Postgres prompt.

2 . Create a new table called phonebook, that contains fields for the phone number, first and last name and the city:

CREATE TABLE phonebook(phone VARCHAR(32), firstname VARCHAR(32), lastname VARCHAR(32), city VARCHAR(64));

3 . Insert data into the table. You may repeat this command if you want to add more than one set of data:

INSERT INTO phonebook(phone, firstname, lastname, city) VALUES('+33 1 23 45 67 89', 'Paul', 'Smith', 'Paris, FR');

4 . Select data from the table. Run the following command to retrieve all data sets, sorted by the lastname

SELECT * FROM phonebook ORDER BY lastname;

This will give you an result like the following:

bill=# SELECT * FROM phonebook ORDER BY lastname;
       phone       | firstname | lastname |     city
-------------------+-----------+----------+---------------
 +33 1 23 45 67 89 | Paul      | Smith    | Paris, FR
 +33 9 87 65 43 21 | Jessica   | White    | Marseille, FR
(2 rows)

To retrieve all entries in the phonebook where the family name is Smith, run:

SELECT * FROM phonebook WHERE lastname = 'Smith';

You will get an result as the following:

bill=# SELECT * FROM phonebook WHERE lastname = 'Smith';
       phone       | firstname | lastname |   city
-------------------+-----------+----------+-----------
 +33 1 23 45 67 89 | Paul      | Smith    | Paris, FR
(1 row)

5 . Update the city of a user:

UPDATE phonebook SET city = 'London, UK', phone = '+44 7123 456789' WHERE firstname = 'Paul' AND lastname = 'Smith';

6 . Delete the data of a user:

DELETE FROM phonebook WHERE firstname = 'Paul' AND lastname = 'Smith';

How to Configure Flask on Ubuntu Xenial

2018-08-23T00:00:00+00:00

Flask Overview

Flask is a web application framework written in Python. Flask is easy to get started with as a beginner because there is little boilerplate code for getting a simple app up and running.

Requirements:

You have an account and are logged into cloud.scaleway.com

You have an Ubuntu Xenial server

You have configured your SSH Key

Installing Python

If you don’t have Python installed on your computer, download the installer from the Python official website.

To make sure your Python installation is functional, you can open a terminal window and type python3, or if that does not work, just python.

python3

which returns

Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.

To exit the interactive prompt, you can type exit() and press Enter. On the Linux and Mac OS X versions of Python you can also exit the interpreter by pressing Ctrl-D. On Windows, the exit shortcut is Ctrl-Z followed by Enter.

Setting up Flask

In Python, packages such as Flask are available in a public repository, from where anybody can download and install them. The official Python package repository is called PyPI, which stands for Python Package Index.

1 . (Optional) If you do not have pip installed, launch

apt install python-pip

2 . Install Flask

pip install Flask

3 . Create a folder called FlaskApp.

mkdir FlaskApp

4 . Navigate to the FlaskApp folder and create a file called app.py.

cd FlaskApp
nano app.py

5 . Paste the following content to the app.py file.

from flask import Flask
app = Flask(__name__)
@app.route("/")
def main():
    return "Welcome to the first Flask App!"
if __name__ == "__main__":
    app.run(host='0.0.0.0')

6 . Save the changes and execute app.py.

~/FlaskApp# python app.py

which returns

* Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production
   environment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)

7 . Open you web browser and type the server_ip:port

It should display

Note: If you run the server you will notice that the server is only accessible from your own computer, not from any other in the network. This is the default because in debugging mode a user of the application can execute arbitrary Python code on your computer. If you have the debugger disabled or trust the users on your network, you can make the server publicly available simply by adding –host=0.0.0.0 to the command line or editting your app.py to match app.run(host='0.0.0.0')` This tells your operating system to listen on all public IPs.

Creating URL routes

URL Routing makes URLs in your Web app easy to remember.

We will create several URLs routes:

/hello
/writers/
/writers/tutorials/

1 . Copy the code below and save it as app.py

app = Flask(__name__)

@app.route("/")
def index():
    return "Index!"

@app.route("/hello")
def hello():
    return "Hello Cloud Riders!"

@app.route("/writers")
def writers():
    return "Thanks to the Scaleway Tech Writers!"

@app.route("/writers/<string:name>/")
def getWriters(name):
    return name

if __name__ == "__main__":
    app.run(host='0.0.0.0')

2 . Restart the application using

python app.py

Try the URLs in your browser:

http://server_ip:5000/
http://server_ip:5000/hello
http://server_ip:5000/writers

Each route should display what is defined in the app.py above, for instance http://server_ip:5000/hello displays:

Styling Flask Pages

In Flask, templates are written as separate files, stored in a templates folder that is inside the application package. So after making sure that you are in the FlaskApp directory, create the directory where templates will be stored.

As explained above, we created a new application called hello.py running on port 80, with the following configuration:

from flask import Flask, flash, redirect, render_template, request,
session, abort

app = Flask(__name__)

@app.route("/")
def index():
    return "Flask App!"

@app.route("/hello/<string:name>/")
def hello(name):
    return render_template(
        'test.html',name=name)

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=80)

1 . Create a directory called templates

mkdir templates

2 . Open a test.html file and paste the following

nano test.html

3 . Paste the following


{% extends "layout.html" %}
{% block body %}

<div class="block1">
<h1>Hello {{name}}!</h1>
  <h2>Discover a New Cloud Experience </h2>
  <p>
"The Disruptive Cloud Computing Platform: Deploy SSD Cloud Servers in
seconds!!"
  </p>
<img src="https://www.universfreebox.com/UserFiles/image/Capture%20d%E2%0%9e%CC%81cran%202017-12-01%20a%CC%80%2012_17_02.png">
</div>
{% endblock %}

4 . Open a layout.html file and paste the following

<html>
<head>
    <title>Website</title>
<style>
@import url(http://fonts.googleapis.com/css?family=Amatic+SC:700);

body{
    text-align: center;
}
h1{
    font-family: 'Amatic SC', cursive;
    font-weight: normal;
    color: #FD6C9E;
    font-size: 2.5em;
}

</style>

</head>
<body>

 {% block body %}{% endblock %}

5 . Launch the application

python hello.py

the application should display

Passing Variables

Let’s try and display random Scaleway catch phrase instead of always the same one. We will need to pass both the name variable and the quote variable.

1 . In the templates directory, edit the test.html to match the following


{% extends "layout.html" %}
{% block body %}

<div class="block1">
<h1>Hello {{name}}!</h1>
  <h2>Discover a New Cloud Experience </h2>
  <p>
{{quote}}
  </p>
<img src="https://www.universfreebox.com/UserFiles/image/Capture%20d%E2%0%9e%CC%81cran%202017-12-01%20a%CC%80%2012_17_02.png">
</div>
{% endblock %}

2 . Save and exit

3 . In the application called hello.py update the configuration to look like this:

from flask import Flask, flash, redirect, render_template, request, session, abort
from random import randint

app = Flask(__name__)

@app.route("/")
def index():
    return "Flask App!"

#@app.route("/hello/<string:name>")
@app.route("/hello/<string:name>/")
def hello(name):
#    return name
  quotes = [    "Pay as You Go - Enjoy a new cloud experience starting at 0.004euros per hour.",
               "Multiple Datacenters - Maximize your services reliability by running your infrastructure through autonomous facilities spread across multiple regions.",
               "Over 5 Tb/s of internet bandwidth - Deliver your content anywhere thanks to our multiple high-end transit providers and the best peerings.",
               "Limitless Infrastructure Combinations - Additional volumes, movable IPs, security groups and hot snapshots are available on all our servers.",
               "Developer Tools - Interact with Scaleway and take control of the cloud in minutes with our many tools, resources and third-party applications.",
               "Hourly Billing - All our cloud resources are billed per hour with monthly capping. Scaleway pricing is predictable and transparent, with no hidden costs."$
  randomNumber = randint(0,len(quotes)-1)
  quote = quotes[randomNumber]

  return render_template(
        'test.html',**locals())

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=80)

As you can notice, we updated the quotes variable with an array of multiples quotes. These can be accessed as quote[0], quote[1], quote[2] and so on. The function randint() returns a random number between 0 and the total number of quotes, one is subtracted because we start counting from zero.

We also added a locals() function which always returns a dictionary of the current namespace.

4 . Save and exit 5 . Run the application hello.py. It will return one of these quotes at random.

python hello.py

Protect a server with Fail2Ban

2018-08-22T00:00:00+00:00

Fail2Ban Overview

Fail2Ban is a useful tool that analyses server log files for recurring patterns of failures. This allows to block IP’s trying to run bruteforce attacks against a server.

In this Tutorial you will learn how to configure the service on an Ubuntu Bionic server to protect the SSH service. Fail2Ban can be used with all services generating log files.

Requirements:

You have an account and are logged into cloud.scaleway.com

You have an Ubuntu Bionic server

You have configured your SSH Key

You have sudo privileges or access to the root user.

Installing Fail2Ban

1 . The required packages are available in the repositories of Ubuntu and can be installed with apt:

sudo apt-get install fail2ban postfix

2 . Choose Internet Site when asked for the configuration:

3 . Once the installation has completed, open the file /etc/aliases and add the following line:

root: me@mydomain.tld

Make sure to replace me@mydomain.tld with your actual email address.

Note: To receive notifications by email, it is required that the email ports are unlocked.

Configuring Fail2Ban

1 . Start by copying the configuration file:

cd /etc/fail2ban && sudo cp jail.conf jail.local

The file jail.conf contains the default parameters. If a file jail.local is available, it will have priority over jail.conf if parameters are modified.

2 . Edit the file /etc/fail2ban/jail.local with your preferred editor.

Following the parameters which should be modified:

ignoreip = 127.0.0.1/8 By default the IPs of localhost are ignored, self-banning would not be very useful. It is possible to exclude other IPs from being banned.

bantime = 600 The duration of an ban. By default it is set to 10 Minutes. The value has to be specified in seconds and it is recommended to set it at least to one hour, or one day.

findtime = 600 The timespan which will be considered for maxretry. If you want for example to ban somebody who made more than 3 malicious attempts during the last hour or, as here, in the last 10 minutes.

maxretry = 3 Amount of attempts before being banned.

destemail = root@localhost The recipient of the mail. As an alias for root has been set during the installation, this value can be left as it is.

sendername = Fail2Ban The name of the sender of the mail.

action = %(action_)s This defines the action to execute when a limit is reached. By default it will only block the user.

To receive an email at each ban, set it to:

action = %(action_mw)

To receive the logs with the mail, set it to:

action = %(action_mwl)

Further down in the configuration file it comes to the “Jails”. These are configurable blocks per service to filter logs and ban in cases where patterns are matched.

As minimum it is recommend to activate the jail ssh as follows:

[ssh]

enabled  = true
port     = ssh
filter   = sshd
logpath  = /var/log/auth.log

Note: If your SSH daemon is listening on multiple ports or on a different port, you have to modify the line port with the correct parameters: Here an example: port = ssh,1234 Fail2Ban analyses the logs and will ban the users who made several intrusion attempts on ports 22 (SSH by default) & 1234.

3 . Save the file once you have edited it.

Fail2Ban uses filters, pre-made configuration files indicating what to parse in a log.

They can be found in /etc/fail2ban/filter.d. You can create your own filters in case you have need to.

4 . Restart the service to take the actions into effect:

sudo service fail2ban restart

The service will now analyze the connetions made to the SSH service. The logs of Fail2Ban are located in the file var/log/fail2ban.log.