google cloud/google cloud with gcloud/gsutil exploring cloud storage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# gsutil: exploring cloud storage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: gsutil [-D] [-DD] [-h header]... [-m] [-o] [-q] [command [opts...] args...]\r\n",
      "Available commands:\r\n",
      "  acl             Get, set, or change bucket and/or object ACLs\r\n",
      "  cat             Concatenate object content to stdout\r\n",
      "  compose         Concatenate a sequence of objects into a new composite object.\r\n",
      "  config          Obtain credentials and create configuration file\r\n",
      "  cors            Get or set a CORS JSON document for one or more buckets\r\n",
      "  cp              Copy files and objects\r\n",
      "  defacl          Get, set, or change default ACL on buckets\r\n",
      "  defstorageclass Get or set the default storage class on buckets\r\n",
      "  du              Display object size usage\r\n",
      "  hash            Calculate file hashes\r\n",
      "  help            Get help about commands and topics\r\n",
      "  iam             Get, set, or change bucket and/or object IAM permissions.\r\n",
      "  label           Get, set, or change the label configuration of a bucket.\r\n",
      "  lifecycle       Get or set lifecycle configuration for a bucket\r\n",
      "  logging         Configure or retrieve logging on buckets\r\n",
      "  ls              List providers, buckets, or objects\r\n",
      "  mb              Make buckets\r\n",
      "  mv              Move/rename objects and/or subdirectories\r\n",
      "  notification    Configure object change notification\r\n",
      "  perfdiag        Run performance diagnostic\r\n",
      "  rb              Remove buckets\r\n",
      "  rewrite         Rewrite objects\r\n",
      "  rm              Remove objects\r\n",
      "  rsync           Synchronize content of two buckets/directories\r\n",
      "  setmeta         Set metadata on already uploaded objects\r\n",
      "  signurl         Create a signed url\r\n",
      "  stat            Display object status\r\n",
      "  test            Run gsutil tests\r\n",
      "  update          Update to the latest gsutil release\r\n",
      "  version         Print version info about gsutil\r\n",
      "  versioning      Enable or suspend versioning for one or more buckets\r\n",
      "  web             Set a main page and/or error page for one or more buckets\r\n",
      "\r\n",
      "Additional help topics:\r\n",
      "  acls            Working With Access Control Lists\r\n",
      "  anon            Accessing Public Data Without Credentials\r\n",
      "  apis            Cloud Storage APIs\r\n",
      "  crc32c          CRC32C and Installing crcmod\r\n",
      "  creds           Credential Types Supporting Various Use Cases\r\n",
      "  csek            Supplying Your Own Encryption Keys\r\n",
      "  dev             Contributing Code to gsutil\r\n",
      "  encoding        Filename encoding and interoperability problems\r\n",
      "  metadata        Working With Object Metadata\r\n",
      "  naming          Object and Bucket Naming\r\n",
      "  options         Top-Level Command-Line Options\r\n",
      "  prod            Scripting Production Transfers\r\n",
      "  projects        Working With Projects\r\n",
      "  retries         Retry Handling Strategy\r\n",
      "  security        Security and Privacy Considerations\r\n",
      "  subdirs         How Subdirectories Work\r\n",
      "  support         Google Cloud Storage Support\r\n",
      "  throttling      Throttling gsutil\r\n",
      "  versions        Object Versioning and Concurrency Control\r\n",
      "  wildcards       Wildcard Names\r\n",
      "\r\n",
      "Use gsutil help <command or topic> for detailed help."
     ]
    }
   ],
   "source": [
    "#some help\n",
    "!gsutil --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://a-brand-new-bucket-toto/\r\n",
      "gs://billing-future-sonar-168815/\r\n"
     ]
    }
   ],
   "source": [
    "#list my buckets \n",
    "#(projectId has been set in the gcloud config \n",
    "# you can use the parameter -p projectId instead )\n",
    "!gsutil ls "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#more details\n",
    "!gsutil ls -L"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## let's create a new bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mNAME\u001b[0;0m\r\n",
      "  mb - Make buckets\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mSYNOPSIS\u001b[0;0m\r\n",
      "\r\n",
      "  gsutil mb [-c class] [-l location] [-p proj_id] url...\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mDESCRIPTION\u001b[0;0m\r\n",
      "  The mb command creates a new bucket. Google Cloud Storage has a single\r\n",
      "  namespace, so you are not allowed to create a bucket with a name already\r\n",
      "  in use by another user. You can, however, carve out parts of the bucket name\r\n",
      "  space corresponding to your company's domain name (see \"gsutil help naming\").\r\n",
      "\r\n",
      "  If you don't specify a project ID using the -p option, the bucket is created\r\n",
      "  using the default project ID specified in your gsutil configuration file\r\n",
      "  (see \"gsutil help config\"). For more details about projects see \"gsutil help\r\n",
      "  projects\".\r\n",
      "\r\n",
      "  The -c and -l options specify the storage class and location, respectively,\r\n",
      "  for the bucket. Once a bucket is created in a given location and with a\r\n",
      "  given storage class, it cannot be moved to a different location, and the\r\n",
      "  storage class cannot be changed. Instead, you would need to create a new\r\n",
      "  bucket and move the data over and then delete the original bucket.\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mBUCKET STORAGE CLASSES\u001b[0;0m\r\n",
      "  You can specify one of the `storage classes\r\n",
      "  <https://cloud.google.com/storage/docs/storage-classes>`_ for a bucket\r\n",
      "  with the -c option.\r\n",
      "\r\n",
      "  Example:\r\n",
      "\r\n",
      "    gsutil mb -c nearline gs://some-bucket\r\n",
      "\r\n",
      "  See online documentation for\r\n",
      "  `pricing <https://cloud.google.com/storage/pricing>`_ and\r\n",
      "  `SLA <https://cloud.google.com/storage/sla>`_ details.\r\n",
      "\r\n",
      "  If you don't specify a -c option, the bucket is created with the\r\n",
      "  default storage class Standard Storage, which is equivalent to Multi-Regional\r\n",
      "  Storage or Regional Storage, depending on whether the bucket was created in\r\n",
      "  a multi-regional location or regional location, respectively.\r\n",
      "\r\n",
      "\u001b[1mBUCKET LOCATIONS\u001b[0;0m\r\n",
      "  You can specify one of the 'available locations\r\n",
      "  <https://cloud.google.com/storage/docs/bucket-locations>`_ for a bucket\r\n",
      "  with the -l option.\r\n",
      "\r\n",
      "  Examples:\r\n",
      "\r\n",
      "    gsutil mb -l asia gs://some-bucket\r\n",
      "\r\n",
      "    gsutil mb -c regional -l us-east1 gs://some-bucket\r\n",
      "\r\n",
      "  If you don't specify a -l option, the bucket is created in the default\r\n",
      "  location (US).\r\n",
      "\r\n",
      "\u001b[1mOPTIONS\u001b[0;0m\r\n",
      "  -c class          Specifies the default storage class. Default is \"Standard\".\r\n",
      "\r\n",
      "  -l location       Can be any multi-regional or regional location. See\r\n",
      "                    https://cloud.google.com/storage/docs/storage-classes\r\n",
      "                    for a discussion of this distinction. Default is US.\r\n",
      "                    Locations are case insensitive.\r\n",
      "\r\n",
      "  -p proj_id        Specifies the project ID under which to create the bucket.\r\n",
      "\r\n",
      "  -s class          Same as -c."
     ]
    }
   ],
   "source": [
    "#some help\n",
    "!gsutil mb --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating gs://a-brand-new-bucket-toto/...\n",
      "ServiceException: 409 Bucket a-brand-new-bucket-toto already exists.\n"
     ]
    }
   ],
   "source": [
    "#creating a new bucket class regional en region wurope-west2\n",
    "!gsutil mb -c regional -l europe-west2 gs://a-brand-new-bucket-toto/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://a-brand-new-bucket-toto/\r\n",
      "gs://billing-future-sonar-168815/\r\n"
     ]
    }
   ],
   "source": [
    "#checking\n",
    "!gsutil ls "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## everything is better with labels :-)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mNAME\u001b[0;0m\r\n",
      "  label - Get, set, or change the label configuration of a bucket.\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mSYNOPSIS\u001b[0;0m\r\n",
      "  gsutil label set label-json-file url...\r\n",
      "  gsutil label get url\r\n",
      "  gsutil label ch <label_modifier>... url...\r\n",
      "\r\n",
      "  where each <label_modifier> is one of the following forms:\r\n",
      "\r\n",
      "    -l <key>:<value>\r\n",
      "    -d <key>\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mDESCRIPTION\u001b[0;0m\r\n",
      "  Gets, sets, or changes the label configuration (also called the tagging\r\n",
      "  configuration by other storage providers) of one or more buckets. An example\r\n",
      "  label JSON document looks like the following:\r\n",
      "\r\n",
      "    {\r\n",
      "      \"your_label_key\": \"your_label_value\",\r\n",
      "      \"your_other_label_key\": \"your_other_label_value\"\r\n",
      "    }\r\n",
      "\r\n",
      "  The label command has three sub-commands:\r\n",
      "\r\n",
      "\u001b[1mGET\u001b[0;0m\r\n",
      "  The \"label get\" command gets the\r\n",
      "  `labels <https://cloud.google.com/storage/docs/key-terms#bucket-labels>`_\r\n",
      "  applied to a bucket, which you can save and edit for use with the \"label set\"\r\n",
      "  command.\r\n",
      "\r\n",
      "\u001b[1mSET\u001b[0;0m\r\n",
      "  The \"label set\" command allows you to set the labels on one or more\r\n",
      "  buckets. You can retrieve a bucket's labels using the \"label get\" command,\r\n",
      "  save the output to a file, edit the file, and then use the \"label set\"\r\n",
      "  command to apply those labels to the specified bucket(s). For\r\n",
      "  example:\r\n",
      "\r\n",
      "    gsutil label get gs://bucket > labels.json\r\n",
      "\r\n",
      "  Make changes to labels.json, such as adding an additional label, then:\r\n",
      "\r\n",
      "    gsutil label set labels.json gs://example-bucket\r\n",
      "\r\n",
      "  Note that you can set these labels on multiple buckets at once:\r\n",
      "\r\n",
      "    gsutil label set labels.json gs://bucket-foo gs://bucket-bar\r\n",
      "\r\n",
      "\u001b[1mCH\u001b[0;0m\r\n",
      "  The \"label ch\" command updates a bucket's label configuration, applying the\r\n",
      "  label changes specified by the -l and -d flags. You can specify multiple\r\n",
      "  label changes in a single command run; all changes will be made atomically to\r\n",
      "  each bucket.\r\n",
      "\r\n",
      "\u001b[1mCH EXAMPLES\u001b[0;0m\r\n",
      "  Examples for \"ch\" sub-command:\r\n",
      "\r\n",
      "  Add the label \"key-foo:value-bar\" to the bucket \"example-bucket\":\r\n",
      "\r\n",
      "    gsutil label ch -l key-foo:value-bar gs://example-bucket\r\n",
      "\r\n",
      "  Change the above label to have a new value:\r\n",
      "\r\n",
      "    gsutil label ch -l key-foo:other-value gs://example-bucket\r\n",
      "\r\n",
      "  Add a new label and delete the old one from above:\r\n",
      "\r\n",
      "    gsutil label ch -l new-key:new-value -d key-foo gs://example-bucket\r\n",
      "\r\n",
      "\u001b[1mCH OPTIONS\u001b[0;0m\r\n",
      "  The \"ch\" sub-command has the following options\r\n",
      "\r\n",
      "    -l          Add or update a label with the specified key and value.\r\n",
      "\r\n",
      "    -d          Remove the label with the specified key."
     ]
    }
   ],
   "source": [
    "#some help\n",
    "!gsutil label --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setting label configuration on gs://a-brand-new-bucket-toto/...\r\n"
     ]
    }
   ],
   "source": [
    "#setting label\n",
    "!gsutil label ch -l env:test gs://a-brand-new-bucket-toto/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"env\": \"test\"\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "#getting the labels\n",
    "!gsutil label get gs://a-brand-new-bucket-toto/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload a nice foto there\n",
    "```![Formentera.JPG](Formentera.JPG)```\n",
    "![Formentera.JPG](Formentera.JPG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mNAME\u001b[0;0m\r\n",
      "  cp - Copy files and objects\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mSYNOPSIS\u001b[0;0m\r\n",
      "\r\n",
      "  gsutil cp [OPTION]... src_url dst_url\r\n",
      "  gsutil cp [OPTION]... src_url... dst_url\r\n",
      "  gsutil cp [OPTION]... -I dst_url\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mDESCRIPTION\u001b[0;0m\r\n",
      "  The gsutil cp command allows you to copy data between your local file\r\n",
      "  system and the cloud, copy data within the cloud, and copy data between\r\n",
      "  cloud storage providers. For example, to copy all text files from the\r\n",
      "  local directory to a bucket you could do:\r\n",
      "\r\n",
      "    gsutil cp *.txt gs://my-bucket\r\n",
      "\r\n",
      "  Similarly, you can download text files from a bucket by doing:\r\n",
      "\r\n",
      "    gsutil cp gs://my-bucket/*.txt .\r\n",
      "\r\n",
      "  If you want to copy an entire directory tree you need to use the -r option:\r\n",
      "\r\n",
      "    gsutil cp -r dir gs://my-bucket\r\n",
      "\r\n",
      "  If you have a large number of files to transfer you might want to use the\r\n",
      "  gsutil -m option, to perform a parallel (multi-threaded/multi-processing)\r\n",
      "  copy:\r\n",
      "\r\n",
      "    gsutil -m cp -r dir gs://my-bucket\r\n",
      "\r\n",
      "  You can pass a list of URLs (one per line) to copy on stdin instead of as\r\n",
      "  command line arguments by using the -I option. This allows you to use gsutil\r\n",
      "  in a pipeline to upload or download files / objects as generated by a program,\r\n",
      "  such as:\r\n",
      "\r\n",
      "    some_program | gsutil -m cp -I gs://my-bucket\r\n",
      "\r\n",
      "  or:\r\n",
      "\r\n",
      "    some_program | gsutil -m cp -I ./download_dir\r\n",
      "\r\n",
      "  The contents of stdin can name files, cloud URLs, and wildcards of files\r\n",
      "  and cloud URLs.\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mHOW NAMES ARE CONSTRUCTED\u001b[0;0m\r\n",
      "  The gsutil cp command strives to name objects in a way consistent with how\r\n",
      "  Linux cp works, which causes names to be constructed in varying ways depending\r\n",
      "  on whether you're performing a recursive directory copy or copying\r\n",
      "  individually named objects; and whether you're copying to an existing or\r\n",
      "  non-existent directory.\r\n",
      "\r\n",
      "  When performing recursive directory copies, object names are constructed that\r\n",
      "  mirror the source directory structure starting at the point of recursive\r\n",
      "  processing. For example, if dir1/dir2 contains the file a/b/c then the\r\n",
      "  command:\r\n",
      "\r\n",
      "    gsutil cp -r dir1/dir2 gs://my-bucket\r\n",
      "\r\n",
      "  will create the object gs://my-bucket/dir2/a/b/c.\r\n",
      "\r\n",
      "  In contrast, copying individually named files will result in objects named by\r\n",
      "  the final path component of the source files. For example, again assuming\r\n",
      "  dir1/dir2 contains a/b/c, the command:\r\n",
      "\r\n",
      "    gsutil cp dir1/dir2/** gs://my-bucket\r\n",
      "\r\n",
      "  will create the object gs://my-bucket/c.\r\n",
      "\r\n",
      "  The same rules apply for downloads: recursive copies of buckets and\r\n",
      "  bucket subdirectories produce a mirrored filename structure, while copying\r\n",
      "  individually (or wildcard) named objects produce flatly named files.\r\n",
      "\r\n",
      "  Note that in the above example the '**' wildcard matches all names\r\n",
      "  anywhere under dir. The wildcard '*' will match names just one level deep. For\r\n",
      "  more details see \"gsutil help wildcards\".\r\n",
      "\r\n",
      "  There's an additional wrinkle when working with subdirectories: the resulting\r\n",
      "  names depend on whether the destination subdirectory exists. For example,\r\n",
      "  if gs://my-bucket/subdir exists as a subdirectory, the command:\r\n",
      "\r\n",
      "    gsutil cp -r dir1/dir2 gs://my-bucket/subdir\r\n",
      "\r\n",
      "  will create the object gs://my-bucket/subdir/dir2/a/b/c. In contrast, if\r\n",
      "  gs://my-bucket/subdir does not exist, this same gsutil cp command will create\r\n",
      "  the object gs://my-bucket/subdir/a/b/c.\r\n",
      "\r\n",
      "  Note: If you use the\r\n",
      "  `Google Cloud Platform Console <https://console.cloud.google.com>`_\r\n",
      "  to create folders, it does so by creating a \"placeholder\" object that ends\r\n",
      "  with a \"/\" character. gsutil skips these objects when downloading from the\r\n",
      "  cloud to the local file system, because attempting to create a file that\r\n",
      "  ends with a \"/\" is not allowed on Linux and MacOS. Because of this, it is\r\n",
      "  recommended that you not create objects that end with \"/\" (unless you don't\r\n",
      "  need to be able to download such objects using gsutil).\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mCOPYING TO/FROM SUBDIRECTORIES; DISTRIBUTING TRANSFERS ACROSS MACHINES\u001b[0;0m\r\n",
      "  You can use gsutil to copy to and from subdirectories by using a command\r\n",
      "  like:\r\n",
      "\r\n",
      "    gsutil cp -r dir gs://my-bucket/data\r\n",
      "\r\n",
      "  This will cause dir and all of its files and nested subdirectories to be\r\n",
      "  copied under the specified destination, resulting in objects with names like\r\n",
      "  gs://my-bucket/data/dir/a/b/c. Similarly you can download from bucket\r\n",
      "  subdirectories by using a command like:\r\n",
      "\r\n",
      "    gsutil cp -r gs://my-bucket/data dir\r\n",
      "\r\n",
      "  This will cause everything nested under gs://my-bucket/data to be downloaded\r\n",
      "  into dir, resulting in files with names like dir/data/a/b/c.\r\n",
      "\r\n",
      "  Copying subdirectories is useful if you want to add data to an existing\r\n",
      "  bucket directory structure over time. It's also useful if you want\r\n",
      "  to parallelize uploads and downloads across multiple machines (potentially\r\n",
      "  reducing overall transfer time compared with simply running gsutil -m\r\n",
      "  cp on one machine). For example, if your bucket contains this structure:\r\n",
      "\r\n",
      "    gs://my-bucket/data/result_set_01/\r\n",
      "    gs://my-bucket/data/result_set_02/\r\n",
      "    ...\r\n",
      "    gs://my-bucket/data/result_set_99/\r\n",
      "\r\n",
      "  you could perform concurrent downloads across 3 machines by running these\r\n",
      "  commands on each machine, respectively:\r\n",
      "\r\n",
      "    gsutil -m cp -r gs://my-bucket/data/result_set_[0-3]* dir\r\n",
      "    gsutil -m cp -r gs://my-bucket/data/result_set_[4-6]* dir\r\n",
      "    gsutil -m cp -r gs://my-bucket/data/result_set_[7-9]* dir\r\n",
      "\r\n",
      "  Note that dir could be a local directory on each machine, or it could be a\r\n",
      "  directory mounted off of a shared file server; whether the latter performs\r\n",
      "  acceptably will depend on a number of factors, so we recommend experimenting\r\n",
      "  to find out what works best for your computing environment.\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mCOPYING IN THE CLOUD AND METADATA PRESERVATION\u001b[0;0m\r\n",
      "  If both the source and destination URL are cloud URLs from the same\r\n",
      "  provider, gsutil copies data \"in the cloud\" (i.e., without downloading\r\n",
      "  to and uploading from the machine where you run gsutil). In addition to\r\n",
      "  the performance and cost advantages of doing this, copying in the cloud\r\n",
      "  preserves metadata (like Content-Type and Cache-Control). In contrast,\r\n",
      "  when you download data from the cloud it ends up in a file, which has\r\n",
      "  no associated metadata. Thus, unless you have some way to hold on to\r\n",
      "  or re-create that metadata, downloading to a file will not retain the\r\n",
      "  metadata.\r\n",
      "\r\n",
      "  Copies spanning locations and/or storage classes cause data to be rewritten\r\n",
      "  in the cloud, which may take some time (but still will be faster than\r\n",
      "  downloading and re-uploading). Such operations can be resumed with the same\r\n",
      "  command if they are interrupted, so long as the command parameters are\r\n",
      "  identical.\r\n",
      "\r\n",
      "  Note that by default, the gsutil cp command does not copy the object\r\n",
      "  ACL to the new object, and instead will use the default bucket ACL (see\r\n",
      "  \"gsutil help defacl\"). You can override this behavior with the -p\r\n",
      "  option (see OPTIONS below).\r\n",
      "\r\n",
      "  One additional note about copying in the cloud: If the destination bucket has\r\n",
      "  versioning enabled, by default gsutil cp will copy only live versions of the\r\n",
      "  source object(s). For example:\r\n",
      "\r\n",
      "    gsutil cp gs://bucket1/obj gs://bucket2\r\n",
      "\r\n",
      "  will cause only the single live version of gs://bucket1/obj to be copied to\r\n",
      "  gs://bucket2, even if there are archived versions of gs://bucket1/obj. To also\r\n",
      "  copy archived versions, use the -A flag:\r\n",
      "\r\n",
      "    gsutil cp -A gs://bucket1/obj gs://bucket2\r\n",
      "\r\n",
      "  The gsutil -m flag is disallowed when using the cp -A flag, to ensure that\r\n",
      "  version ordering is preserved.\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mCHECKSUM VALIDATION\u001b[0;0m\r\n",
      "  At the end of every upload or download the gsutil cp command validates that\r\n",
      "  the checksum it computes for the source file/object matches the checksum\r\n",
      "  the service computes. If the checksums do not match, gsutil will delete the\r\n",
      "  corrupted object and print a warning message. This very rarely happens, but\r\n",
      "  if it does, please contact gs-team@google.com.\r\n",
      "\r\n",
      "  If you know the MD5 of a file before uploading you can specify it in the\r\n",
      "  Content-MD5 header, which will cause the cloud storage service to reject the\r\n",
      "  upload if the MD5 doesn't match the value computed by the service. For\r\n",
      "  example:\r\n",
      "\r\n",
      "    % gsutil hash obj\r\n",
      "    Hashing     obj:\r\n",
      "    Hashes [base64] for obj:\r\n",
      "            Hash (crc32c):          lIMoIw==\r\n",
      "            Hash (md5):             VgyllJgiiaRAbyUUIqDMmw==\r\n",
      "\r\n",
      "    % gsutil -h Content-MD5:VgyllJgiiaRAbyUUIqDMmw== cp obj gs://your-bucket/obj\r\n",
      "    Copying file://obj [Content-Type=text/plain]...\r\n",
      "    Uploading   gs://your-bucket/obj:                                182 b/182 B\r\n",
      "\r\n",
      "    If the checksum didn't match the service would instead reject the upload and\r\n",
      "    gsutil would print a message like:\r\n",
      "\r\n",
      "    BadRequestException: 400 Provided MD5 hash \"VgyllJgiiaRAbyUUIqDMmw==\"\r\n",
      "    doesn't match calculated MD5 hash \"7gyllJgiiaRAbyUUIqDMmw==\".\r\n",
      "\r\n",
      "  Even if you don't do this gsutil will delete the object if the computed\r\n",
      "  checksum mismatches, but specifying the Content-MD5 header has several\r\n",
      "  advantages:\r\n",
      "\r\n",
      "      1. It prevents the corrupted object from becoming visible at all, whereas\r\n",
      "      otherwise it would be visible for 1-3 seconds before gsutil deletes it.\r\n",
      "\r\n",
      "      2. If an object already exists with the given name, specifying the\r\n",
      "      Content-MD5 header will cause the existing object never to be replaced,\r\n",
      "      whereas otherwise it would be replaced by the corrupted object and then\r\n",
      "      deleted a few seconds later.\r\n",
      "\r\n",
      "      3. It will definitively prevent the corrupted object from being left in\r\n",
      "      the cloud, whereas the gsutil approach of deleting after the upload\r\n",
      "      completes could fail if (for example) the gsutil process gets ^C'd\r\n",
      "      between upload and deletion request.\r\n",
      "\r\n",
      "      4. It supports a customer-to-service integrity check handoff. For example,\r\n",
      "      if you have a content production pipeline that generates data to be\r\n",
      "      uploaded to the cloud along with checksums of that data, specifying the\r\n",
      "      MD5 computed by your content pipeline when you run gsutil cp will ensure\r\n",
      "      that the checksums match all the way through the process (e.g., detecting\r\n",
      "      if data gets corrupted on your local disk between the time it was written\r\n",
      "      by your content pipeline and the time it was uploaded to GCS).\r\n",
      "\r\n",
      "  Note: The Content-MD5 header is ignored for composite objects, because such\r\n",
      "  objects only have a CRC32C checksum.\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mRETRY HANDLING\u001b[0;0m\r\n",
      "  The cp command will retry when failures occur, but if enough failures happen\r\n",
      "  during a particular copy or delete operation the cp command will skip that\r\n",
      "  object and move on. At the end of the copy run if any failures were not\r\n",
      "  successfully retried, the cp command will report the count of failures, and\r\n",
      "  exit with non-zero status.\r\n",
      "\r\n",
      "  Note that there are cases where retrying will never succeed, such as if you\r\n",
      "  don't have write permission to the destination bucket or if the destination\r\n",
      "  path for some objects is longer than the maximum allowed length.\r\n",
      "\r\n",
      "  For more details about gsutil's retry handling, please see\r\n",
      "  \"gsutil help retries\".\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mRESUMABLE TRANSFERS\u001b[0;0m\r\n",
      "  gsutil automatically performs a resumable upload whenever you use the cp\r\n",
      "  command to upload an object that is larger than 8 MiB. You do not need to\r\n",
      "  specify any special command line options to make this happen. If your upload\r\n",
      "  is interrupted you can restart the upload by running the same cp command that\r\n",
      "  you ran to start the upload. Until the upload has completed successfully, it\r\n",
      "  will not be visible at the destination object and will not replace any\r\n",
      "  existing object the upload is intended to overwrite. However, see the section\r\n",
      "  on PARALLEL COMPOSITE UPLOADS, which may leave temporary component objects in\r\n",
      "  place during the upload process.\r\n",
      "\r\n",
      "  Similarly, gsutil automatically performs resumable downloads (using standard\r\n",
      "  HTTP Range GET operations) whenever you use the cp command, unless the\r\n",
      "  destination is a stream. In this case, a partially downloaded temporary file\r\n",
      "  will be visible in the destination directory. Upon completion, the original\r\n",
      "  file is deleted and overwritten with the downloaded contents.\r\n",
      "\r\n",
      "  Resumable uploads and downloads store state information in files under\r\n",
      "  ~/.gsutil, named by the destination object or file. If you attempt to resume a\r\n",
      "  transfer from a machine with a different directory, the transfer will start\r\n",
      "  over from scratch.\r\n",
      "\r\n",
      "  See also \"gsutil help prod\" for details on using resumable transfers\r\n",
      "  in production.\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mSTREAMING TRANSFERS\u001b[0;0m\r\n",
      "  Use '-' in place of src_url or dst_url to perform a streaming\r\n",
      "  transfer. For example:\r\n",
      "\r\n",
      "    long_running_computation | gsutil cp - gs://my-bucket/obj\r\n",
      "\r\n",
      "  Streaming uploads using the JSON API (see \"gsutil help apis\") are buffered in\r\n",
      "  memory part-way back into the file and can thus retry in the event of network\r\n",
      "  or service problems.\r\n",
      "\r\n",
      "  Streaming transfers using the XML API do not support resumable\r\n",
      "  uploads/downloads. If you have a large amount of data to upload (say, more\r\n",
      "  than 100 MiB) it is recommended that you write the data to a local file and\r\n",
      "  then copy that file to the cloud rather than streaming it (and similarly for\r\n",
      "  large downloads).\r\n",
      "\r\n",
      "  WARNING: When performing streaming transfers gsutil does not compute a\r\n",
      "  checksum of the uploaded or downloaded data. Therefore, we recommend that\r\n",
      "  users either perform their own validation of the data or use non-streaming\r\n",
      "  transfers (which perform integrity checking automatically).\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mSLICED OBJECT DOWNLOADS\u001b[0;0m\r\n",
      "  gsutil uses HTTP Range GET requests to perform \"sliced\" downloads in parallel\r\n",
      "  when downloading large objects from Google Cloud Storage. This means that disk\r\n",
      "  space for the temporary download destination file will be pre-allocated and\r\n",
      "  byte ranges (slices) within the file will be downloaded in parallel. Once all\r\n",
      "  slices have completed downloading, the temporary file will be renamed to the\r\n",
      "  destination file. No additional local disk space is required for this\r\n",
      "  operation.\r\n",
      "\r\n",
      "  This feature is only available for Google Cloud Storage objects because it\r\n",
      "  requires a fast composable checksum (CRC32C) that can be used to verify the\r\n",
      "  data integrity of the slices. And because it depends on CRC32C, using sliced\r\n",
      "  object downloads also requires a compiled crcmod (see \"gsutil help crcmod\") on\r\n",
      "  the machine performing the download. If compiled crcmod is not available,\r\n",
      "  a non-sliced object download will instead be performed.\r\n",
      "\r\n",
      "  Note: since sliced object downloads cause multiple writes to occur at various\r\n",
      "  locations on disk, this mechanism can degrade performance for disks with slow\r\n",
      "  seek times, especially for large numbers of slices. While the default number\r\n",
      "  of slices is set small to avoid this problem, you can disable sliced object\r\n",
      "  download if necessary by setting the \"sliced_object_download_threshold\"\r\n",
      "  variable in the .boto config file to 0.\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mPARALLEL COMPOSITE UPLOADS\u001b[0;0m\r\n",
      "  gsutil can automatically use\r\n",
      "  `object composition <https://cloud.google.com/storage/docs/composite-objects>`_\r\n",
      "  to perform uploads in parallel for large, local files being uploaded to Google\r\n",
      "  Cloud Storage. If enabled (see below), a large file will be split into\r\n",
      "  component pieces that are uploaded in parallel and then composed in the cloud\r\n",
      "  (and the temporary components finally deleted). A file can be broken into as\r\n",
      "  many as 32 component pieces; until this piece limit is reached, the maximum\r\n",
      "  size of each component piece is determined by the variable\r\n",
      "  \"parallel_composite_upload_component_size,\" specified in the [GSUtil] section\r\n",
      "  of your .boto configuration file (for files that are otherwise too big,\r\n",
      "  components are as large as needed to fit into 32 pieces). No additional local\r\n",
      "  disk space is required for this operation.\r\n",
      "\r\n",
      "  Using parallel composite uploads presents a tradeoff between upload\r\n",
      "  performance and download configuration: If you enable parallel composite\r\n",
      "  uploads your uploads will run faster, but someone will need to install a\r\n",
      "  compiled crcmod (see \"gsutil help crcmod\") on every machine where objects are\r\n",
      "  downloaded by gsutil or other Python applications. Note that for such uploads,\r\n",
      "  crcmod is required for downloading regardless of whether the parallel\r\n",
      "  composite upload option is on or not. For some distributions this is easy\r\n",
      "  (e.g., it comes pre-installed on MacOS), but in other cases some users have\r\n",
      "  found it difficult. Because of this, at present parallel composite uploads are\r\n",
      "  disabled by default. Google is actively working with a number of the Linux\r\n",
      "  distributions to get crcmod included with the stock distribution. Once that is\r\n",
      "  done we will re-enable parallel composite uploads by default in gsutil.\r\n",
      "\r\n",
      "  Warning: Parallel composite uploads should not be used with NEARLINE or\r\n",
      "  COLDLINE storage class buckets, because doing so incurs an early deletion\r\n",
      "  charge for each component object.\r\n",
      "\r\n",
      "  To try parallel composite uploads you can run the command:\r\n",
      "\r\n",
      "    gsutil -o GSUtil:parallel_composite_upload_threshold=150M cp bigfile gs://your-bucket\r\n",
      "\r\n",
      "  where bigfile is larger than 150 MiB. When you do this notice that the upload\r\n",
      "  progress indicator continuously updates for several different uploads at once\r\n",
      "  (corresponding to each of the sections of the file being uploaded in\r\n",
      "  parallel), until the parallel upload completes. If after trying this you want\r\n",
      "  to enable parallel composite uploads for all of your future uploads\r\n",
      "  (notwithstanding the caveats mentioned earlier), you can uncomment and set the\r\n",
      "  \"parallel_composite_upload_threshold\" config value in your .boto configuration\r\n",
      "  file to this value.\r\n",
      "\r\n",
      "  Note that the crcmod problem only impacts downloads via Python applications\r\n",
      "  (such as gsutil). If all users who need to download the data using gsutil or\r\n",
      "  other Python applications can install crcmod, or if no Python users will\r\n",
      "  need to download your objects, it makes sense to enable parallel composite\r\n",
      "  uploads (see above). For example, if you use gsutil to upload video assets,\r\n",
      "  and those assets will only ever be served via a Java application, it would\r\n",
      "  make sense to enable parallel composite uploads on your machine (there are\r\n",
      "  efficient CRC32C implementations available in Java).\r\n",
      "\r\n",
      "  If a parallel composite upload fails prior to composition, re-running the\r\n",
      "  gsutil command will take advantage of resumable uploads for the components\r\n",
      "  that failed, and the component objects will be deleted after the first\r\n",
      "  successful attempt. Any temporary objects that were uploaded successfully\r\n",
      "  before gsutil failed will still exist until the upload is completed\r\n",
      "  successfully. The temporary objects will be named in the following fashion:\r\n",
      "\r\n",
      "    <random ID>/gsutil/tmp/parallel_composite_uploads/for_details_see/gsutil_help_cp/<hash>\r\n",
      "\r\n",
      "  where <random ID> is a numerical value, and <hash> is an MD5 hash (not related\r\n",
      "  to the hash of the contents of the file or object).\r\n",
      "\r\n",
      "  To avoid leaving temporary objects around, you should make sure to check the\r\n",
      "  exit status from the gsutil command.  This can be done in a bash script, for\r\n",
      "  example, by doing:\r\n",
      "\r\n",
      "    if ! gsutil cp ./local-file gs://your-bucket/your-object; then\r\n",
      "      << Code that handles failures >>\r\n",
      "    fi\r\n",
      "\r\n",
      "  Or, for copying a directory, use this instead:\r\n",
      "\r\n",
      "    if ! gsutil cp -c -L cp.log -r ./dir gs://bucket; then\r\n",
      "      << Code that handles failures >>\r\n",
      "    fi\r\n",
      "\r\n",
      "  One important caveat is that files uploaded using parallel composite uploads\r\n",
      "  are subject to a maximum number of components limit. For example, if you\r\n",
      "  upload a large file that gets split into 10 components, and try to compose it\r\n",
      "  with another object with 1015 components, the operation will fail because it\r\n",
      "  exceeds the 1024 component limit. If you wish to compose an object later and the\r\n",
      "  component limit is a concern, it is recommended that you disable parallel\r\n",
      "  composite uploads for that transfer.\r\n",
      "\r\n",
      "  Also note that an object uploaded using parallel composite uploads will have a\r\n",
      "  CRC32C hash, but it will not have an MD5 hash (and because of that, users who\r\n",
      "  download the object must have crcmod installed, as noted earlier). For details\r\n",
      "  see \"gsutil help crc32c\".\r\n",
      "\r\n",
      "  Parallel composite uploads can be disabled by setting the\r\n",
      "  \"parallel_composite_upload_threshold\" variable in the .boto config file to 0.\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mCHANGING TEMP DIRECTORIES\u001b[0;0m\r\n",
      "  gsutil writes data to a temporary directory in several cases:\r\n",
      "\r\n",
      "  - when compressing data to be uploaded (see the -z and -Z options)\r\n",
      "  - when decompressing data being downloaded (when the data has\r\n",
      "    Content-Encoding:gzip, e.g., as happens when uploaded using gsutil cp -z\r\n",
      "    or gsutil cp -Z)\r\n",
      "  - when running integration tests (using the gsutil test command)\r\n",
      "\r\n",
      "  In these cases it's possible the temp file location on your system that\r\n",
      "  gsutil selects by default may not have enough space. If gsutil runs out of\r\n",
      "  space during one of these operations (e.g., raising\r\n",
      "  \"CommandException: Inadequate temp space available to compress <your file>\"\r\n",
      "  during a gsutil cp -z operation), you can change where it writes these\r\n",
      "  temp files by setting the TMPDIR environment variable. On Linux and MacOS\r\n",
      "  you can do this either by running gsutil this way:\r\n",
      "\r\n",
      "    TMPDIR=/some/directory gsutil cp ...\r\n",
      "\r\n",
      "  or by adding this line to your ~/.bashrc file and then restarting the shell\r\n",
      "  before running gsutil:\r\n",
      "\r\n",
      "    export TMPDIR=/some/directory\r\n",
      "\r\n",
      "  On Windows 7 you can change the TMPDIR environment variable from Start ->\r\n",
      "  Computer -> System -> Advanced System Settings -> Environment Variables.\r\n",
      "  You need to reboot after making this change for it to take effect. (Rebooting\r\n",
      "  is not necessary after running the export command on Linux and MacOS.)\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mCOPYING SPECIAL FILES\u001b[0;0m\r\n",
      "\r\n",
      "gsutil cp does not support copying special file types such as sockets, device\r\n",
      "files, named pipes, or any other non-standard files intended to represent an\r\n",
      "operating system resource. You should not run gsutil cp with sources that\r\n",
      "include such files (for example, recursively copying the root directory on\r\n",
      "Linux that includes /dev ). If you do, gsutil cp may fail or hang.\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "\u001b[1mOPTIONS\u001b[0;0m\r\n",
      "  -a canned_acl  Sets named canned_acl when uploaded objects created. See\r\n",
      "                 \"gsutil help acls\" for further details.\r\n",
      "\r\n",
      "  -A             Copy all source versions from a source buckets/folders.\r\n",
      "                 If not set, only the live version of each source object is\r\n",
      "                 copied. Note: this option is only useful when the destination\r\n",
      "                 bucket has versioning enabled.\r\n",
      "\r\n",
      "  -c             If an error occurs, continue to attempt to copy the remaining\r\n",
      "                 files. If any copies were unsuccessful, gsutil's exit status\r\n",
      "                 will be non-zero even if this flag is set. This option is\r\n",
      "                 implicitly set when running \"gsutil -m cp...\". Note: -c only\r\n",
      "                 applies to the actual copying operation. If an error occurs\r\n",
      "                 while iterating over the files in the local directory (e.g.,\r\n",
      "                 invalid Unicode file name) gsutil will print an error message\r\n",
      "                 and abort.\r\n",
      "\r\n",
      "  -D             Copy in \"daisy chain\" mode, i.e., copying between two buckets\r\n",
      "                 by hooking a download to an upload, via the machine where\r\n",
      "                 gsutil is run. This stands in contrast to the default, where\r\n",
      "                 data are copied between two buckets \"in the cloud\", i.e.,\r\n",
      "                 without needing to copy via the machine where gsutil runs.\r\n",
      "\r\n",
      "                 By default, a \"copy in the cloud\" when the source is a\r\n",
      "                 composite object will retain the composite nature of the\r\n",
      "                 object. However, Daisy chain mode can be used to change a\r\n",
      "                 composite object into a non-composite object. For example:\r\n",
      "\r\n",
      "                     gsutil cp -D -p gs://bucket/obj gs://bucket/obj_tmp\r\n",
      "                     gsutil mv -p gs://bucket/obj_tmp gs://bucket/obj\r\n",
      "\r\n",
      "                 Note: Daisy chain mode is automatically used when copying\r\n",
      "                 between providers (e.g., to copy data from Google Cloud Storage\r\n",
      "                 to another provider).\r\n",
      "\r\n",
      "  -e             Exclude symlinks. When specified, symbolic links will not be\r\n",
      "                 copied.\r\n",
      "\r\n",
      "  -I             Causes gsutil to read the list of files or objects to copy from\r\n",
      "                 stdin. This allows you to run a program that generates the list\r\n",
      "                 of files to upload/download.\r\n",
      "\r\n",
      "  -L <file>      Outputs a manifest log file with detailed information about\r\n",
      "                 each item that was copied. This manifest contains the following\r\n",
      "                 information for each item:\r\n",
      "\r\n",
      "                 - Source path.\r\n",
      "                 - Destination path.\r\n",
      "                 - Source size.\r\n",
      "                 - Bytes transferred.\r\n",
      "                 - MD5 hash.\r\n",
      "                 - UTC date and time transfer was started in ISO 8601 format.\r\n",
      "                 - UTC date and time transfer was completed in ISO 8601 format.\r\n",
      "                 - Upload id, if a resumable upload was performed.\r\n",
      "                 - Final result of the attempted transfer, success or failure.\r\n",
      "                 - Failure details, if any.\r\n",
      "\r\n",
      "                 If the log file already exists, gsutil will use the file as an\r\n",
      "                 input to the copy process, and will also append log items to\r\n",
      "                 the existing file. Files/objects that are marked in the\r\n",
      "                 existing log file as having been successfully copied (or\r\n",
      "                 skipped) will be ignored. Files/objects without entries will be\r\n",
      "                 copied and ones previously marked as unsuccessful will be\r\n",
      "                 retried. This can be used in conjunction with the -c option to\r\n",
      "                 build a script that copies a large number of objects reliably,\r\n",
      "                 using a bash script like the following:\r\n",
      "\r\n",
      "                   until gsutil cp -c -L cp.log -r ./dir gs://bucket; do\r\n",
      "                     sleep 1\r\n",
      "                   done\r\n",
      "\r\n",
      "                 The -c option will cause copying to continue after failures\r\n",
      "                 occur, and the -L option will allow gsutil to pick up where it\r\n",
      "                 left off without duplicating work. The loop will continue\r\n",
      "                 running as long as gsutil exits with a non-zero status (such a\r\n",
      "                 status indicates there was at least one failure during the\r\n",
      "                 gsutil run).\r\n",
      "\r\n",
      "                 Note: If you're trying to synchronize the contents of a\r\n",
      "                 directory and a bucket (or two buckets), see\r\n",
      "                 \"gsutil help rsync\".\r\n",
      "\r\n",
      "  -n             No-clobber. When specified, existing files or objects at the\r\n",
      "                 destination will not be overwritten. Any items that are skipped\r\n",
      "                 by this option will be reported as being skipped. This option\r\n",
      "                 will perform an additional GET request to check if an item\r\n",
      "                 exists before attempting to upload the data. This will save\r\n",
      "                 retransmitting data, but the additional HTTP requests may make\r\n",
      "                 small object transfers slower and more expensive.\r\n",
      "\r\n",
      "  -p             Causes ACLs to be preserved when copying in the cloud. Note\r\n",
      "                 that this option has performance and cost implications when\r\n",
      "                 using  the XML API, as it requires separate HTTP calls for\r\n",
      "                 interacting with ACLs. (There are no such performance or cost\r\n",
      "                 implications when using the -p option with the JSON API.) The\r\n",
      "                 performance issue can be mitigated to some degree by using\r\n",
      "                 gsutil -m cp to cause parallel copying. Note that this option\r\n",
      "                 only works if you have OWNER access to all of the objects that\r\n",
      "                 are copied.\r\n",
      "\r\n",
      "                 You can avoid the additional performance and cost of using\r\n",
      "                 cp -p if you want all objects in the destination bucket to end\r\n",
      "                 up with the same ACL by setting a default object ACL on that\r\n",
      "                 bucket instead of using cp -p. See \"gsutil help defacl\".\r\n",
      "\r\n",
      "                 Note that it's not valid to specify both the -a and -p options\r\n",
      "                 together.\r\n",
      "\r\n",
      "  -P             Causes POSIX attributes to be preserved when objects are\r\n",
      "                 copied. With this feature enabled, gsutil cp will copy fields\r\n",
      "                 provided by stat. These are the user ID of the owner, the group\r\n",
      "                 ID of the owning group, the mode (permissions) of the file, and\r\n",
      "                 the access/modification time of the file. For downloads, these\r\n",
      "                 attributes will only be set if the source objects were uploaded\r\n",
      "                 with this flag enabled.\r\n",
      "\r\n",
      "                 On Windows, this flag will only set and restore access time and\r\n",
      "                 modification time. This is because Windows doesn't have a\r\n",
      "                 notion of POSIX uid/gid/mode.\r\n",
      "\r\n",
      "  -R, -r         The -R and -r options are synonymous. Causes directories,\r\n",
      "                 buckets, and bucket subdirectories to be copied recursively.\r\n",
      "                 If you neglect to use this option for an upload, gsutil will\r\n",
      "                 copy any files it finds and skip any directories. Similarly,\r\n",
      "                 neglecting to specify this option for a download will cause\r\n",
      "                 gsutil to copy any objects at the current bucket directory\r\n",
      "                 level, and skip any subdirectories.\r\n",
      "\r\n",
      "  -s <class>     The storage class of the destination object(s). If not\r\n",
      "                 specified, the default storage class of the destination bucket\r\n",
      "                 is used. Not valid for copying to non-cloud destinations.\r\n",
      "\r\n",
      "  -U             Skip objects with unsupported object types instead of failing.\r\n",
      "                 Unsupported object types are Amazon S3 Objects in the GLACIER\r\n",
      "                 storage class.\r\n",
      "\r\n",
      "  -v             Requests that the version-specific URL for each uploaded object\r\n",
      "                 be printed. Given this URL you can make future upload requests\r\n",
      "                 that are safe in the face of concurrent updates, because Google\r\n",
      "                 Cloud Storage will refuse to perform the update if the current\r\n",
      "                 object version doesn't match the version-specific URL. See\r\n",
      "                 \"gsutil help versions\" for more details.\r\n",
      "\r\n",
      "  -z <ext,...>   Applies gzip content-encoding to file uploads with the given\r\n",
      "                 extensions. This is useful when uploading files with\r\n",
      "                 compressible content (such as .js, .css, or .html files)\r\n",
      "                 because it saves network bandwidth and space in Google Cloud\r\n",
      "                 Storage, which in turn reduces storage costs.\r\n",
      "\r\n",
      "                 When you specify the -z option, the data from your files is\r\n",
      "                 compressed before it is uploaded, but your actual files are\r\n",
      "                 left uncompressed on the local disk. The uploaded objects\r\n",
      "                 retain the Content-Type and name of the original files but are\r\n",
      "                 given a Content-Encoding header with the value \"gzip\" to\r\n",
      "                 indicate that the object data stored are compressed on the\r\n",
      "                 Google Cloud Storage servers.\r\n",
      "\r\n",
      "                 For example, the following command:\r\n",
      "\r\n",
      "                   gsutil cp -z html -a public-read cattypes.html gs://mycats\r\n",
      "\r\n",
      "                 will do all of the following:\r\n",
      "\r\n",
      "                 - Upload as the object gs://mycats/cattypes.html (cp command)\r\n",
      "                 - Set the Content-Type to text/html (based on file extension)\r\n",
      "                 - Compress the data in the file cattypes.html (-z option)\r\n",
      "                 - Set the Content-Encoding to gzip (-z option)\r\n",
      "                 - Set the ACL to public-read (-a option)\r\n",
      "                 - If a user tries to view cattypes.html in a browser, the\r\n",
      "                   browser will know to uncompress the data based on the\r\n",
      "                   Content-Encoding header, and to render it as HTML based on\r\n",
      "                   the Content-Type header.\r\n",
      "\r\n",
      "                 Note that if you download an object with Content-Encoding:gzip\r\n",
      "                 gsutil will decompress the content before writing the local\r\n",
      "                 file.\r\n",
      "\r\n",
      "  -Z             Applies gzip content-encoding to file uploads. This option\r\n",
      "                 works like the -z option described above, but it applies to\r\n",
      "                 all uploaded files, regardless of extension.\r\n",
      "\r\n",
      "                 Warning: If you use this option and some of the source files\r\n",
      "                 don't compress well (e.g., that's often true of binary data),\r\n",
      "                 this option may result in files taking up more space in the\r\n",
      "                 cloud than they would if left uncompressed."
     ]
    }
   ],
   "source": [
    "#what does the help say?\n",
    "!gsutil cp --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying file://Formentera.JPG [Content-Type=image/jpeg]...\n",
      "| [1 files][  2.2 MiB/  2.2 MiB]                                                \n",
      "Operation completed over 1 objects/2.2 MiB.                                      \n"
     ]
    }
   ],
   "source": [
    "#let's copy Formentera.JPG there\n",
    "!gsutil cp \"Formentera.JPG\" gs://a-brand-new-bucket-toto/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://a-brand-new-bucket-toto/Formentera.JPG\r\n"
     ]
    }
   ],
   "source": [
    "#checking the result\n",
    "!gsutil ls gs://a-brand-new-bucket-toto/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   2275011  2017-06-30T20:48:39Z  gs://a-brand-new-bucket-toto/Formentera.JPG\r\n",
      "TOTAL: 1 objects, 2275011 bytes (2.17 MiB)\r\n"
     ]
    }
   ],
   "source": [
    "#the file is there let's have a look\n",
    "!gsutil ls -l gs://a-brand-new-bucket-toto/Formentera.JPG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#with more details\n",
    "!gsutil ls -L gs://a-brand-new-bucket-toto/Formentera.JPG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated ACL on gs://a-brand-new-bucket-toto/Formentera.JPG\r\n"
     ]
    }
   ],
   "source": [
    "#let's make it public!\n",
    "!gsutil acl ch -u AllUsers:R gs://a-brand-new-bucket-toto/Formentera.JPG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Photo is there on the internet!\n",
    "```![Formentera.JPG](https://storage.googleapis.com/a-brand-new-bucket-toto/Formentera.JPG)```\n",
    "![Formentera.JPG](https://storage.googleapis.com/a-brand-new-bucket-toto/Formentera.JPG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying gs://a-brand-new-bucket-toto/Formentera.JPG [Content-Type=image/jpeg]...\n",
      "Removing gs://a-brand-new-bucket-toto/Formentera.JPG...                         \n",
      "\n",
      "Operation completed over 1 objects/2.2 MiB.                                      \n"
     ]
    }
   ],
   "source": [
    "#let's move it\n",
    "!gsutil mv gs://a-brand-new-bucket-toto/Formentera.JPG gs://a-brand-new-bucket-toto/formentera.jpeg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://a-brand-new-bucket-toto/formentera.jpeg\r\n"
     ]
    }
   ],
   "source": [
    "#check the result\n",
    "!gsutil ls gs://a-brand-new-bucket-toto"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Removing gs://a-brand-new-bucket-toto/formentera.jpeg...\n",
      "/ [1 objects]                                                                   \n",
      "Operation completed over 1 objects.                                              \n"
     ]
    }
   ],
   "source": [
    "#delete the file\n",
    "!gsutil rm gs://a-brand-new-bucket-toto/formentera.jpeg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Removing gs://a-brand-new-bucket-toto/...\r\n"
     ]
    }
   ],
   "source": [
    "#deleting the bucket\n",
    "!gsutil rm -r gs://a-brand-new-bucket-toto"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://billing-future-sonar-168815/\r\n"
     ]
    }
   ],
   "source": [
    "#check the result\n",
    "!gsutil ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}