飞纯技术
  • 主页
  • 相册
  • 关于我
KEEP IN TOUCH

Posts tagged s3

删除S3上文件的脚本s3del

七04
2010
1 Comment Written by Filia.Tao

之间写过一个讲文件备份到Amazon S3 的文章。 之后就有新的需求比如定期清理很久之前的备份(比如一个月前)。
Boto 库里面有一个s3put 的脚本,但是没有删除文件的脚本。 于是自己写了个s3del, 代码如下 。

#!/home/ftao/env/bin/python
# Copyright (c) 2010 Tao Fei http://ftao.org/
# Based on boto's s3put scirpt 
 
# Copyright (c) 2006,2007,2008 Mitch Garnaat http://garnaat.org/
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish, dis-
# tribute, sublicense, and/or sell copies of the Software, and to permit
# persons to whom the Software is furnished to do so, subject to the fol-
# lowing conditions:
#
# The above copyright notice and this permission notice shall be included
# in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABIL-
# ITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
# SHALL THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, 
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
# IN THE SOFTWARE.
#
import getopt, sys, os
import boto
from boto.exception import S3ResponseError
 
usage_string = """
SYNOPSIS
    s3del [-a/--access_key <access_key>] [-s/--secret_key <secret_key>]
          -b/--bucket <bucket_name> 
          [-d/--debug <debug_level>]
          [-n/--no_op] [-q/--quiet]
          path
 
    Where
        access_key - Your AWS Access Key ID.  If not supplied, boto will
                     use the value of the environment variable
                     AWS_ACCESS_KEY_ID
        secret_key - Your AWS Secret Access Key.  If not supplied, boto
                     will use the value of the environment variable
                     AWS_SECRET_ACCESS_KEY
        bucket_name - The name of the S3 bucket the file(s) should be
                      delete from.
        path - A path to a file or folder (must ends with '/') on s3 
        debug_level - 0 means no debug output (default), 1 means normal
                      debug output from boto, and 2 means boto debug output
                      plus request/response output from httplib
 
     If the -n option is provided, no files will be deleted from S3 but
     informational messages will be printed about what would happen.
"""
def usage():
    print usage_string
    sys.exit()
 
def main():
    try:
        opts, args = getopt.getopt(sys.argv[1:], 'a:b::d:hn:qs:v',
                                   ['access_key', 'bucket', 'debug', 'help', 
                                    'no_op', 'quiet', 'secret_key' ])
    except:
        usage()
    aws_access_key_id = None
    aws_secret_access_key = None
    bucket_name = ''
    debug = 0
    quiet = False
    no_op = False
    for o, a in opts:
        if o in ('-h', '--help'):
            usage()
            sys.exit()
        if o in ('-a', '--access_key'):
            aws_access_key_id = a
        if o in ('-b', '--bucket'):
            bucket_name = a
        if o in ('-d', '--debug'):
            debug = int(a)
        if o in ('-n', '--no_op'):
            no_op = True
        if o in ('-q', '--quiet'):
            quiet = True
        if o in ('-s', '--secret_key'):
            aws_secret_access_key = a
    if len(args) != 1:
        usage()
    path = args[0]
    if bucket_name:
        c = boto.connect_s3(aws_access_key_id=aws_access_key_id,
                            aws_secret_access_key=aws_secret_access_key)
        c.debug = debug
        b = c.get_bucket(bucket_name)
        keys = []
        #a folder 
        if path.endswith("/") and b.get_key(path) is None:
            for key in b.list():
                if key.name.startswith(path):
                    keys.append(key.name)
        else:
            keys = [path]
        for key in keys:
            print "remove " + key
            if not no_op:
                b.delete_key(key)
    else:
        usage()
 
if __name__ == "__main__":
    main()
Posted in python - Tagged amazon, boto, delete, python

使用Python 和Boto 库将文件备份到AmazonS3

五26
2010
1 Comment Written by Filia.Tao

AmazonS3 是一个很好用的云存储服务。 够便宜。 速度也不错。是个备份的好地方。
Boto 是Amazon WebService (包含S3)的Python绑定, 很好用。 (唯一的问题是稍微有一点滞后,比如S3现在已经有了在新加坡的区域。这个还没有更新)
Boto 提供一个很实用的命令行工具s3put, 可以很方便的将一个目录/文件存储到S3上去。
比如下面的脚本将/home/ftao/backup/daily 下面的文件保存在ftaobackup 这个bucket 上面去。

#!/bin/sh
export AWS_ACCESS_KEY_ID=YOUR-AWS-ACCESS-KEY-ID
export AWS_SECRET_ACCESS_KEY=YOUR-AWS-SECRET-ACCESS-KEY
BUCKET_NAME=ftaobackup
/home/ftao/env/bin/s3put -b $BUCKET_NAME --no_overwrite -p /home/ftao/backup/  /home/ftao/backup/daily

在给出一个创建bucket 的代码例子。

def create_or_get_bucket(bucket_name, location):
    import boto
    from boto.exception import S3CreateError
    conn = boto.connect_s3()
    bucket = conn.lookup(bucket_name)
    if bucket is None:
        bucket = conn.create_bucket(bucket_name, None, location)
    else:
        if bucket.get_location() != location:
            raise Exception('bad bucket location')
    return bucket
 
if __name__ == "__main__":
    from boto.s3.connection import Location
    bucket = create_or_get_bucket('ftaobackup', "us-west-1")
    print bucket
Posted in python, 编程开发 - Tagged amazon, boto, python

授权方式

Creative Commons License
本站作品采用
知识共享署名-非商业性使用-相同方式共享 3.0 许可协议
进行许可。

最近评论

  • carlos 发表在《yacc,ast and graphviz》
  • xiang 发表在《关于我》
  • healthy green tea 发表在《debian 同步系统时间》
  • Filia.Tao 发表在《Kinper – A Kindle Helper Service》
  • pensz 发表在《厦门行简单记录》

My Tweets

RSS My KnowHowSpot

标签

指令 汇编 算法 计算机科学 2008 amazon android ast boto C++ C/C++ compiler Computer design-pattern DFA Django ezengage Firefox github google GSoc http imagedownload iterator javascript jquery kindle kinper lex life Linux locationbar Mix opensource proxy python s3 S5Creator shanghai slide STL vector vista web Web开发

分类

  • ideas (2)
  • job (2)
  • life (2)
  • notes (1)
  • opensource (38)
    • Firefox (17)
    • GSoc (7)
    • Linux (13)
  • project (3)
  • 生活 (3)
  • 编程开发 (67)
    • C/C++ (4)
    • GAE (1)
    • http (2)
    • javascript (24)
    • python (20)
    • Web开发 (12)
    • 端口映射工具的实现 (6)
  • 计算机科学 (23)
    • compiler (17)
      • lex (11)
    • 算法 (5)
  • 随便写写 (67)

文章归档

Blogroll

  • 11′s SKY
  • 86's world
  • Filia’s Summer Of Code
  • limodou的学习记录
  • Loki
  • MyAllBlue
  • perol’s blog
  • Realazy
  • 一个藏袍
  • 人猿星球
  • 冰古Blog
  • 刀枪Blue
  • 懶懶喵日記
  • 桑林志
  • 白菜
  • 车东[Blog^2]
  • 释翼的天空
  • 阿文的自留地

开源网站

  • beagle
  • linuxsir
  • sourceforge
  • 中国Linux 公社
  • 啄木鸟社区

我的项目

  • ezEngage
  • KnowHowSpot

EvoLve theme by Theme4Press  •  Powered by WordPress 飞纯技术