業務背景
約定五天前的HDFS數據為過期版本數據，寫一個awk腳本自動刪除過期版本數據
$ Hadoop fs -ls /user/pms/workspace/ouyangyewei/data
Found 9 items
drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-01
drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-02
drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-03
drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-04
drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-05
drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-06
drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-07
drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-08
drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-09
腳本實現
# ---------------------------------------------------------
#
# 刪除歷史版本（五天前的為過期版本數據）
#
# ---------------------------------------------------------
old_version=$(hadoop fs -ls /user/pms/workspace/ouyangyewei/data | awk 'BEGIN{ five_days_ago=strftime("%F", systime()-5*24*3600) }{ split($8,arr,"/"); if(arr[7]<five_days_ago){printf "%s\n", $8} }')
arr=(${old_version// / })
for version in ${arr[@]}
do
hadoop fs -rmr $version
done
執行以後
$ hadoop fs -ls /user/pms/workspace/ouyangyewei/data
Found 4 items
drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-06
drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-07
drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-08
drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-09
awk賦值操作符 http://www.linuxidc.com/Linux/2015-06/118887.htm
awk轉義序列與算術操作符 http://www.linuxidc.com/Linux/2015-06/118886.htm
AWK簡介及使用實例 http://www.linuxidc.com/Linux/2013-12/93519.htm
AWK 簡介和例子 http://www.linuxidc.com/Linux/2012-12/75441.htm
Shell腳本之AWK文本編輯器語法 http://www.linuxidc.com/Linux/2013-11/92787.htm
正則表達式中AWK的學習和使用 http://www.linuxidc.com/Linux/2013-10/91892.htm
文本數據處理之AWK 圖解 http://www.linuxidc.com/Linux/2013-09/89589.htm
如何在Linux中使用awk命令 http://www.linuxidc.com/Linux/2014-10/107542.htm
文本分析工具-awk http://www.linuxidc.com/Linux/2014-12/110939.htm
