tag:blogger.com,1999:blog-10317598919705198892024-03-20T13:18:09.390-07:00尋找Google的創新k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.comBlogger111125tag:blogger.com,1999:blog-1031759891970519889.post-60090387665881035782017-02-28T21:45:00.004-08:002017-02-28T21:45:58.065-08:00有趣的Google container-vm<div style="box-sizing: border-box; color: #333333; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
研究Google在GCE執行datalab的範例中,發現了個頗有趣的事情... 不久前Google針對Optimized Container出了一個container-vm,雖然目前已經預計要deprecate這個image,不過在datalab執行於GCE的範例中M還是使用這個image來執行...</div>
<div style="box-sizing: border-box; color: #333333; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
有趣的地方是,datalab-service.yaml是一個k8s的pod描述檔... 下面是instance執行的gcloud腳本:</div>
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow: visible; padding: 0px; word-break: normal; word-wrap: normal;">gcloud compute instances create "instance-name" \
--project "project-id" \
--zone "zone" \
--network "datalab-network" \
--image-family "container-vm" \
--image-project "google-containers" \
--metadata-from-file "google-container-manifest=datalab-server.yaml" \
--machine-type "n1-highmem-2" \
--scopes "cloud-platform"
</code></pre>
<div style="box-sizing: border-box; color: #333333; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
打開datalab-server.yaml檢視,內容如下:</div>
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow: visible; padding: 0px; word-break: normal; word-wrap: normal;">apiVersion: v1
kind: Pod
metadata:
name: 'datalab-server'
spec:
containers:
- name: datalab
image: gcr.io/cloud-datalab/datalab:local
command: ['/datalab/run.sh']
imagePullPolicy: Always
ports:
- containerPort: 8080
hostPort: 8080
hostIP: 127.0.0.1
env:
- name: DATALAB_ENV
value: GCE
volumeMounts:
- name: home
mountPath: /content
- name: logger
image: gcr.io/google_containers/fluentd-gcp:1.18
env:
- name: FLUENTD_ARGS
value: -q
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
volumes:
- name: home
hostPath:
path: /root
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
</code></pre>
<div style="box-sizing: border-box; color: #333333; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 16px;">
這本身是個k8s pod的描述檔,本來需要透過kubernetes cluster才能夠執行,但是卻可以在container-vm中執行... 檢查了一下,原來是透過kubelet來執行... 研究一下kubelet執行container的方式,目前找到可以在/etc/kubernetes/manifests下面直接放置yaml檔案,然後重啟kubelet(/etc/init.d/kubelet restart)即可。</div>
<h2 style="border-bottom: 1px solid rgb(238, 238, 238); box-sizing: border-box; color: #333333; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; line-height: 1.25; margin-bottom: 16px; margin-top: 24px; padding-bottom: 0.3em;">
<a aria-hidden="true" class="anchor" href="https://github.com/peihsinsu/notebook/blob/master/google-container-vm.md#reference" id="user-content-reference" style="box-sizing: border-box; color: #4078c0; float: left; line-height: 1; margin-left: -20px; padding-right: 4px; text-decoration: none;"><svg aria-hidden="true" class="octicon octicon-link" height="16" version="1.1" viewbox="0 0 16 16" width="16"><path d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z" fill-rule="evenodd"></path></svg></a>Reference</h2>
<ul style="box-sizing: border-box; color: #333333; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; margin-bottom: 0px !important; margin-top: 0px; padding-left: 2em;">
<li style="box-sizing: border-box;">Datalab run on GCE doc: <a href="https://cloud.google.com/datalab/docs/quickstarts/quickstart-gcp-frontend" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">https://cloud.google.com/datalab/docs/quickstarts/quickstart-gcp-frontend</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;">Google container-vm doc: <a href="https://cloud.google.com/compute/docs/containers/container_vms" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">https://cloud.google.com/compute/docs/containers/container_vms</a></li>
<li style="box-sizing: border-box; margin-top: 0.25em;">New version of container optimized vm: <a href="https://cloud.google.com/container-optimized-os/docs/" style="box-sizing: border-box; color: #4078c0; text-decoration: none;">https://cloud.google.com/container-optimized-os/docs/</a></li>
</ul>
k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com1tag:blogger.com,1999:blog-1031759891970519889.post-23809906523723210512017-02-27T07:53:00.000-08:002017-02-27T07:53:06.257-08:00為GCS加上HTTP(S) Load Balancer與CDNGCS可以串連HLB了,也可以附加CDN功能了!也就是說,未來幫任何的GCS設定frontend存取的位置,甚至加上SSL都已經不是夢想,還在嫌架設主機太麻煩嗎?<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/jZ13GGJW1U4/0.jpg" frameborder="0" height="320" src="https://www.youtube.com/embed/jZ13GGJW1U4?feature=player_embedded" width="480"></iframe></div>
<br />k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-65209497332141620022017-02-27T07:48:00.003-08:002017-02-27T07:50:23.209-08:00GCS website bucket YouTube這是GCS Website Bucket的設定過程,動手比較有感覺,給大家參考一下~<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/zn7vWGjsms0/0.jpg" frameborder="0" height="320" src="https://www.youtube.com/embed/zn7vWGjsms0?feature=player_embedded" width="480"></iframe></div>
<br />k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-71119954277170399572017-02-27T03:08:00.004-08:002017-02-27T03:08:52.437-08:00第一次接觸Google API就上手很久以前分享的文章,關於使用Google API的一些做法,因為最近看到比較多人開始考慮使用Google Machine Learning相關API,想說再貼出來給大家參考一下~<br />
<br />
http://www.ithome.com.tw/guest-post/98590k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-31425514940292227122016-08-31T09:01:00.002-07:002016-08-31T09:01:15.028-07:00PubSub入門<h1 style="-webkit-user-select: text; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; letter-spacing: 0.2px; line-height: 1.1; margin-bottom: 0.85em; margin-left: 0px; margin-right: 0px; margin-top: 0px !important;">
<span style="font-size: 14px; font-weight: normal; letter-spacing: 0.2px; line-height: 23.8px; orphans: 3; widows: 3;">Cloud PubSub是Google Cloud Platform上的一個Publish Subscriber的服務,讓使用者可以透過API將資料放到PubSub上,並且透過建立Subscriber讓其他的城市可以讀取該資料做進一步的處理...</span></h1>
<div style="-webkit-user-select: text; background-color: white; box-sizing: border-box; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; letter-spacing: 0.2px; line-height: 23.8px; margin-bottom: 0.85em; orphans: 3; widows: 3;">
PubSub的特性如下:</div>
<ul style="-webkit-user-select: text; background-color: white; box-sizing: border-box; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; letter-spacing: 0.2px; line-height: 23.8px; margin: 0px 0px 0.85em; padding: 0px 0px 0px 2em;">
<li style="-webkit-user-select: text; box-sizing: border-box; cursor: text; font-size: inherit;">作爲應用程式服務的中介,供連接異質性來源的系統作為資料傳遞之用</li>
<li style="-webkit-user-select: text; box-sizing: border-box; cursor: text; font-size: inherit;">支援Push與Pull兩種方式,其中Push可以讓使用者設定Push URL,讓系統可以直接再接收到訊息時,呼叫push url</li>
<li style="-webkit-user-select: text; box-sizing: border-box; cursor: text; font-size: inherit;">提供“at least once” delivery的送達保證,並且資料在傳輸過程中均有加密保護</li>
<li style="-webkit-user-select: text; box-sizing: border-box; cursor: text; font-size: inherit;">PubSub是一個全球化的服務,並且會依照使用者之彈性需求自動調整資源</li>
</ul>
<div style="-webkit-user-select: text; background-color: white; box-sizing: border-box; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; letter-spacing: 0.2px; line-height: 23.8px; margin-bottom: 0.85em; orphans: 3; widows: 3;">
<img alt="pubsub push vs pull model" src="https://cloud.google.com/pubsub/images/push_pull.svg" style="-webkit-user-select: text; border: 0px; box-sizing: border-box; break-inside: avoid; cursor: text; font-size: inherit; max-width: 100%;" />概念上,topic是讓您放您資料的位置,然後需要在該topic下面建立subscriber,讓程式可以在該subscriber下讀取到放置在該topic的資料,再透過ACK這個動作回覆系統,確認已經被讀取...</div>
<h2 style="-webkit-user-select: text; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.75em; letter-spacing: 0.2px; line-height: 1.1; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
建立topic</h2>
<pre style="-webkit-user-select: text; background: rgb(247, 247, 247); border-radius: 4px; border: none; box-sizing: border-box; break-inside: avoid; color: #333333; cursor: text; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 14px; letter-spacing: 0.2px; line-height: 1.42857; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; word-break: break-all; word-wrap: normal;"><code style="-webkit-user-select: text; background: 0px 0px; border-radius: 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; cursor: text; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px;">$ gcloud --format=json alpha pubsub topics create my-topic
[
{
"reason": "",
"success": true,
"topicId": "projects/sunny-573/topics/my-topic"
}
]
</code></pre>
<h2 style="-webkit-user-select: text; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.75em; letter-spacing: 0.2px; line-height: 1.1; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
列表topics</h2>
<pre style="-webkit-user-select: text; background: rgb(247, 247, 247); border-radius: 4px; border: none; box-sizing: border-box; break-inside: avoid; color: #333333; cursor: text; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 14px; letter-spacing: 0.2px; line-height: 1.42857; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; word-break: break-all; word-wrap: normal;"><code style="-webkit-user-select: text; background: 0px 0px; border-radius: 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; cursor: text; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px;">$ gcloud alpha pubsub topics list
---
topic: projects/sunny-573/topics/my-topic
topicId: my-topic
...
</code></pre>
<h2 style="-webkit-user-select: text; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.75em; letter-spacing: 0.2px; line-height: 1.1; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
Push訊息到pubsub</h2>
<pre style="-webkit-user-select: text; background: rgb(247, 247, 247); border-radius: 4px; border: none; box-sizing: border-box; break-inside: avoid; color: #333333; cursor: text; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 14px; letter-spacing: 0.2px; line-height: 1.42857; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; word-break: break-all; word-wrap: normal;"><code style="-webkit-user-select: text; background: 0px 0px; border-radius: 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; cursor: text; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px;">$ gcloud alpha pubsub topics publish my-topic '{"aaa":123,"bbb":223}'
</code></pre>
<h2 style="-webkit-user-select: text; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.75em; letter-spacing: 0.2px; line-height: 1.1; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
為topic建立一個subscriber</h2>
<pre style="-webkit-user-select: text; background: rgb(247, 247, 247); border-radius: 4px; border: none; box-sizing: border-box; break-inside: avoid; color: #333333; cursor: text; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 14px; letter-spacing: 0.2px; line-height: 1.42857; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; word-break: break-all; word-wrap: normal;"><code style="-webkit-user-select: text; background: 0px 0px; border-radius: 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; cursor: text; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px;">$ gcloud alpha pubsub subscriptions create sub002 --topic my-topic
---
ackDeadlineSeconds: 10
pushEndpoint: null
reason: ''
subscriptionId: projects/sunny-573/subscriptions/sub002
success: true
topic: projects/sunny-573/topics/my-topic
type: push
</code></pre>
<h2 style="-webkit-user-select: text; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.75em; letter-spacing: 0.2px; line-height: 1.1; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
列表subscriber</h2>
<pre style="-webkit-user-select: text; background: rgb(247, 247, 247); border-radius: 4px; border: none; box-sizing: border-box; break-inside: avoid; color: #333333; cursor: text; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 14px; letter-spacing: 0.2px; line-height: 1.42857; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; word-break: break-all; word-wrap: normal;"><code style="-webkit-user-select: text; background: 0px 0px; border-radius: 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; cursor: text; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px;">$ gcloud alpha pubsub subscriptions list
┌───────────┬──────────────┬──────────┬──────┬──────────────┐
│ PROJECT │ SUBSCRIPTION │ TOPIC │ TYPE │ ACK_DEADLINE │
├───────────┼──────────────┼──────────┼──────┼──────────────┤
│ sunny-573 │ sub001 │ my-topic │ PULL │ 10 │
│ sunny-573 │ sub002 │ my-topic │ PULL │ 10 │
│ sunny-573 │ sub01 │ test01 │ PUSH │ 10 │
│ sunny-573 │ sunny │ sunny │ PULL │ 10 │
└───────────┴──────────────┴──────────┴──────┴──────────────┘
</code></pre>
<h2 style="-webkit-user-select: text; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.75em; letter-spacing: 0.2px; line-height: 1.1; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
從subscribers取資料</h2>
<div style="-webkit-user-select: text; background-color: white; box-sizing: border-box; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; letter-spacing: 0.2px; line-height: 23.8px; margin-bottom: 0.85em; orphans: 3; widows: 3;">
取資料可以分成兩個動作:讀取、回覆ACK,當只有讀取動作時,其他針對同subscriber的查詢,仍可以查詢得到資料... 其中,該筆訊息會帶ack_id,可以讓未來針對該ack_id來做回覆ack的動作。</div>
<pre style="-webkit-user-select: text; background: rgb(247, 247, 247); border-radius: 4px; border: none; box-sizing: border-box; break-inside: avoid; color: #333333; cursor: text; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 14px; letter-spacing: 0.2px; line-height: 1.42857; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; word-break: break-all; word-wrap: normal;"><code style="-webkit-user-select: text; background: 0px 0px; border-radius: 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; cursor: text; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px;">$ gcloud alpha pubsub subscriptions pull sub001
┌───────────────────────┬────────────────┬────────────┬────────────────────────────────┐
│ DATA │ MESSAGE_ID │ ATTRIBUTES │ ACK_ID │
├───────────────────────┼────────────────┼────────────┼────────────────────────────────┤
│ {"aaa":123,"bbb":223} │ 43961024144056 │ │ MTJFQV5AEkw6...4cqZhg9XxJLLD5- │
└───────────────────────┴────────────────┴────────────┴────────────────────────────────┘
</code></pre>
<div style="-webkit-user-select: text; background-color: white; box-sizing: border-box; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; letter-spacing: 0.2px; line-height: 23.8px; margin-bottom: 0.85em; orphans: 3; widows: 3;">
下面是針對ack_id做回覆的動作:</div>
<pre style="-webkit-user-select: text; background: rgb(247, 247, 247); border-radius: 4px; border: none; box-sizing: border-box; break-inside: avoid; color: #333333; cursor: text; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 14px; letter-spacing: 0.2px; line-height: 1.42857; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; word-break: break-all; word-wrap: normal;"><code style="-webkit-user-select: text; background: 0px 0px; border-radius: 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; cursor: text; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px;">$ gcloud alpha pubsub subscriptions ack sub001 MkVBXkASTDo...JLLD5-MQ
ackIds:
- MkVBXkASTDo...JLLD5-MQ
subscriptionId: projects/sunny-573/subscriptions/sub001
</code></pre>
<div style="-webkit-user-select: text; background-color: white; box-sizing: border-box; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; letter-spacing: 0.2px; line-height: 23.8px; margin-bottom: 0.85em; orphans: 3; widows: 3;">
如果嫌麻煩... 在gcloud指令有支援--auto-ack,可以針對request data時候,直接在request完成回覆ack,也就是告訴該subscriber這筆資料已經接收,其他人就無法在讀取該筆資料... 下面是針對subscriber加上auto-ack的結果,原則上ack完之後,ack_id將會消失。</div>
<pre style="-webkit-user-select: text; background: rgb(247, 247, 247); border-radius: 4px; border: none; box-sizing: border-box; break-inside: avoid; color: #333333; cursor: text; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 14px; letter-spacing: 0.2px; line-height: 1.42857; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; word-break: break-all; word-wrap: normal;"><code style="-webkit-user-select: text; background: 0px 0px; border-radius: 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; cursor: text; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px;">$ gcloud alpha pubsub subscriptions pull sub001 --auto-ack
┌───────────────────────┬────────────────┬────────────┐
│ DATA │ MESSAGE_ID │ ATTRIBUTES │
├───────────────────────┼────────────────┼────────────┤
│ {"aaa":123,"bbb":423} │ 43961235494393 │ │
└───────────────────────┴────────────────┴────────────┘
</code></pre>
<h2 style="-webkit-user-select: text; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.75em; letter-spacing: 0.2px; line-height: 1.1; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
刪除測試資源</h2>
<div style="-webkit-user-select: text; background-color: white; box-sizing: border-box; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; letter-spacing: 0.2px; line-height: 23.8px; margin-bottom: 0.85em; orphans: 3; widows: 3;">
在測試完成後,建議就順手將建立的資源刪除,下面是刪除subscriber與topic的指令:</div>
<div style="-webkit-user-select: text; background-color: white; box-sizing: border-box; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; letter-spacing: 0.2px; line-height: 23.8px; margin-bottom: 0.85em; orphans: 3; widows: 3;">
刪除subscriber:</div>
<pre style="-webkit-user-select: text; background: rgb(247, 247, 247); border-radius: 4px; border: none; box-sizing: border-box; break-inside: avoid; color: #333333; cursor: text; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 14px; letter-spacing: 0.2px; line-height: 1.42857; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; word-break: break-all; word-wrap: normal;"><code style="-webkit-user-select: text; background: 0px 0px; border-radius: 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; cursor: text; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px;">$ gcloud --format=json alpha pubsub subscriptions delete sub002
[
[
"projects/sunny-573/subscriptions/sub002"
],
[]
]
</code></pre>
<div style="-webkit-user-select: text; background-color: white; box-sizing: border-box; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; letter-spacing: 0.2px; line-height: 23.8px; margin-bottom: 0.85em; orphans: 3; widows: 3;">
刪除topic:</div>
<pre style="-webkit-user-select: text; background: rgb(247, 247, 247); border-radius: 4px; border: none; box-sizing: border-box; break-inside: avoid; color: #333333; cursor: text; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 14px; letter-spacing: 0.2px; line-height: 1.42857; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; word-break: break-all; word-wrap: normal;"><code style="-webkit-user-select: text; background: 0px 0px; border-radius: 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; cursor: text; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px;">$ gcloud --format=json alpha pubsub topics delete my-topic
[
{
"reason": "",
"success": true,
"topicId": "projects/sunny-573/topics/my-topic"
}
]
</code></pre>
<h2 style="-webkit-user-select: text; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.75em; letter-spacing: 0.2px; line-height: 1.1; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
注意事項</h2>
<ul style="-webkit-user-select: text; background-color: white; box-sizing: border-box; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; letter-spacing: 0.2px; line-height: 23.8px; margin: 0px 0px 0.85em; padding: 0px 0px 0px 2em;">
<li style="-webkit-user-select: text; box-sizing: border-box; cursor: text; font-size: inherit;">由於pubsub服務的特性是保證至少一次獨取,因此不排除會發生最終同步前重複獨取料的問題。如果擔心這類問題,建議透過中介服務來擷取pubsub資料,然後再做第二層的分派處理。</li>
<li style="-webkit-user-select: text; box-sizing: border-box; cursor: text; font-size: inherit;">由於pubsub支援一對多的應用,如果在一個topic中設定一個以上的subscriber時候,則每個subscriber會收到一樣的訊息。</li>
</ul>
<h2 style="-webkit-user-select: text; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.75em; letter-spacing: 0.2px; line-height: 1.1; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
參考</h2>
<ul style="-webkit-user-select: text; background-color: white; box-sizing: border-box; color: #333333; cursor: text; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; letter-spacing: 0.2px; line-height: 23.8px; margin-bottom: 0px !important; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding: 0px 0px 0px 2em;">
<li style="-webkit-user-select: text; box-sizing: border-box; cursor: text; font-size: inherit;">PubSub簡介:<a href="https://www.gitbook.com/book/gcpug-tw/google-cloud-platform-in-practice/edit#" style="-webkit-user-select: text; background: 0px 0px; box-sizing: border-box; color: #4183c4; cursor: text; font-size: inherit; text-decoration: none;">https://cloud.google.com/pubsub/</a></li>
<li style="-webkit-user-select: text; box-sizing: border-box; cursor: text; font-size: inherit;">PubSub SLA: <a href="https://www.gitbook.com/book/gcpug-tw/google-cloud-platform-in-practice/edit#" style="-webkit-user-select: text; background: 0px 0px; box-sizing: border-box; color: #4183c4; cursor: text; font-size: inherit; text-decoration: none;">https://cloud.google.com/pubsub/sla</a></li>
</ul>
k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-88673130352543643522016-07-14T01:55:00.005-07:002016-07-14T01:55:58.193-07:00最精簡的K8S執行環境 - MiniKube<h1 id="最精簡的k8s執行環境---minikube" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; letter-spacing: 0.2px; margin-bottom: 0.85em; margin-left: 0px; margin-right: 0px; margin-top: 0px !important;">
<span style="font-size: 16px; font-weight: normal; letter-spacing: 0.2px; line-height: 27.2px; orphans: 3; widows: 3;">kubernetes.io發佈了一個供本地端單機跑kubernetes最精簡的環境minikube,透過minikube可以很快速的使用與測試k8s的功能唷~</span></h1>
<h2 id="安裝minikube" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.75em; letter-spacing: 0.2px; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
安裝minikube</h2>
<div class="comments-section" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 0.85em; orphans: 3; position: relative; widows: 3;">
安裝的部分要看您的主機是哪種OS,下面是Mac OS的安裝方式</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: rgb(247, 247, 247); border: none; box-sizing: border-box; break-inside: avoid; color: #333333; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: 0px 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre;">curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.6.0/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
</code></pre>
<div class="comments-section" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 0.85em; orphans: 3; position: relative; widows: 3;">
其他的操作部分都是使用kubectl,可以參考kubernetes的指令...</div>
<h2 id="其他一功能" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.75em; letter-spacing: 0.2px; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
其他一功能</h2>
<div class="comments-section" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 0.85em; orphans: 3; position: relative; widows: 3;">
透過minikube的指令,可以看到在Mac上,其實是使用Virtualbox來啟動裝有kubernetes的boot2docker環境。</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: rgb(247, 247, 247); border: none; box-sizing: border-box; break-inside: avoid; color: #333333; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: 0px 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre;"># ps -ef | grep minikube
502 14260 14236 0 11:49上午 ?? 21:11.73 /Applications/VirtualBox.app/Contents/MacOS/VBoxHeadless --comment minikubeVM --startvm cb568998-d8b4-40d4-a6e7-c92aea5aee3d --vrde config
502 18986 4258 0 3:59下午 ttys008 0:00.00 grep --color=auto minikube
</code></pre>
<div class="comments-section" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 0.85em; orphans: 3; position: relative; widows: 3;">
minikube除了提供k8s環境之外,而且也連結了這些環境提供一些不錯的操控方式,這邊紀錄幾個看到還不錯用的功能...</div>
<h3 id="檢視版本資訊" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.5em; letter-spacing: 0.2px; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
檢視版本資訊</h3>
<div class="comments-section" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 0.85em; orphans: 3; position: relative; widows: 3;">
minikube version</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: rgb(247, 247, 247); border: none; box-sizing: border-box; break-inside: avoid; color: #333333; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: 0px 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre;"># minikube version
minikube version: v0.6.0
</code></pre>
<div class="comments-section" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 0.85em; orphans: 3; position: relative; widows: 3;">
k8s version</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: rgb(247, 247, 247); border: none; box-sizing: border-box; break-inside: avoid; color: #333333; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: 0px 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre;"># minikube get-k8s-versions
The following Kubernetes versions are available:
- v1.3.0
</code></pre>
<h3 id="minikube的儀表板" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.5em; letter-spacing: 0.2px; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
Minikube的儀表板</h3>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: rgb(247, 247, 247); border: none; box-sizing: border-box; break-inside: avoid; color: #333333; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: 0px 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre;">minikube dashboard
</code></pre>
<div class="comments-section" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 0.85em; orphans: 3; position: relative; widows: 3;">
此時,桌面會跳出browser的畫面,裡面是kubernetes的dashboard,這版本的dashboard已經提供了直接在網頁介面Deploy App的功能,除此之外,也可以透過yaml來作部署的動作...</div>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 0.85em; orphans: 3; widows: 3;">
<img alt="minikube dashboard" src="https://peihsinsu.gitbooks.io/docker-note-book/content/assets/minikube-k8s-dashboard.png" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border: 0px; box-sizing: border-box; break-inside: avoid; font-size: inherit; max-width: 100%;" /></div>
<h3 id="minikube-docker-env" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.5em; letter-spacing: 0.2px; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
minikube docker env</h3>
<div class="comments-section" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 0.85em; orphans: 3; position: relative; widows: 3;">
這個指令提供顯示目前minikube的docker環境的連線參數,可以讓外部的docker client透過這些參數來連線minikube環境做進階操控。</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: rgb(247, 247, 247); border: none; box-sizing: border-box; break-inside: avoid; color: #333333; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: 0px 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre;"># minikube docker-env
export DOCKER_TLS_VERIFY=1
export DOCKER_HOST=tcp://192.168.99.100:2376
export DOCKER_CERT_PATH=/Users/peihsinsu/.minikube/certs
# Run this command to configure your shell:
# eval $(minikube docker-env)
</code></pre>
<h3 id="關閉與重新啟動" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.5em; letter-spacing: 0.2px; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
關閉與重新啟動</h3>
<div class="comments-section" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 0.85em; orphans: 3; position: relative; widows: 3;">
停掉minikube</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: rgb(247, 247, 247); border: none; box-sizing: border-box; break-inside: avoid; color: #333333; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: 0px 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre;"># minikube stop
Stopping local Kubernetes cluster...
Machine stopped.
</code></pre>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 0.85em; orphans: 3; widows: 3;">
確認狀態</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: rgb(247, 247, 247); border: none; box-sizing: border-box; break-inside: avoid; color: #333333; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: 0px 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre;"># minikube status
Stopped
</code></pre>
<div class="comments-section" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 0.85em; orphans: 3; position: relative; widows: 3;">
啟動minikube</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: rgb(247, 247, 247); border: none; box-sizing: border-box; break-inside: avoid; color: #333333; direction: ltr; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 1.275em; overflow: auto; padding: 0.85em 1em; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: 0px 0px; border: none; box-sizing: border-box; break-inside: avoid; color: inherit; direction: ltr; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 0.85em; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre;"># minikube start
Starting local Kubernetes cluster...
Kubernetes is available at https://192.168.99.100:8443.
Kubectl is now configured to use the cluster.
</code></pre>
<h3 id="ssh連線到minikube虛擬主機中" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; break-after: avoid; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 1.5em; letter-spacing: 0.2px; margin-bottom: 0.85em; margin-top: 1.275em; orphans: 3; widows: 3;">
ssh連線到minikube虛擬主機中</h3>
<div class="comments-section" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; margin-bottom: 0.85em; orphans: 3; position: relative; widows: 3;">
要連線到minikube主機時,可以透過minikube ssh的指令...</div>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; orphans: 3; widows: 3;">
<img alt="minikube ssh" src="https://peihsinsu.gitbooks.io/docker-note-book/content/assets/minikube-ssh.png" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border: 0px; box-sizing: border-box; break-inside: avoid; font-size: inherit; max-width: 100%;" /></div>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; orphans: 3; widows: 3;">
<br /></div>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: white; box-sizing: border-box; color: #333333; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0.2px; line-height: 27.2px; orphans: 3; widows: 3;">
文章來自:https://peihsinsu.gitbooks.io/docker-note-book/content/the-mini-env-of-k8s---minikube.html</div>
k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-27550204482434953132016-06-22T10:02:00.001-07:002016-06-22T10:02:29.951-07:00Google Cloud上的大資料工具探討 - Lambda Architecture原來這麼簡單<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="font-family: "Trebuchet MS"; font-size: 21.3333px; line-height: 1.38; white-space: pre-wrap;">大資料的環境需求</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">步上雲端的時代,運算與儲存空間隨著莫爾定律的實踐慢慢變得越來越便宜,而大資料的議題也越來越被重視,在Google這麼一個以大資料起家的企業,從一開始的分散式儲存體GFS(Google File System)到大資料分析工具MapReduce的概念,讓我們知道在大資料處理上,儲存(Input)與讀取(Output)兩個不可或缺的環節。</span></div>
<b style="font-weight: normal;"><br /></b>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17.333333333333332px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">分散式儲存</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">在傳統的架構上,存在一顆硬碟或資料庫的資料,要讓多台主機同時存取並分析資料會有許多衝突問題需要解決。而且,在雲端的概念中,要訴求一台超級電腦似乎永遠是成本遠高於利潤的作法,在雲端分毫必爭的資源環境,Google是以小而多的方式來取勝。而GFS正式以此方式讓儲存可以無限延伸,不只空間的部分得以延伸,連存取該空間所需要用到的運算能力,也可以在這個架構上得到滿足。</span></div>
<b style="font-weight: normal;"><br /></b>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17.333333333333332px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">有效率的查詢方式</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">MapReduce是一種沒有index的查詢方式,從頭開始看資料,再透過所給訂的條件作過濾... 原本看似毫無效能的運作,在GFS之後漸漸變成可能... 由於GFS提供了平行的存取架構,讓原本查詢需要在一台機器上運作的傳統被打破。而基於GFS的可過展儲存方式,同一份資料可以由眾多的主機同時讀取,因此Google實作了MapReduce的架構,可以同時間讓多個運算資源存取同一份資料,因此只要使用正比於資料的大小的機器來運算,就可以把處理的時間壓縮到可以忍受的範圍。</span></div>
<b style="font-weight: normal;"><br /></b>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17.333333333333332px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">更多的資源調用方法</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">當最困難的問題都已經解決,接下來的問題就相對單純了許多,而且,在現階段不論是網路環境、運算資源、儲存資源等等都持續不斷提升之下,資源的價格也在不斷的降低,這意味著適合大資料的環境已經具備... 接下來,開發者們可以集中在思考資料有意義的地方。但在資源無虞的情況下,使用資源的方式則需要再好好思考,如何自動化、如何因應量大的儲存、處理...</span></div>
<b style="font-weight: normal;"><br /></b>
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">部署管理:如何在最短的時間內部署您的服務,讓他可以承受最大的request、儲存最多的資料,並且具備HA的架構...</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">自動化拓展(Auto Scale):延伸部署管理的另一個應用層面,如何在不影響服務的層面下,自動化擴展您的伺服器群,並且可以在離峰時間將資源釋放。</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">全球網路:雲端時代在步入全球存取的狀況下,我們在思考如何讓全世界的使用者可以存取無礙,已經成為雲端的課題之一。</span></div>
</li>
</ol>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">剩下的,就是轉換傳統的思維,讓我們開始思考哪些是現存的資料,哪些是預計可以再收入的資料,結合這些資料可以得到哪些幫助。最重要的是,儲存、分析的成本,是否可以在未來進一步的轉換成為獲益!</span></div>
<b style="font-weight: normal;"><br /></b>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">什麼是Lambda Architecture</span></h1>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">資料量大,意味著讀取分析的Cost變得更高,在傳統的技術稱作Data Warehouse(DW),資料在資料中心以許多大型儲存裝備儲存著,而在分析前,需要先行依照使用到的資料來建立Cube,來減少資料量,以方便後續進行分析使用。延伸這樣的概念,我們可以想像在搜集大量資料時,可以將資料的輸入分別存入批次區、即時區儲存體。在需要長時間統計時候,可以參考批次區的資料,他的運作會花較多時間,Cost比較高;而因應短期分析需要,我們可以從即時區參考,他的資料量比較少,查詢比較快,Cost相對比較低... 如此,可以讓分析的動作得到完整與即時兩個項面,這就是Lambda Architecture的精髓。</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="342" src="https://lh6.googleusercontent.com/SadfJwkgju9767k880rSvNiR6mUapdalhdjpladp7glgxSuXztebLbrXG1CcXnCJB_nsxUqCKGwAA0Ni0byN84vKpnuT0v-J_a65qdvQ2evoNNs0ErVOShAqC2RBy8mVUksXtoE" style="border: none; transform: rotate(0rad);" width="600" /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Lambda Architecture (</span><a href="http://lambda-architecture.net/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://lambda-architecture.net/</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">)</span></div>
<b style="font-weight: normal;"><br /></b><br />
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Google Cloud Platform上有哪些相關工具</span></h1>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">對應到正在蓬勃發展中的Cloud,Google Cloud Platform(GCP)以自己的經驗轉化成服務,提供了相對應的技術讓您可以收容這些巨大資料,並且在幾近無限的運算資源輔助之下,您可以在GCP上使用最經濟實惠的方式完成大資料分析!</span></div>
<b style="font-weight: normal;"><br /></b>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17.333333333333332px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Google Cloud Storage</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">在儲存面,基於GFS的概念,Google Cloud Storage(GCS)的建構以分散為概念,透過Replica的方式將資料複製到一個以上的實體儲存裝置上,並將存取服務的方式包裝成API介面,供其他系統做介接存取。Cloud Storage在大資料服務上,佔有一個非常重要的地位,他提供以物件為單位的存放方式、物件的儲存沒有上線、支援多種存取的方式(包括:API, SDK, Fuse等等),並且在大量讀取時,存取的Endpoint具備快取及自動擴展的特性。</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="197" src="https://lh3.googleusercontent.com/osHea1ZpShzjk6FKpFvXEoLnGrcIFiDmi4xgBrG6xseBJbIk1-k0yvHSq-H-0xSurdCCyxHFQgzuHgGZ_7RivKPKUsf4uuE31OwOSANo5J786nb79OEOvErDMNCiHvo7sU2ehn0" style="border: none; transform: rotate(0rad);" width="640" /></span></div>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17.333333333333332px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Cloud Pub/Sub</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Cloud Pub/Sub可以想像成是雲端的儲存佇列(Queue),具備發佈跟訂閱的特性,讓開發者可以不用思考收資料的端點(Endpoint)的擴展,並且可以結合Google Cloud內部的許多應用來訂閱消化進入到該佇列的資料,並進行進一步的資料處理。</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="159" src="https://lh5.googleusercontent.com/eEmWDcCRla77WWqqqAZyccxapPzNjU3q54frTYIBgaJ7eAqhWB-D2lLyKaZy1ib8e532SwNps9DrjBeuBkROCK19iWL3RMPoGjPvqtT4HmKz9Em11gu7YFSMM8J0_h70lfT5iKk" style="border: none; transform: rotate(0rad);" width="640" /></span></div>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17.333333333333332px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">BigQuery</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">BigQuery是Google以MapReduce概念所實作的大資料儲存與查詢服務,在概念上,BigQuery以Column Base的方式來儲存所有的資料,提供SQL like的查詢介面,並且以大量的主機同時Full Table Scan的方法快速的過濾完所給定的條件,再做資料的整合,以資料集的方式回覆給查詢端。在BigQuery中,具備前端的資料載入端點(Streaming Insert, Cloud Storage Insert)、無限的儲存空間以及快速的反應速度。開發者可以使用最簡單的方式I/O BigQuery的服務!</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="214" src="https://lh6.googleusercontent.com/QrTHJo8Hbj7F6RIhyLAFK7fPfzCEuho5aaqP96dClYFEfW6OEr8YATV2k_s-1a6tDQUPRQQdJxe3pjZt9CJ4lHhuBC9nSO40NMr0IZVWhXJBtnCwuj5CuhyIoick3OYA1Id2z2c" style="border: none; transform: rotate(0rad);" width="640" /></span></div>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17.333333333333332px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Cloud Logging</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Cloud Logging是Google儲存大量記錄檔案的服務,接收實作Fluentd的Client端所傳回的資料,讓所有以Fluentd做ETL資料轉換的服務可以無縫接軌到Google的雲端環境,Cloud Logging的儲存僅是一個方便資料查詢的暫存區,提供一些簡單的過濾方式讓資料可以被查詢... 性質上,他可以作為Lambda Architecture中的即時資料呈現層,讓您可以檢索近一個月所載入的記錄資訊。</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="211" src="https://lh6.googleusercontent.com/7xWAk5pylciEBX8qess1ST12K7TIB3C7UkyRu4KM-ftZSS5RjE9SIPFbmjo6XLcO9Vp1IDJW5aXfC4fOs2ut1O_K4zuRiP3uFpLfcJX8GXo7rA7wGv9kvA0k1KmBAw8aIK8xrKA" style="border: none; transform: rotate(0rad);" width="640" /></span></div>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17.333333333333332px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Cloud Monitor</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Cloud Monitor是Google所提供的Cloud Monitor API與Stackdriver所提供的Monitor Console所整合提供的一個監控服務,除了監控的數值、圖表之外,也包含了客製化告警通知功能。另外,Cloud Monitor也支援透過RESTful API來抓取監控的資料來進一步處理,以及可以把自訂的資料輸入到Custom Metrics中,達到進階監控的目的。</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="315" src="https://lh6.googleusercontent.com/D5a30YWZ4EENrJJ64EXceBpWigh2RASHGqBhgMQprEP9kPjeMbiP44c7JtKYoBjTtmaTiHAg2PBBgxy1UGUIcQ-HHDkc2kzjTgS39Jj89w5QaZoWgq3RxVaLFHXMHxgmt4kCVD0" style="border: none; transform: rotate(0rad);" width="640" /></span></div>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17.333333333333332px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Dataflow</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Dataflow是Google新推出的大資料處理方案,是一套承襲MapReduce概念的系統,以Java程式來建構整個大資料處理流程。在Dataflow,資料來源可以承接Cloud Pub/Sub、Cloud Storage以及BigQuery的資料,並且在處理完成後,回寫回Cloud Storage與BigQuery,再搭配其他的大資料處理工具進行進一步的分析作業。而整體的運作過程,由Google幫您解決處理程式的Scale out問題,以及分散運算後的資料整合問題。</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="346" src="https://lh4.googleusercontent.com/JPyvOIPX70tXTRixkRTI6jZTOmV_6ej9gVesxf3_EyLFQQq0-zxN0EDPVdaOJMdExuInNqUdrbXJv88k4m_rLtbjihnVlxiz7hAiaNXz6kHXN6XvRuD6nuD_RGh-sXZW6q83l3Y" style="border: none; transform: rotate(0rad);" width="640" /></span></div>
<b style="font-weight: normal;"><br /></b><br />
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">一步一步建立Google上的Lambda Architecture</span></h1>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">在了解Google上的服務之後,稍微整理一下我們可以間單應用Google的服務在這些大資料的需求中...下面是一個簡單的資料蒐集平台,該平台希望做到大量的資料蒐集,並可以在最少的系統維運的狀態下運作,因此我們希望:</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">平台的資料進入點可以簡單設定即可蒐集資料</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">末端儲存裝置可以收容無限的資料,並提供簡單的分析方式</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">呈現端可以以服務方式呈現,以設定的方式提供給使用者</span></div>
</li>
</ul>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">依據上面的需求,我們透過最通用的資料傳輸工具 - Fluentd來進行末端的資料蒐集作業,搭配Cloud Logging為核心作為資料蒐集的中介層,然後設定將資料載入到BigQuery服務中,最後透過Cloud Monitor服務作為Dashboard平台來達到圖表呈現與告警的目的。對照Lambda Architecture,我們可以分類如下:</span></div>
<b style="font-weight: normal;"><br /></b>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Speed Layer:</span></div>
</li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">使用Cloud Logging的即時呈現功能,可以顯示近30天的資料,即時呈現於Cloud Logging中。</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">在fluentd client端以延伸fluentd功能設定之方式,讓資料可以同時間複製一份給即時呈現層,呈現即時報表。</span></div>
</li>
</ul>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Batch Layer</span></div>
</li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">以BigQuery為大資料儲存的最後儲存目的地,並可以透過Table Decorator[1]做小範圍的資料集合或透過Daily Table[2]等方式作為區間資料集,也可以依您的定義蒐集各種不同的記錄檔資料、交易資料等等。</span></div>
</li>
</ul>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Serving Layer</span></div>
</li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">實作排呈服務,以查詢BigQuery的各項資料集,並定期將查詢結果匯入呈現層,供呈現批次報表。在此,呈現層與Serving Layer我們可以使用Google的免費工具”Apps Script”來實作。</span></div>
</li>
</ul>
</ul>
<b style="font-weight: normal;"><br /></b><br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="496" src="https://docs.google.com/drawings/d/sxrYjdBlNncg2rz-n4cSsNg/image?w=789&h=612&rev=682&ac=1" style="border: none; transform: rotate(0rad);" width="640" /></span></div>
<b style="font-weight: normal;"><br /></b>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17.333333333333332px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">實作</span></h2>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">在上面架構中,需要實作的部分包含:</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Fluentd client的設定檔,讓資料可以存入Cloud Logging,並且同時複製資料儲存到呈現層,供即時圖表繪製。</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Query engine,實作排程查詢BigQuery的方式,並將可以將資料做即時圖表呈現。</span></div>
</li>
</ul>
<b style="font-weight: normal;"><br /></b>
<h3 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">安裝google-fluentd</span></h3>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Google fluentd的安裝展示在他的說明文件中: </span><a href="https://cloud.google.com/logging/docs/agent/installation" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">https://cloud.google.com/logging/docs/agent/installation</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">,我們可以透過該說明完成google-fluentd的安裝。</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Step 1: 下載 google-fluentd 安裝腳本</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 971px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 8pt; margin-top: 8pt;">
<span style="background-color: #fafafa; color: #007000; font-family: Verdana; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">curl -sSO https://storage.googleapis.com/signals-agents/logging/google-fluentd-install.sh</span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Step 2: 執行安裝腳本</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 971px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 8pt; margin-top: 8pt;">
<span style="background-color: #fafafa; color: #007000; font-family: Verdana; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">sudo bash google-fluentd-install.sh</span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /></b>
<h3 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">設定credential</span></h3>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">如果您的fluentd client主機是位於GCE內,則上面說明安裝完成,已經可以存取Cloud Logging的服務,您只需要在/etc/google-fluentd中修改您的Input設定檔即可。如您是在GCE以外的主機安裝google-fluentd,則可以透過下面方式啟用google-fluentd</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Step 1: 在您的Cloud Console建立service account 並下載 json credential 檔案</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="292" src="https://lh5.googleusercontent.com/ucn66ewHXQBke9QnQ09QzaWMmnyiygG5a8z2wVdGqgRNZU8OavoQWoOk2xp2jp5EgXZu6hTGRtVybI4TJHr1rAnFn9BvOs-dnUKlvJc7N6GSBEAz1YRTmcBmZ8uB8YORJ_ipQn0" style="border: none; transform: rotate(0rad);" width="640" /></span></div>
<b style="font-weight: normal;"><br /></b><br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Step 2: 設定json檔案到client路徑中</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">複製所下載的json檔案到:“</span><span style="background-color: transparent; color: red; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">/root/.config/gcloud/application_default_credentials.json</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">”,則google-fluentd可以使用此credential執行資料傳輸動作。</span></div>
<b style="font-weight: normal;"><br /></b><br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Step 3: 修改google-fluentd設定檔</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">最後只需要將下面設定檔案更新到/etc/google-fluentd/google-fluentd.conf,並將project_id換成您的Google Cloud Platform專案即可。</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 971px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">@include config.d/*.conf</span></div>
<br /><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><match fluent.**></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> type null</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></match></span></div>
<br /><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><match **></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> type copy</span></div>
<br /><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> <store></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> type stdout</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </store></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> <store></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> type google_cloud</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> buffer_chunk_limit 512K</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> flush_interval 5s</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> max_retry_wait 300</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> disable_retry_limit</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">project_id</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> mitac-cp300-taipei101</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> zone micloud</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> vm_id dns-master</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </store></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></match></span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Step 4: 重新啟動google-fluentd agent</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 971px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">$ sudo /etc/init.d/google-fluentd restart</span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /></b><br />
<h3 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 8pt;">
<span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">實作Query Engine</span></h3>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">在Query Engine中,我們可以按照Google所發布的Open Source專案:</span><a href="https://github.com/GoogleCloudPlatform/lambda-dashboard" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">https://github.com/GoogleCloudPlatform/lambda-dashboard</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 來建置我們的Dashboard。</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Step1: 複製Lambda Dashboard</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Apps Script建置的部分,需要兩個Script檔案,我們可以直接從</span><span style="background-color: white; color: #333333; font-family: Arial; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><a href="https://www.youtube.com/watch?v=EZkw5TDcCGw" style="text-decoration: none;"><span style="background-color: white; color: #4078c0; font-family: Arial; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Lambda Dashboard Demo Video</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 中複製這個Google Sheet,同時就會複製該Apps Script腳本,裡面的兩個Script分別是fluent_listener.gs,負責開啟HTTPS port來聆聽由fluentd client所傳來的request;bq_query.gs,提供排程查詢BigQuery的方式,並將查詢結果寫入SpreadSheet中,順便繪製圖表。</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="172" src="https://lh3.googleusercontent.com/jnPje8m5scgt35WbZANf2qubdUAb__wDWAHrBiFQOW9Dju8Rq72cv1FicmQUzxXhWh_OyaC4g4ovMVBLEZ97TaeagcL33iAhr-briwf9WDFcsNECfC548ujT_MDbMWDSzATVk94" style="border: none; transform: rotate(0rad);" width="526" /></span></div>
<b style="font-weight: normal;"><br /></b><br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Step 2: 設定fluentd client</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">而在fluentd client的設定檔案中,我們需要加入https_json的一段設定到match tag中,</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; width: 971px;"><colgroup><col width="*"></col></colgroup><tbody>
<tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><match **></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> type copy</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> <store></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> type stdout</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </store></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> <store></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> type google_cloud</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> buffer_chunk_limit 512K</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> flush_interval 5s</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> max_retry_wait 300</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> disable_retry_limit</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">project_id</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> mitac-cp300-taipei101</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> zone micloud</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> vm_id dns-master</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </store></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: blue; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><store></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> type https_json</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> use_https true</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> buffer_path /tmp/buffer</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> buffer_chunk_limit 256m</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> buffer_queue_limit 128</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> flush_interval 3s</span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> endpoint <<ENDPOINT URL>></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: blue; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </store></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></match></span></div>
</td></tr>
</tbody></table>
</div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">其中<<ENDPOINT URL>>需要置換成您的Apps Script所發佈的存取URL,讓資料可以從這個點分流到Apps Script中的Endpoint。</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Step 3: 設定排程器</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">然後,我們需要在Apps Script的排程器上面設定Query的執行時間間隔,讓Apps Script每固定時間確認一次,然後執行需要執行的排程。</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="240" src="https://lh5.googleusercontent.com/nc3VzPAZJ0lkv_ZuuXkfs9VQBLmihCSFET1bRwZHYeVVYEFMA3n86SkfYxujbF-_dPEWI9YY5pzNgysifyO0PYflgJQBM5pX9zbBThCdk-EoVIIOLLlyoBBpE_zUyutjwBmUD00" style="border: none; transform: rotate(0rad);" width="640" /></span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Step 4: 自定Query語法</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">最後我們只需要在Lambda Dashboard的Spreadsheet中的BQ Queries這個Tab加入我們所需要的查詢以及設定排程的interval時間,</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="355" src="https://lh5.googleusercontent.com/CRwqgXVH5QEVqu6iELToU-RP6PDwPXyjHGAYLB3XDLmM1tu0AVfFXCFJnP0b2SWA9hN_I7SdI1xgnRHcb2LX8oFhEIpJhHkqFZzapIn7-bVCUCf74KIrWxrc5VoVQnCYtbXrCc4" style="border: none; transform: rotate(0rad);" width="640" /></span></div>
<b style="font-weight: normal;"><br /></b><br />
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">進階的Lambda Architecture</span></h1>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">最後,給大家一個Google上所有大資料相關工具的概觀,未來可以以Lambda Dashboard的方式來實作更穩定、更健全的Lambda Architecture:</span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="459" src="https://docs.google.com/drawings/d/s9bD-tPANaLXt3m_HgZ47nA/image?w=789&h=567&rev=1&ac=1" style="border: none; transform: rotate(0rad);" width="640" /></span></div>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">上面的這些系統包含:</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">資料持續接收層:以Google Managed的服務方式存在,使用者只要會使用它,可以達到極小化管理的目的。</span></div>
</li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Cloud Logging</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Cloud Storage</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Cloud Pub/Sub</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">BigQuery Streaming Insert</span></div>
</li>
</ul>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">大資料查詢層:以無限量儲存為目的,讓資料持續存放,並且可以提供查詢能力,並在有限的時間內回覆結果。</span></div>
</li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">BigQuery</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Dataflow</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">客製化查詢的Query Engine</span></div>
</li>
</ul>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">展示層:以Google管理的方式,讓使用者可以透過設定的方法來產生圖表,並且最終可以結合自動化通知機制,做到及時反應。</span></div>
</li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Cloud Monitor</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 13.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">StackDriver</span></div>
</li>
</ul>
</ul>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">以上,是Google目前所提供的大資料工具介紹,您也可以在最小的開發、維運的狀態下開始進行大資料的蒐集了!接下來的工作,就是開始從這些資料面去挖掘更有意義的資料,透過BigQuery來關聯查詢您期望的結果,這一切已經讓Google簡化了!</span></div>
<b style="font-weight: normal;"><br /></b>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 10pt;">
<span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21.333333333333332px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">附註</span></h1>
<b style="font-weight: normal;"><br /></b>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">[1]. Table Decorator: 參考 </span><a href="https://cloud.google.com/bigquery/table-decorators" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">https://cloud.google.com/bigquery/table-decorators</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> ,BigQuery在查詢中提供Hot Area的資料查詢,該區域的資料依照給定的範圍不一。</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">[2]. Daily Table: BigQuery中支援再from條件中以萬用字元一次查詢多個表格,在應用上,我們會以日期來切割表格,在查詢時再串連在一起做查詢。</span></div>
<div>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 14.666666666666666px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-11400930018545127412015-09-02T18:05:00.001-07:002015-09-02T18:05:47.522-07:00Wow... Google Docs有Template了耶~<div dir="ltr"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_lD1jryzyvDJtSppyw74LeU-jWzIm1hR4x-t_ocNNcZzRbYUx6xZgH0odiNKqxR4YXyr8WRHi7-3JBgtSHM9MNaIr5m5MO0ncwXzFmI1EkjMF28kTnnUfKo1WD-9oO5bXROg4r3-iFsI/s1600/image-747523.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_lD1jryzyvDJtSppyw74LeU-jWzIm1hR4x-t_ocNNcZzRbYUx6xZgH0odiNKqxR4YXyr8WRHi7-3JBgtSHM9MNaIr5m5MO0ncwXzFmI1EkjMF28kTnnUfKo1WD-9oO5bXROg4r3-iFsI/s320/image-747523.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6190088751970181346" /></a><br> </div> k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-90441827179310824382015-08-15T10:05:00.001-07:002015-08-15T10:06:11.594-07:00gcloud-kvstore module,讓存取cloud datastore更方便!<h1 id="gcloud-kvstore-module" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 2.25em; line-height: 1.2; margin-bottom: 16px; margin-left: 0px; margin-right: 0px; margin-top: 0px !important; padding-bottom: 0.3em; position: relative;">
gcloud-kvstore module</h1>
<h1 id="gcloud-kvstore-module" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 2.25em; line-height: 1.2; margin-bottom: 16px; margin-left: 0px; margin-right: 0px; margin-top: 0px !important; padding-bottom: 0.3em; position: relative;">
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
gcloud-kvstore是一套基於gcloud模組來存取<a href="https://cloud.google.com/datastore/" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; box-sizing: border-box; color: #4183c4; text-decoration: none;" target="_blank">Google Cloud Datastore</a>的方法,讓存取Datastore的部分可以更直覺。而該資料的儲存,最後會存入到"KeyValue"這個Kind中,以簡化所有儲存的動作。</div>
</h1>
<h2 id="github-repository" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em; position: relative;">
Github repository</h2>
<h1 style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 2.25em; line-height: 1.2; margin-bottom: 16px; margin-left: 0px; margin-right: 0px; margin-top: 0px !important; padding-bottom: 0.3em; position: relative;">
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
<a href="https://github.com/stephenplusplus/gcloud-kvstore" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; box-sizing: border-box; color: #4183c4; text-decoration: none;" target="_blank">https://github.com/stephenplusplus/gcloud-kvstore</a></div>
</h1>
<h2 id="installation" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em; position: relative;">
Installation</h2>
<h1 style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 2.25em; line-height: 1.2; margin-bottom: 16px; margin-left: 0px; margin-right: 0px; margin-top: 0px !important; padding-bottom: 0.3em; position: relative;">
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: #f7f7f7; border-radius: 3px; border: 0px; box-sizing: border-box; color: #333333; direction: ltr; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; font-weight: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; direction: ltr; display: inline; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre; word-wrap: normal;">npm install gcloud-kvstore
</code></pre>
</h1>
<h2 id="sample-usage" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em; position: relative;">
Sample Usage</h2>
<h3 id="%E8%AA%8D%E8%AD%89" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.5em; line-height: 1.43; margin-bottom: 16px; margin-top: 1em; position: relative;">
認證</h3>
<h1 style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 2.25em; line-height: 1.2; margin-bottom: 16px; margin-left: 0px; margin-right: 0px; margin-top: 0px !important; padding-bottom: 0.3em; position: relative;">
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
這個模組是使用gcloud的認證方式,所以開始需要有service account所給予的json file,然後搭配可以存取的project id</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: #f7f7f7; border-radius: 3px; border: 0px; box-sizing: border-box; color: #333333; direction: ltr; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; font-weight: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; direction: ltr; display: inline; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre; word-wrap: normal;">var dataset = gcloud.datastore.dataset({
projectId: 'your-project-id',
keyFilename: '/path/to/key.json'
});
</code></pre>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
認證過後的datastore物件就可以拿來使用...</div>
</h1>
<h3 id="%E5%9F%B7%E8%A1%8C" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.5em; line-height: 1.43; margin-bottom: 16px; margin-top: 1em; position: relative;">
執行</h3>
<h1 style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 2.25em; line-height: 1.2; margin-bottom: 16px; margin-left: 0px; margin-right: 0px; margin-top: 0px !important; padding-bottom: 0.3em; position: relative;">
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
下面是一段可以執行的code,我們可以透過安插一些時間點,來做執行間的監控,瞭解一下Cloud Datastore的反應速度。</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: #f7f7f7; border-radius: 3px; border: 0px; box-sizing: border-box; color: #333333; direction: ltr; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; font-weight: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; direction: ltr; display: inline; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre; word-wrap: normal;">var gcloud = require('gcloud');
var kvstore = require('gcloud-kvstore');
//初始化cloud datastore物件
var dataset = gcloud.datastore.dataset({
projectId: 'your-project-id',
keyFilename: '/path/to/key.json'
});
var t0 = new Date().getTime();
//將初始化的dataset提供給kvstore使用
var store = kvstore(dataset);
//
store.set('todos', ['eat', 'sleep', 'repeat'], function(err, key) {});
console.log('After insert time cost:', new Date().getTime() - t0);
var cnt = 10;
function doit() {
var ts = new Date().getTime();
store.get('todos', function(err, todos) {
var te = new Date().getTime();
console.log(err || todos);
console.log('cost: ', te - ts);
cnt--;
if(cnt > 0) doit();
});
}
doit();
</code></pre>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
另外,作者針對資料存取的動作,一共提供了下面三個簡單的操作:</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: #f7f7f7; border-radius: 3px; border: 0px; box-sizing: border-box; color: #333333; direction: ltr; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; font-weight: normal; line-height: 1.45; overflow: auto; padding: 16px; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; direction: ltr; display: inline; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre; word-wrap: normal;">//以key維id存一份資料
store.set('key', 'value', function() {});
//取出id=key得值
store.get('key', function() {});
//刪除id=key的物件
store.delete('key', function() {});</code></pre>
</h1>
k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-80967351146166806262015-07-30T10:24:00.001-07:002015-07-30T10:24:35.481-07:00簡單的script搭配gsutil做備份<div dir="ltr"><div>嫌cloud storage rsync每次要scan並checksum整個folder太慢嗎...</div><div>可以使用find這個指令搭配mtime或mmin的參數(可以參考:<a href="http://blog.wu-boy.com/2009/01/linuxfreebsd-find-%E6%8C%87%E4%BB%A4%E7%94%A8%E6%B3%95%E6%95%99%E5%AD%B8/)">http://blog.wu-boy.com/2009/01/linuxfreebsd-find-%E6%8C%87%E4%BB%A4%E7%94%A8%E6%B3%95%E6%95%99%E5%AD%B8/)</a></div><div>這樣可以直接找出異動的檔案</div><div>然後搭配shell做檔案的列表</div><div><br></div><div>ex: 找今天內變更的資料做備份</div><div><br></div><div>for name in `find . -mtime -1 ` ; do</div><div><br></div><div># do gsutil to cloud storage</div><div>gsutil cp $name gs://your-bucket-name/</div><div><br></div><div>done</div><div><br></div><div>這樣就可以簡單的備份差異到cloud storage上唷!</div> </div> k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-33358533506454440182015-07-30T09:42:00.000-07:002015-07-30T09:43:12.587-07:00Cloud Storage權限存取的奧妙<div dir="ltr"><div>今天介紹一下Cloud Storage如何使用專案角色認證來給予權限... 首先,我們需要到要請求存取的專案中,找出一些專案的資訊... 包含專案ID與專案的Role ID,這兩個直可以對應在Bucket權限設定中的Project與Group兩個Entity上,簡單的設定說明如下:</div><div><br> <div>讀取專案ID,然後複製下來...</div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmUpVGHyaE2AzpxsCvemxcLNBdIOd0YqrecBsvtOILoBs6ms1s3BspQd-CvbbYBLn_TaVCI3wJEzJar_uNyT3K2_kf4t_vtheG7aE7842hdh-Dz5QFSgirt3qYKkoYVkQT6dCcUif3k1U/s1600/Storage_settings_-_mitac-cp300-taipei101-792588.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmUpVGHyaE2AzpxsCvemxcLNBdIOd0YqrecBsvtOILoBs6ms1s3BspQd-CvbbYBLn_TaVCI3wJEzJar_uNyT3K2_kf4t_vtheG7aE7842hdh-Dz5QFSgirt3qYKkoYVkQT6dCcUif3k1U/s320/Storage_settings_-_mitac-cp300-taipei101-792588.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6177342345396060162" /></a><br></div><div><br></div><div>在權限授與的地方,可以用文字owners-, editors-, viewers-等字眼,後面加上project id作為授與權限的對象,再給予Access的權限對應,即可將bucket授與對象專案的相對角色相關權限。</div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoLBJTNaU080BuPhfohugBhOkVL0TZNayPhSZqijPVw1CMehQ8ELyvnhnqhNoGuPnQXPdOSx6GA7PKuRuiXju3bTmv7IA_smqhwqW84EuL5SlWY3QjBp8cnafWl30Hy4OQkQ_UfGuoQFg/s1600/Storage_browser_-_mitac-cp300-797105.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoLBJTNaU080BuPhfohugBhOkVL0TZNayPhSZqijPVw1CMehQ8ELyvnhnqhNoGuPnQXPdOSx6GA7PKuRuiXju3bTmv7IA_smqhwqW84EuL5SlWY3QjBp8cnafWl30Hy4OQkQ_UfGuoQFg/s320/Storage_browser_-_mitac-cp300-797105.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6177342360824496770" /></a>\<br></div></div><div><br></div><div>例如,給予project 288********的所有editor角色具備write的權限,則可以使用</div><div><br></div><div>ENTITY = project</div><div>NAME = 288********</div><div>ACCESS = Writer</div><div><br></div><div>這樣的一組設定來完成</div><div><br></div><div>另外,如果透過專案的Role ID來做權限授與的話,也可以達到相似的效果,這邊可以在Storage的setting頁面找到這個專案的對應role id</div><div><br></div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghoiU4WTCrfJSqolVIeEliQlPN02l_NTOPHus130blWlP9xOTmmUdd8R4rB0t0gNTOrw_mmtpzW5KvTIsztrHaWpDaeOJJtmT35uIxUmL1ify0SDec1g-53C3vgnOykcKs4Lj_kHcZ55Y/s1600/Storage_settings_-_mitac-cp300-taipei101-700149.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghoiU4WTCrfJSqolVIeEliQlPN02l_NTOPHus130blWlP9xOTmmUdd8R4rB0t0gNTOrw_mmtpzW5KvTIsztrHaWpDaeOJJtmT35uIxUmL1ify0SDec1g-53C3vgnOykcKs4Lj_kHcZ55Y/s320/Storage_settings_-_mitac-cp300-taipei101-700149.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6177342370069570514" /></a><br></div><div><br></div><div>這邊可以直接複製role id直接貼到NAME欄位,然後ENTITY選擇Group,在給予對應的Access權限...</div><div><br></div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVpK2qOBJAcbUwJRlsVJn44EsyjNtjRjVR0ckJbKh3T-PQlDhupetUnG7I42P5LX2n0SAJ8kaiG6roQBHpJ3nZUNXJy91mFcfrLXSdIjF776W0ZmQfc3fwRZMNkKTpQmis07lcyXirya8/s1600/Storage_browser_-_mitac-cp300-703691.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVpK2qOBJAcbUwJRlsVJn44EsyjNtjRjVR0ckJbKh3T-PQlDhupetUnG7I42P5LX2n0SAJ8kaiG6roQBHpJ3nZUNXJy91mFcfrLXSdIjF776W0ZmQfc3fwRZMNkKTpQmis07lcyXirya8/s320/Storage_browser_-_mitac-cp300-703691.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6177342387678527826" /></a><br></div><div><br></div><div>在存完檔案之後,重新瀏覽後,可以發現該設定會被轉換成專案的對應權限...</div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjy1cN1tcAZPShwPio17-4s1eNoq02Ob5hdt258zg4GHIiXgZXiaq6P5zabuxIYq2Vd6h3LgRXW2vvFbAXWYGoPioV574Ks0wk283i2uzUQee72Xdti5vZEvewkX6y-3_0qwGGd-tZ3ll8/s1600/Storage_browser_-_mitac-cp300-706624.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjy1cN1tcAZPShwPio17-4s1eNoq02Ob5hdt258zg4GHIiXgZXiaq6P5zabuxIYq2Vd6h3LgRXW2vvFbAXWYGoPioV574Ks0wk283i2uzUQee72Xdti5vZEvewkX6y-3_0qwGGd-tZ3ll8/s320/Storage_browser_-_mitac-cp300-706624.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6177342399169906210" /></a><br></div><div><br></div><div>透過這樣的設定,我們可以簡單的讓專案之間彼此可以互相傳遞資料,而不用特別去auth個人的帳戶給執行主機,也可以避免一些存取上的風險 :D</div><div><br></div><div><br></div><div>最後,可能有人會想到使用vm instance的service account來做權限授與...</div><div>下面是一個GCE instance,目前開放auth使用instance內建的service account (有掛active那組)</div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-H03devRvXLSxyJnqkRp4P0RjwflMn-MhzU0JlRXZStc_gwMLl41fEW4iX6rkmf9LdgVwSRgWzb-bB3BYdI4gjYVHythV_x5t4u3ooGVP4cBKyDp98eXYkPyefJpadPczVb4TjOgeIeQ/s1600/%253D%253FUTF-8%253FB%253Fc2ltb25zdV9pbnN0YW5jZS0xX19fX%252BWSjF9TdG9yYWdlX2Jyb3dzZXJfLV9taXRhYy1jcDMwMC5wbg%253D%253D%253F%253D%2509%253D%253FUTF-8%253FB%253FZw%253D%253D%253F%253D-711599"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-H03devRvXLSxyJnqkRp4P0RjwflMn-MhzU0JlRXZStc_gwMLl41fEW4iX6rkmf9LdgVwSRgWzb-bB3BYdI4gjYVHythV_x5t4u3ooGVP4cBKyDp98eXYkPyefJpadPczVb4TjOgeIeQ/s320/%253D%253FUTF-8%253FB%253Fc2ltb25zdV9pbnN0YW5jZS0xX19fX%252BWSjF9TdG9yYWdlX2Jyb3dzZXJfLV9taXRhYy1jcDMwMC5wbg%253D%253D%253F%253D%2509%253D%253FUTF-8%253FB%253FZw%253D%253D%253F%253D-711599" border="0" alt="" id="BLOGGER_PHOTO_ID_6177342417534248578" /></a><br></div><div><br></div><div>而在另一個專案的bucket給予這個instance內建的account權限之後:</div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTkfTMY3WlexW2chPznkG3RWHesmb859cM-BSXg4AAhgJhND-0RPiNv8oWt59IUOgIm0JOJoaWKdmwFeCE7QqxVdGOoF_4kR0TwIU9Y4pOiarSJzfnwTQZSFuvocagJnxe-Q4_ihdNKSM/s1600/Storage_browser_-_mitac-cp300-714258.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTkfTMY3WlexW2chPznkG3RWHesmb859cM-BSXg4AAhgJhND-0RPiNv8oWt59IUOgIm0JOJoaWKdmwFeCE7QqxVdGOoF_4kR0TwIU9Y4pOiarSJzfnwTQZSFuvocagJnxe-Q4_ihdNKSM/s320/Storage_browser_-_mitac-cp300-714258.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6177342433600881730" /></a></div><div><br></div><div>經測試可以正常存取該bucket</div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihguUqZMZiRrTZPwAz6r5pS5Uua6Ss_5XOA0JIiZLNLjDnD5iuSygI9IG9O1gK8_qYgDDwPCTY8-BL7OzXD2wxIjquzzd930SumAeld4I0MY-t3m0tjHz8lva_5m8MJ5USikDqzySaCfQ/s1600/image-718030.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihguUqZMZiRrTZPwAz6r5pS5Uua6Ss_5XOA0JIiZLNLjDnD5iuSygI9IG9O1gK8_qYgDDwPCTY8-BL7OzXD2wxIjquzzd930SumAeld4I0MY-t3m0tjHz8lva_5m8MJ5USikDqzySaCfQ/s320/image-718030.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6177342449214075650" /></a><br></div><div><br></div><div>但如果使用A專案所建立的Service Account去認證Cloud SDK(參考<a href="https://cloud.google.com/sdk/gcloud/reference/auth/activate-service-account">這邊</a>),則最後無法去存取已經授與權限的Service Account....</div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxD14SiZ7vvk_h2aGtvocWf9xlgC4hqasKIqWyqtkeiDr8lGrcL50knX_LRWIud8wfW5FXJQlXYiO-nGeI8uuPyrmy4osRv4EifFB39q1oaZNG2e-X9P8qSS7lbAjbolVLxUwS4Q78tsY/s1600/Storage_browser_-_mitac-cp300-720612.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxD14SiZ7vvk_h2aGtvocWf9xlgC4hqasKIqWyqtkeiDr8lGrcL50knX_LRWIud8wfW5FXJQlXYiO-nGeI8uuPyrmy4osRv4EifFB39q1oaZNG2e-X9P8qSS7lbAjbolVLxUwS4Q78tsY/s320/Storage_browser_-_mitac-cp300-720612.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6177342455600507106" /></a><br></div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmwktHctj0pDLglyITv2r6Kt8782XLHR0HLbcIzfTx_rTNyf7UvuyVuPOf9ocR2gyAclqZW-Qb1b1bFZz9uWpHxRoUXKtZhp68fLKMTm-YuKRXDsxYFA_ZuXolIcVz1isfptYxCedVA2A/s1600/%253D%253FUTF-8%253FB%253Fc2ltb25zdV9pbnN0YW5jZS0xX19fX%252BWSjF9TdG9yYWdlX2Jyb3dzZXJfLV9taXRhYy1jcDMwMC5wbg%253D%253D%253F%253D%2509%253D%253FUTF-8%253FB%253FZw%253D%253D%253F%253D-725027"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmwktHctj0pDLglyITv2r6Kt8782XLHR0HLbcIzfTx_rTNyf7UvuyVuPOf9ocR2gyAclqZW-Qb1b1bFZz9uWpHxRoUXKtZhp68fLKMTm-YuKRXDsxYFA_ZuXolIcVz1isfptYxCedVA2A/s320/%253D%253FUTF-8%253FB%253Fc2ltb25zdV9pbnN0YW5jZS0xX19fX%252BWSjF9TdG9yYWdlX2Jyb3dzZXJfLV9taXRhYy1jcDMwMC5wbg%253D%253D%253F%253D%2509%253D%253FUTF-8%253FB%253FZw%253D%253D%253F%253D-725027" border="0" alt="" id="BLOGGER_PHOTO_ID_6177342477115412786" /></a><br></div><div><br></div><div>結論:</div><div><br></div><div>1. 我們可以使用project與role的權限授與方式來給予其他專案存取此專案Bucket的權限</div><div>2. Service Account會跟隨專案,即使其他專案給予這個Service Account存取權限,也無法跨專案存取Bucket</div><div><br></div><div>以上,給大家參考</div><div><br></div><div><br></div></div> k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-90638922496768005362015-07-13T23:30:00.001-07:002015-07-13T23:30:29.642-07:00GCP專案介面,有趣的新改變...<div dir="ltr">在GCP專案中,發現多樂個"Edit labels",可以提供專案上添加label<div>而Label的設定設,直接提供key, value欄位供選擇</div><div>這是個滿有趣而且滿有用的設定方式</div><div>通常對專案多到一個不行的管理人員,有致勝的幫助!<br><div><br clear="all"><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9jlkvZws_BjIi87HGqA0QjKcs1B3oxF4yTopaz1BxfFuNNQbASr1nHpbSzK_A2NMADsvwskv4FGfMyr1Q2_W1s5jcX3OkfdDU0NHDPaeHwguzKSO9QhmGyx2AI4swXyaOCW3OB79SJRw/s1600/image-729642.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9jlkvZws_BjIi87HGqA0QjKcs1B3oxF4yTopaz1BxfFuNNQbASr1nHpbSzK_A2NMADsvwskv4FGfMyr1Q2_W1s5jcX3OkfdDU0NHDPaeHwguzKSO9QhmGyx2AI4swXyaOCW3OB79SJRw/s320/image-729642.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6171247083590100834" /></a><br></div><div><br></div><div>經過選擇幾個專案,然後添加name的label後</div><div>上方的filter提供透過label來過濾的方式,讓專案列表頓時間乾淨了許多!</div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2tR_vzNO3uRtzq7C51_CTjhyphenhyphenBzxgiq3syVA4zGSdi6srXOdkVeT5N0W_npZGhnMOgCbUbvIHkLQqocaUC-i1DroqqYFQLYHrCi75FNAn1PpNJ1VwBODSUMTBpfnqNfz0JgJxhET4exDA/s1600/image-732546.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2tR_vzNO3uRtzq7C51_CTjhyphenhyphenBzxgiq3syVA4zGSdi6srXOdkVeT5N0W_npZGhnMOgCbUbvIHkLQqocaUC-i1DroqqYFQLYHrCi75FNAn1PpNJ1VwBODSUMTBpfnqNfz0JgJxhET4exDA/s320/image-732546.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6171247090356727314" /></a><br></div><div><br></div><div>未來的Google雲端專案管理上,如果善用labels,真的可以達到很清楚的管理</div><div>大資料的世界裡面,爆多專案看來也不是問題了唷!</div> </div></div></div> k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-460205874297042572015-07-13T18:22:00.000-07:002015-07-13T18:35:53.021-07:00Cloud APP上傳檔案與下載策略<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
在Cloud APP上傳檔案與下載時,因為顧及到主機Scale out時候檔案存放位置會影響到Scale out出來的主機如何存取新上傳的檔案,在雲端,透過Cloud Storage是一個不錯的解決方案...</div>
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
<br /></div>
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
Node.js跟許多語言在file upload的處理方式有點不同,以Express為例,我們可以透過app.use()來導入其他的模組作為控管檔案上傳的部分,而我最常使用的是<a href="https://github.com/expressjs/multer" style="color: #1155cc;">multr</a>這個模組... 這次透過一些改寫,希望基於multr讓上傳、下載的動作直接結合Cloud Storage來作為存取的位置。</div>
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
<br /></div>
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
下面這是整個上傳與下載的示意圖:</div>
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
<br /></div>
<div style="text-align: center;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBSiaiyLT46hzNtVkcDQDrJqdT2sIK5mqjIGKKdEWeQJYgHejvISvrJI0DlAGZu66AV-xyQxRi4Ln14ecYcgyUopyg_sz9eVr0hmj87ZjJD5f58iRGa39NZLti_JMKaXuoSYTt96whdho/s1600/express-gcs-uploader.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="444" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBSiaiyLT46hzNtVkcDQDrJqdT2sIK5mqjIGKKdEWeQJYgHejvISvrJI0DlAGZu66AV-xyQxRi4Ln14ecYcgyUopyg_sz9eVr0hmj87ZjJD5f58iRGa39NZLti_JMKaXuoSYTt96whdho/s640/express-gcs-uploader.png" width="640" /></a></div>
<br /></div>
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
<br /></div>
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
上圖分成兩個部分:</div>
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
<ul>
<li>上傳的部分,無論如何直接將一份副本複製到Cloud Storage</li>
<li>下載的部分,主要依循幾個策略:</li>
<ul>
<li>如果本地端有檔案,則直接使用本地端檔案回覆</li>
<li>如果本地端無檔案,系統直接到Cloud Storage拉檔案</li>
<li>承上,如果使用者允許cache,可以在拉完Cloud Storage檔案後,在存放一個副本在本地端,之後就可以支援從本地端存取功能</li>
</ul>
</ul>
<div>
上面所述,因為效能關係,可能會開啟cache的功能,如果上傳的檔案名稱不是唯一,則可能重複上傳後再本地端會有舊資料問題,為了避免這個問題,在設定檔中可以指定上傳時候不維持原檔名,這樣每次上傳會以重新編碼檔名,讓每一個物件名稱不會重複。</div>
</div>
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
<br /></div>
<div style="font-family: arial, sans-serif; font-size: small;">
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
安裝</h2>
<pre style="background-color: #f7f7f7; border-radius: 3px; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background: transparent; border-radius: 3px; border: 0px; display: inline; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; word-break: normal; word-wrap: normal;">npm install express-gcs-uploader --save
</code></pre>
<h2 style="border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
<a aria-hidden="true" class="" href="https://github.com/peihsinsu/express-gcs-uploader#setup" id="user-content-setup" style="color: #4078c0; display: block; line-height: 1; margin-left: -30px; padding-left: 30px; padding-right: 6px; text-decoration: none;"></a>設定</h2>
<h3 style="color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 1.5em; line-height: 1.43; margin-bottom: 16px; margin-top: 1em;">
<a aria-hidden="true" class="" href="https://github.com/peihsinsu/express-gcs-uploader#step1-setup-the-auth-and-related-configuration" id="user-content-step1-setup-the-auth-and-related-configuration" style="color: #4078c0; display: block; line-height: 1.2; margin-left: -30px; padding-left: 30px; padding-right: 6px; text-decoration: none;"></a>Step1: 與Cloud Storage認證部分設定</h3>
<pre style="background-color: #f7f7f7; border-radius: 3px; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background-attachment: initial; background-clip: initial; background-color: transparent; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-image-outset: initial; border-image-repeat: initial; border-image-slice: initial; border-image-source: initial; border-image-width: initial; border-radius: 3px; border: 0px; display: inline; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; word-break: normal; word-wrap: normal;"><span style="color: #333333;">var gcsuplder = require('express-gcs-uploader');
gcsuplder.auth({
rootdir: __dirname, </span><span style="color: blue;">//設定專案目錄位置,在這邊是為了對應uploads目錄用</span><span style="color: #333333;">
upload_url: '/uploads', </span><span style="color: blue;">//上傳檔案的資料夾</span><span style="color: #333333;">
download_url: '/download', </span><span style="color: blue;">//檔案下載的路由</span><span style="color: #333333;">
cdn_url: 'http://your.bucket.com.tw', </span><span style="color: red;">//option: 如果GCS設定為public或是website buket,則可以直接由此位置提供服務</span><span style="color: #333333;">
keep_filename: true, </span><span style="color: red;">//option: 是否保留原本檔案名稱</span><span style="color: #333333;">
cache: true, </span><span style="color: red;">//option: 是否每次下載完儲存一份在本地端</span><span style="color: #333333;">
bucket: 'your.bucket.com.tw', </span><span style="color: blue;">//GCS bucket名稱</span><span style="color: #333333;">
projectId: 'your-project-id', </span><span style="color: blue;">//GCP專案名稱</span><span style="color: #333333;">
keyFilename: '/path/to/your/key.json' </span><span style="color: blue;">//Service Account的json key位置</span><span style="color: #333333;">
});
</span></code></pre>
<div style="color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 16px; line-height: 25.6000003814697px; margin-bottom: 16px;">
<span style="font-size: 1.5em; line-height: 1.43;">Step2: 設定下載的route使用downloadproxy</span></div>
<pre style="background-color: #f7f7f7; border-radius: 3px; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background: transparent; border-radius: 3px; border: 0px; display: inline; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; word-break: normal; word-wrap: normal;">app.use('/downloads/:id', gcsuplder.downloadproxy);
</code></pre>
<div style="color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 16px; line-height: 25.6000003814697px; margin-bottom: 16px;">
如上設定,則可以在下面網址存取一個上傳的檔案: http://localhost:3000/downloads/e13b6a98a0d50b5990123a83eb87f2a8.png </div>
<div style="color: #333333; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 16px; line-height: 25.6000003814697px; margin-bottom: 16px;">
其中,“:id”到時候帶入的會是檔案名稱。</div>
</div>
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
<br /></div>
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
上述模組,發佈在Github給大家參考:<a href="https://github.com/peihsinsu/express-gcs-uploader">https://github.com/peihsinsu/express-gcs-uploader</a></div>
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
<br /></div>
k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-71543366780074077832015-07-08T03:54:00.000-07:002015-07-09T03:31:14.020-07:00使用SNMP監控Linux CPU並整合Google Cloud Monitor<div dir="ltr">
<h2 id="google-cloud-monitor%E6%98%AF%E4%BB%80%E9%BA%BC" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em; position: relative;">
Google Cloud Monitor是什麼</h2>
<h2 id="google-cloud-monitor%E6%98%AF%E4%BB%80%E9%BA%BC" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em; position: relative;">
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
Google Cloud Monitor簡單的說是Cloud Monitor API與StackDriver服務的總和。Google透過Cloud Monitor提供所有Google Cloud上的一些操作記錄以及讓開發者可以自訂自己的Monitor來在同一個平台上監控。</div>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
下面展示的是透過SNMP來蒐集主機的CPU資訊,並且透過Google Cloud Monitor來蒐集這些資訊,呈現在StackDriver的Dashboard上。</div>
</h2>
<h2 id="snmp-cpu%E8%B3%87%E8%A8%8A%E8%A8%AD%E5%AE%9A" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em; position: relative;">
SNMP CPU資訊設定</h2>
<h2 style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em; position: relative;">
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
如前面SNMP章節所建,我們可以在snmpd.conf中加入CPU load average的設定,讓SNMP可以讀到CPU的資訊。</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: #f7f7f7; border-radius: 3px; border: 0px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; font-weight: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre; word-wrap: normal;"># vim /etc/snmp/snmpd.conf
(skip)
1 minute Load: .1.3.6.1.4.1.2021.10.1.3.1
5 minute Load: .1.3.6.1.4.1.2021.10.1.3.2
15 minute Load: .1.3.6.1.4.1.2021.10.1.3.3
(skip)
</code></pre>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
修改完成後,請restart snmp daemon已更新執行狀態。</div>
</h2>
<h2 id="create-monitor-to-metrics-script" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em; position: relative;">
Create monitor to metrics script</h2>
<h2 style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em; position: relative;">
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
在轉入資料到Cloud Monitor中的部分,需要透過一些程式來進行,下面是欲透過fluentd exec模組來塞資料進到Cloud Monitor的程式片段,目的是從fluentd exec收到欲轉入到Cloud Monitor的資料,並且將資料輸入到Cloud Monitor...</div>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
接收到的資料範例:</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: #f7f7f7; border-radius: 3px; border: 0px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; font-weight: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre; word-wrap: normal;">{"name":"UCD-SNMP-MIB::laLoad.1","value":0.0}
{"name":"UCD-SNMP-MIB::laLoad.2","value":0.0}
{"name":"UCD-SNMP-MIB::laLoad.3","value":0.0}
</code></pre>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
test.js的部分:</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: #f7f7f7; border-radius: 3px; border: 0px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; font-weight: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre; word-wrap: normal;">//file: test.js
var fs = require('fs');
var metric = require('gcmetrics');
var txt = fs.readFileSync(process.argv[2],'utf8');
var project_id = 'mitac-cp300-taipei101';
metric.init({
scope: 'https://www.googleapis.com/auth/monitoring',
client_email: "your-service-account@developer.gserviceaccount.com",
key_pem: "/path/to/your/pem"
});
function insert(metricid, cid, value) {
metric.writeTimeSeries(
project_id,
metricid,
value,
{"custom.cloudmonitoring.googleapis.com/laLoad" : cid},
null,
null,
function(err, req, doc){
if(err) console.log('ERROR:', err);
console.log(doc);
});
}
if(txt)
txt.split('\n').forEach(function(line){
if(line.length > 0) {
var vo = JSON.parse(line);
console.log('vo:', vo);
if(vo)
insert('cpuloadavg', vo.name, vo.value);
}
})
</code></pre>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
這段程式是透過<a href="http://nodejs.org/" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; box-sizing: border-box; color: #4183c4; text-decoration: none;" target="_blank">Node.js</a>撰寫,因此如果未安裝Node.js程式,則需要先行安裝... 安裝後需要另外安裝gcmetrics模組,可以在您的程式路徑執行下面指令安裝:</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: #f7f7f7; border-radius: 3px; border: 0px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; font-weight: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre; word-wrap: normal;">$ npm init
...(skip)
$ npm install gcmetrics --save
</code></pre>
</h2>
<h2 id="fluentd%E8%A8%AD%E5%AE%9A" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em; position: relative;">
Fluentd設定</h2>
<h2 style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em; position: relative;">
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
完成上面程式後,在fluentd端,可以按照下面設定:</div>
<pre style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background-color: #f7f7f7; border-radius: 3px; border: 0px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; font-weight: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; white-space: pre-wrap; word-wrap: normal;"><code style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6000003814697px; line-height: inherit; margin: 0px; max-width: initial; overflow: initial; padding: 0px; white-space: pre; word-wrap: normal;"><source>
type snmp
tag snmp.server1
nodes name, value
host localhost
community idv90we3rnov90wer # 這是config檔案中的community的設定
mib laLoad.1, laLoad.2, laLoad.3
mib_modules UCD-SNMP-MIB
method_type get
polling_time 5
polling_type async_run
</source>
<match **>
type copy
<store>
type stdout
output_type json
</store>
<store>
type exec
command /opt/node/bin/node /path/to/your/test.js
format json
time_format %Y-%m-%d %H:%M:%S
flush_interval 5s
buffer_path /tmp/
</store>
</match>
</code></pre>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
上面的設定,原則上會每5秒鐘到snmp端口查詢snmp資料,並且把資料丟給test.js做處理...</div>
</h2>
<h2 id="google-cloud-monitor-stackdriver%E8%A8%AD%E5%AE%9A" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em; position: relative;">
Google Cloud Monitor - StackDriver設定</h2>
<h2 style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border-bottom-color: rgb(238, 238, 238); border-bottom-style: solid; border-bottom-width: 1px; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em; position: relative;">
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
完成了Fluentd與Cloud Monitor程式的設定後,接下來我們可以透過Google Cloud Monitor(StackDriver)來建置即時監控的報表:</div>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
首先,可以在StackDriver中找到自訂Dashboard(Dashboards > Create...),然後點選右上角的"Add Chart"來建立自己的圖表:</div>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
<img alt="Google Cloud Monitor Dashboard" src="http://peihsinsu.gitbooks.io/fluentd-in-action/content/Custom_Metrics_Dashboard_%E2%80%93_Dashboard.png" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border: 0px; box-sizing: border-box; max-width: 100%; page-break-inside: avoid;" /></div>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
如果一切無誤,則可以看到類似這樣的圖表:</div>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
<img alt="StackDriver Chart" src="http://peihsinsu.gitbooks.io/fluentd-in-action/content/Custom_Metrics_Dashboard_%E2%80%93_Dashboard2.png" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border: 0px; box-sizing: border-box; max-width: 100%; page-break-inside: avoid;" /></div>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px; margin-bottom: 16px;">
有了圖表,接下來可以建立自己的告警功能,透過StackDriver功能列Alerting > Create Alerting Policy可以建立通知告警!</div>
<div style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; box-sizing: border-box; color: #333333; font-family: 'Open Sans', 'Clear Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 16px; font-weight: normal; line-height: 25.6000003814697px;">
<img alt="StackDriver Create Alert Policy" src="http://peihsinsu.gitbooks.io/fluentd-in-action/content/StackDriver-Create_Policy.png" style="-webkit-font-smoothing: antialiased; -webkit-tap-highlight-color: transparent; border: 0px; box-sizing: border-box; max-width: 100%; page-break-inside: avoid;" /></div>
</h2>
<h2 style="border-bottom-color: rgb(238,238,238); border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: 'Open Sans','Clear Sans','Helvetica Neue',Helvetica,Arial,sans-serif; font-size: 1.75em; line-height: 1.225; margin-bottom: 16px; margin-top: 1em; padding-bottom: 0.3em;">
</h2>
<h1 style="border-bottom-color: rgb(238,238,238); border-bottom-style: solid; border-bottom-width: 1px; color: #333333; font-family: 'Open Sans','Clear Sans','Helvetica Neue',Helvetica,Arial,sans-serif; font-size: 2.25em; line-height: 1.2; margin-bottom: 16px; margin-left: 0px; margin-right: 0px; margin-top: 0px!important; padding-bottom: 0.3em;">
</h1>
</div>
k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-42494914145028140192015-07-07T05:43:00.000-07:002015-07-07T05:44:10.177-07:00GCE can specify root disk size when boot, now!<div dir="ltr">忙太久,沒發現,GCE現在可以在開機器的時候指定root disk大小了唷!<br> <div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqbXd11Lf7RPbQxXVs6p6UDm-SpAS6L9eSAqytO8rN6voSqAJyuS6gym3trFZIlaWlaj4d3qzGl4cDxdsprZqVbI94jL97cQ4RhnnyExzf6K1wWAqXnxxl6KGgcbSnIFqaIRXglSaMvog/s1600/image-750177.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqbXd11Lf7RPbQxXVs6p6UDm-SpAS6L9eSAqytO8rN6voSqAJyuS6gym3trFZIlaWlaj4d3qzGl4cDxdsprZqVbI94jL97cQ4RhnnyExzf6K1wWAqXnxxl6KGgcbSnIFqaIRXglSaMvog/s320/image-750177.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6168745785244641154" /></a><br></div></div> k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-64011003415769810012015-06-24T08:32:00.001-07:002015-06-24T08:32:21.849-07:00Google Cloud Monitor integrate with Custom Metrics<div dir="ltr"><div>最近在玩Google Cloud Monitor API的Custom Metric,簡單的說就是可以客制輸入到Monitor系統的數據,讓我們可以客製化監控的功能... 為了讓這一些更簡單,我寫了個簡單的工具來當作指令以輔助Custom Metric輸入資料,有了Custom Metric的資料,未來整合Cloud Monitor的圖表跟告警功能就更無往不利了!</div><div><h1 style="font-size:2.25em;margin:1em 0px 16px;line-height:1.2;padding-bottom:0.3em;border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(238,238,238);color:rgb(51,51,51);font-family:'Helvetica Neue',Helvetica,'Segoe UI',Arial,freesans,sans-serif">Installation</h1><div>在安裝的部分,需要先有node.js環境,然後就可以安裝gcmetrics指令:</div><div><br></div><pre style="overflow:auto;font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;font-size:13.6000003814697px;margin-top:0px;margin-bottom:16px;font-stretch:normal;line-height:1.45;padding:16px;border-radius:3px;word-wrap:normal;color:rgb(51,51,51);background-color:rgb(247,247,247)"><code style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;font-size:13.6000003814697px;padding:0px;margin:0px;border-radius:3px;word-break:normal;border:0px;display:inline;max-width:initial;overflow:initial;line-height:inherit;word-wrap:normal;background:transparent">npm install gcmetrics -g </code></pre><h1 style="font-size:2.25em;margin:1em 0px 16px;line-height:1.2;padding-bottom:0.3em;border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(238,238,238);color:rgb(51,51,51);font-family:'Helvetica Neue',Helvetica,'Segoe UI',Arial,freesans,sans-serif"><a id="user-content-auth" class="" href="https://github.com/peihsinsu/gcmetrics/blob/master/sample.md#auth" style="color:rgb(64,120,192);text-decoration:none;display:block;padding-right:6px;padding-left:30px;line-height:1;background-color:transparent"></a>Auth</h1><div>認證部分,需要先有Google project,並且申請service account,再透過下面指令來進行認證:</div><div><br></div><pre style="overflow:auto;font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;font-size:13.6000003814697px;margin-top:0px;margin-bottom:16px;font-stretch:normal;line-height:1.45;padding:16px;border-radius:3px;word-wrap:normal;color:rgb(51,51,51);background-color:rgb(247,247,247)"><code style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;font-size:13.6000003814697px;padding:0px;margin:0px;border-radius:3px;word-break:normal;border:0px;display:inline;max-width:initial;overflow:initial;line-height:inherit;word-wrap:normal;background:transparent">gcmetric -t auth -s <a href="mailto:288173501788-9fh14iojjmehakjtgkhoubdiaguppn1h@developer.gserviceaccount.com">288173501788-9fh14iojjmehakjtgkhoubdiaguppn1h@developer.gserviceaccount.com</a> \ -k $HOME/.gcpkeys/mitac-cp300-taipei101/mitac-cp300-taipei101-8c0662095ef9.pem \ -p mitac-cp300-taipei101 </code></pre><h1 style="font-size:2.25em;margin:1em 0px 16px;line-height:1.2;padding-bottom:0.3em;border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(238,238,238);color:rgb(51,51,51);font-family:'Helvetica Neue',Helvetica,'Segoe UI',Arial,freesans,sans-serif"><a id="user-content-create-a-metric" class="" href="https://github.com/peihsinsu/gcmetrics/blob/master/sample.md#create-a-metric" style="color:rgb(64,120,192);text-decoration:none;display:block;padding-right:6px;padding-left:30px;line-height:1;background-color:transparent"></a>Create a metric</h1><div>接下來可以建立一個lightweight metric,來準備收資料...</div><div><br></div><pre style="overflow:auto;font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;font-size:13.6000003814697px;margin-top:0px;margin-bottom:16px;font-stretch:normal;line-height:1.45;padding:16px;border-radius:3px;word-wrap:normal;color:rgb(51,51,51);background-color:rgb(247,247,247)"><code style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;font-size:13.6000003814697px;padding:0px;margin:0px;border-radius:3px;word-break:normal;border:0px;display:inline;max-width:initial;overflow:initial;line-height:inherit;word-wrap:normal;background:transparent">gcmetric -t init -n mymetric -i firstid -d "test first id" </code></pre><h1 style="font-size:2.25em;margin:1em 0px 16px;line-height:1.2;padding-bottom:0.3em;border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(238,238,238);color:rgb(51,51,51);font-family:'Helvetica Neue',Helvetica,'Segoe UI',Arial,freesans,sans-serif"><a id="user-content-insert-data-to-metric" class="" href="https://github.com/peihsinsu/gcmetrics/blob/master/sample.md#insert-data-to-metric" style="color:rgb(64,120,192);text-decoration:none;display:block;padding-right:6px;padding-left:30px;line-height:1;background-color:transparent"></a>Insert data to metric</h1><div>傳資料的部分,可以透過: gcmetric -t simple -n mymetric -d [value] 來指定數值傳入Cloud Monitor,既然有了command,那寫個簡單的bash script就可以一直持續的塞資料唷!</div><div><br></div><pre style="overflow:auto;font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;font-size:13.6000003814697px;margin-top:0px;font-stretch:normal;line-height:1.45;padding:16px;border-radius:3px;word-wrap:normal;color:rgb(51,51,51);margin-bottom:0px!important;background-color:rgb(247,247,247)"><code style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;font-size:13.6000003814697px;padding:0px;margin:0px;border-radius:3px;word-break:normal;border:0px;display:inline;max-width:initial;overflow:initial;line-height:inherit;word-wrap:normal;background:transparent">while [ true ] ; do gcmetric -t simple -n mymetric -d `free | grep Mem | awk '{print $3}'`; sleep 3 ; done</code></pre></div><div><br></div><div><br></div><div>最後透過Cloud Monitor的自訂Dashboard功能,可以輕易的定義自己的圖表唷~</div><div><br></div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeAJfPvraLBJ4rO4lSmXZnVDwqitZQ2kug_EWNJnraNhaekHThbnzd9qvgmgiQUwee9yV0FdoOMSDUU4GyKQavpo9Kg7PD917etdXplSaGvK7XVEhBAPkVg4yGmdyvdoZBrvlWXHMi0a8/s1600/image-741850.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeAJfPvraLBJ4rO4lSmXZnVDwqitZQ2kug_EWNJnraNhaekHThbnzd9qvgmgiQUwee9yV0FdoOMSDUU4GyKQavpo9Kg7PD917etdXplSaGvK7XVEhBAPkVg4yGmdyvdoZBrvlWXHMi0a8/s320/image-741850.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6163965019259933538" /></a><br></div><div><br></div><div>之後,圖表、告警就都可以在上面直接設定了ㄛ!</div><div><br></div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEh7frBR5H88r9yJFQwJrqMJ0FQITX_i-jgMns4JJR8kGo9lXLTMJ_ZhnWq7KuS8GHmup9T7pmLvvdOGGTc6cdID1EeCfMswFY60OTI2Z4KdLi8ea74oWjN_j34cpJZrQaJlx79DIvNvw/s1600/%253D%253FUTF-8%253FB%253FQ3VzdG9tX01ldHJpY3NfRGFzaGJvYXJkX%252BKAk19EYXNoYm9hcmQucG5n%253F%253D-744557"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEh7frBR5H88r9yJFQwJrqMJ0FQITX_i-jgMns4JJR8kGo9lXLTMJ_ZhnWq7KuS8GHmup9T7pmLvvdOGGTc6cdID1EeCfMswFY60OTI2Z4KdLi8ea74oWjN_j34cpJZrQaJlx79DIvNvw/s320/%253D%253FUTF-8%253FB%253FQ3VzdG9tX01ldHJpY3NfRGFzaGJvYXJkX%252BKAk19EYXNoYm9hcmQucG5n%253F%253D-744557" border="0" alt="" id="BLOGGER_PHOTO_ID_6163965028294515650" /></a><br></div><div><br></div><div>專案已經發佈在:<a href="https://github.com/peihsinsu/gcmetrics">https://github.com/peihsinsu/gcmetrics</a></div><div>歡迎有興趣的朋友自行取用 :D</div><div><br></div><div><br></div><div><br></div><div><br></div></div> k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-30893486746480567422015-06-11T06:19:00.001-07:002015-06-11T06:19:36.036-07:00Google Cloud Monitor Customer Metrics<div dir="ltr"><div>Google Cloud Monitor在結合<a href="https://app.google.stackdriver.com">StackDriver</a>之後,提供了一個強大且彈性的Cloud Monitor API,並且允許客制監控,Google稱為<a href="https://cloud.google.com/monitoring/custom-metrics/">Customer Metrics</a>...</div><div>透過Cloud Monitor API可以以RESTful方式輸入Customer Metrics,然後在StackDriver內取用來作為圖表以及監控的資料來源...</div><div><br></div><div>在Customer Metrics在實作上分成Lightweight custom metric與Labeled custom metric兩種,分別在資料內容上有些許的差別:</div><div><br></div><div>Lightweight custom metric以數值+時間為主,資料內容大致如下:<br></div><div><div>$3,918.33 on March 20.</div><div>$4,107.68 on March 21.</div><div>$3,798.00 on March 22.</div></div><div><br></div><div>而Labeled custom metric以數值+時間+描述為主,資料內容大致如下:</div><div><div>$23,213.97 at the "Columbus" store on January 10.</div><div>$26,087.44 at the "Magellan" store on January 10.</div><div>$19,650.00 at the "Columbus" store on January 11.</div></div><div><br></div><div>因此只要能夠抽取需要監控的數值與當下的時間,就可以開始監控 :D</div><div>下面是針對Customer Metrics的Lightweight customer metric來做資料輸入的片段Node.js程式:</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>var request = require('request');</div></div><div><div>var auth = require('google-api-utility')</div></div><div><div> , request = auth.request</div></div><div><div> , util = require('util')</div></div><div><div> , project = 'your-project-id';</div></div><div><div><br></div></div><div><div>auth.init({</div></div><div><div> scope: '<a href="https://www.googleapis.com/auth/monitoring">https://www.googleapis.com/auth/monitoring</a>',</div></div><div><div> client_email: '<a href="mailto:your-service-account-id@developer.gserviceaccount.com">your-service-account-id@developer.gserviceaccount.com</a>',</div></div><div><div> key_pem: '/path/to/your/pemkey/xxx.pem'</div></div><div><div>});</div></div><div><div><br></div></div><div><div>var url = {</div></div><div><div> writeTimeSeries: '<a href="https://www.googleapis.com/cloudmonitoring/v2beta2/projects/">https://www.googleapis.com/cloudmonitoring/v2beta2/projects/</a>' + project + '/timeseries:write'</div></div><div><div>}</div></div><div><div><br></div></div><div><div>var data = {</div></div><div><div> "timeseries": [</div></div><div><div> {</div></div><div><div> "timeseriesDesc": {</div></div><div><div> "project": project,</div></div><div><div> "metric": "<a href="http://custom.cloudmonitoring.googleapis.com/">custom.cloudmonitoring.googleapis.com/</a><b>daily_sales</b>"</div></div><div><div> },</div></div><div><div> "point": {</div></div><div><div> "start": "2015-06-11T10:52:00.000Z",</div></div><div><div> "end": "2015-06-11T10:52:00.000Z",</div></div><div><div> "doubleValue": 2228.74</div></div><div><div> }</div></div><div><div> }</div></div><div><div> ]</div></div><div><div>}</div></div><div><div><br></div></div><div><div>request({</div></div><div><div> url: url.writeTimeSeries,</div></div><div><div> json: data,</div></div><div><div> method: 'POST'</div></div><div><div>}, function(e, r, d){</div></div><div><div> if(e) console.log('[ERROR] ',e);</div></div><div><div> console.log(d);</div></div><div><div>});</div></div></blockquote><div><br></div><div><br></div><div>其中data部分是寫入的資料格式內容,我們定義這個metric名稱為daily_sales,並且在point中指定某個數值與發生時間區間...</div><div>以上面的程式再多寫幾筆資料後,我們就可以到StackDriver中拉出該Metric的報表(可以開個Dashboard並且在其中Add Chart...)</div><div><br></div><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNZCuVDM8rOXKMAz8yaSkS8MOEbvVdgk96iA7wV0EuWDqGWx4NRa7X15dhtH0FWjYK_RgIxIsLW9rrhKksSvKe_iB4OVw1I1pfuXUUZGfxRFSvrHGEi9Lww1e86j-1uf1ovdz3J_DPpTc/s1600/image-776037.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNZCuVDM8rOXKMAz8yaSkS8MOEbvVdgk96iA7wV0EuWDqGWx4NRa7X15dhtH0FWjYK_RgIxIsLW9rrhKksSvKe_iB4OVw1I1pfuXUUZGfxRFSvrHGEi9Lww1e86j-1uf1ovdz3J_DPpTc/s320/image-776037.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6159106702859421826" /></a></div></blockquote></div><div><br></div><div>然後再出現的圖表中,就可以看到該Metric數值所呈現的圖表:</div><div><br></div><div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEij38_QXOZLHMiRaIY_bCX1GgEUfE0fBoEOcFOuFtRIwbM2LHjN35xvvCgMtacre67xCrKu-FUdBtrMJoA3hwmOXLFnANysB3Or7MBYj5wmpztNw8nfgc78QyQisMFqrBhJtYmgXTKOiFU/s1600/image-779588.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEij38_QXOZLHMiRaIY_bCX1GgEUfE0fBoEOcFOuFtRIwbM2LHjN35xvvCgMtacre67xCrKu-FUdBtrMJoA3hwmOXLFnANysB3Or7MBYj5wmpztNw8nfgc78QyQisMFqrBhJtYmgXTKOiFU/s320/image-779588.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6159106713790169570" /></a></div></blockquote></div><div><br></div><div>再進一步就是去設定一些Alert了... 在StackDriver中支援Email, SMS, PageDuty...等數種通知的方式,可以好好規劃一下哪些資訊需要Alert了唷!</div><div><br></div></div> k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-63780908627355543152015-06-11T06:00:00.001-07:002015-06-11T06:00:50.722-07:00Fluentd exec output測試<div dir="ltr">由於Google Cloud Logging是透過fluentd實作,加上fluentd介接BigQuery也非常簡單...<div>有這個機會跟fluentd多親近一下... :D</div><div><br><div><div><div>在玩了fluentd一陣子後,雖然他提供了不少的input, output, filter plugin,但是總是有一些無法滿足的地方<div>加上fluentd是ruby friendly的系統...@@,沒學過ruby總是難以在上面發揮...@@</div><div><br></div><div>看了一陣子,終於下手測試output exec plugin... 期待可以藉由他來用第三方程式語言去介接其他output plugin所未提供的服務...</div><div>下面是一段將syslog source輸入,然後output到node.js程式的範例config<br><div><br></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><source></div></div></div><div><div><div> type syslog</div></div></div><div><div><div> protocol_type udp</div></div></div><div><div><div> port 514</div></div></div><div><div><div> bind 0.0.0.0</div></div></div><div><div><div> tag system</div></div></div><div><div><div></source></div></div></div><div><div><div><source></div></div></div><div><div><div> type syslog</div></div></div><div><div><div> protocol_type tcp</div></div></div><div><div><div> port 514</div></div></div><div><div><div> bind 0.0.0.0</div></div></div><div><div><div> tag system</div></div></div><div><div><div></source></div></div></div><div><div><div><br></div></div></div><div><div><div><match **></div></div></div><div><div><div> type copy</div></div></div><div><div><div> <store></div></div></div><div><div><div> type stdout</div></div></div><div><div><div> output_type json</div></div></div><div><div><div> </store></div></div></div><div><div><div> <store></div></div></div><div><div><div> type exec</div></div></div><div><div><div> command /opt/node/bin/node /home/simonsu/data/test.js</div></div></div><div><div><div> format json</div></div></div><div><div><div> time_format %Y-%m-%d %H:%M:%S</div></div></div><div><div><div> flush_interval 5s</div></div></div><div><div><div> buffer_path /tmp</div></div></div><div><div><div> </store></div></div></div><div><div><div></match></div></div></div></blockquote><div><div><br></div><div><br></div><div>然後其中"/home/simonsu/data/test.js"則是我們預計要執行的node.js程式</div><div>在exec中,以node.js會在指令列接收到類似:/tmp.20150611.q5183cc42694434d7.log的字串</div><div>該字串是buffer中未flush的資料檔,以上面設定而言,資料會以json的方式收錄</div><div>內容長得像這樣:</div><div><br></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><div>{"host":"simon-centos6","ident":"run-parts","message":"(/etc/cron.hourly)[2050 starting 0anacron"}</div></div></div><div><div><div>{"host":"simon-centos6","ident":"anacron","pid":"2061","message":"Anacron started on 2015-06-11"}</div></div></div><div><div><div>{"host":"simon-centos6","ident":"run-parts","message":"(/etc/cron.hourly)[2063 finished 0anacron"}</div></div></div><div><div><div>{"host":"simon-centos6","ident":"anacron","pid":"2061","message":"Will run job `cron.daily' in 17 min."}</div></div></div><div><div><div>{"host":"simon-centos6","ident":"anacron","pid":"2061","message":"Will run job `cron.weekly' in 37 min."}</div></div></div><div><div><div>{"host":"simon-centos6","ident":"anacron","pid":"2061","message":"Will run job `cron.monthly' in 57 min."}</div></div></div><div><div><div>{"host":"simon-centos6","ident":"anacron","pid":"2061","message":"Jobs will be executed sequentially"}</div></div></div></blockquote><div><div><br></div><div>先不討論關於接收到檔案後要做的事情,我們可以直接在test.js中透過fs把該檔案內容讀出來:</div><div><br></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div>var fs = require('fs');</div></div></div><div><div><div>var txt = fs.readFileSync(process.argv[2],'utf8');</div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>//ex: console.log('[OUT]', txt);</div></blockquote><div><div><br></div><div>後續可以針對txt來做後續的處理動作...</div><div>附註:如果處理的當下有exception,fluentd會將內容暫存在buffer_path內(也就是前面所接到的參數)</div><div><br></div><div>接下來啟動測試,透過fluentd或td-agent執行(ex: td-agent -c xxx.conf)後,如果node.js程式當中有做console.log,則在console output內可以看到console.log的結果喔!</div><div><br></div><div><br></div><div><br></div></div></div></div></div></div></div> k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-11661939503447753112015-06-10T04:37:00.000-07:002015-06-10T04:38:08.501-07:00從藏在GKE後面的秘密網站偷看Docker...<div dir="ltr"> <div>Google Container Engine(GKE)在推出後,讓許多Docker運作的麻煩輕鬆了不少... 最近發現了在Master Node隱藏的Kubernetes管理平台...</div><div>在GKE平台建立後,使用者可以透過Master Node的Public IP (<a href="https://master-node-ip-address/static/app/#/dashboard/">https://master-node-ip-address/static/app/#/dashboard/</a>) 連線到這個管理平台,雖然平台的功能還很陽春,但是卻已經偷偷告知未來在K8S上的一些操作將會更加簡單...<br></div><div><br></div><div>Cluster群的狀態:</div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBp2rJ_XAO4UJIlE10Ks7tvKOqlDgEXR4BdJR65tuY70hC3V-SHRg_8jbm1N2k25CoVpwVmZM0eczhJJIZIvidmuCzsXE27YGHqIYAb4V4uBx9LMoHPz-WhNaqN7sbNkVGhTbgX1EKgsw/s1600/image-788502.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBp2rJ_XAO4UJIlE10Ks7tvKOqlDgEXR4BdJR65tuY70hC3V-SHRg_8jbm1N2k25CoVpwVmZM0eczhJJIZIvidmuCzsXE27YGHqIYAb4V4uBx9LMoHPz-WhNaqN7sbNkVGhTbgX1EKgsw/s320/image-788502.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6158709466484466786" /></a><br></div><div><br></div><div>透過表單可以知道大致上Pods, Replication Controller, Services... 都可以透過介面來查看他的狀態...</div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKn1eBov0o1D6NTOapXCENuj_Pq9KeswWJbzb30XJZDwDZiVtf3_PEMUGCjc6dC3TxjkO-xkZ-8rwEGUv6eDU3xdwKP6ldOfbYUicvgdVa-cuND_n3zqx58a6SmVldm3XAIL2RDWoGdZU/s1600/image-790904.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKn1eBov0o1D6NTOapXCENuj_Pq9KeswWJbzb30XJZDwDZiVtf3_PEMUGCjc6dC3TxjkO-xkZ-8rwEGUv6eDU3xdwKP6ldOfbYUicvgdVa-cuND_n3zqx58a6SmVldm3XAIL2RDWoGdZU/s320/image-790904.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6158709476816638946" /></a><br></div><div><br></div><div>透過上面的圖片,不難想像在一個密你的Datacenter中建置自己服務的情境... 而除了這些,這次的Web Console升版,也開始讓使用者看到GCP project based repository有增加一些用心,雖然只是簡單的幾個畫面,但相信不久就會出現許多docker hub相關的功能(吧... 期待!)</div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRp-6Ec0TBMncnPmfGl51Kqm00v_dSOZeKcgHk4_LRM3sHju1zU96Ep9TBft-tCBHBMa3l7ZHYDHeeldxDuJfdG4SdO5K-ZnqkWdVK9X3jsiRnYe9zCp6asXQq4RqOgMABiRqEFeYESj0/s1600/image-793021.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRp-6Ec0TBMncnPmfGl51Kqm00v_dSOZeKcgHk4_LRM3sHju1zU96Ep9TBft-tCBHBMa3l7ZHYDHeeldxDuJfdG4SdO5K-ZnqkWdVK9X3jsiRnYe9zCp6asXQq4RqOgMABiRqEFeYESj0/s320/image-793021.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6158709483673335266" /></a><br></div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiODR7jmZo82eOk8gIz9n3hyyifYrQvT829vl5Ga3Xn7H8hDUvdqD-bQBHKzACjL4kyOrK2uzyxL-tw4YE58ebaPj8VC4zijOaVPsHJnriD5dpUnONfIdciNHNwJsp2_GcloHcCKeJz_3Q/s1600/image-795320.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiODR7jmZo82eOk8gIz9n3hyyifYrQvT829vl5Ga3Xn7H8hDUvdqD-bQBHKzACjL4kyOrK2uzyxL-tw4YE58ebaPj8VC4zijOaVPsHJnriD5dpUnONfIdciNHNwJsp2_GcloHcCKeJz_3Q/s320/image-795320.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6158709494111074274" /></a><br></div><div><br></div><div>如果對GKE還不慎了解,可以從Kubernetes (K8S)開始,其中K8S平台所提供的幾個重點元件包含:</div><div><br></div><div>- Pods: Container執行的一個群集,可以運作於一台或是Cluster起來的數台主機上,允許內部定義Volumes來共用某個空間... 而Pods中運作的Container也會受到K8S node agent的監控,任何掛掉的狀況,會被自動重啟回復。</div><div>- Replication Controller: 可以操控同一個Pod設定中的Container擴增,並可以在Container fail時候重新啟動Container。</div><div>- Services: Service為K8S中一個群集的抽象層,透過label selector為一個動態Pods提供一個抽象的結合,而Service則負責提供該集合對外的IP與DNS名稱等存取端口。</div><div><br></div><div>註:上面解釋局部翻譯於:<a href="https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/overview.md">https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/overview.md</a></div><div><br></div><div>透過這些設定,在K8S上,目前可以看到幾個優點:</div><div><br></div><div>- Nodes可以整合,對於使用者來說,可以不用再針對多個平台做操控管理</div><div>- Replica可以設定,在該設定範圍內,Container將會受到監控,任何fail的狀態,可以自動回復服務</div><div>- Rolling Update... 在程式改版後,管理者可以透過Rolling Update的功能一次將所有的運作Container做版本升級</div><div><br></div><div>而如果您的運作環境在GKE上</div><div><br></div><div>- Service的設定中的"createExternalLoadBalancer"將可以讓整個建制結合Google Network Loadbalancer做發布,讓入口端點依賴在Google的服務之下運作。</div><div>- Firewall的相關設定可以協助在運作環境上加強安全性管理...</div><div><br></div><div>整體而言,操作上... GKE的環境已經整合kubectl這個指令,讓K8S的使用者在某個層面可以透過一樣的指令存取自建的K8S或是GKE,唯有Network Load Balancer在GKE上,可以直接使用Network Load Balancer來提供服務,會優於自建K8S上必須要自己處理Service層的IP或網路對應。</div><div><br></div><div>而,未來以Docker來提供服務如果相依於K8S或GKE平台的話,針對設定的部分可以幾乎無痛轉移,也可以在Auto Scale上相對放心...</div><div><br></div><div>以上,提供給想要參考Docker作為未來production服務平台的人一些參考 :D</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div> k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-20607236039433033002015-05-22T20:44:00.001-07:002015-05-22T20:44:32.367-07:00StackDriver for Ubuntu 15<div dir="ltr">StackDriver出了Ubuntu 15的映像檔,但StackDriver的安裝過程有些錯誤,經詢問Support後,下面是完整的Ubuntu 15的安裝流程,給大家參考<br> <div><br></div><div><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">1. sudo su -</span><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">2. curl -o /etc/apt/sources.list.d/</span><span style="font-size:12.8000001907349px">stackdriver.list </span><a href="https://repo.stackdriver.com/trusty.list" target="_blank" style="font-size:12.8000001907349px">https://repo.stackdriver.com/trusty.list</a><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">3. curl --silent </span><a href="https://app.google.stackdriver.com/RPM-GPG-KEY-stackdriver" target="_blank" style="font-size:12.8000001907349px">https://app.google.stackdriver.com/RPM-GPG-KEY-stackdriver</a><span style="font-size:12.8000001907349px"> |apt-key add -</span><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">4. apt-get update</span><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">5. apt-get install stackdriver-agent</span><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">6. This will update the installation, prompt you for the API key, and start the agent. If you are installing the agent in an automated fashion [1], you can preset the API key in debconf to avoid being prompted for the key by running the following command, substituting your API key [2] for STACKDRIVER_API_KEY:</span><br style="font-size:12.8000001907349px"><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">sudo echo "stackdriver-agent stackdriver-agent/apikey string STACKDRIVER_API_KEY" | debconf-set-selections</span><br style="font-size:12.8000001907349px"><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">7. Continue with Verifying successful Linux installation [3].</span><br style="font-size:12.8000001907349px"><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">[1]: </span><a href="https://cloud.google.com/monitoring/agent/install-agent#automated_installation" target="_blank" style="font-size:12.8000001907349px">https://cloud.google.com/monitoring/agent/install-agent#automated_installation</a><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">[2]: </span><a href="https://cloud.google.com/monitoring/agent/install-agent#api-key" target="_blank" style="font-size:12.8000001907349px">https://cloud.google.com/monitoring/agent/install-agent#api-key</a><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">[3]: </span><a href="https://cloud.google.com/monitoring/agent/install-agent#verify" target="_blank" style="font-size:12.8000001907349px">https://cloud.google.com/monitoring/agent/install-agent#verify</a><br style="font-size:12.8000001907349px"></div></div> k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-60380948302080166592015-05-22T11:19:00.000-07:002015-05-22T11:24:02.053-07:00GCE Network Load Balancer也可以輕易達到AutoScaler的功能...<div dir="ltr">
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
截至目前為止,Network Load Balancer尚未能夠結合Instance Group的Auto Scaler設定,但是在創意無限的MiTAC GCP團隊裡,我們(感謝Sunny)化腐朽為神奇.... 下面是透過Auto Scaler加上Instance template中設定startup/shutdown script來動態的加入或移除Network Load Balancer的建置步驟,給大家參考 :D</div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
<br /></div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
1. 首先去要先設定network load balancer , target pool . 在這邊pool名稱我們設定為:lb-pool, 須注意,pool name必須在之後的startup與shutdown script中帶入...</div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
<br /></div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
2. 增加forwarding rule,並選擇 lb-pool 作為他的target pool</div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
<br /></div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
3. 設定instance template , 並且在template中增加startup-script / shutdown-script</div>
<div style="margin: 0px;">
<br /></div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium; text-align: center;">
<div style="margin: 0px; text-align: left;">
<div style="text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiqS0ycMc1gXCxNkXmHeSx8OB_SKXWY-T_iFYhJno4AO_xqn_HlUBVBuxJhmqoe8C-kxsPYUFCvl00o1XBQmlcpcc8p2PMJEBiU-0VhvvmyLrh45ehZEbeTw-RFhgG5eXo-sVQbv67W1r4/s1600/image-701880.png"><img alt="" border="0" height="172" id="BLOGGER_PHOTO_ID_6151762414125896706" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiqS0ycMc1gXCxNkXmHeSx8OB_SKXWY-T_iFYhJno4AO_xqn_HlUBVBuxJhmqoe8C-kxsPYUFCvl00o1XBQmlcpcc8p2PMJEBiU-0VhvvmyLrh45ehZEbeTw-RFhgG5eXo-sVQbv67W1r4/s640/image-701880.png" width="640" /></a></div>
</div>
<div style="margin: 0px; text-align: left;">
<div style="text-align: center;">
<img height="169" style="height: auto;" width="640" /></div>
</div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
為了方便貼上(<span style="color: red;">關鍵就在這裡!</span>):</div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
<b><br /></b></div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
<b>startup-script</b></div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
gcloud compute target-pools add-instances lb-pool --instances `hostname` --zone asia-east1-c</div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
<br /></div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
<b>shutdown-script</b></div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
gcloud compute target-pools remove-instances lb-pool --instances `hostname` --zone asia-east1-c</div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
<br /></div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
由於屆時主機需要具備compute engine的操作權限(startup/shutdown script用的...),我們必須在主機權限的部分設定該template所產生的主機具備Compute的"Read Write"的權限... 如果instance中有需要用到其他服務,也需要在這邊一併設定...</div>
<div style="margin: 0px;">
<br /></div>
<div style="margin: 0px;">
<div style="text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKNX8I-MDWkmuI_8KOIv94C60o0OetKVNRCVXRFa7-KXTu0jgLB_rg6XKG2MiNZs4J9tPBzLnHFFcIfnmqkpGqdk3EnhhLINxChrz99G3w2GFxo62xwZ_O-LfofSLqNdcjgnATBolp7Qs/s1600/image-703987.png"><img alt="" border="0" height="114" id="BLOGGER_PHOTO_ID_6151762422662905474" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKNX8I-MDWkmuI_8KOIv94C60o0OetKVNRCVXRFa7-KXTu0jgLB_rg6XKG2MiNZs4J9tPBzLnHFFcIfnmqkpGqdk3EnhhLINxChrz99G3w2GFxo62xwZ_O-LfofSLqNdcjgnATBolp7Qs/s640/image-703987.png" width="640" /></a></div>
</div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium; text-align: center;">
<div style="margin: 0px;">
<img height="115" style="height: auto;" width="640" /></div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
<br /></div>
<div style="margin: 0px;">
4. 建立instance groups,並確認autoscaling的設定打開,設定必須以CPU usage為監控項目,讓主機依照CPU設定的門檻做自動增加及減少的動作...</div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSXz9LfdLuWtnZ9yTCFPiP2Y_mBLJ61lOV9mxmr8r84-RN_ehLrYyoY9IAyUw5JWSB6hiR9wCoMWi5-v6ROROl_3wA9-elufzFAoZ1iy0uBYQ2TC-l9JI5j-qsBE8v3Z23l3Msq1-GJgg/s1600/image-706181.png" style="margin-left: 1em; margin-right: 1em;"><img alt="" border="0" height="400" id="BLOGGER_PHOTO_ID_6151762430946417042" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSXz9LfdLuWtnZ9yTCFPiP2Y_mBLJ61lOV9mxmr8r84-RN_ehLrYyoY9IAyUw5JWSB6hiR9wCoMWi5-v6ROROl_3wA9-elufzFAoZ1iy0uBYQ2TC-l9JI5j-qsBE8v3Z23l3Msq1-GJgg/s400/image-706181.png" width="386" /></a></div>
<div style="margin: 0px;">
</div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium; text-align: center;">
<div style="margin: 0px;">
<img height="400" style="height: auto;" width="376" /></div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium; margin: 0px;">
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
<br /></div>
<div style="margin: 0px;">
5. 一切設定妥當,就可以壓力測試一下服務啦...</div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
<div style="margin: 0px;">
<br /></div>
</div>
<div style="color: black; font-family: 'Heiti TC'; font-size: medium;">
</div>
</div>
k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-87593745313037444432015-05-11T04:13:00.001-07:002015-05-11T04:13:33.869-07:00Cloud Logging for Operation Logs, and BigQuery / CloudStorage export support<div dir="ltr"><div>一直以來只用到Cloud Logging的GAE Log顯示、查詢功能... 有一天... 發現他查得到GAE的紀錄... 不只這樣,還包含VPN Service跟防火牆設定等等的紀錄喔....</div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqVAUjqTRRMVnMzXXSpS9y71Q7JNARfk1Qk1-cEA5hUJawT9oG5dVkmsZOT185Y5R4uDJEgGVMpTAfEcLYm-CQO_1GxVlDxmzaW_Q1DxKQMRoVzKRRbwNEOdOoTT_78ZGwxPgIyajdkLY/s1600/image-713870.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqVAUjqTRRMVnMzXXSpS9y71Q7JNARfk1Qk1-cEA5hUJawT9oG5dVkmsZOT185Y5R4uDJEgGVMpTAfEcLYm-CQO_1GxVlDxmzaW_Q1DxKQMRoVzKRRbwNEOdOoTT_78ZGwxPgIyajdkLY/s320/image-713870.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6147570582418784722" /></a><br></div><div><br></div><div>而現在,GCP的Operation Log可以更準確的查詢了.... </div><div><br></div><div>最近Cloud Logging的功能加強了,可以在Monitoring > Logs裡面可以找到Exports的設定,透過這個介面可以設定App Engine與Compte Engine的Log到BigQuery或是Cloud Storage中...</div><div><br></div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjX1gmLyFcU07oWKohSmxJWZCz3FUS_oM-1dW0m1yXkUi8uBdbi9jnAPrx4e1ltPvr1DO6v_I3Zyie-ojXIn0urx17Gr9fHWNWBnJRJ0-TTZwEm1FOQa8Vc9d3lNUF1K_Feck7iXW9Ggn4/s1600/image-717987.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjX1gmLyFcU07oWKohSmxJWZCz3FUS_oM-1dW0m1yXkUi8uBdbi9jnAPrx4e1ltPvr1DO6v_I3Zyie-ojXIn0urx17Gr9fHWNWBnJRJ0-TTZwEm1FOQa8Vc9d3lNUF1K_Feck7iXW9Ggn4/s320/image-717987.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6147570592991947378" /></a><div><br></div><div>然後,就可以在指定的dataset中查詢到更詳細的Operation Log了...<br> <div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhO6wu7f95dnVhvSCp4ZEs-Yz7Dg8ZV80C7xjy56FlwE3DboE_8eGWuA_nVM7IPtFMfkDvppvEMV2IiHo-y6EwZHcnqgkjKub5T_LH7h6J5vz88Tfj5ORuBc2cok6PRuHdStsgXvgtNcs4/s1600/image-720892.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhO6wu7f95dnVhvSCp4ZEs-Yz7Dg8ZV80C7xjy56FlwE3DboE_8eGWuA_nVM7IPtFMfkDvppvEMV2IiHo-y6EwZHcnqgkjKub5T_LH7h6J5vz88Tfj5ORuBc2cok6PRuHdStsgXvgtNcs4/s320/image-720892.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6147570607320863394" /></a><br></div></div><div><br></div><div>維持Google一貫的程式人員風格,資料還是收容再以non-structure的json格式中,在BigQuery中會以repeat的方式存在,而整個原始資料則是存在"textPayload"中...</div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjgZzhhcTzdePzuoTm45O9NLtH6DoWCdjk7k8ProaRz4ycwgoEzas4c5c-cUVxCsEcGVyMnxXjiuHwcVqO-CE8vHSDh6DrtjhVvDPPGvIgA3ihHA9sdXhhrX3rPyo95x_NNj2bs6UWJww/s1600/image-723983.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjgZzhhcTzdePzuoTm45O9NLtH6DoWCdjk7k8ProaRz4ycwgoEzas4c5c-cUVxCsEcGVyMnxXjiuHwcVqO-CE8vHSDh6DrtjhVvDPPGvIgA3ihHA9sdXhhrX3rPyo95x_NNj2bs6UWJww/s320/image-723983.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6147570618999763538" /></a><br></div><div>有了這些設定,不論在帳務的掌控,或是安全性的稽核,都有更完整的參考資料了喔!</div></div> k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com1tag:blogger.com,1999:blog-1031759891970519889.post-91682372473084613582015-05-10T05:29:00.001-07:002015-05-10T05:29:59.966-07:00You can find BigTable and Deployments UI now!!<div dir="ltr"><div>BigTable在儲存(Storage)這個項目下:</div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgis7mac7DKIX-l0iZkHsflBMpG11NL6uJQmmnummr90MXK2uk4SdAKeVEw6J5ns2EJVMx5cg5i6xx8OqvyrfRb6dcaMAC3tVjj5UwVIbKadZVmGo-ilhlHJVqWBFj4MofAYCJU56tdFqg/s1600/image-799968.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgis7mac7DKIX-l0iZkHsflBMpG11NL6uJQmmnummr90MXK2uk4SdAKeVEw6J5ns2EJVMx5cg5i6xx8OqvyrfRb6dcaMAC3tVjj5UwVIbKadZVmGo-ilhlHJVqWBFj4MofAYCJU56tdFqg/s320/image-799968.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6147219192964542802" /></a><br></div><div><br></div><div>Deployment在部署與管理(Deploy & Manage)這個項目下:</div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiikjsgzpgQDOvaaF6jwAvqbXiDa5Ms9efqkOaUEQvMPElmPNlTLAL5Oe00IapZETo49livPuCRt4TkXI6ckiX0IDakElyCs3I2tjxYOOSLQQhM4XScYmVX-pKrVGFgdRqAMfPQqOqyJTw/s1600/image-704552.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiikjsgzpgQDOvaaF6jwAvqbXiDa5Ms9efqkOaUEQvMPElmPNlTLAL5Oe00IapZETo49livPuCRt4TkXI6ckiX0IDakElyCs3I2tjxYOOSLQQhM4XScYmVX-pKrVGFgdRqAMfPQqOqyJTw/s320/image-704552.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6147219206400145970" /></a><br></div><div><br></div><div>怎麼玩?!待小弟好好找個時間測測 :D</div> </div> k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-24418822130970952092015-05-04T08:05:00.000-07:002015-05-04T08:06:11.848-07:00Google Cloud Storage object url CNAME mapping with SSL<div dir="ltr"><div>繼Cloud Storage Website Bucket之後,目前GCS上的Object也可以在不轉變為website bucket情況下,簡單的mapping物件的url</div><div>關鍵是需要把bucket的名稱以dns命名規則建立,然後開始上傳您的物件...</div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAAPWJNQvbJJm7L70i_PmNI4VdC_fv3hZb7IXAYEi3ZtMvcqAJo5mjUdfwWe6lU-ILmQ0uyVXm0ofImCBkeBHpqrm_9dfZOC-G9T-E0MqeWwMDNcmZ8UCn5aaJkDDuYv2VH5cRi782l_w/s1600/image-771850.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAAPWJNQvbJJm7L70i_PmNI4VdC_fv3hZb7IXAYEi3ZtMvcqAJo5mjUdfwWe6lU-ILmQ0uyVXm0ofImCBkeBHpqrm_9dfZOC-G9T-E0MqeWwMDNcmZ8UCn5aaJkDDuYv2VH5cRi782l_w/s320/image-771850.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6145032931197718898" /></a><br></div><div>然後我們需要到DNS server指定CNAME record (<a href="http://gcsweb.micloud.tw">gcsweb.micloud.tw</a> --> <a href="http://c.storage.googleapis.com">c.storage.googleapis.com</a>),讓網域名稱與bucket名稱可以得到對應...</div><div>就是這麼簡單,上傳之後,物件其實就可以以<a href="http://gcsweb.micloud.tw/GCPUG-1.png">http://gcsweb.micloud.tw/GCPUG-1.png</a>來瀏覽...</div><div><br></div><div>而,加上SSL的關鍵,其實是使用CloudFlare的CDN功能...</div><div>在CloudFlare上,當Domain Name過CDN,在會自動增加SSL的連線能力,讓http跟https都可以通...</div><div><br></div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEga1QaI0xhpUl8Acp0Cws4b1zb6PtXiBKYmH1kRIHsIzm_iBBYxm0IuOJ42gx24rCCNNEt2dzbQ70Nw79TI6LgA4RpHhWYvgoUYjAfpo0_Phgjh-mo5zR0BOPpTfPLAo6mtiuby8LGnXRM/s1600/image-774624.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEga1QaI0xhpUl8Acp0Cws4b1zb6PtXiBKYmH1kRIHsIzm_iBBYxm0IuOJ42gx24rCCNNEt2dzbQ70Nw79TI6LgA4RpHhWYvgoUYjAfpo0_Phgjh-mo5zR0BOPpTfPLAo6mtiuby8LGnXRM/s320/image-774624.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6145032943378680482" /></a><br> <div><br></div><div>只要您的DNS是hosting在Cloudflare,則可以在打開CDN設定後,讓連線可以透過HTTPS連線</div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDjSLFysttWDy-XPgRRHhg6ODjYS_LMH2LVDAprLXLakKNcRsATeX_2ljIIN8bzTi4_OmRRs4cd6K73uIoAjLSOVYGfGGws7LwpugvohdaXCYm0uAwwyU8Ea9rNYjlrWxjKbGwUpqqYQ8/s1600/image-776756.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDjSLFysttWDy-XPgRRHhg6ODjYS_LMH2LVDAprLXLakKNcRsATeX_2ljIIN8bzTi4_OmRRs4cd6K73uIoAjLSOVYGfGGws7LwpugvohdaXCYm0uAwwyU8Ea9rNYjlrWxjKbGwUpqqYQ8/s320/image-776756.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6145032946701263570" /></a><br></div><div><br></div><div><a href="https://gcsweb.micloud.tw/GCPUG-1.png" target="_blank">https://gcsweb.micloud.tw/GCPUG-1.png</a><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8qD0Rf85_3uVI-cuka15BKNcC_59sU-Xiy8twAHSKEZW6dVArMsFEPaAsdFZ3RwgQgaTqFwU7_pCK-bDRgrJ9KT6QdknMaK-es1ITBEuDGMhVTgDZgqXppmH4G6rz_FsKkmAFeyYa_44/s1600/image-779187.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8qD0Rf85_3uVI-cuka15BKNcC_59sU-Xiy8twAHSKEZW6dVArMsFEPaAsdFZ3RwgQgaTqFwU7_pCK-bDRgrJ9KT6QdknMaK-es1ITBEuDGMhVTgDZgqXppmH4G6rz_FsKkmAFeyYa_44/s320/image-779187.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6145032959506909746" /></a><br></div><div><br></div><div><br></div><div>寫到這邊,突然覺得可怕... 如果GCS蟑螂出現,開始book大家的Domain.... 還好Google在建立bucket時候,有檢查網域是否屬於您所擁有... 如果不是您的網域,則會得到下面的錯誤訊息... 必須要經過Bucket Name Verification來認證您是bucket的擁有者喔 :D</div><div><br></div><div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiNa693ucNmhtGSPLy9oSyyM-KtY6FfMKZAHl0pjHYX0rn5JPhp_KDm80Rh53VhFje9EoYS7bu0nz25v2oshokLp3pP8WeHzKNtfeS0v128StJBsMg3xMO-oVNYUO1xMkXGbLRcGx01ic/s1600/image-781677.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiNa693ucNmhtGSPLy9oSyyM-KtY6FfMKZAHl0pjHYX0rn5JPhp_KDm80Rh53VhFje9EoYS7bu0nz25v2oshokLp3pP8WeHzKNtfeS0v128StJBsMg3xMO-oVNYUO1xMkXGbLRcGx01ic/s320/image-781677.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6145032969984725586" /></a><br></div><div><br></div><div><br></div><div>Reference: <a href="https://cloud.google.com/storage/docs/reference-uris" target="_blank">https://cloud.google.com/storage/docs/reference-uris</a><br></div></div> k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0tag:blogger.com,1999:blog-1031759891970519889.post-69722445810313677682015-04-27T18:42:00.001-07:002015-04-27T18:42:25.222-07:00GCP Web Console再一異動!<div dir="ltr">哇~開主機畫面變漂亮了 :D<br> <div><br></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_xLNPNMpDRFdb1J93JM7N_qjhp-o_VPZXqwILwcUpNr5bbUHEdrTF1HYrFPflBB_5AQg2DaV2nAmXHht2_I0JumjSHel-lTuomDadgmFg2-aml0alDD3dvZLbGeyhx6aLOy5fbbFotGY/s1600/image-745223.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_xLNPNMpDRFdb1J93JM7N_qjhp-o_VPZXqwILwcUpNr5bbUHEdrTF1HYrFPflBB_5AQg2DaV2nAmXHht2_I0JumjSHel-lTuomDadgmFg2-aml0alDD3dvZLbGeyhx6aLOy5fbbFotGY/s320/image-745223.png" border="0" alt="" id="BLOGGER_PHOTO_ID_6142599289663395090" /></a><br></div></div> k8s-studyhttp://www.blogger.com/profile/15291073043196262423noreply@blogger.com0