Google Cloud have a nice Log Service, with some cool features (like Traces, I wrote before), but it lacks real analytics based on this logs. Like Kibana.
Fortunately Google Cloud can export logs to Cloud Storage. What’s cool is that this logs are in JSON format, so we could easily import them into ELK, without any complex Logstash configuration (honestly I cannot say that JSON schema fits well ELK, but still it’s easy to import).
I’ve prepared a basic Docker container with Elasticsearch, Logstash and Kibana configured for Appengine logs. Run ELK container with:
(you can get sources of this Docker image there).
To download logs run following (you have to replace
logs_bucket with bucket that contains your logs):
One important note: you need to download logs after ELK Docker is started, or Logstash will ignore any existing data, and start processing only data added after.
And also, don’t expect it to index all your logs immediately, it takes a time. Maybe hours. Google Cloud creates
%YEAR/%MONTH%/%DAY/ path for every log file, that’s pretty useful, since you can load only logs for some concrete day (or month):
As you’ve already started ELK container, downloaded logs, now you can go to Kibana at http://192.168.59.103:5601/. Where 192.168.59.103 is IP of a docker vm on my machine, but it could be different on your. If you’re using boot2docker run
boot2docker ip to get such IP
At first run you’ll get Kibana Settings page, where you have to set index pattern (keep default
logstash-*) and a field used for log timestamp (use
And that’s it. Use all power of Kibana to make awesome reports from your logs.
If you’ll decide to make your own ELK setup for Apppengine there is most important part of logstash configuration:
PS If you are unfamiliar with Kibana, you should definitely try it now :) it’s great tool to make useful metrics. Like chart for average response time, for whole app, or maybe for some particular area (like based on urls of some specific API endpoints, etc). It allows you to compare workload from different client types, or different versions. And many many other things. Also useful if your webapp have an API, and most of API calls cannot be tracked by using a 3rd party analytics service, such as Google Analytics.