flask - Google App Engine: unexpected cost and non-transparent billing status (app engine dashboard disagree with billing export results)

Question

I have been exploring the App Engine settings for a small data science web application for 2 weeks. Since it is a personal project that bills my own wallet, I tried a few different parameters in app.yaml to reduce the "frontend instances" cost. Several changes in, I got unexpected ~10x cost surge!!! It was painful!!! In order to not waste it, I decided to learn something here to understand the behaviour :)... Don't worry, I had temporarily shut down my app ;)

Version 1 app.yaml:

service: my-app
runtime: python37
instance_class: F4
env: standard
automatic_scaling:
  min_idle_instances: 1
  max_idle_instances: 1
default_expiration: "1m"

inbound_services:
- warmup

entrypoint: gunicorn -b 0.0.0.0:8080 main:server

Version 1, billing result (usage.amount_in_pricing_units exported from billing account): ~100hr/day, the same as Front end Instance Hours shown from App Engine billing status. This is understandable, because I had a F4 instance constantly runing idle that would translate into 24*4=96 frontend instance hours. Adding the instance usage from actual requests (from me only), ~100hr/day seems reasonable.

Version 2, where I intended to lower the instance class and number of instances and also made longer the default_expiration and hoping it would help the app to start quicker and some other stuff that I thought wouldn't affect much....

service: my-app
runtime: python37
instance_class: F2
env: standard
automatic_scaling:
  min_instances: 1
  max_instances: 1
  target_cpu_utilization: 0.85
  max_concurrent_requests: 80
  max_pending_latency: 6s
default_expiration: "3h"

inbound_services:
- warmup

entrypoint: gunicorn -b 0.0.0.0:8080 main:server

Version 2, billing result (usage.amount_in_pricing_units exported from billing account): ~800hr+/day, ouch!!! In contrast, the Front end Instance Hours from App Engine dashboard billing status is less than 60hr/day as expected. This is where I got lost:

1. Why the usage from billing is so much larger than the App Engine Dashboard where do those usage come from?

2. Where to find and track indicators of those unaccounted usage in App Engine Dashboard etc?

Answer 1

Two days ago while I was waiting for Google Billing Support to come back to me, I found this: Pricing of Google App Engine Flexible env, a $500 lesson

Namely, the past deployed versions of the app also eating frontend instance hours, which needed real world confirmation. (To my surprise, this has nothing to do with app.yaml file!!) So I deleted all the past versions of the app and let it run for two days while observing instance hours and billing records with the following app.yaml file.

service: my-app
runtime: python37
instance_class: F2
env: standard
automatic_scaling:
  min_instances: 1
  max_instances: 2
  max_idle_instances: 1
  target_cpu_utilization: 0.85
  max_concurrent_requests: 80
  max_pending_latency: 6s
default_expiration: "1m"

inbound_services:
- warmup

entrypoint: gunicorn -b 0.0.0.0:8080 main:server 

This should have always one F2 instance running and goes up to maximum 2 instances. This time both app engine and exported billing usage hours agreed on 50 hours frontend instance hours. Yes!!! The daily cost is cut down to 1/16.

This solves the cost question #1, but #2 remains to be answered. It is very problematic that app engine dashboard is not showing all the billed usage of frontend instances. Yesterday I heard from Google Billing Support Team, the answer is not helpful (mainly talking about instance numbers in app.yaml, which doesn't help), they seem oblivious about this issue, I will have to let them know.