Image archive, analysis, and report generation with Google APIs

How can Google Cloud help? Like Drive, Cloud Storage provides file (and generic blob) storage in the cloud. (More on the differences between Drive & Cloud Storage can be found in this video.)

Cloud Storage provides several storage classes depending on how often you expect to access your archived files. The less often files are accessed, the “colder” the storage, and the lower the cost. As users progress from one project to another, they’re not as likely to need older Drive folders and those make great candidates to backup to Cloud Storage.

First challenge: determine the security model. When working with Google Cloud APIs, you generally select OAuth client IDs to access data owned by users and service accounts for data owned by applications/projects. The former is typically used with Workspace APIs while the latter is the primary way to access Google Cloud APIs. Since we’re using APIs from both product groups, we need to make a decision (for now and change later if desired).

Since the goal is a simple proof-of-concept, user auth suffices. OAuth client IDs are standard for Drive & Sheets API access, and the Vision API only needs API keys so the more-secure OAuth client ID is more than enough. The only IAM permissions to acquire are for the user running the script to get write access to the destination Cloud Storage bucket. Lastly, Workspace APIs don’t have their own product client libraries (yet), so the lower-level Google APIs “platform” client libraries serve as a “lowest common denominator” to access all four REST APIs. Those who have written Cloud Storage or Vision code using the Cloud client libraries will see something different.

The prototype is a command-line script. In real life, it would likely be an application in the cloud, executing as a Cloud Function or a Cloud Task running as determined by Cloud Scheduler. In that case, it would use a service account with Workspace domain-wide delegation to act on behalf of an employee to backup their files. See this page in the documentation describing when you’d use this type of delegation and when not to.

Our simple prototype targets individual image files, but you can continue to evolve it to support multiple files, movies, folders, and ZIP archives if desired. Each function calls a different API, creating a “service pipeline” with which to process the images. The first pair of functions are drive_get_file() and gcs_blob_upload(). The former queries for the image on Drive, grabs pertinent metadata (filename, ID, MIMEtype, size), downloads the binary “blob” and returns all of that to the caller. The latter uploads the binary along with relevant metadata to Cloud Storage. The script was written in Python for brevity, but the client libraries support most popular languages. Below is the aforementioned function pseudocode:

What's your reaction?

In Love
Not Sure

You may also like

More in:GOOGLE

Leave a reply

Your email address will not be published. Required fields are marked *