Google Drive

This page was recently updated. What do you think about it? Let us know!.

Connect Google Drive to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.

The requirements are as follows.

1. A Google Cloud service account and its related credentials.json key file or its contents in JSON format. Learn how.

2. To ensure maximum compatibility across Unstructured service offerings, you should give the service account key information to Unstructured as a single-line string that contains the contents of the downloaded service account key file (and not the service account key file itself). To print this single-line string without line breaks, suitable for copying, you can run one of the following commands from your Terminal or Command Prompt. In this command, replace <path-to-downloaded-key-file> with the path to the credentials.json key file that you downloaded by following the preceding instructions.

   • For macOS or Linux:

     tr -d '\n' < <path-to-downloaded-key-file>
     

   • For Windows:

     (Get-Content -Path "<path-to-downloaded-key-file>" -Raw).Replace("`r`n", "").Replace("`n", "")
     

3. Give the service account’s email address access to the Google Drive folder. Learn more.

4. Note the folder ID. This is a part of the URL for your Google Drive folder represented in the following URL as {folder_id}: https://drive.google.com/drive/folders/{folder-id}.

The Google Drive connector dependencies:

CLI, Python

pip install "unstructured-ingest[google-drive]"

You might also need to install additional dependencies, depending on your needs. Learn more.

The following environment variables:

• GOOGLE_DRIVE_FOLDER_ID - The folder ID, represented by --drive-id (CLI) or drive_id (Python).

• One of the following:

  • GCP_SERVICE_ACCOUNT_KEY_FILEPATH - The path to the credentials.json key file, represented by --service-account-key-path (CLI) or service_account_key_path (Python).
  • GCP_SERVICE_ACCOUNT_KEY_STRING - The contents of the credentials.json key file as a string, represented by --service-account-key (CLI) or service_account_key (Python).

These environment variables:

• UNSTRUCTURED_API_KEY - Your Unstructured API key value.
• UNSTRUCTURED_API_URL - Your Unstructured API URL.

Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:

Output a key file’s contents as a string

If you need to convert the contents of a credentials.json file into a string, you could use a Python script such as the following. This script takes the local path to the key file as input and outputs the key file’s contents as a string.

Python

# Filename: json_file_to_string.py

import json
import sys

def json_file_to_string(file_path):
    try:
        # Read the JSON file.
        with open(file_path, 'r') as file:
            # Load the JSON data.
            data = json.load(file)
        
        # Convert the JSON data back to a string, with no whitespace.
        json_string = json.dumps(data, separators=(',', ':'))
        
        return json_string
    
    except FileNotFoundError:
        print(f"Error: File '{file_path}' not found.")
        return None
    except json.JSONDecodeError:
        print(f"Error: '{file_path}' is not a valid JSON file.")
        return None
    except Exception as e:
        print(f"An unexpected error occurred: {e}")
        return None

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python json_file_to_string.py <path_to_json_file>")
        sys.exit(1)
    
    file_path = sys.argv[1]
    result = json_file_to_string(file_path)
    
    if result:
        print(result)