Load data to Google BigQuery using csv file upload and append data from Google Cloud Storage (GCS) bucket, Amazon S3 bucket using Big Query Data Transfer Service and view data in Looker Studio. Then finally add more data using Python to the table.
In another project based on a real-world scenario, I acted as a Cloud Engineer in a company that uses Google Cloud to load data to Google BigQuery using a csv and also from Google Cloud Storage(GCS) and AWS S3 services and then use Looker Studio to create a report. At the end appended 4 more records to the table by using Python code. The Looker Studio report shows the complete data once it is refreshed.
Below are few screenshots:
Create a dataset in BigQuery
- Upload a csv file from local machine to the table in BigQuery
Create a GCS bucket and load a csv file to it.
2. Append GCS file to the table in BigQuery
Create a file in Amazon S3 bucket
3. Append Amazon S3 bucket file to the table in BigQuery
Launced Looker Studio to view the data
Add a service account
Python code to append 4 more additional record to the BigQuery table.
Ran the Python code successfully and able to view the data in BigQuery table
Open the Looker Studio report that was created earlier and refresh the data and we can see the additional 4 records that were added by Python code in the report.
