Data Ingestion to Cloud SQL from GCS using Google’s Dataflow with Apache Beam (Python)

Create a Cloud Storage Bucket

$ gsutil mb gs://mybucket

Create a Cloud SQL Instance

Create a Database and a Table with the required Schema

$ gcloud sql databases create mysqldb --instance=myinstance
$ gcloud sql connect myinstance --user=root
USE mysqldb;
CREATE TABLE beer_data (sr INT, abv FLOAT, name VARCHAR(255), style VARCHAR(255), ounces FLOAT, id INT NOT NULL, PRIMARY KEY(id));

Build a Pipeline to run in Dataflow

Building a ParDo Function to Implement Cloud SQL Write Transform

Conclusion

References:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store