![]() You'd rather have the object represented in the native type system that BQ provides instead of raw json.īQ does have some json extraction syntax for dealing with "raw" json. ┆Issue is synchronized with this Asana task by Unito Thanks for looking into this! Let me know if there is anything further I can provide to help make this happen. Arguably, the "most raw" form is the object represented by the JSON, not a row containing JSON serialized to a string. ![]() But it would make a lot more sense for Airbyte to simply load the data in its true, raw form. In theory, I could work around the problem by exporting all the data from the _airbyte_data column and then going through the same procedure of generating a schema and then manually uploading the data. Then I use the BigQuery console to load the data it takes JSONL as one of its source formats and accepts a JSON schema, and that works nicely. The way I've usually done this is to export the data from my data source into a big JSONL file (newline-delimited JSON), then run bigquery-schema-generator over it to produce a JSON schema file. Describe the alternative you’ve considered or used ![]() bigquery-schema-generator will scan the whole file and come up with a schema that can accommodate every record. The –autodetect option to bq won't cut it, because it only scans 100 rows, so if a new field appears in some other row, the entire import will fail. I would recommend using, which will scan the file and produce a schema. Note that it's necessary to provide a schema. The usual bq command can do this, for example: (b) Use Basic Normalization, which doesn't handle hierarchical data Describe the solution you’d likeīigQuery is already built to handle hierarchical data, so I'd love to have JSON simply load into BigQuery as structured records. (a) Load the JSON data as a big string in one _airbyte_data column, which I can't query However, with Airbyte, I only have two options: Google AdWords) to be available in my destination, BigQuery, so that I can run meaningful queries on the data. I'd like the structure of the JSON in my data source (e.g. Tell us about the problem you're trying to solve
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |