Cómo pivotar una tabla en BigQuery

Estoy usando Google Big Query y estoy intentando obtener un resultado dinámico a partir de un conjunto de datos de muestra públicos.

Una consulta simple a una tabla existente es:

SELECT * 
FROM publicdata:samples.shakespeare
LIMIT 10;

Esta consulta devuelve el siguiente conjunto de resultados.

Ahora lo que estoy tratando de hacer es obtener los resultados de la tabla de tal manera que si la palabra es valiente, seleccione "BRAVE" como columna_1 y si la palabra es atendida, seleccione "ATENIDO" como columna_2 y agregue el recuento de palabras. para estos 2.

Aquí está la consulta que estoy usando.

SELECT
(CASE WHEN word = 'brave' THEN 'BRAVE' ELSE '' END) AS column_1,
(CASE WHEN word = 'attended' THEN 'ATTENDED' ELSE '' END) AS column_2,
SUM (word_count)
FROM publicdata:samples.shakespeare
WHERE (word = 'brave' OR word = 'attended')
GROUP BY column_1, column_2
LIMIT 10;

Pero esta consulta devuelve los datos.

Lo que estaba buscando es

Sé que este pivote para este conjunto de datos no tiene sentido. Pero sólo estoy tomando esto como un ejemplo para explicar el problema. Sería genial si pudieras darme algunas instrucciones.

EDITADO: También me referí a ¿Cómo simular una tabla dinámica con BigQuery? y parece que también tiene el mismo problema que mencioné aquí.

Jan 01 '70 08:01 user1401472

Actualización 2020:

Simplemente llama fhoffa.x.pivot(), como se detalla en esta publicación:

• https://medium.com/@hoffa/easy-pivot-in-bigquery-one-step-5a1f13c6c710

Para el ejemplo de 2019, por ejemplo:

CREATE OR REPLACE VIEW `fh-bigquery.temp.a` AS (
 SELECT * EXCEPT(SensorName), REGEXP_REPLACE(SensorName, r'.*/', '') SensorName
 FROM `data-sensing-lab.io_sensor_data.moscone_io13`
);

CALL fhoffa.x.pivot(
  'fh-bigquery.temp.a'
  , 'fh-bigquery.temp.delete_pivotted' # destination table
  , ['MoteName', 'TIMESTAMP_TRUNC(Timestamp, HOUR) AS hour'] # row_ids
  , 'SensorName' # pivot_col_name
  , 'Data' # pivot_col_value
  , 8 # max_columns
  , 'AVG' # aggregation
  , 'LIMIT 10' # optional_limit
);

Actualización 2019:

Dado que esta es una pregunta popular, permítanme actualizar a #standardSQL y un caso más general de pivotación. En este caso tenemos varias filas y cada sensor analiza un tipo diferente de propiedad. Para pivotarlo, haríamos algo como:

#standardSQL
SELECT MoteName
  , TIMESTAMP_TRUNC(Timestamp, hour) hour
  , AVG(IF(SensorName LIKE '%altitude', Data, null)) altitude
  , AVG(IF(SensorName LIKE '%light', Data, null)) light
  , AVG(IF(SensorName LIKE '%mic', Data, null)) mic
  , AVG(IF(SensorName LIKE '%temperature', Data, null)) temperature
FROM `data-sensing-lab.io_sensor_data.moscone_io13`
WHERE MoteName = 'XBee_40670F5F'
GROUP BY 1, 2

Como alternativa AVG()puedes probar MAX(), ANY_VALUE(), etc.

Previamente :

No estoy seguro de lo que estás intentando hacer, pero:

SELECT NTH(1, words) WITHIN RECORD column_1, NTH(2, words) WITHIN RECORD column_2, f0_
FROM (
  SELECT NEST(word) words, SUM(c)  
  FROM (
    SELECT word, SUM(word_count) c
    FROM publicdata:samples.shakespeare
    WHERE word in ('brave', 'attended')
    GROUP BY 1
  )
)

ACTUALIZACIÓN: Mismos resultados, consulta más simple:

SELECT NTH(1, word) column_1, NTH(2, word) column_2, SUM(c)
FROM (
    SELECT word, SUM(word_count) c
    FROM publicdata:samples.shakespeare
    WHERE word in ('brave', 'attended')
    GROUP BY 1
)

Oct 10 '2014 04:10 Felipe Hoffa

También inspirado en ¿Cómo simular una tabla dinámica con BigQuery? , la siguiente solicitud mediante subselección produce el resultado exacto deseado:

SELECT
  MAX(column_1),
  MAX(column_2),
  SUM(wc),
FROM (
  SELECT
  (CASE WHEN word = 'brave' THEN 'BRAVE' ELSE '' END) AS column_1,
  (CASE WHEN word = 'attended' THEN 'ATTENDED' ELSE '' END) AS column_2,
  SUM (word_count) AS wc
  FROM publicdata:samples.shakespeare
  WHERE (word = 'brave' OR word = 'attended')
  GROUP BY column_1, column_2
  LIMIT 10
)

El truco es que es MAX(NULL, 'ATTENDED', NULL, ...)igual 'ATTENDED'.

May 21 '2015 14:05 Deleplace

Usar declaraciones case/if para crear columnas pivotadas es una forma de hacerlo. Pero se vuelve muy molesto si el número de columnas pivotadas comienza a crecer. Para solucionar esto, creé un módulo de Python usando python pandas que genera automáticamente la consulta SQL que luego se puede ejecutar en BigQuery. He aquí una pequeña introducción:

https://yashuseth.blog/2018/06/06/how-to-pivot-large-tables-in-bigquery

Código de github relevante en caso de que github se caiga:

import re
import pandas as pd

class BqPivot():
    """
    Class to generate a SQL query which creates pivoted tables in BigQuery.

    Example
    -------

    The following example uses the kaggle's titanic data. It can be found here -
    `https://www.kaggle.com/c/titanic/data`

    This data is only 60 KB and it has been used for a demonstration purpose.
    This module comes particularly handy with huge datasets for which we would need
    BigQuery(https://en.wikipedia.org/wiki/BigQuery).

    >>> from bq_pivot import BqPivot
    >>> import pandas as pd
    >>> data = pd.read_csv("titanic.csv").head()
    >>> gen = BqPivot(data=data, index_col=["Pclass", "Survived", "PassengenId"],
                      pivot_col="Name", values_col="Age",
                      add_col_nm_suffix=False)
    >>> print(gen.generate_query())

    select Pclass, Survived, PassengenId, 
    sum(case when Name = "Braund, Mr. Owen Harris" then Age else 0 end) as braund_mr_owen_harris,
    sum(case when Name = "Cumings, Mrs. John Bradley (Florence Briggs Thayer)" then Age else 0 end) as cumings_mrs_john_bradley_florence_briggs_thayer,
    sum(case when Name = "Heikkinen, Miss. Laina" then Age else 0 end) as heikkinen_miss_laina,
    sum(case when Name = "Futrelle, Mrs. Jacques Heath (Lily May Peel)" then Age else 0 end) as futrelle_mrs_jacques_heath_lily_may_peel,
    sum(case when Name = "Allen, Mr. William Henry" then Age else 0 end) as allen_mr_william_henry
    from <--insert-table-name-here-->
    group by 1,2,3

    """
    def __init__(self, data, index_col, pivot_col, values_col, agg_fun="sum",
                 table_name=None, not_eq_default="0", add_col_nm_suffix=True, custom_agg_fun=None,
                 prefix=None, suffix=None):
        """
        Parameters
        ----------

        data: pandas.core.frame.DataFrame or string
            The input data can either be a pandas dataframe or a string path to the pandas
            data frame. The only requirement of this data is that it must have the column
            on which the pivot it to be done.

        index_col: list
            The names of the index columns in the query (the columns on which the group by needs to be performed)

        pivot_col: string
            The name of the column on which the pivot needs to be done.

        values_col: string
            The name of the column on which aggregation needs to be performed.

        agg_fun: string
            The name of the sql aggregation function.

        table_name: string
            The name of the table in the query.

        not_eq_default: numeric, optional
            The value to take when the case when statement is not satisfied. For example,
            if one is doing a sum aggregation on the value column then the not_eq_default should
            be equal to 0. Because the case statement part of the sql query would look like - 

            ... ...
            sum(case when <pivot_col> = <some_pivot_col_value> then values_col else 0)
            ... ...

            Similarly if the aggregation function is min then the not_eq_default should be
            positive infinity.

        add_col_nm_suffix: boolean, optional
            If True, then the original values column name will be added as suffix in the new 
            pivoted columns.

        custom_agg_fun: string, optional
            Can be used if one wants to give customized aggregation function. The values col name 
            should be replaced with {}. For example, if we want an aggregation function like - 
            sum(coalesce(values_col, 0)) then the custom_agg_fun argument would be - 
            sum(coalesce({}, 0)). 
            If provided this would override the agg_fun argument.

        prefix: string, optional
            A fixed string to add as a prefix in the pivoted column names separated by an
            underscore.

        suffix: string, optional
            A fixed string to add as a suffix in the pivoted column names separated by an
            underscore.        
        """
        self.query = ""

        self.index_col = list(index_col)
        self.values_col = values_col
        self.pivot_col = pivot_col

        self.not_eq_default = not_eq_default
        self.table_name = self._get_table_name(table_name)

        self.piv_col_vals = self._get_piv_col_vals(data)
        self.piv_col_names = self._create_piv_col_names(add_col_nm_suffix, prefix, suffix)

        self.function = custom_agg_fun if custom_agg_fun else agg_fun + "({})"

    def _get_table_name(self, table_name):
        """
        Returns the table name or a placeholder if the table name is not provided.
        """
        return table_name if table_name else "<--insert-table-name-here-->"

    def _get_piv_col_vals(self, data):
        """
        Gets all the unique values of the pivot column.
        """
        if isinstance(data, pd.DataFrame):
            self.data = data
        elif isinstance(data, str):
            self.data = pd.read_csv(data)
        else:
            raise ValueError("Provided data must be a pandas dataframe or a csv file path.")

        if self.pivot_col not in self.data.columns:
            raise ValueError("The provided data must have the column on which pivot is to be done. "\
                             "Also make sure that the column name in the data is same as the name "\
                             "provided to the pivot_col parameter.")

        return self.data[self.pivot_col].astype(str).unique().tolist()

    def _clean_col_name(self, col_name):
        """
        The pivot column values can have arbitrary strings but in order to 
        convert them to column names some cleaning is required. This method 
        takes a string as input and returns a clean column name.
        """

        # replace spaces with underscores
        # remove non alpha numeric characters other than underscores
        # replace multiple consecutive underscores with one underscore
        # make all characters lower case
        # remove trailing underscores
        return re.sub("_+", "_", re.sub('[^0-9a-zA-Z_]+', '', re.sub(" ", "_", col_name))).lower().rstrip("_")

    def _create_piv_col_names(self, add_col_nm_suffix, prefix, suffix):
        """
        The method created a list of pivot column names of the new pivoted table.
        """
        prefix = prefix + "_" if prefix else ""
        suffix = "_" + suffix if suffix else ""

        if add_col_nm_suffix:
            piv_col_names = ["{0}{1}_{2}{3}".format(prefix, self._clean_col_name(piv_col_val), self.values_col.lower(), suffix)
                             for piv_col_val in self.piv_col_vals]
        else:
            piv_col_names = ["{0}{1}{2}".format(prefix, self._clean_col_name(piv_col_val), suffix)
                             for piv_col_val in self.piv_col_vals]

        return piv_col_names

    def _add_select_statement(self):
        """
        Adds the select statement part of the query.
        """
        query = "select " + "".join([index_col + ", " for index_col in self.index_col]) + "\n"
        return query

    def _add_case_statement(self):
        """
        Adds the case statement part of the query.
        """
        case_query = self.function.format("case when {0} = \"{1}\" then {2} else {3} end") + " as {4},\n"

        query = "".join([case_query.format(self.pivot_col, piv_col_val, self.values_col,
                                           self.not_eq_default, piv_col_name)
                         for piv_col_val, piv_col_name in zip(self.piv_col_vals, self.piv_col_names)])

        query = query[:-2] + "\n"
        return query

    def _add_from_statement(self):
        """
        Adds the from statement part of the query.
        """
        query =  "from {0}\n".format(self.table_name)
        return query

    def _add_group_by_statement(self):
        """
        Adds the group by part of the query.
        """
        query = "group by " + "".join(["{0},".format(x) for x in range(1, len(self.index_col) + 1)])
        return query[:-1]

    def generate_query(self):
        """
        Returns the query to create the pivoted table.
        """
        self.query = self._add_select_statement() +\
                     self._add_case_statement() +\
                     self._add_from_statement() +\
                     self._add_group_by_statement()

        return self.query

    def write_query(self, output_file):
        """
        Writes the query to a text file.
        """
        text_file = open(output_file, "w")
        text_file.write(self.generate_query())
        text_file.close()

Jun 05 '2018 22:06 Yashu Seth