r - 使用SparkR查找成为主键的变量

这是我的玩具数据:

df <- tibble::tribble(
  ~var1, ~var2, ~var3, ~var4, ~var5, ~var6, ~var7,
    "A",   "C",    1L,    5L,  "AA",  "AB",    1L,
    "A",   "C",    2L,    5L,  "BB",  "AC",    2L,
    "A",   "D",    1L,    7L,  "AA",  "BC",    2L,
    "A",   "D",    2L,    3L,  "BB",  "CC",    1L,
    "B",   "C",    1L,    8L,  "AA",  "AB",    1L,
    "B",   "C",    2L,    6L,  "BB",  "AC",    2L,
    "B",   "D",    1L,    9L,  "AA",  "BC",    2L,
    "B",   "D",    2L,    6L,  "BB",  "CC",    1L)

我在以下链接上的原始问题
https://stackoverflow.com/a/53110342/6762788是:
如何获得最小数量的变量的组合，这些变量可以唯一地标识数据帧中的观测值，即哪些变量可以共同构成主键？非常感谢thelatemail，以下答案/代码完全可以正常工作。

nms <- unlist(lapply(seq_len(length(df)), combn, x=names(df), simplify=FALSE), rec=FALSE)
out <- data.frame(
  vars = vapply(nms, paste, collapse=",", FUN.VALUE=character(1)),
  counts = vapply(nms, function(x) nrow(unique(df[x])), FUN.VALUE=numeric(1))
)

现在，要使其适用于大数据，我想将其带到SparkR。利用此答案，如何在SparkR中翻译此代码？如果在SparkR中比较困难，那么我可以使用sparklyr。

最佳答案

我将上述问题分解为小段，并尝试了以下SparkR代码。但是，“counts

library(SparkR); library(tidyverse)

df_spark <- mtcars %>% as.DataFrame()

num_m <- seq_len(ncol(df_spark))

nam_list <- SparkR::colnames(df_spark)

combinations <- function(num_m) {
  combn(num_m, x=nam_list, simplify=FALSE)
}

nms <- spark.lapply(num_m, combinations) %>% unlist(rec=FALSE)

vars = map_chr(nms, ~paste(.x, collapse = ","))

counts <- lapply(nms, function(x) df_spark %>% SparkR::select(x) %>% SparkR::distinct() %>% SparkR::count()) %>% unlist()

out <- data.frame(
  vars = vars,
  counts = counts
)

primarykeys <- out %>%
  dplyr::mutate(n_vars = str_count(vars, ",")+1) %>%
  dplyr::filter(counts==nrow(df)) %>%
  dplyr::filter(n_vars==min(n_vars))

primarykeys

关于r - 使用SparkR查找成为主键的变量，我们在Stack Overflow上找到一个类似的问题：https://stackoverflow.com/questions/53326497/

sparkR

r - 使用SparkR查找成为主键的变量