本文介绍了如何读取csv文件作为数据帧?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我想读的火花CSV,我想将其存储在一个火花数据帧。 https://databricks.com/blog/2015/02/17/introducing-dataframes-in-spark-for-large-scale-data-science.html
I would like to read a csv in spark and I would like to store it in a spark dataframe. https://databricks.com/blog/2015/02/17/introducing-dataframes-in-spark-for-large-scale-data-science.html
我曾尝试:
scala> val df = sqlContext.load("/home/donbeo/Documents/dataset/winequality-red.csv")
java.lang.RuntimeException: file:/home/donbeo/Documents/dataset/winequality-red.csv is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [49, 59, 54, 10]
at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:418)
at org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:277)
at org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$refresh$6.apply(newParquet.scala:276)
at scala.collection.parallel.mutable.ParArray$Map.leaf(ParArray.scala:658)
at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:54)
at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:53)
at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:53)
at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:56)
at scala.collection.parallel.mutable.ParArray$Map.tryLeaf(ParArray.scala:650)
at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:165)
at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:514)
at scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
但它不工作。
什么是正确的命令?
but it is not working.What is the right command?
推荐答案
星火SQL只提供3种类型的数据源的内置支持:
Spark SQL provides inbuilt support for only 3 types of data sources:
- 实木复合地板(这是默认值)
- 的Json
- JDBC
有关CSV,有一个独立的库:
For CSV, there is a separate library: spark-csv
这是 CsvContext
类提供了可以用来加载CSV csvFile
方法。
It's CsvContext
class provides csvFile
method which can be used to load csv.
val cars = sqlContext.csvFile("cars.csv") // uses implicit class CsvContext
这篇关于如何读取csv文件作为数据帧?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!