Zeppelin0.6.2+sparkR2.0.2环境搭建

0.序

  先吐槽一下网上旧版本的Zeppelin和R的装置,让自身折腾了多少个钟头。

  可是最终仍旧调通了也不易于,其实自己现在一点R都未曾学呢,只是刚看了一节课,可是这一个工具既然现身在了Spark中,我想它依然蛮流行和好用的。

  在此之前安排了Zeppelin的Spark的结合,不过这尚未配置R的一部分,然则Zeppelin既然能够有这么多的解释器,在我看来就是呈现他的一个此外软件所不抱有的优势:这就是在一个note里你可以应用多种解释器来编排文字,生成图像,分析一个题材。

  再增长方便的导入导出和储存效用,我个人感觉将来盛行只是时间问题。

  废话尽量少说,进入正题:

 
Unmarshalling是Akka-http内把网上可传输格式的数额转变成程序高级结构话数据的历程,比如把Json数据转换成某个自定义类型的实例。按实际流程来说就是先把Json转换成可传输格式数据如:MessageEntity,HttpRequest,HttpReponse等,然后再转换成程序高级结构数据如classXX实例。Unmarshalling对一个A类实例到B类实例的更换是通过Unmarshaller[A,B]来实现的:

1.安装R及有关的包:

在意自身利用的是CentOS7的序列,假使您不是本身的系统可能下载的包会不同等。但是请不要担心,我会尽量表明白怎么着找到呼应系统的R的安装包。

率先走访官网https://www.r-project.org/

然后我们找到这个:

图片 1

接下来找到China那一栏,随便点一个地址就好。

下一场我们挑选这一个:

图片 2

那下你应有能找到了呢,里面包含两种常用Linux发行版,我的是CentOS,自然是采取RedHat体系了。

下一场我们依次点下去,直到点开README

图片 3

很好,这里似乎要设置一个EPEL的源来完成我们的R包的装置,那里我们进来划横线的网址:

图片 4

箭头所指是那些源的RPM包,下载并设置之!

wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
rpm -ivh epel-release-latest-7.noarch.rpm

接下去自己打开这一个网址:https://zeppelin.apache.org/docs/0.6.2/interpreter/r.html

安装相关R以及有关倚重:

yum install R R-devel libcurl-devel openssl-devel

安装时间相比较长,耐心等待,去喝杯咖啡或者看会其余,再回去吧!

只是你假如向来密切看的话,它竟然在拍卖倚重关系的时候自动给安装上了open-jdk,没事,我们利用以下方法解决:

wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u112-b15/jdk-8u112-linux-x64.rpm 
rpm -ivh jdk-8u112-linux-x64.rpm 

再给它安装回去-。-醉了。

设置一些官网所说的依赖:

R -e “install.packages(‘devtools’, repos = ‘http://cran.rstudio.com‘)”
R -e “install.packages(‘knitr’, repos = ‘http://cran.rstudio.com‘)”
R -e “install.packages(‘ggplot2’, repos = ‘http://cran.rstudio.com‘)”
R -e “install.packages(c(‘devtools’,’mplot’, ‘googleVis’), repos =
http://cran.rstudio.com‘); require(devtools);
install_github(‘ramnathv/rCharts’)”

trait Unmarshaller[-A, B] extends akka.http.javadsl.unmarshalling.Unmarshaller[A, B] {...}
object Unmarshaller
  extends GenericUnmarshallers
  with PredefinedFromEntityUnmarshallers
  with PredefinedFromStringUnmarshallers {

  // format: OFF

  //#unmarshaller-creation
  /**
   * Creates an `Unmarshaller` from the given function.
   */
  def apply[A, B](f: ExecutionContext ⇒ A ⇒ Future[B]): Unmarshaller[A, B] =
    withMaterializer(ec => _ => f(ec))
...}

2.配置Zeppelin

复制官网的一段话:

从Unmarshaller的构建函数apply可以推断它的效果应该与函数A=>Future[B]很相似。A代表网上可传输类型如MessageEntity、HttpRequest,B代表某种程序高级数据类型。因为A到B的转移是non-blocking的,所以可以立刻重回Future类型结果。Akka-http按被转换对象类型分类命名了下面这个序列别名:

Configuration

To run Zeppelin with the R Interpreter, the SPARK_HOME environment
variable must be set. The best way to do this is by
editing conf/zeppelin-env.sh. If it is not set, the R Interpreter will
not be able to interface with Spark.

You should also
copy conf/zeppelin-site.xml.template to conf/zeppelin-site.xml. That
will ensure that Zeppelin sees the R Interpreter the first time it
starts up.

多少个步骤:

  • 在conf/zeppelin-env.sh中加入SPARK_HOME那个环境变量
  •  cp conf/zeppelin-site.xml.template  conf/zeppelin-site.xml

这里不再赘言。

type FromEntityUnmarshaller[T] = Unmarshaller[HttpEntity, T]
type FromMessageUnmarshaller[T] = Unmarshaller[HttpMessage, T]
type FromResponseUnmarshaller[T] = Unmarshaller[HttpResponse, T]
type FromRequestUnmarshaller[T] = Unmarshaller[HttpRequest, T]
type FromByteStringUnmarshaller[T] = Unmarshaller[ByteString, T]
type FromStringUnmarshaller[T] = Unmarshaller[String, T]
type FromStrictFormFieldUnmarshaller[T] = Unmarshaller[StrictForm.Field, T]

3.重启Zeppelin:

/opt/zeppelin-0.6.2-bin-all/bin/zeppelin-daemon.sh restart

Akka-http对以下系列提供了电动的Unmarshalling转换: 

4.验证条件是否装好:

开拓浏览器进入Zeppelin的地点,然后notebook -> R Tutorial

如下图

图片 5

 

等候编译完成,然后自己截取了几张图纸放在下边:

图片 6

图片 7

图片 8

好,我们的SparkR环境搭建好了,Zeppelin又多了一个新的工具,又有力了许多。

PredefinedFromStringUnmarshallers
Byte
Short
Int
Long
Float
Double
Boolean
PredefinedFromEntityUnmarshallers
Array[Byte]
ByteString
Array[Char]
String
akka.http.scaladsl.model.FormData
GenericUnmarshallers
Unmarshaller[T, T] (identity unmarshaller)
Unmarshaller[Option[A], B], if an Unmarshaller[A, B] is available
Unmarshaller[A, Option[B]], if an Unmarshaller[A, B] is available

也就是说Akka-http提供了这一个U类型的Unmarshaller[U,B]隐式实例。Akka-http也提供了工具类型Unmarshal:

object Unmarshal {
  def apply[T](value: T): Unmarshal[T] = new Unmarshal(value)
}

class Unmarshal[A](val value: A) {
  /**
   * Unmarshals the value to the given Type using the in-scope Unmarshaller.
   *
   * Uses the default materializer [[ExecutionContext]] if no implicit execution context is provided.
   * If you expect the marshalling to be heavy, it is suggested to provide a specialized context for those operations.
   */
  def to[B](implicit um: Unmarshaller[A, B], ec: ExecutionContext = null, mat: Materializer): Future[B] = {
    val context: ExecutionContext = if (ec == null) mat.executionContext else ec

    um(value)(context, mat)
  }
}

咱俩得以因此Unmarshal.to[B]把Unmarshal[A]转换成Future[B]。注意:这一步只囊括了从网上可传输类型到程序类型转换这一进程,不包括具体贯彻时的Json转换。上边是一些Unmarshal的用例:

import akka.actor._
import akka.stream._
import akka.http.scaladsl.unmarshalling.Unmarshal
import akka.http.scaladsl.model._
import akka.http.scaladsl.server.Directives._

object Unmarshalling {
  implicit val httpSys = ActorSystem("httpSystem")
  implicit val httpMat = ActorMaterializer()
  implicit val httpEC = httpSys.dispatcher

  val futInt = Unmarshal(43).to[Int]
  val futBoolean = Unmarshal("0").to[Boolean]
  val futString = Unmarshal(HttpEntity("Hello")).to[String]
  val futHello = Unmarshal(HttpRequest(method = HttpMethods.GET, entity = HttpEntity("hello")))

}

以上都是已知类型之间转换,可能没什么实际用途,不像marshalling:中间层Marshalling有实在转换的需要。Unmarshalling可以一向开展Json到自定义类型之间的更换,如:

 val route = (path("User") & post) { entity(as[User]){ user =>
    complete(Future(s"inserting user: $user"))
  }} ~
    (path("Item"/IntNumber) & put) { id => entity(as[Item]){ item =>
      complete(Future(s"update item $id: $item"))
    }}

如上是透过Directive
as[???]实现的:

 /**
   * Returns the in-scope [[FromRequestUnmarshaller]] for the given type.
   *
   * @group marshalling
   */
  def as[T](implicit um: FromRequestUnmarshaller[T]) = um

这亟需把FromRequestUmarshaller[T]身处可视域内,FromRequestUmarshaller[T]实际是Unmarshaller[T,B]的别名: 

type FromRequestUnmarshaller[T] = Unmarshaller[HttpRequest, T]

在上篇研究我们介绍了Akka-http的Marshalling是type-class形式的。其中重大可以参见上篇探讨。现在我们需要这一个Unmarshaller的隐式实例:

trait Formats extends SprayJsonSupport with DefaultJsonProtocol
object Converters extends Formats {
  case class User(id: Int, name: String)
  case class Item(id: Int, name: String, price: Double)
  implicit val itemFormat = jsonFormat3(Item.apply)
  implicit val userFormat = jsonFormat2(User.apply)
}

object Unmarshalling {
  import Converters._
...

一经采取Json4s的贯彻形式,大家需要如下提供那么些隐式实例:

trait JsonCodec extends Json4sSupport {
  import org.json4s.DefaultFormats
  import org.json4s.ext.JodaTimeSerializers
  implicit val serilizer = jackson.Serialization
  implicit val formats = DefaultFormats ++ JodaTimeSerializers.all
}
object JsConverters extends JsonCodec

Json4s的实际用例如下:

  import scala.collection.mutable._
  case class User(id: Int, name: String)
  class Item(id: Int, name: String, price: Double)
  object AnyPic {
    val area = 10
    val title = "a picture"
    val data = ArrayBuffer[Byte](1,2,3)
  }

  val route = (path("User") & post) { entity(as[User]){ user =>
    complete(Future(s"inserting user: $user"))
  }} ~
    (path("Item"/IntNumber) & put) { id => entity(as[Item]){ item =>
      complete(Future(s"update item $id: $item"))
    }} ~
    (path("Picture") & put) { entity(as[AnyPic.type]){ pic =>
      complete(Future(s"insert picture: $pic"))
    }}

从功用上和发挥灵活性来讲,Json4s的贯彻格局要占优。

下边就是此次研讨的示范源代码:

Unmarshalling

import akka.actor._
import akka.stream._
import akka.http.scaladsl.unmarshalling.Unmarshal
import akka.http.scaladsl.model._
import akka.http.scaladsl.server.Directives._
import scala.concurrent._
import akka.http.scaladsl.marshallers.sprayjson._
import spray.json._

trait Formats extends SprayJsonSupport with DefaultJsonProtocol
object Converters extends Formats {
  case class User(id: Int, name: String)
  case class Item(id: Int, name: String, price: Double)
  implicit val itemFormat = jsonFormat3(Item.apply)
  implicit val userFormat = jsonFormat2(User.apply)
}

object Unmarshalling {
  import Converters._
  implicit val httpSys = ActorSystem("httpSystem")
  implicit val httpMat = ActorMaterializer()
  implicit val httpEC = httpSys.dispatcher

  val futInt = Unmarshal(43).to[Int]
  val futBoolean = Unmarshal("0").to[Boolean]
  val futString = Unmarshal(HttpEntity("Hello")).to[String]
  val futHello = Unmarshal(HttpRequest(method = HttpMethods.GET, entity = HttpEntity("hello")))

  val route = (path("User") & post) { entity(as[User]){ user =>
    complete(Future(s"inserting user: $user"))
  }} ~
    (path("Item"/IntNumber) & put) { id => entity(as[Item]){ item =>
      complete(Future(s"update item $id: $item"))
    }}

}

Json4sUnmarshalling

import akka.actor._
import akka.stream._
import akka.http.scaladsl.server.Directives._
import de.heikoseeberger.akkahttpjson4s.Json4sSupport
import org.json4s.jackson
import scala.concurrent._
trait JsonCodec extends Json4sSupport {
  import org.json4s.DefaultFormats
  import org.json4s.ext.JodaTimeSerializers
  implicit val serilizer = jackson.Serialization
  implicit val formats = DefaultFormats ++ JodaTimeSerializers.all
}
object JsConverters extends JsonCodec


object Json4sUnmarshalling {
  import JsConverters._
  implicit val httpSys = ActorSystem("httpSystem")
  implicit val httpMat = ActorMaterializer()
  implicit val httpEC = httpSys.dispatcher

  import scala.collection.mutable._
  case class User(id: Int, name: String)
  class Item(id: Int, name: String, price: Double)
  object AnyPic {
    val area = 10
    val title = "a picture"
    val data = ArrayBuffer[Byte](1,2,3)
  }

  val route = (path("User") & post) { entity(as[User]){ user =>
    complete(Future(s"inserting user: $user"))
  }} ~
    (path("Item"/IntNumber) & put) { id => entity(as[Item]){ item =>
      complete(Future(s"update item $id: $item"))
    }} ~
    (path("Picture") & put) { entity(as[AnyPic.type]){ pic =>
      complete(Future(s"insert picture: $pic"))
    }}
}