带有SAP HANA Spark控制器的HortonWorks Hadoop

2020-08-31 06:32发布

点击此处---> 群内免费提供SAP练习系统(在群公告中)加入QQ群:457200227(SAP S4 HANA技术交流) 群内免费提供SAP练习系统(在群公告中)你好 我们在顶部运行带有Spa...

         点击此处--->   EasySAP.com群内免费提供SAP练习系统(在群公告中)

加入QQ群:457200227(SAP S4 HANA技术交流) 群内免费提供SAP练习系统(在群公告中)


你好

我们在顶部运行带有Spark2 v2.3.0和SAP HANA SparkController v2.3.5的沙箱2.6.5。 Ambari仪表板显示所有服务均已启动并正在运行,一切似乎都很好:


现在,我们希望 通过SparkController从我们的HANA数据库(运行在IP 10.118.169.32上)连接到Hadoop。 我们使用以下属性在HANA Studio客户端工具中配置了连接:

连接成功建立,我们可以看到我们想要访问的表:

但是,当我们运行一个简单的选择查询时,它只运行了大约10分钟,直到连接关闭。 没有数据返回:

  1. 内部错误:打开用于远程数据库的游标时出错。连接已关闭。对于来自" foodmart"的查询SELECT COUNT(*)。" customer"

在hana_controller.log中,我们可以看到已接受SQL查询并已处理了执行计划=>

  1. 19/06/0611:39:28调试NetworkUtils:InboundMessage =>'消息类型=>'PING'; 消息ID =>'693201651063748'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a59';'
  2. 19/06/0611:39:28调试HanaMessageHandler:ReceivedRequestfrom:'172.18.0.3'; MessageType =>'PING'
  3. 19/06/0611:39:28调试NetworkUtils:OutboundMessage =>'消息类型=>'PONG'; 消息ID =>'693201670667961'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a59';'
  4. 19/06/0611:39:28调试NetworkUtils:InboundMessage =>'消息类型=>'START_SESSION'; 消息ID =>'693201689176436'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a5b'; 会话ID =>'3878625230891646998-12696546365132577370';'
  5. 19/06/0611:39:28调试HanaMessageHandler:ReceivedRequestfrom:'172.18.0.3'; MessageType =>'START_SESSION'
  6. 19/06/0611:39:28 INFO CommandRouterDefault:CurrentAssignments => NONE; IncomingAssignment =>'3878625230891646998-12696546365132577370#35d3a519-002a-0016-b033-31a660011a5b'
  7. 19/06/0611:39:28 INFO RequestHandler:BoundMessage发送给Orchestrator:'35d3a519-002a-0016-b033-31a660011a5b'
  8. 19/06/0611:39:28 INFO RequestHandler:RetrievingBoundMessage:'35d3a519-002a-0016-b033-31a660011a5b'
  9. 19/06/0611:39:28调试RequestHandler:RelayingMessage到客户端:'CONTINUE_SESSION'
  10. 19/06/0611:39:28调试NetworkUtils:OutboundMessage =>'消息类型=>'CONTINUE_SESSION'; 消息ID =>'693201720489241'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a5b'; 会话ID =>'3878625230891646998-12696546365132577370';'
  11. 19/06/0611:39:28调试NetworkUtils:InboundMessage =>'消息类型=>'CONTINUE_SESSION'; 消息ID =>'693201726088539'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a5b'; 会话ID =>'3878625230891646998-12696546365132577370';'
  12. 19/06/0611:39:28调试HanaMessageHandler:ReceivedRequestfrom:'172.18.0.3'; MessageType =>'CONTINUE_SESSION'
  13. 19/06/0611:39:28 INFO CommandRouter默认值:CurrentAssignments => 3878625230891646998-12696546365132577370#35d3a519-002a-0016-b033-31a660011a5b-> Actor [akka://SAPHanaSpark/user/Controller/HanaSparkRouter/$ a# 1995820509]; 传入分配=>'3878625230891646998-12696546365132577370#35d3a519-002a-0016-b033-31a660011a5b'
  14. 19/06/0611:39:28 INFO RequestHandler:RetrievingBoundMessage:'35d3a519-002a-0016-b033-31a660011a5b'
  15. 19/06/0611:39:28 INFO HanaSparkFacade:内部评估会议:hanaes(作者:SIMPLE)
  16. 19/06/0611:39:28调试会话:ConnectingUser:'root'
  17. 19/06/0611:39:28调试RequestHandler:RelayingMessage到客户端:'SESSION_OPENED'
  18. 19/06/0611:39:28调试NetworkUtils:OutboundMessage =>'消息类型=>'SESSION_OPENED'; 消息ID =>'693201738786757'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a5b'; 会话ID =>'3878625230891646998-12696546365132577370';'
  19. 19/06/0611:39:28调试RequestHandler:完成执行:'START_SESSION'。返回池
  20. 19/06/0611:39:28 DEBUG ChannelManager:NewSession已添加:'3878625230891646998-12696546365132577370'
  21. 19/06/0611:39:28调试NetworkUtils:InboundMessage =>'消息类型=>'PING'; 消息ID =>'693201743893250'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a5c';'
  22. 19/06/0611:39:28调试HanaMessageHandler:ReceivedRequestfrom:'172.18.0.3'; MessageType =>'PING'
  23. 19/06/0611:39:28调试NetworkUtils:OutboundMessage =>'消息类型=>'PONG'; 消息ID =>'693201746082223'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a5c';'
  24. 19/06/0611:39:28调试NetworkUtils:InboundMessage =>'消息类型=>'PING'; 消息ID =>'693201748509338'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a5d';'
  25. 19/06/0611:39:28调试HanaMessageHandler:ReceivedRequestfrom:'172.18.0.3'; MessageType =>'PING'
  26. 19/06/0611:39:28调试NetworkUtils:OutboundMessage =>'消息类型=>'PONG'; 消息ID =>'693201750304135'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a5d';'
  27. 19/06/0611:39:28调试NetworkUtils:InboundMessage =>'消息类型=>'PING'; 消息ID =>'693201752596927'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a5e';'
  28. 19/06/0611:39:28调试HanaMessageHandler:ReceivedRequestfrom:'172.18.0.3'; MessageType =>'PING'
  29. 19/06/0611:39:28调试NetworkUtils:OutboundMessage =>'消息类型=>'PONG'; 消息ID =>'693201753461377'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a5e';'
  30. 19/06/0611:39:28调试NetworkUtils:InboundMessage =>'消息类型=>'QUERY_EXECUTE'; 消息ID =>'693201755190134'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a5f'; 会话ID =>'3878625230891646998-12696546365132577370';'
  31. 19/06/0611:39:28调试HanaMessageHandler:ReceivedRequestfrom:'172.18.0.3'; MessageType =>'QUERY_EXECUTE'
  32. 19/06/0611:39:28 INFO CommandRouterDefault:CurrentAssignments => NONE; IncomingAssignment =>'3878625230891646998-12696546365132577370#35d3a519-002a-0016-b033-31a660011a5f'
  33. 19/06/0611:39:28 INFO RequestHandler:BoundMessage发送给Orchestrator:'35d3a519-002a-0016-b033-31a660011a5f'
  34. 19/06/0611:39:28 INFO RequestHandler:RetrievingBoundMessage:'35d3a519-002a-0016-b033-31a660011a5f'
  35. 19/06/0611:39:28调试HanaExecPlan: SELECT" target =" _blank"> http://www.sap.com/hana"> 从" foodmart"中选择COUNT(*)AS CA_10261569202013848859。"客户"" Hadoop_customer" 0色 4
  36. 19/06/0611:39:28调试RequestHandler:ChosenEngine ID:'sparksql'
  37. 19/06/0611:39:28 DEBUG DataRouteRegistry:RegisteringQueryExecution和requestId:'147BF43D222C974CC039F0B3C6E6356C'
  38. 19/06/0611:39:28信息AsyncExecutor:RunningwithUserPrivilege'root'
  39. 19/06/0611:39:28调试RequestHandler:UnblockedRelay
  40. 19/06/0611:39:28调试RequestHandler:RelayingMessage到客户端:'QUERY_ACCEPTED'
  41. 19/06/0611:39:28调试NetworkUtils:OutboundMessage =>'消息类型=>'QUERY_ACCEPTED'; 消息ID =>'693201797735164'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a5f'; 会话ID =>'3878625230891646998-12696546365132577370';'
  42. 19/06/0611:39:28信息HanaSparkFacade:HanaSelectTask(SQL):从`foodmart`中选择COUNT(*)作为CA_10261569202013848859.customer``Hadoop_customer`
  43. 19/06/0611:39:29信息SparkCompositeCatalog:查找表:'foodmart.customer'
  44. 19/06/0611:39:30调试HanaHiveSQLContext:resolveRelations23-hti.table ='customer'; rel =''SubqueryAlias客户
  45. +-" UnresolvedCatalogRelation`foodmart`.`customer`,org.apache.hadoop.hive.ql.io.orc.OrcSerde
  46. '
  47. 19/06/0611:39:30 DEBUG DistributedDataSetFactoryImpl:查询计划:
  48. 'Project ['COUNT(1)AS CA_10261569202013848859#39]
  49. +-'SubqueryAlias Hadoop_customer
  50. +-'SubqueryAlias客户
  51. +-'SubqueryAlias客户
  52. +-'unresolvedCatalogRelation`foodmart..customer`,org.apache.hadoop.hive.ql.io.orc.OrcSerde
  53. 19/06/0611:39:30 INFO SparkFacade $ anon $ 1:从0创建广播0
  54. 19/06/0611:39:31信息SparkFacade $ anon $ 1:开始工作:SparkFacade.scala:299上的transferDatafromPartitions
  55. 19/06/0611:39:31信息SparkFacade $ anon $ 1:从DAGScheduler.scala:1039的广播创建广播1
  56. 19/06/0611:39:33信息SparkFacade $ anon $ 1:从DAGScheduler.scala:1039广播中创建广播2。
  57. 19/06/0611:39:35调试SmartReverseProxyHandler:StartingChannelTag:' sandbox-hdp。 hortonworks.com :49372'
  58. 19/06/0611:39:35调试SmartReverseProxyHandler:MessageType =>'DATA_SOURCE_LOCATION'
  59. 19/06/0611:39:35 DEBUG DataRouteRegistry:RegisteringEndPoint for requestId:'147BF43D222C974CC039F0B3C6E6356C'('3878625230891891646998-12696546365132577370','35d3a519-002a-0016-b033-31a660011a5f'),transferId:'Result:0 li>
  60. 19/06/0611:39:35调试NetworkUtils:InboundMessage =>'消息类型=>'DATA_SOURCE_LOCATION'; 消息ID =>'693208692846518'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a5f'; 会话ID =>'3878625230891646998-12696546365132577370';'
  61. 19/06/0611:39:35调试HanaMessageHandler:ReceivedRequestfrom:' sandbox-hdp。 hortonworks.com '; MessageType =>'DATA_SOURCE_LOCATION'
  62. 19/06/0611:39:35 INFO CommandRouter默认值:CurrentAssignments => 3878625230891646998-12696546365132577370#35d3a519-002a-0016-b033-31a660011a5f-> Actor [akka://SAPHanaSpark/user/Controller/HanaSparkRouter/$ b#- 336533065]; 传入分配=>'3878625230891646998-12696546365132577370#35d3a519-002a-0016-b033-31a660011a5f'
  63. 19/06/0611:39:35 INFO RequestHandler:RetrievingBoundMessage:'35d3a519-002a-0016-b033-31a660011a5f'
  64. 19/06/0611:39:35 INFO RequestHandler:ForwardingDataVent:'QUERY_DATA_SOURCE';方向:'0'
  65. 19/06/0611:39:35调试RequestHandler:DataTransferEndPoint:'172.18.0.2:7861'
  66. 19/06/0611:39:35调试给客户端的RequestHandler:RelayingMessage:'QUERY_DATA_SOURCE'
  67. 19/06/0611:39:35调试NetworkUtils:OutboundMessage =>'消息类型=>'QUERY_DATA_SOURCE'; 消息ID =>'693208699630324'; 请求ID =>'35d3a519-002a-0016-b033-31a660011a5f'; 会话ID =>'3878625230891646998-12696546365132577370';'
  68. 19/06/0611:49:35 DEBUG HanaMessageHandler:Channel未注册...关闭所有会话
  69. 19/06/0611:49:35调试ChannelManager:ClosingAllSessions
  70. 19/06/0611:49:35调试ChannelManager:Session已关闭:'3878625230891646998-12696546365132577370'

我们需要帮助弄清楚为什么结果没有回来。 这是我们的第一个Hadoop安装,我们不知道根本原因可能是什么。 也许在"数据传输端点"行中提到的IP或端口有问题?

非常感谢您就如何查明和解决根本原因提出建议。

非常感谢您,

伯恩哈德

2条回答
能不能别闹
2020-08-31 07:03

我正面临着同样的问题。 我的猜测是,问题与以下事实有关:从Hortonworks 3.0开始,Spark und Hive不再共享相同的数据库架构。 记录在

https://docs。 hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/integrating-hive/content/hive_configure_a_spark_hive_connection.html

不幸的是,上面链接中提到的解决方案无济于事-我在Ambari中为Spar和SAP Spark Controller添加了配置。 没有效果。

任何帮助将不胜感激。

一周热门 更多>